EdgeGallery Ansible Offline Installation

This guide is for EdgeGallery (EG) offline installation when there is no public network with the environment.

The same as online installation, the offline installation is also based on Ubuntu OS and Kubernetes/k3s, supports x86_64 and ARM64 as well.

1. The Dependencies and How to Set Nodes

EdgeGallery supports Multi Node and All-In-One (AIO) deployment now.

1.1 AIO Deployment

1.1.1 Need One Ansible Controller Node

There is a kindly suggestion to choose a machine which can access Internet as the Ansible controller node, that will be helpful for installing Python, pip3 and Ansible, as well as downloading the EdgeGallery Offline Packages.

The components in the following table should be installed on Ansible controller node in advance. If the Ansible controller node has no Internet access, you could refer to section 2.2 for how to install Ansible offline.

The following versions are the suggested ones which have been proved to work well by EG developers and testers.

Module	Version	Arch
Ubuntu	18.04	ARM 64 & X86_64
Python	3.6.9	ARM 64 & X86_64
pip3	9.0.1	ARM 64 & X86_64
Ansible	2.10.7	ARM 64 & X86_64

1.1.2. Need One Master Node

The Master Node should only install Ubuntu 18.04 and with the following hardware resources:

4CPU
16G RAM
100G Storage
Single or Multi NIC

INFO: The Ansible controller node and the Master Node could be the same node.

1.2 Multi-Node Deployment

1.2.1 Need One Ansible Controller Node

The requirements of the Ansible controller node here are the same as the AIO deployment in the previous section. There only need one single Ansible controller node which can work as the control center to deploy several k8s clusters and EG PaaS.

1.2.2 Need Several Master and Worker Nodes

The k8s Cluster could be one Master node (only support one master node now) and several Worker nodes (less than 12). And the requirements of these Nodes are all the same as Section 1.1.2.

2. How to Config the Ansible Controller Node

The commands in the following sections are all executed on Ansible controller node and there is no commands that need to be executed on any k8s/k3s nodes.

2.1 Login Ansible Controller Node

The Ansible controller node should already install ububntu 18.04, python3.6 and pip3 in advance.

2.2 Install Ansible

Ansible Online Installation

# Recommend to install Ansible with python3
apt install -y python3-pip
pip3 install ansible

Ansible Offline Installation

Download X86 Ansible package or ARM64 Ansible package on a machine that can access Internet.
Copy the package to Ansible controller node, e.g. /home

Do the following commands to install Ansible

# Here take the x86 as example
cd /home
tar -xvf ansible-offline-install-python3-x86.tar.gz

# Install Ansible
pip3 install -f ansible-offline-install-python3-x86 --no-index ansible

# Check Whether Ansible installed successfully
ansible --version

2.3 Download EdgeGallery Offline Package

All EG offline packages could be found on EdgeGallery Download Page. Users need to choose the package with exact architecture (x86 or arm64) and EG Mode (edge, controller or all).

The following guide takes x86 architecture and "all" mode (edge + controller) as the example to introduce how to deploy EG in both single node and multi nodes cases.

Download EG offline package "all" mode on x86 on a machine that can access Internet, and copy it to Ansible controller node, e.g. /home
```
cd /home
tar -xvf EdgeGallery-v1.5.0-all-x86.tar.gz
```
Set password-less ssh from Ansible controller node to other nodes

2.1. sshpass required：
```
# Check whether sshpass installed
sshpass -V

# If not, install sshpass
cd /home/EdgeGallery-v1.5.0-all-x86
dpkg -i -G -E sshpass_1.06-1_amd64.deb

# Check whether sshpass installed successfully
sshpass -V
```
2.2 There should be id_rsa and id_rsa.pub under /root/.ssh/, if not, do the following to generate them:
```
ssh-keygen -t rsa
```
2.3 Do the following to set the password-less ssh, execute the command several times for all master and worker nodes one by one where <master-or-worker-node-ip> is the private IP, <master-or-worker-node-root-password> is the password of root user of that node and <ssh-port> is the port used to ssh which default is 22.
```
sshpass -p <master-or-worker-node-root-password> ssh-copy-id -p <ssh-port> -o StrictHostKeyChecking=no root@<master-or-worker-node-ip>
```

3. EdgeGallery Deployment

Currently, the Ansible scripts support deploying both IaaS (k8s or k3s) and PaaS (EG).

The Harbor deployment in role eg_prepare could only work on x86 machines, and it will be deployed automatically during the EG deployment on x86.

However, when deploying on arm64 machines, you should prepare a x86 machine and install Harbor on it manually before deploying EG on those arm64 machines. That's because Harbor hasn't provided arm64 Docker images now. Please refer to Section 6 to install Harbor manually before deploying EG.

The following table gives all deployment scenarios pre-defineded in the EG offline package (under /home/EdgeGallery-v1.5.0-all-x86/install/ directory), and you can use them directly to deploy EG.

EG_MODE	NODE_MODE	install yml	uninstall yml
all	aio	eg_all_aio_install.yml	eg_all_aio_uninstall.yml
	muno	eg_all_muno_install.yml	eg_all_muno_uninstall.yml
controller	aio	eg_controller_aio_install.yml	eg_controller_aio_uninstall.yml
	muno	eg_controller_muno_install.yml	eg_controller_muno_uninstall.yml
edge	aio	eg_edge_aio_install.yml	eg_edge_aio_uninstall.yml
	muno	eg_edge_muno_install.yml	eg_edge_muno_uninstall.yml

3.1. How to Config Ansible Inventory

Ansible inventory is used to set the master and worker nodes info which used to ssh to these nodes by Ansible. Please refer to the files, hosts-aio and hosts-muno under /home/EdgeGallery-v1.5.0-all-x86/install, to do the Ansible inventory configuration.

AIO Inventory, replace the exactly master node IP in file host-aio:
```
[master]
xxx.xxx.xxx.xxx
```
Multi Node Inventory, refer to file hosts-muno and replace the master and worker nodes IPs:
```
[master]
xxx.xxx.xxx.xxx

[worker]
xxx.xxx.xxx.xxx
xxx.xxx.xxx.xxx
```
If Ansible Controller Node and the cluster master node are the same one, can use localhost instead of IP which can speed up the deployment.
```
[master]
localhost ansible_connection=local
```

SSH port is not the default value 22, should add some more info about the ssh port

[master]
xxx.xxx.xxx.xxx
[master:vars]
ansible_ssh_port=xx

[worker]
xxx.xxx.xxx.xxx
xxx.xxx.xxx.xxx
[worker:vars]
ansible_ssh_port=xx

SSH user must be root, if failed with the log "Timeout (12s) waiting for privilege escalation prompt: ", then need to set the user to be root.

[master]
xxx.xxx.xxx.xxx
[master:vars]
ansible_ssh_port=xx
ansible_ssh_user=root

[worker]
xxx.xxx.xxx.xxx
xxx.xxx.xxx.xxx
[worker:vars]
ansible_ssh_port=xx
ansible_ssh_user=root

NOTE: only supports one master and several worker (less than 12) nodes. Also the Ansible controller node can also act as one of the master or worker node.

3.2. How to Set the Parameters

All parameters that user could set are in file install/var.yml.

# Set the regex name of the network interface for calico
NETWORK_INTERFACE: eth.*

# Could be true or false
# true: Deploy k8s NFS Server to keep the persistence of all pods' data
# false: No need to keep the persistence of all pods' data
# for k3s, could only be false because k3s doesn't support persistence storage now
ENABLE_PERSISTENCE: true

# One IP of the cluster master node
MASTER_IP: xxx.xxx.xxx.xxx

# ip for portals, will be set to private IP of master node default or reset it to be the public IP of master node here
# PORTAL_IP: xxx.xxx.xxx.xxx

# IP of the Controller master which is used for Edge to connect
# If you deploy Controller and Edge together in one cluster, then ther is no need to set this param
# CONTROLLER_MASTER_IP: xxx.xxx.xxx.xxx

# NIC name of master node
# If master node is with single NIC, not need to set it here and will get the default NIC name during the run time
# If master node is with multiple NICs, should set it here to be 2 different NICs
# EG_NODE_EDGE_MP1: eth0
# EG_NODE_EDGE_MM5: eth0

# Email Server Config for User Mgmt
usermgmt_mail_enabled: false
# If usermgmt_mail_enabled is true, then the following 4 params need to be set
# usermgmt_mail_host: xxxxx
# usermgmt_mail_port: xxxxx
# usermgmt_mail_sender: xxxxx
# usermgmt_mail_authcode: xxxxx

3.3 How to Config Access of EdgeGallery with Proxy

If you want to access EdgeGallery portal with proxy, you can following the doc EdgeGallery_ProxyAccessConfig_Guide-cn.md to do the configuration.

3.4 How to Set the Passwords

All passwords needed are in file install/password-var.yml. The Ansible scripts don't provide any default password now, and all passwords are needed to be given by users before deploying.

Note: All passwords must include capital letters, lowercase letters, numbers and special characters and whose length must be no less than 8 characters. Also there should be no special characters & in it. Otherwise, the deployment will failed because of these simple passwords.

# Set the Password of Harbor admin account, no default value, must set by users here
HARBOR_ADMIN_PASSWORD: xxxxx

# postgresPassword is used for all postgres DB of all roles, no default value, must set by users here
postgresPassword: xxxxx

# oauth2ClientPassword is used for user mgmt, no default value, must set by users here
oauth2ClientPassword: xxxxx

# Redis Password used by user mgmt, no default value, must set by users here
userMgmtRedisPassword: xxxxx

# certPassword is used for generating SSL keys
certPassword: xxxxx

3.5 How to Collaborate with 3rd party IAM

If you want to use the 3rd party IAM system instead of EdgeGallery default one, you can config like below:

# External IAM Config
ENABLE_EXTERNAL_IAM: true

# If ENABLE_EXTERNAL_IAM is true, then the following need to be set
EXTERNAL_IAM_ENDPOINT: https://xxx.xxx.xxx.xxx

In this case, the 3rd party IAM system is required to realize the APIs according to the requirements of EdgGallery. Please refer to the API list.

3.6 How to Deploy

It only needs to specify the inventory file (host-aio or host-muno) and the scenario file when deploying.

cd /home/EdgeGallery-v1.5.0-all-x86/install

# AIO Deployment
ansible-playbook --inventory hosts-aio eg_all_aio_install.yml

# Multi Node Deployment
ansible-playbook --inventory hosts-muno eg_all_muno_install.yml

4. Uninstall EdgeGallery

Please refer to the following table to find out which uninstall scenario file should be chosen to do the uninstall operation according to the install scenario file chosen before.

EG_MODE	NODE_MODE	install yml	uninstall yml
all	aio	eg_all_aio_install.yml	eg_all_aio_uninstall.yml
	muno	eg_all_muno_install.yml	eg_all_muno_uninstall.yml
controller	aio	eg_controller_aio_install.yml	eg_controller_aio_uninstall.yml
	muno	eg_controller_muno_install.yml	eg_controller_muno_uninstall.yml
edge	aio	eg_edge_aio_install.yml	eg_edge_aio_uninstall.yml
	muno	eg_edge_muno_install.yml	eg_edge_muno_uninstall.yml

# Uninstall AIO Deployment
ansible-playbook --inventory hosts-aio eg_all_aio_uninstall.yml

# Uninstall Multi Node Deployment
ansible-playbook --inventory hosts-muno eg_all_muno_uninstall.yml

5. How to Install EdgeGallery with Self-defined Demand

Besides using the scenario files given in the offline packages, users can choose which roles to deploy and which don't with the ansible-playbook command lines by using options --tags and --skip-tags.

The following commands skip roles mep and mecm-mepm when deploying multi-node EG with all (edge+controller) mode.

ansible-playbook --inventory hosts-muno eg_all_muno_install.yml --skip-tags=mep,mecm-mepm

All roles in the deployment are list below. Users could choose some of them according to your own demand.

init (mandatory): Unarchive EG offline package and copy related files to master or worker nodes
k8s (mandatory): If you want to install k3s instead pf k8s, please refer Section 7 install EG based on k3s
eg_prepare (mandatory): Setup for deploying EG which do some network configuration, install Harbor, create some k8s resources et al.
mep (mandatory): Independent with other EG modules, could deploy or not deploy
mecm-mepm (optional): Independent with other EG modules except mep
user-mgmt (mandatory): The dependency module of all the following modules which used to do the User Management for all web portals
mecm-meo (optional): Independent with other EG modules except the user-mgmt
mecm-fe (optional): Independent with other EG modules except the user-mgmt
appstore (optional): Independent with other EG modules except the user-mgmt
developer (optional): Independent with other EG modules except the user-mgmt
atp (optional): Independent with other EG modules except the user-mgmt
eg_check (optional): Independent with all other EG modules and only check the deployed modules and print the Web portal URL

In summary, all modules are optional except init, eg_prepare, mep and user-mgmt. mep is the mandatory role for edge and user-mgmt is the mandatory one for controller.

6. How to Install Harbor Manually on X86

When you want to deploy EG on arm64 machines, you need to install Harbor manually on a x86 machine before deploying EG.

6.1 Deploy Harbor on X86

Install Docker and docker-compose because Harbor is rely on them
Config /etc/docker/daemon.json, add the following section. xxx.xxx.xxx.xxx is the private or public IP of this machine and can be accessed by the arm64 cluster. If there is no this file, create it.
```
{
    "insecure-registries" : ["xxx.xxx.xxx.xxx"]
}
```
Restart Docker Service
```
systemctl restart docker.service
```
Download Harbor offline install package and put it on /home

Install Harbor, you can directly copy and paste all commands except the xxx.xxx.xxx.xxx and which should be the IP given in step 2 and the password you want to set for Harbor

cd /root
openssl rand -writerand .rnd

cd /home
mkdir harbor
tar -xvf harbor.tar.gz -C harbor

export HARBOR_ROOT=/home/harbor
export HARBOR_DATA_VOLUME=/root/harbor/data_volume
export HARBOR_IP=xxx.xxx.xxx.xxx
export HARBOR_ADMIN_PASSWORD=<password>

cd $HARBOR_ROOT/cert
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Guangzhou/L=Guangzhou/O=example/CN="$HARBOR_IP -key ca.key -out ca.crt
openssl x509 -inform PEM -in ca.crt -out ca.cert

mkdir -p /etc/docker/certs.d/$HARBOR_IP:443
cp $HARBOR_ROOT/cert/ca.cert /etc/docker/certs.d/$HARBOR_IP:443

sed -i "s/hostname: .*/hostname: $HARBOR_IP/g" $HARBOR_ROOT/harbor.yml
sed -i "s#certificate: .*#certificate: $HARBOR_ROOT/cert/ca.crt#g" $HARBOR_ROOT/harbor.yml
sed -i "s#private_key: .*#private_key: $HARBOR_ROOT/cert/ca.key#g" $HARBOR_ROOT/harbor.yml
sed -i "s#data_volume: .*#data_volume: $HARBOR_DATA_VOLUME#g" $HARBOR_ROOT/harbor.yml
sed -i "s/harbor_admin_password: .*/harbor_admin_password: $HARBOR_ADMIN_PASSWORD/g" $HARBOR_ROOT/harbor.yml

cd $HARBOR_ROOT
bash install.sh

Login Harbor, login successfully means Harbor has been installed successfully
```
docker login -u admin -p $HARBOR_ADMIN_PASSWORD $HARBOR_IP
```

6.2 Config Harbor on Ansible Controller Node

The params related to Harbor should be set on Ansible controller Node before deploying EG. Please set HarborIP in the end of file install/default-var.yml as the IP given in step 2 in the previous section.

# If harbor is setup in a remote system, then mention the remote system IP as harbor IP
#HarborIP: xxx.xxx.xxx.xxx

6.3 Deploy EG on Ansible Controller Node

After installing Harbor successfully on that x86 machine, go on to refer to Section 3 to deploy EG on Ansible Controller Node.

7. Install EG Based on k3s

EdgeGallery supports to be installed based on k8s as well as k3s. The configuration of them are totally the same. Please refer to Section 3 to deploy EG, and then following the next steps to install k3s and EdgeGallery.

7.1 Init EG, unarchive EG offline package

# AIO
ansible-playbook --inventory hosts-aio eg_all_aio_install.yml --tags=init

# Multi-node
ansible-playbook --inventory hosts-muno eg_all_muno_install.yml --tags=init

7.2 Download k3s offline packages and put them in certain path

All the k3s installation packages should be put under the directory define in file default-var.yml with section TARBALL_PATH. The default value is /home/edgegallery-offline.

cd /home/edgegallery-offline
wget https://k3s-offline-install.obs.cn-north-4.myhuaweicloud.com/k3s
wget https://k3s-offline-install.obs.cn-north-4.myhuaweicloud.com/k3s-install.sh
wget https://k3s-offline-install.obs.cn-north-4.myhuaweicloud.com/k3s-airgap-images-amd64.tar

7.3 Install all other roles

Edit file g_all_aio_install.yml or eg_all_muno_install.yml and change 'k8s' into 'k3s'. Then run the following commands:

# AIO
ansible-playbook --inventory hosts-aio eg_all_aio_install.yml --skip-tags=init

# Multi-node
ansible-playbook --inventory hosts-muno eg_all_muno_install.yml --skip-tags=init

8. Trouble Shoot

error: fatal: [1.2.3.4]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

solution: deploy eg as a 'root' user

error: ESTABLISH SSH CONNECTION FOR USER: None

solution: append '-e "ansible_user=root"' to ansible-playbook command
eg: ansible-playbook --inventory hosts-aio eg_all_aio_uninstall.yml -e "ansible_user=root"

Info: For more verbosity add '-v' flag to ansible-playbook command

eg1: ansible-playbook --inventory hosts-aio eg_all_aio_uninstall.yml -vvv
eg2: ansible-playbook --inventory hosts-aio eg_all_aio_uninstall.yml -v

OSCHINA-MIRROR/edgegallery-installer

EdgeGallery Ansible Offline Installation

1. The Dependencies and How to Set Nodes

1.1 AIO Deployment

1.1.1 Need One Ansible Controller Node

1.1.2. Need One Master Node

1.2 Multi-Node Deployment

1.2.1 Need One Ansible Controller Node

1.2.2 Need Several Master and Worker Nodes

2. How to Config the Ansible Controller Node

2.1 Login Ansible Controller Node

2.2 Install Ansible

2.3 Download EdgeGallery Offline Package

3. EdgeGallery Deployment

3.1. How to Config Ansible Inventory

3.2. How to Set the Parameters

3.3 How to Config Access of EdgeGallery with Proxy

3.4 How to Set the Passwords

3.5 How to Collaborate with 3rd party IAM

3.6 How to Deploy

4. Uninstall EdgeGallery

5. How to Install EdgeGallery with Self-defined Demand

6. How to Install Harbor Manually on X86

6.1 Deploy Harbor on X86

6.2 Config Harbor on Ansible Controller Node

6.3 Deploy EG on Ansible Controller Node

7. Install EG Based on k3s

7.1 Init EG, unarchive EG offline package

7.2 Download k3s offline packages and put them in certain path

7.3 Install all other roles

8. Trouble Shoot

Опубликовать ( 0 )

Введение

Обновления

Участники

Недавние действия

OSCHINA-MIRROR/edgegallery-installer .gitee-modal { min-width: 500px !important; } .gitee-modal .close { right: 0.6rem !important; }

EdgeGallery Ansible Offline Installation

1. The Dependencies and How to Set Nodes

1.1 AIO Deployment

1.1.1 Need One Ansible Controller Node

1.1.2. Need One Master Node

1.2 Multi-Node Deployment

1.2.1 Need One Ansible Controller Node

1.2.2 Need Several Master and Worker Nodes

2. How to Config the Ansible Controller Node

2.1 Login Ansible Controller Node

2.2 Install Ansible

2.3 Download EdgeGallery Offline Package

3. EdgeGallery Deployment

3.1. How to Config Ansible Inventory

3.2. How to Set the Parameters

3.3 How to Config Access of EdgeGallery with Proxy

3.4 How to Set the Passwords

3.5 How to Collaborate with 3rd party IAM

3.6 How to Deploy

4. Uninstall EdgeGallery

5. How to Install EdgeGallery with Self-defined Demand

6. How to Install Harbor Manually on X86

6.1 Deploy Harbor on X86

6.2 Config Harbor on Ansible Controller Node

6.3 Deploy EG on Ansible Controller Node

7. Install EG Based on k3s

7.1 Init EG, unarchive EG offline package

7.2 Download k3s offline packages and put them in certain path

7.3 Install all other roles

8. Trouble Shoot

Опубликовать ( 0 )

Введение

Обновления

Участники

Недавние действия

OSCHINA-MIRROR/edgegallery-installer