准备bastion节点，离线包下载

1 bastion节点安装操作系统redhat7.8LE版本

2 设置主机名

hostnamectl set-hostname bastion.ocp4.example.com

3 安装相关工具软件

yum -y install podman httpd httpd-tools vim 

4 安装配置DNS和DHCP (option)

yum -y  install dnsmasq

配置通过ocp4/helpernode完成

5 安装配置HAProxy(option)

yum -y  install haproxy

配置通过ocp4/helpernode完成

6  创建内部镜像仓库

此仓库用于存放用于部署openshift平台所需使用的镜像，仓库本身也是用容器启动的。安装所需软件包，及创建所需目录，这些文件夹将挂载到registry容器中。

DNS本地解析

cat <<EOF>> /etc/hosts

172.16.167.175  bastion.ocp4.example.com

172.16.167.175  registry.ocp4.example.com

EOF

创建给本地镜像库使用的相关目录

mkdir-p/opt/registry/{auth,certs,data}

创建证书输入你相关信息

注意：Common Name (eg, your name or your server's hostname) 字段要填写镜像仓库的域名，不能使用IP。其他的可以空着，这一步我错了几次，如下：

cd /opt/registry/certs

openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt

创建密码文件

htpasswd -bBc /opt/registry/auth/htpasswd admin admin

更新Linux系统证书

cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/

update-ca-trust

使用一个镜像来部署私有镜像仓库：

podman run --name mirror-registry -p 5000:5000 \

      -v /opt/registry/data:/var/lib/registry:z \

      -v /opt/registry/auth:/auth:z \

      -e "REGISTRY_AUTH=htpasswd" \

      -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \

      -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \

      -v /opt/registry/certs:/certs:z \

      -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \

      -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \

      -d docker.io/ppc64le/registry:2

加入防火墙规则

firewall-cmd --add-port=5000/tcp --zone=internal --permanent

firewall-cmd --add-port=5000/tcp --zone=public  --permanent

firewall-cmd --reload

测试镜像仓库是否能正常访问：

curl -u admin:admin -k https://registry.ocp4.example.com:5000/v2/_catalog

{"repositories":[]}

将镜像仓库的用户名密码镜像base64加密

echo -n 'admin:admin' | base64 -w0

YWRtaW46YWRtaW4=

下载openshift的镜像文件所需的pull_sercet文件

登录cloud.redhat.com使用账号和密码登录

1 下载pull secret for OpenShift registry

2  下载最新的oc命令。进入下载链接，

https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp/stable-4.3/

下载

openshift-install-linux-4.3.22.tar.gz

openshift-client-linux-4.3.22.tar.gz

3 安装工具

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

yum -y install jq

4 解压openshift client和openshift install工具

tar -xzf openshift-client-linux-4.3.22.tar.gz -C /usr/local/bin/

tar -xzf openshift-install-linux-4.3.22.tar.gz -C /usr/local/bin/

将下载的pull_secret.txt格式化json文件pull_secret

mkdir -p ~/.openshift

cat ./pull-secret.txt | jq .  >  pull-secret

然后添加一下私有镜像仓库的内容，用于将quay的镜像同步到私有镜像仓库过程中需要做认证，这就是个认证密码文件，类似如下。

{ "auths": {

    "registry.ocp4.example.com:5000": {

      "auth": "YWRtaW46YWRtaW4=",

      "email": "noemail@example.com"

  }

}

}

完整的pull_secret如下

6 复制镜像

export BUILDNUMBER=4.3.22-ppc64le

echo ${BUILDNUMBER}

export OCP_RELEASE=${BUILDNUMBER}

export LOCAL_REG='registry.ose.example.com:5000'

export LOCAL_REPO='ocp4/openshift4'

export UPSTREAM_REPO='openshift-release-dev'

export LOCAL_SECRET_JSON="/home/ocp4.3/pull-secret.json"

export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE}

export RELEASE_NAME="ocp-release"

oc adm release mirror -a ${LOCAL_SECRET_JSON} \

--from=quay.io/${UPSTREAM_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \

--to-release-image=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE} \

--to=${LOCAL_REG}/${LOCAL_REPO}

镜像下载成功后显示如下信息

Success

Update image:  registry.ocp4.example.com:5000/ocp4/openshift4:4.3.22-ppc64le

Mirror prefix: registry.ocp4.example.com:5000/ocp4/openshift4

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:

- mirrors:

  - registry.ocp4.example.com:5000/ocp4/openshift4

  source: quay.io/openshift-release-dev/ocp-release

- mirrors:

  - registry.ocp4.example.com:5000/ocp4/openshift4

  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1

kind: ImageContentSourcePolicy

metadata:

  name: example

spec:

  repositoryDigestMirrors:

  - mirrors:

    - registry.ocp4.example.com:5000/ocp4/openshift4

    source: quay.io/openshift-release-dev/ocp-release

  - mirrors:

    - registry.ocp4.example.com:5000/ocp4/openshift4

    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

运行命令查看所有的镜像

curl -u admin:admin -k https://registry.ocp4.example.com:5000/v2/ocp4/openshift4/tags/list |jq

{

  "name": "ocp4/openshift4",

  "tags": [

    "4.3.22-ovn-kubernetes",

    "4.3.22-keepalived-ipfailover",

    "4.3.22-prometheus-node-exporter",

    "4.3.22-oauth-proxy",

    "4.3.22-thanos",

    "4.3.22-jenkins-agent-nodejs",

    "4.3.22-multus-admission-controller",

    "4.3.22-deployer",

    "4.3.22-tests",

    "4.3.22-cluster-kube-apiserver-operator",

    "4.3.22-baremetal-machine-controllers",

    "4.3.22-ironic-machine-os-downloader",

    "4.3.22-docker-builder",

    "4.3.22-ironic-inspector",

    "4.3.22-cluster-dns-operator",

    "4.3.22-cluster-samples-operator",

    "4.3.22-machine-config-operator",

    "4.3.22-prometheus-operator",

    "4.3.22-ironic-static-ip-manager",

    "4.3.22-machine-api-operator",

    "4.3.22-cluster-node-tuned",

    "4.3.22-cloud-credential-operator",

    "4.3.22-configmap-reloader",

    "4.3.22-cluster-network-operator",

    "4.3.22-cluster-config-operator",

    "4.3.22-coredns",

    "4.3.22-cluster-autoscaler-operator",

    "4.3.22-kube-client-agent",

    "4.3.22-kube-proxy",

    "4.3.22-cluster-authentication-operator",

    "4.3.22-prometheus-config-reloader",

    "4.3.22-installer",

    "4.3.22-ironic-ipa-downloader",

    "4.3.22-cluster-kube-scheduler-operator",

    "4.3.22-haproxy-router",

    "4.3.22-ironic",

    "4.3.22-docker-registry",

    "4.3.22-multus-route-override-cni",

    "4.3.22-cluster-node-tuning-operator",

    "4.3.22-operator-registry",

    "4.3.22-k8s-prometheus-adapter",

    "4.3.22-console-operator",

    "4.3.22-telemeter",

    "4.3.22-aws-machine-controllers",

    "4.3.22-service-ca-operator",

    "4.3.22-baremetal-operator",

    "4.3.22-kube-state-metrics",

    "4.3.22-openshift-state-metrics",

    "4.3.22-azure-machine-controllers",

    "4.3.22-operator-marketplace",

    "4.3.22-openshift-apiserver",

    "4.3.22-libvirt-machine-controllers",

    "4.3.22-mdns-publisher",

    "4.3.22-cli",

    "4.3.22-baremetal-installer",

    "4.3.22-insights-operator",

    "4.3.22-cluster-svcat-apiserver-operator",

    "4.3.22-service-catalog",

    "4.3.22-must-gather",

    "4.3.22-openstack-machine-controllers",

    "4.3.22-operator-lifecycle-manager",

    "4.3.22-cluster-ingress-operator",

    "4.3.22-installer-artifacts",

    "4.3.22-cluster-openshift-controller-manager-operator",

    "4.3.22-kube-rbac-proxy",

    "4.3.22-cluster-openshift-apiserver-operator",

    "4.3.22-pod",

    "4.3.22-ppc64le",

    "4.3.22-console",

    "4.3.22-cluster-storage-operator",

    "4.3.22-kuryr-controller",

    "4.3.22-cluster-kube-controller-manager-operator",

    "4.3.22-oauth-server",

    "4.3.22-cluster-autoscaler",

    "4.3.22-prometheus-alertmanager",

    "4.3.22-local-storage-static-provisioner",

    "4.3.22-container-networking-plugins",

    "4.3.22-prometheus",

    "4.3.22-kube-etcd-signer-server",

    "4.3.22-gcp-machine-controllers",

    "4.3.22-machine-os-content",

    "4.3.22-multus-whereabouts-ipam-cni",

    "4.3.22-baremetal-runtimecfg",

    "4.3.22-jenkins",

    "4.3.22-cli-artifacts",

    "4.3.22-grafana",

    "4.3.22-cluster-policy-controller",

    "4.3.22-hyperkube",

    "4.3.22-cluster-machine-approver",

    "4.3.22-cluster-update-keys",

    "4.3.22-prom-label-proxy",

    "4.3.22-cluster-monitoring-operator",

    "4.3.22-sdn",

    "4.3.22-cluster-bootstrap",

    "4.3.22-cluster-version-operator",

    "4.3.22-etcd",

    "4.3.22-ironic-hardware-inventory-recorder",

    "4.3.22-jenkins-agent-maven",

    "4.3.22-multus-cni",

    "4.3.22-cluster-image-registry-operator",

    "4.3.22-cluster-svcat-controller-manager-operator",

    "4.3.22-openshift-controller-manager",

    "4.3.22-kuryr-cni"

  ]

}

在bastion节点上用redhat 部署工具ocp4_helpnoder安装配置相关服务DNS和DHCP ，HAProxy

1  安装ansible和git

yum install ansible git

注意要使用新版本的ansible，要大于2.9.6

2 

git clone https://github.com/RedHatOfficial/ocp4-helpernode

cd ocp4-helpernode

3 cp docs/examples/vars-ppc64le.yaml .

配置文件修改如下

---

disk: sda

helper:

  name: "helper"

  ipaddr: "172.16.167.175"

dns:

  domain: "example.com"

  clusterid: "ocp4"

  forwarder1: "172.16.167.175"

  forwarder2: "172.16.167.175"

dhcp:

  router: "172.16.167.1"

  bcast: "172.16.167.255"

  netmask: "255.255.255.0"

  poolstart: "172.16.167.177"

  poolend: "172.16.167.183"

  ipid: "172.16.167.0"

  netmaskid: "255.255.255.0"

bootstrap:

  name: "bootstrap"

  ipaddr: "172.16.167.177"

  macaddr: "56:6f:e5:d2:00:14"

masters:

  - name: "master1"

    ipaddr: "172.16.167.178"

    macaddr: "56:6f:e5:d2:00:15"

  - name: "master2"

    ipaddr: "172.16.167.179"

    macaddr: "56:6f:e5:d2:00:16"

  - name: "master3"

    ipaddr: "172.16.167.180"

    macaddr: "56:6f:e5:d2:00:17"

workers:

  - name: "worker1"

    ipaddr: "172.16.167.182"

    macaddr: "56:6f:e5:d2:00:18"

  - name: "worker2"

    ipaddr: "172.16.167.183"

    macaddr: "56:6f:e5:d2:00:19"

ppc64le: true

ocp_bios: "file:///home/ocp4.3/rhcos-4.3.18-ppc64le-metal.ppc64le.raw.gz"

ocp_initramfs: "file:///home/ocp4.3/rhcos-4.3.18-installer-initramfs.ppc64le.img"

ocp_install_kernel: "file:///home/ocp4.3/rhcos-4.3.18-installer-kernel-ppc64le"

ocp_client: "file:///home/ocp4.3/openshift-client-linux-4.3.22.tar.gz"

ocp_installer: "file:///home/ocp4.3/openshift-install-linux-4.3.22.tar.gz"

注：

rhcos-4.3.18-ppc64le-metal.ppc64le.raw.gz

rhcos-4.3.18-installer-initramfs.ppc64le.img

rhcos-4.3.18-installer-kernel-ppc64le

可以在下面链接下载

https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos/latest/latest/

4 执行

ansible-playbook -e @vars.yaml tasks/main.yml

执行完成后 

检查服务状态

helpernodecheck 

helpernodecheck services

在named里面添加一条解析registry的dns的A记录

vim /var/named/zonefile.db

$TTL 1W

@  IN  SOA ns1.ocp4.poc.com.  root (

            2020042803  ; serial

            3H      ; refresh (3 hours)

            30M    ; retry (30 minutes)

            2W      ; expiry (2 weeks)

            1W )        ; minimum (1 week)

    IN  NS  ns1.ocp4.poc.com.

    IN  MX 10  smtp.ocp4.poc.com.

;

;

ns1 IN  A  172.31.20.100

smtp    IN  A  172.31.20.100

;

helper  IN  A  172.31.20.100

registry    IN  A  172.31.20.100

;

重启systemctl restart named

5 开始安装

5.1 创建Ignition文件

mkdir ~/ocp4

cd ~/ocp4

mkdir -p ~/.openshift

查看helper_rsa文件是否存在

现在的ansible会帮助我们创建密钥文件

[root@bastion ocp4]# ls -l ~/.ssh/helper_rsa

-rw-------. 1 root root 3243 Jun  5 02:28 /root/.ssh/helper_rsa

准备创建install-config文件

首先获取每台虚拟机的mac地址

如果是libvirt 使用virt命令创建的虚拟机获取虚拟机mac地址的脚本

for i in bootstrap master{0..2} worker{0..1}

do

  echo -ne "${i}\t" ; virsh dumpxml ocp4-${i} | grep "mac address" | cut -d\' -f2

done

详细参考https://github.com/RedHatOfficial/ocp4-helpernode/blob/master/docs/quickstart-ppc64le.md

[root@bastion ocp4]# cd ~/ocp4

执行命令创建install-config文件

cat <<EOF > install-config.yaml

apiVersion: v1

baseDomain: example.com

compute:

- hyperthreading: Enabled

  name: worker

  replicas: 0

controlPlane:

  hyperthreading: Enabled

  name: master

  replicas: 3

metadata:

  name: ocp4

networking:

  clusterNetworks:

  - cidr: 10.254.0.0/16

    hostPrefix: 24

  networkType: OpenShiftSDN

  serviceNetwork:

  - 172.30.0.0/16

platform:

  none: {}

pullSecret: '$(< ~/pull-secret)'

sshKey: '$(< ~/.ssh/helper_rsa.pub)'

EOF

需要更新install-config.yaml中的additionalTrustBundle,和/opt/registry/certs/domain.crt一致

注意additionalTrustBundle证书那里需要空两格。就是  -----BEGIN CERTIFICATE-----开始下面的内容 如下

additionalTrustBundle: |

  -----BEGIN CERTIFICATE-----

  MIIGBTCCA+2gAwIBAgIJAPgD7df5XhjgMA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD

  VQQGEwJjbjEQMA4GA1UECAwHYmVpamluZzEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5

  MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBhbnkgTHRkMSIwIAYDVQQDDBlyZWdpc3Ry

  eS5vY3A0LmV4YW1wbGUuY29tMR4wHAYJKoZIhvcNAQkBFg9ub2VhbWlsQGlibS5j

  b20wHhcNMjAwNjA1MTU1NzUwWhcNMjEwNjA1MTU1NzUwWjCBmDELMAkGA1UEBhMC

  Y24xEDAOBgNVBAgMB2JlaWppbmcxFTATBgNVBAcMDERlZmF1bHQgQ2l0eTEcMBoG

  A1UECgwTRGVmYXVsdCBDb21wYW55IEx0ZDEiMCAGA1UEAwwZcmVnaXN0cnkub2Nw

  NC5leGFtcGxlLmNvbTEeMBwGCSqGSIb3DQEJARYPbm9lYW1pbEBpYm0uY29tMIIC

  IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwXdl5pAczvLWw1kTorE1ih/v

  bdtVzV5dw4QSq59Z2C9g4zBwmqADADCfDsCMSnWb4ZjezSdk9OrlnYQVmHZERr8F

  0rVuP+q9RPCZSRz9SREBH5wIyiYMYFpSkvn0y8wt7kAN2QGduht1kzVmfE7689xr

  uR80BnSGC9rPPGPrv7r7IifrB+UeAaSdot6oA/5XrTsyHUi4fF23iis6ZBEHzYCZ

  bRKdtWsb6zf81F6SL52ot1hgXV9b4/TD+NNqQ81wf7fF53tzV0zjN4UvtRYniluJ

  ex5QL0fOjFaT/hScYKDDI13sGIEZVXB8BUuFidpUtHwKTvoUBJmzvHU7VQn/jfS8

  F7qjdYbt09FGDAghj8LCwngT+eZzP5SoIDTtsjZ2ahTGZ1XmBIOGHrmcghr+xm+X

  7iJaYJbMpAMja85hzAzqUf0aTOTVzuRLvQp3xgN7j4/J2hMXQXjW+5N/mMXuZABh

  5c3udlnS2AlHaQkdZDD1qr74XcWiP43y5/OvFWHiE5RvqTWhwf54Oldcz12kreYB

  kvLBFgVDuLUxYYvDvB2zzwP86JVIrCUujJ5HjcD9E8D8gTDRApCFxRUtti5KUF/R

  /sr3ladiejRIjsPlI/KVvyxxeyF5GvoiE2JT6Ka6foXEoampzHgMZ8zPGYNtgcDR

  I0L3tQ9+epTjLoyLgdECAwEAAaNQME4wHQYDVR0OBBYEFEVYgD1VekTKSim8KJIv

  zjpRuVJfMB8GA1UdIwQYMBaAFEVYgD1VekTKSim8KJIvzjpRuVJfMAwGA1UdEwQF

  MAMBAf8wDQYJKoZIhvcNAQELBQADggIBAGe53qwPZX1z4RNG3tbbtWsIKqDdWn4u

  +mfvXNy6YPdFJuoVyGYkkMyH7wVJVRhlnnPkOM/O8HR7dHX6J0Pt/hByNVK0WDGm

  UddcAUvhWlu81njRlupLQKMHRRngSE67kzunfygAHhFOW0zLAZnjzUyZEfItOhnj

  K+jqqhv0KLsZHzIM6QHUooLKcRXVTnBaVw2nhNrgC/RAIHNL7Ku1KJQxhrPpgi61

  TfaBjWXNY8ZWYQJMpJpDUurZUIa1NFalVxBau3DnE39nBwEb2OJVSJmsPwxaJCyn

  M8efSm8O+atjX77xB2SBVIdwgDJMzt7uZe0xRZPBK9L+t0kB4DGa2aOrJlOm09mK

  bfd1vAU6EMNn7WlfLAh3XUNBaAuQywvgGNecoe+Q3eElVF3mqEYdlDh2EqgK+v+r

  pKO9QUPbe5I4WloWgWHxh2/xi5RdtQTR7ujDFlbtHYQRRZG4N6+LSV8KUd79EhOz

  uokrPcoPuYpS0x6HzQ7ruN7N0pWuwoCMzIvDDA4DVsUqXREZC3HduBdEGUA779IW

  7nHdRzKZXSnMowfcjK4sok1FWFkvnVvHYbPMv0ckGcJxjQBy+ncIQh+UFVks82b+

  pZbN7r4iP6qmqp51v0SucRU2/EZ4KYW7oBw11oywAzvNlmmaqMrzDoUc3CDOESlV

  aEOt96JrTYen

  -----END CERTIFICATE-----

imageContentSources:

- mirrors:

  - registry.ocp4.example.com:5000/ocp4/openshift4

  source: quay.io/openshift-release-dev/ocp-release

- mirrors:

  - registry.ocp4.example.com:5000/ocp4/openshift4

  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

创建manifests

openshift-install create manifests

将master设置为不可调度

sed-i's/mastersSchedulable:true/mastersSchedulable:false/g' manifests/cluster-scheduler-02-config.yml

创建ignition文件：

[root@bastion ocp4]# openshift-install create ignition-configs

INFO Consuming Worker Machines from target directory

INFO Consuming Common Manifests from target directory

INFO Consuming Master Machines from target directory

INFO Consuming OpenShift Install (Manifests) from target directory

INFO Consuming Openshift Manifests from target directory

放到对应路径：

cp~/ocp4/*.ign /var/www/html/ignition/

restorecon -vR /var/www/html/

chmod o+r /var/www/html/ignition/*.ign

开始安装

openshift-install wait-for bootstrap-complete --log-level debug

在虚拟机管理界面使用以PXE启动虚拟机

在HAproxy界面可以观测节点状态，绿色表示正常，红色表示还未正常

http://172.16.167.175:9000

命令行观察安装过程

openshift-install wait-for bootstrap-complete --log-level debug

openshift-install wait-for install-complete

安装完成后准备配置文件

export  KUBECONFIG=/root/ocp4/auth/kubeconfig

# 或      cp~/ocp4/auth/kubeconfig ~/.kube/config

查看节点状态

oc get node

Web Console 登录

ocp4的web console 入口走router了，所以找下域名

首先找到我们的域名，然后在我们自己电脑上 hosts添加解析，指向到worker-0，这样就能够访问openshift 的web 控制台了

oc get route --all-namespaces |grep console-openshift

把172.16.167.175 oauth-openshift.apps.ocp4.example.com console-openshift-console.apps.ocp4.example.com 写入hosts文件

然后浏览器访问console

https://console-openshift-console.apps.ocp4.example.com

用户名是 kubeadmin

密码在这个文件里

cat /root/ocp4/auth/kubeadmin-password

或者通过下面命令拿到集群登录信息

openshift-install wait-for install-complete

trouble shooting

1 安装过程中master节点和work 节点使用的虚拟机配置文件是GMT+8 时区

RHCOS在PXE安装后是UTC时区

所以在安装基础架构节点的时候操作系统要设置时区为UTC

timedatectl  set-timezone UTC

注：各节点时间一定要一致

2  证书过期或者需要重新安装的时候

执行下面的命令 ，重新生成文件

rm  -rf ~/ocp4

openshift-install create ignition-configs

生成文件后重新复制到相关路径覆盖原文件

cp ~/ocp4/*.ign /var/www/html/ignition/

restorecon -vR /var/www/html/

chmod o+r /var/www/html/ignition/*.ign 

再次以PXE方式启动虚拟机开始重新安装

3 debug bootstrap

在bastion2节点上登录bootstrap节点

ssh core@bootstrap

# 检查方法:

# 查看网络信息

hostname -I

ip route show

cat /etc/resolv.conf

# 查看运行容器，sudo podman ps, 应该有运行中容器

# 如果没有在运行的容器， ps -ef 看下是否有podman pull 的进程

# 通过curl命令检查跟镜像仓库的连通性，我遇到过连不上仓库，在部署机上重启了下仓库后好了

sudo podman ps

sudo ps -ef

# 查看进程端口

netstat -an 查看 6443、22623

# 也可以通过haproxy的页面，就是192.168.2.10:9000 可以看到bootstrap的状态变成了绿色

# 说明这个时候bootstrap 已经部署成功

# 查看服务状态的命令，ssh进去的时候就会提示这条命令

journalctl -b -f -u bootkube.service

参考链接

https://www.jianshu.com/p/0b4eadc854a4

https://www.jianshu.com/p/11dc00f1d93f

https://www.talkwithtrend.com/Article/246491

https://www.cnblogs.com/ericnie/p/11764124.html

https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.2.disconnect.install.md