前言

本环境采用kubeadm安装(为了简单,非二进制环境安装),master节点未做高可用.经典的1master/2node模式.

dawdler-series 终究还是要是要容器化.大势所趋所以被迫搞一下子.

全球第一个关于buildkit+nexus私有库实现的部署文档.

1.环境准备

1.1 四台虚拟机

操作系统	cpu	内存大小	ip地址	描述
Fedora-Server-dvd-x86_64-36-1.5	4u	4G	192.168.43.144	k8s-master
Fedora-Server-dvd-x86_64-36-1.5	4u	4G	192.168.43.146	k8s-node1
Fedora-Server-dvd-x86_64-36-1.5	4u	4G	192.168.43.147	k8s-node2
Fedora-Server-dvd-x86_64-36-1.5	4u	8G	192.168.43.137	nexus

1.2 设置基础环境

三台k8s的主机分别做以下设置

1.2.1 设置host

vim /etc/hosts

192.168.43.144 k8s-master1 192.168.43.146 k8s-node1 192.168.43.147 k8s-node2 192.168.43.137 my.nexus.org 

也可以用 hostnamectl set-hostname k8s-master1

1.2.2 关闭防火墙与selinux

systemctl stop firewalld   systemctl disable firewalld   setenforce 0  #表示临时关闭selinux防火墙 

永久关闭selinux vim /etc/selinux/config

SELINUX=disabled

1.2.3 LVS的管理工具ipvsadm ipset

yum install -y ipvsadm  ipset 

1.2.4 安装 iproute

yum install iproute-tc.x86_64 -y 

1.2.5 时间同步

yum install chrony -y   systemctl enable chronyd   systemctl start chronyd  chronyc sources  

1.2.6 设置k8s.conf 二层网络走FORWARD

vim /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1  net.bridge.bridge-nf-call-iptables = 1  net.ipv4.ip_forward = 1  vm.swappiness=0 #vm.swappiness是操作系统控制物理内存交换出去的策略.它允许的值是一个百分比的值,范围在0-100，该值默认为60.vm.swappiness设置为0表示尽量少swap,100表示尽量将inactive的内存页做交换. 

sysctl -p /etc/sysctl.d/k8s.conf 

1.2.7 关闭swap

修改/etc/fstab文件,注释掉 SWAP 的自动挂载.

swapoff -a  sed -i '/swap/s/^/#/' /etc/fstab  systemctl mask dev-zram0.swap  sysctl -p 

使用free -m确认 swap 已经关闭.

1.2.8 内核加载所需模块

mkdir /etc/sysconfig/modules/ 

vim /etc/sysconfig/modules/ipvs.modules

modprobe -- ip_vs  modprobe -- ip_vs_rr  modprobe -- ip_vs_wrr  modprobe -- ip_vs_sh  modprobe -- nf_conntrack modprobe br_netfilter  

使用lsmod | grep -e ip_vs -e nf_conntrack_命令查看是否已经正确加载所需的内核模块.

改变权限并加载ipvs模块

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack 

1.2.9 安装yum和lvm2

 yum install -y yum-utils device-mapper-persistent-data lvm2 

1.2.10 添加docker源

yum-config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo   yum install containerd.io.x86_64 -y  

1.2.11 创建containerd配置文件

mkdir -p /etc/containerd  containerd config default > /etc/containerd/config.toml 

如果不能正常访问互联网(你懂得)的同学则需要替换配置文件

sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g"  /etc/containerd/config.toml sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g"  /etc/containerd/config.toml 

启动Containerd 并设置自启动

systemctl daemon-reload  systemctl enable containerd  systemctl restart containerd 

1.2.12 加入kubernetes源

vim /etc/yum.repos.d/kubernetes.repo

不能正常访问互联网(你懂得)的同学用这个源即可

[kubernetes]  name=Kubernetes  baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64  enabled=1  gpgcheck=0  repo_gpgcheck=0  gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg  

可以正常访问互联网(你懂得)的同学建议用下面的源

[kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg 

1.2.13 安装kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl  systemctl daemon-reload   systemctl enable kubelet && systemctl start kubelet 

1.2.14 设置运行时的endpoint

crictl config runtime-endpoint /run/containerd/containerd.sock 

1.2.15 设置~/.docker/config.json

安装nexus之前可以跳过此步骤,安装之后再配置.

mkdir ~/.docker 

通过base64输出nexus账号密码

echo -n "admin:123456" | base64 

输出: YWRtaW46MTIzNDU2

vim ~/.docker/config.json

{   "auths": {     "https://my.nexus.org:8083": {       "auth": "YWRtaW46MTIzNDU2"     }   } } 

2. kubeadm初始化k8s环境

在master节点执行

2.1 查看kubeadm 需要的镜像列表(可选)

 kubeadm config images list  

2.2 拉取kubeadm需要的镜像列表

可以正常访问互联网(你懂得)的可以去除–image-repository

 kubeadm config images pull --kubernetes-version=v1.24.2 --image-repository=registry.aliyuncs.com/google_containers 

2.3 kubeadm初始化

命令行初始化 可以正常访问互联网(你懂得)的可以去除–image-repository

kubeadm init      --apiserver-advertise-address=192.168.43.144      --image-repository registry.aliyuncs.com/google_containers      --pod-network-cidr=10.244.0.0/16      --service-cidr=10.1.0.0/16      --kubernetes-version=v1.24.2 

配置文件初始化方式

kubeadm config print init-defaults > kubeadm.yaml  

编辑 kubeadm.yaml

不可以正常访问互联网(你懂得)的同学请替换 registry.cn-hangzhou.aliyuncs.com/google_containers

kubeadm init --config=kubeadm.yaml 

执行完成之后会有以下输出

kubeadm join 192.168.43.144:6443 --token ds62l9.scyt0ise8k9qakae          --discovery-token-ca-cert-hash sha256:c1d0577681ad0c5dacd7814fc60afd71fe9adb33b3c6168bf93854645d33e33e 

2.4 开启ipvs

设置ConfigMap的kube-system/kube-proxy中的config.conf配置mode: “ipvs”

kubectl edit cm kube-proxy -n kube-system 

删除pod(会重新启动新的)

kubectl get pod -n kube-system |grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}' 

3. node节点加入k8s集群

运行kubeadm init输出

在node节点中执行

kubeadm join 192.168.43.144:6443 --token ds62l9.scyt0ise8k9qakae          --discovery-token-ca-cert-hash sha256:c1d0577681ad0c5dacd7814fc60afd71fe9adb33b3c6168bf93854645d33e33e 

3.1 查看集群状态

在master中执行

kubectl get nodes 

此时可以在列表中看到k8s-node1、k8s-node2 状态为NotReady.(因为未安装网络插件)

4. 安装calico插件

k8s网络插件有很多,推荐用calico.使用其他的如Flannel请自行google.

在master中执行

wget https://docs.projectcalico.org/manifests/calico.yaml  kubectl apply -f calico.yaml 

安装完成之后可以看下集群状态,状态NotReady变为Ready.

如果有问题请自行排查,如:

kubectl get pod -n kube-system 

5. 配置自动补全命令行(可选)

在master中执行

yum install -y bash-completion  source /usr/share/bash-completion/bash_completion  source <(kubectl completion bash)  echo "source <(kubectl completion bash)" >> ~/.bashrc  

6. 部署nginx并测试

在master中执行

6.1 通过kubectl运行nginx绑定80端口

kubectl run web-nginx --image=nginx --port=80 

6.2 验证nginx

查看镜像列表(默认namespace为default)

kubectl get pods -owide 

响应如下：

NAME        READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES web-nginx   1/1     Running   0          20s     172.17.104.30   node2   <none>           <none> 

wget 172.17.104.30 

响应的index.html 通过cat可以查看为nginx的首页

6.3 通过expose暴露端口在node

kubectl expose pod web-nginx --name=web-nginx  --port=80  --type=NodePort 

查看服务列表

kubectl get service -owide 

响应结果

NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE    SELECTOR kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        7d1h   <none> web-nginx    NodePort    10.96.181.25   <none>        80:32002/TCP   5s     run=web-nginx 

查看所在node

 kubectl get pods -A -owide 

通过 访问 http://nodeIp:32002/ 可验证nginx是否启动成功.

7. 制作证书

7.1 生成caKey与证书

#输入密码 密码要记下来 例: password openssl genrsa -des3 -out ca.key  2048   #输入上面输入的密码 后面按需求输入 openssl req -x509 -key ca.key -out ca.crt 

7.2 pkiCa配置

mkdir -p /etc/pki/CA/newcerts   touch /etc/pki/CA/index.txt   echo '01' > /etc/pki/CA/serial 

7.3 生成nexus证书key

#输入密码 密码要记下来 例: password openssl genrsa -des3 -out nexus.key 2048   

7.4 生成证书请求文件

#输入nexus证书key的密码 后面按需求输入(与ca.crt一致) openssl req -new -key nexus.key -reqexts SAN -config <(cat /etc/pki/tls/openssl.cnf          <(printf "[SAN]nsubjectAltName=DNS:my.nexus.org,DNS:www.nexus.org"))     -out nexus.csr 

7.5 自签署证书

openssl x509 -req -days 3650      -in nexus.csr -out nexus.crt      -CA ca.crt -CAkey ca.key -CAcreateserial      -extensions SAN      -extfile <(cat /etc/pki/tls/openssl.cnf          <(printf "[SAN]nsubjectAltName=DNS:my.nexus.org,DNS:www.nexus.org")) 

注意: 如果多次签署报错请清理通过 vim etc/pki/CA/index.txt编辑文件删除文件内容.

7.6 添加证书信任

三台机器(k8s)全部需要执行相同步骤,ca.crt复制到其他节点上即可.

chmod 644 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem          cat ca.crt >> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem  chmod 444 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 

cp nexus.crt  /etc/pki/ca-trust/source/anchors/my.nexus.org.crt  update-ca-trust 

7.7 导出pfx文件

openssl pkcs12 -export -out nexus.pfx -in nexus.crt -inkey nexus.key 

7.8 通过java代码生成keystore.jks

import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.security.Key; import java.security.KeyStore; import java.security.cert.Certificate; import java.util.Enumeration;  public class CoverP12TokeyStore {  public static final String PKCS12 = "PKCS12";  public static final String JKS = "JKS";  public static final String PFX_KEYSTORE_FILE = "/home/srchen/my.pfx";// pfx文件位置  public static final String PFX_PASSWORD = "password";// 导出为pfx文件的设的密码  public static final String JKS_KEYSTORE_FILE = "/home/srchen/software/nexus/nexus-3.40.1-01/etc/ssl/keystore.jks"; // jks文件位置  public static final String JKS_PASSWORD = "password";// JKS的密码   public static void coverTokeyStore() {   FileInputStream fis = null;   FileOutputStream out = null;   try {    KeyStore inputKeyStore = KeyStore.getInstance("PKCS12");    fis = new FileInputStream(PFX_KEYSTORE_FILE);    char[] pfxPassword = null;    if ((PFX_PASSWORD == null) || PFX_PASSWORD.trim().equals("")) {     pfxPassword = null;    } else {     pfxPassword = PFX_PASSWORD.toCharArray();    }    char[] jksPassword = null;    if ((JKS_PASSWORD == null) || JKS_PASSWORD.trim().equals("")) {     jksPassword = null;    } else {     jksPassword = JKS_PASSWORD.toCharArray();    }     inputKeyStore.load(fis, pfxPassword);    fis.close();    KeyStore outputKeyStore = KeyStore.getInstance("JKS");    outputKeyStore.load(null, jksPassword);    Enumeration<String> enums = inputKeyStore.aliases();    while (enums.hasMoreElements()) {     String keyAlias = enums.nextElement();     if (inputKeyStore.isKeyEntry(keyAlias)) {      Key key = inputKeyStore.getKey(keyAlias, pfxPassword);      Certificate[] certChain = inputKeyStore.getCertificateChain(keyAlias);      outputKeyStore.setKeyEntry(keyAlias, key, jksPassword, certChain);     }    }    out = new FileOutputStream(JKS_KEYSTORE_FILE);    outputKeyStore.store(out, jksPassword);    out.flush();   } catch (Exception e) {    e.printStackTrace();   } finally {    if (fis != null) {     try {      fis.close();     } catch (IOException e) {      e.printStackTrace();     }    }    if (out != null) {     try {      out.close();     } catch (IOException e) {      e.printStackTrace();     }    }   }  }   public static void main(String[] args) {   coverTokeyStore();  } } 

8. 私有镜像库搭建

8.1 下载nexus

wget https://download.sonatype.com/nexus/3/nexus-3.40.1-01-unix.tar.gz  tar -zxvf nexus-3.40.1-01-unix.tar.gz 

8.2 启动nexuss

cd nexus-3.40.1-01/bin  ./nexus run & 

启动完成之后密码生成在sonatype-work/nexus3/admin.password文件中.

访问http://192.168.43.137:8081/ 输入密码即可.

8.3 开启https服务

由于后面采用buildkit来构建镜像推送到私有库中,只能采用https方式,所以需要开启nexus的https服务.

进入nexus配置目录.

cd /home/srchen/software/nexus/nexus-3.40.1-01/etc 

编辑nexus-default.properties配置文件,添加${jetty.etc}/jetty-https.xml.

exus-args=${jetty.etc}/jetty.xml,${jetty.etc}/jetty-https.xml,${jetty.etc}/jetty-http.xml,${jetty.etc}/jetty-requestlog.xml 

编辑/home/srchen/software/nexus/nexus-3.40.1-01/etc/jetty/jetty-https.xml,配置上面设置的jks的密码.

<New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">           <Set name="KeyStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>     <Set name="KeyStorePassword">password</Set>     <Set name="KeyManagerPassword">password</Set>     <Set name="TrustStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>     <Set name="TrustStorePassword">password</Set>     <Set name="EndpointIdentificationAlgorithm"></Set>     <Set name="NeedClientAuth"><Property name="jetty.ssl.needClientAuth" default="false"/></Set>     <Set name="WantClientAuth"><Property name="jetty.ssl.wantClientAuth" default="false"/></Set>     <Set name="IncludeProtocols">       <Array type="java.lang.String">         <Item>TLSv1.2</Item>       </Array>     </Set>   </New> 

重启nexus

./nexus restart 

8.4 开启containerd私有库的支持

登陆 http://192.168.43.137:8081/

访问 http://192.168.43.137:8081/#admin/repository/repositories

点击 Create repository、docker(hosted),

Name输入 Containerd

Online 选中

HTTPS 输入8083端口

剩下全部选中,如果需要清理策略请自行配置.

保存之后会开启https的支持端口为8083.

9. buildkit安装

buildkit 号称下一代构建docker的神器.

在master主机上下载

wget https://github.com/moby/buildkit/releases/download/v0.10.3/buildkit-v0.10.3.linux-arm64.tar.gz  

解压并移到local下

tar -zxvf buildkit-v0.10.3.linux-arm64.tar.gz  mkdir -p  /usr/local/buildkit-v0.10.3  mv bin /usr/local/buildkit-v0.10.3 

vim /etc/profile  #加入 PATH=$PATH:/usr/local/buildkit/bin  source /etc/profile 

10. 配置buildkit

mkdir /etc/buildkit 

vim /etc/buildkit/buildkitd.toml

debug = true [worker.containerd]   namespace = "k8s.io"  [registry."docker.io"]   mirrors = ["my.nexus.org:8083"]   http = false   nsecure = true   ca=["/home/srchen/keys/ca.key"]    [registry."my.nexus.org:8083"]   http = false   insecure = true   ca=["/home/srchen/keys/ca.key"]   [[registry."my.nexus.org".keypair]]     key="/home/srchen/keys/nexus.key"     cert="/home/srchen/keys/nexus.crt" 

10.1 启动buildkitd服务

buildkitd --oci-worker=false --containerd-worker=true &  

10.2 构建自定义镜像

通过dawdler-series来举例子.

wget https://github.com/srchen1987/dawdler-runtime/archive/refs/tags/0.0.2-RELEASES.tar.gz  tar -zxvf 0.0.2-RELEASES.tar.gz  cd dawdler-runtime-0.0.2-RELEASES  

vim Dockerfile

FROM suranagivinod/openjdk8 MAINTAINER suxuan696@gmail.com RUN mkdir /opt/dawdler-server-0.0.2 ADD ./dawdler-server-0.0.2 /opt/dawdler-server-0.0.2 WORKDIR /opt/dawdler-server-0.0.2/bin ENTRYPOINT ["/bin/sh","dawdler.sh","run"] 

10.3 构建并推送到私有镜像库

buildctl build   --no-cache      --frontend=dockerfile.v0      --local context=.      --local dockerfile=.      --output type=image,name=my.nexus.org:8083/dawdler:0.0.2,push=true      --export-cache type=inline 

10.4 通过kubectl发布dawdler服务

这里只是演示所以采用 run方式,如果是生产环境建议采用 deployment或apply -f dawdler.yaml方式来创建.

注意: 用run方式运行 node节点中镜像如果没更新则需要通过以下脚本清理.

#查看列表 crictl images  # 449cdf80490b5是 images查看到容器ID crictl rmi 449cdf80490b5 

也可以通过 imagePullPolicy: Always 配置每次私有仓拉取镜像.

kubectl run dawdler --image=my.nexus.org:8083/dawdler:0.0.2 

10.5 查看服务列表

kubectl get pods -n default -owide 

响应

NAME        READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES dawdler     1/1     Running   0          11h     172.17.104.28   node2   <none>           <none> 

10.6 查看日志

kubectl logs dawdler 

响应

Welcome to use dawdler!    _____              __          __  _____    _        ______   _____  |  __       /              / / |  __   | |      |  ____| |  __   | |  | |    /          /  / /  | |  | | | |      | |__    | |__) |  | |  | |   / /       /  / /   | |  | | | |      |  __|   |  _  /  | |__| |  / ____        /  /    | |__| | | |____  | |____  | |    |_____/  /_/    _     /  /     |_____/  |______| |______| |_|  _   OS arch: amd64 OS availableProcessors: 4 OS name: Linux OS version: 5.17.5-300.fc36.x86_64 Jvm totalMemory: 5.20093696E8(507904.00K) Jvm freeMemory: 5.01973992E8(490208.98K) Heap Memory Usage: init = 536870912(524288K) used = 18119704(17695K) committed = 520093696(507904K) max = 520093696(507904K) Non-Heap Memory Usage: init = 2555904(2496K) used = 11309544(11044K) committed = 11730944(11456K) max = -1(-1K) Java options: [-Xms512m, -Xmx512m, -Xmn128m] ClassPath: .:./asm-7.1.jar:./cglib-3.3.0.jar:./commons-jexl3-3.2.jar:./commons-logging-1.2.jar:./commons-net-3.6.jar:./commons-pool2-2.10.0.jar:./consul-api-1.4.5.jar:./curator-client-5.1.0.jar:./curator-framework-5.1.0.jar:./curator-recipes-5.1.0.jar:./dawdler-core-0.0.2-RELEASES.jar:./dawdler-serialization-0.0.2-RELEASES.jar:./dawdler-server-0.0.2-RELEASES.jar:./dawdler-util-0.0.2-RELEASES.jar:./dom4j-2.1.3.jar:./jaxen-1.2.0.jar:./kryo-4.0.2.jar:./logback-access-1.2.7.jar:./logback-classic-1.2.7.jar:./logback-core-1.2.7.jar:./minlog-1.3.0.jar:./objenesis-2.5.1.jar:./reflectasm-1.11.3.jar:./slf4j-api-1.7.32.jar:./zookeeper-3.6.2.jar:./zookeeper-jute-3.6.2.jar LibraryPath: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Server startup in 0 ms,Listening port: 9527!