部署kubernetesv1.25.3（k8s）基于containerd容器运行时

大家好&＃xff0c;我是秋意临。

今日分享&＃xff0c;kuberneter-v1.25.3版本部署&＃xff08;目前2022年11月最新版&＃xff09;&＃xff0c;由于自 1.24 版起&＃xff0c;Dockershim 已从 Kubernetes 项目中移除&＃xff0c;所以我们的 容器运行时&＃xff08;容器运行时负责运行容器的软件&＃xff09; 已不在是docker。本文将采用containerd作为 容器运行时。

Kubernetes 中几个常见的容器运行时。&＃xff08;具体用法见kubernetes官方文档 &＃xff09;

• containerd
• CRI-O
• Docker Engine
• Mirantis Container Runtime

本文操作配置&＃xff0c;如下&＃xff1a;

最低配置&＃xff1a;CPU核心不低于2个&＃xff0c;RAM不低于2G。

注意命令在那台节点上执行的。

修改主机名

#master节点
hostnamectl set-hostname master
bash
#node节点
hostnamectl set-hostname node
bash


配置hosts映射

cat >> /etc/hosts << EOF
192.168.200.5 master
192.168.200.6 node
EOF


关闭防火墙

systemctl stop firewalld
systemctl disable firewalld


关闭selinux

setenforce 0
sed -i "s/SELINUX&＃61;enforcing/SELINUX&＃61;disabled/g" /etc/selinux/config


关闭交换分区

为了保证 kubelet 正常工作&＃xff0c;必须禁用交换分区。

swapoff -a
sed -i &＃39;s/.*swap.*/#&/&＃39; /etc/fstab


转发 IPv4 并让 iptables 看到桥接流

为了让 Linux 节点的 iptables 能够正确查看桥接流量&＃xff0c;请确认 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 设置为 1。

#转发 IPv4 并让 iptables 看到桥接流量
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
lsmod | grep br_netfilter #验证br_netfilter模块
# 设置所需的 sysctl 参数&＃xff0c;参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables &＃61; 1
net.bridge.bridge-nf-call-ip6tables &＃61; 1
net.ipv4.ip_forward &＃61; 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system


配置 时间同步

#删除centos默认repo包&＃xff0c;配置阿里云Centos-7.repo包
rm -rf /etc/yum.repos.d/*
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
#方式1&＃xff1a;安装配置chrony时间同步
IP&＃61;&＃96;ip addr | grep &＃39;state UP&＃39; -A2 | grep inet | egrep -v &＃39;(127.0.0.1|inet6|docker)&＃39; | awk &＃39;{print $2}&＃39; | tr -d "addr:" | head -n 1 | cut -d / -f1&＃96;
yum install -y chrony
sed -i &＃39;3,6s/^/#/g&＃39; /etc/chrony.conf
sed -i "7s|^|server $IP iburst|g" /etc/chrony.conf
echo "allow all" >> /etc/chrony.conf
echo "local stratum 10" >> /etc/chrony.conf
systemctl restart chronyd
systemctl enable chronyd
timedatectl set-ntp true
sleep 5
systemctl restart chronyd
chronyc sources
#方式2&＃xff1a;时间同步 注意&＃xff1a;系统重启后恢复成原时间
yum install ntpdate -y
ntpdate ntp1.aliyun.com

三、安装containerd&＃xff08;所有节点操作&＃xff09;

3.1、安装containerd

下载containerd包
首先访问https://github.com/&＃xff0c;搜索containerd&＃xff0c;进入项目找到Releases&＃xff0c;下拉找到对应版本的tar包&＃xff0c;如图所示&＃xff1a;

$ tar Cvzxf /usr/local containerd-1.6.9-linux-amd64.tar.gz
# 通过 systemd 启动 containerd
$ vi /etc/systemd/system/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Description&＃61;containerd container runtime
Documentation&＃61;https://containerd.io
After&＃61;network.target local-fs.target
[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment&＃61;"ENABLE_CRI_SANDBOXES&＃61;sandboxed"
ExecStartPre&＃61;-/sbin/modprobe overlay
ExecStart&＃61;/usr/local/bin/containerd
Type&＃61;notify
Delegate&＃61;yes
KillMode&＃61;process
Restart&＃61;always
RestartSec&＃61;5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC&＃61;infinity
LimitCORE&＃61;infinity
LimitNOFILE&＃61;infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax&＃61;infinity
OOMScoreAdjust&＃61;-999
[Install]
WantedBy&＃61;multi-user.target


# 加载配置、启动
systemctl daemon-reload
systemctl enable --now containerd
# 验证
ctr version
#生成配置文件
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
systemctl restart containerd


3.2、安装runc

#下载runc地址&＃xff1a;https://github.com/opencontainers/runc/releases
# 安装
install -m 755 runc.amd64 /usr/local/sbin/runc
# 验证
runc -v


3.3、安装CNI

#下载CNI地址&＃xff1a;https://github.com/containernetworking/plugins/releases
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.1.1.tgz


3.4、配置加速器

#参考&＃xff1a;https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration
#添加 config_path &＃61; "/etc/containerd/certs.d"
sed -i &＃39;s/config_path\ &＃61;.*/config_path &＃61; \"\/etc\/containerd\/certs.d\"/g&＃39; /etc/containerd/config.toml
mkdir /etc/containerd/certs.d/docker.io -p
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server &＃61; "https://docker.io"
[host."https://vh3bm52y.mirror.aliyuncs.com"]
capabilities &＃61; ["pull", "resolve"]
EOF

systemctl daemon-reload && systemctl restart containerd

四、cgroup 驱动&＃xff08;所有节点操作&＃xff09;

在 Linux 上&＃xff0c;控制组&＃xff08;CGroup&＃xff09;用于限制分配给进程的资源。

kubelet 和底层容器运行时都需要对接控制组 为 Pod 和容器管理资源 &＃xff0c;如 CPU、内存这类资源设置请求和限制。
若要对接控制组&＃xff08;CGroup&＃xff09;&＃xff0c;kubelet 和容器运行时需要使用一个 cgroup 驱动。 关键的一点是 kubelet 和容器运行时需使用相同的 cgroup 驱动并且采用相同的配置。

#把SystemdCgroup &＃61; false修改为&＃xff1a;SystemdCgroup &＃61; true
sed -i &＃39;s/SystemdCgroup\ &＃61;\ false/SystemdCgroup\ &＃61;\ true/g&＃39; /etc/containerd/config.toml
#把sandbox_image &＃61; "k8s.gcr.io/pause:3.6"修改为&＃xff1a;sandbox_image&＃61;"registry.aliyuncs.com/google_containers/pause:3.8"
sed -i &＃39;s/sandbox_image\ &＃61;.*/sandbox_image\ &＃61;\ "registry.aliyuncs.com\/google_containers\/pause:3.8"/g&＃39; /etc/containerd/config.toml|grep sandbox_image
systemctl daemon-reload
systemctl restart containerd

五、安装crictl&＃xff08;所有节点操作&＃xff09;

kubernetes中使用crictl管理容器&＃xff0c;不使用ctr。

crictl 是 CRI 兼容的容器运行时命令行接口。 可以使用它来检查和调试 Kubernetes 节点上的容器运行时和应用程序。

#配置crictl对接ctr容器运行时。
tar -vzxf crictl-v1.25.0-linux-amd64.tar.gz
mv crictl /usr/local/bin/
cat >> /etc/crictl.yaml << EOF
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
EOF
systemctl restart containerd

六、kubeadm部署集群

6.1、安装kubeadm、kubelet、kubectl&＃xff08;所有节点操作&＃xff09;

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name&＃61;Kubernetes
baseurl&＃61;https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled&＃61;1
gpgcheck&＃61;1
repo_gpgcheck&＃61;1
gpgkey&＃61;https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install --nogpgcheck kubelet-1.25.3 kubeadm-1.25.3 kubectl-1.25.3 -y
systemctl enable kubelet


yum安装时出现错误&＃xff0c;如下&＃xff1a;


# 详情参考&＃xff1a;https://blog.csdn.net/Dan1374219106/article/details/112450922
[root&＃64;master ~]# yum install --nogpgcheck kubelet
Loaded plugins: fastestmirror
Existing lock /var/run/yum.pid: another copy is running as pid 8721.
Another app is currently holding the yum lock; waiting for it to exit...
The other application is: yum
Memory : 44 M RSS (444 MB VSZ)
Started: Fri Nov 11 20:40:32 2022 - 02:07 ago
State : Traced/Stopped, pid: 8721
#解决方法
[root&＃64;master ~]# rm -f /var/run/yum.pid


6.2、kubeadm初始化&＃xff08;master节点操作&＃xff09;


#查看我们kubeadm版本&＃xff0c;这里为GitVersion:"v1.25.3"
[root&＃64;master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:55:36Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
# 生成默认配置文件
$ kubeadm config print init-defaults > kubeadm.yaml
$ vi kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.200.5 # 修改为宿主机ip
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
name: master # 修改为宿主机名
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 修改为阿里镜像
kind: ClusterConfiguration
kubernetesVersion: 1.25.3 # kubeadm的版本为多少这里就修改为多少
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16 ## 设置pod网段
scheduler: {}
###添加内容&＃xff1a;配置kubelet的CGroup为systemd
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
#下载镜像
$ kubeadm config images pull --image-repository&＃61;registry.aliyuncs.com/google_containers --kubernetes-version&＃61;v1.25.3
#初始化
$ kubeadm init --config kubeadm.yaml
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG&＃61;/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.200.5:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:7d52da1b42af69666db3483b30a389ab143a1a199b500843741dfd5f180bcb3f


# master节点执行
[root&＃64;master ~]# mkdir -p $HOME/.kube
[root&＃64;master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root&＃64;master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# node节点执行
[root&＃64;node ~]# kubeadm join 192.168.200.5:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:7d52da1b42af69666db3483b30a389ab143a1a199b500843741dfd5f180bcb3f


# master节点执行
[root&＃64;master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 3m25s v1.25.4
node NotReady <none> 118s v1.25.4


6.3、部署网络&＃xff08;master节点操作&＃xff09;

6.3.1、说明

本博客测试时&＃xff0c;由于ctr拉取镜像特别慢&＃xff0c;所有我们这里采用docker拉取镜像。首先在node节点安装docker-ce&＃xff0c;拉取calico网络插件需要的镜像&＃xff0c;再使用docker save命令打包后上传镜像到master节点。步骤如下&＃xff1a;

注意&＃xff1a;使用calico.yaml下载地址下载后有可能不能使用&＃xff0c;报错信息如下。打开配置文件后发现镜像版本为 v3.14.2。这个版本本人测试时是不可用的。报错信息&＃xff1a;resource mapping not found for name: "bgpconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
版本“apiextension.k8s.io/v1beta1”中的类型“CustomResourceDefinition”不匹配

#如果遇到上述情况&＃xff0c;请下载使用本博客提供的calico.yaml文件。

yum install -y wget
wget https://docs.projectcalico.org/v3.14/manifests/calico.yaml --no-check-certificate
#部署calico报错&＃xff0c;使用本博客所提供的calico文件部署
[root&＃64;master ~]# kubectl apply -f calico.yaml
configmap/calico-config configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers configured
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node configured
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16&＃43;; use the "priorityClassName" field instead
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
resource mapping not found for name: "bgpconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "bgppeers.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "blockaffinities.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "clusterinformations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "felixconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "globalnetworkpolicies.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "globalnetworksets.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "hostendpoints.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamblocks.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamconfigs.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamhandles.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ippools.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "kubecontrollersconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "networkpolicies.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "networksets.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first


6.3.2、操作&＃xff08;calico下载&＃xff09;

关注微信公众号&＃xff1a;CloudLog无名小歌 或 秋意临&＃xff0c;回复calico获取下载。


node节点安装docker-ce&＃xff0c;并拉取镜像如下

#安装docker-ce
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i &＃39;s&＃43;download.docker.com&＃43;mirrors.aliyun.com/docker-ce&＃43;&＃39; /etc/yum.repos.d/docker-ce.repo
yum -y install docker-ce
#配置加速器
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors":["https://vh3bm52y.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
#拉取镜像、打包镜像
docker pull docker.io/calico/node:v3.24.4
docker save -o calico_node_v3.24.4.tar docker.io/calico/node:v3.24.4
docker pull docker.io/calico/cni:v3.24.4
docker save -o calico_cni_v3.24.4.tar docker.io/calico/cni:v3.24.4
docker pull docker.io/calico/kube-controllers:v3.24.4
docker save -o calico_kube-controllers_v3.24.4.tar docker.io/calico/kube-controllers:v3.24.4


master节点执行

由于container有命名空间的概念&＃xff0c;kubernetes的名称空间为k8s.io。

#导入镜像到k8s.io名称空间
ctr -n k8s.io image import calico_node_v3.24.4.tar
ctr -n k8s.io image import calico_cni_v3.24.4.tar
ctr -n k8s.io image import calico_kube-controllers_v3.24.4.tar
#查看镜像是否导入到k8s.io名称空间
[root&＃64;master ~]# crictl images
...
...
IMAGE TAG IMAGE ID SIZE
docker.io/calico/cni v3.24.4 0b046c51c02a8 198MB
docker.io/calico/kube-controllers v3.24.4 0830ebe059a9e 71.4MB
docker.io/calico/node v3.24.4 32c45127e587f 226MB
registry.aliyuncs.com/google_containers/coredns v1.9.3 5185b96f0becf 14.8MB
registry.aliyuncs.com/google_containers/etcd 3.5.4-0 a8a176a5d5d69 102MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.25.3 0346dbd74bcb9 34.2MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.25.3 6039992312758 31.3MB
registry.aliyuncs.com/google_containers/kube-proxy v1.25.3 beaaf00edd38a 20.3MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.25.3 6d23ec0e8b87e 15.8MB
registry.aliyuncs.com/google_containers/pause 3.8 4873874c08efc 311kB
#下载本博客提供的calico.yaml文件后运行。运行后等待几分钟后即可。
[root&＃64;master ~]# kubectl apply -f calico.yaml
[root&＃64;master ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-c676cc86f-ddp44 1/1 Running 0 87m
kube-system coredns-c676cc86f-mg278 1/1 Running 0 87m
kube-system etcd-master 1/1 Running 0 87m
kube-system kube-apiserver-master 1/1 Running 0 87m
kube-system kube-controller-manager-master 1/1 Running 0 87m
kube-system kube-proxy-75svm 1/1 Running 0 87m
kube-system kube-proxy-7bl66 1/1 Running 0 87m
kube-system kube-scheduler-master 1/1 Running 0 87m
[root&＃64;master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane 87m v1.25.3
node Ready <none> 86m v1.25.3


至此kubernetes集群搭建完毕~~~

我是秋意临&＃xff0c;欢迎大家一键三连、加入云社区

(⊙o⊙)&＃xff0c;我们下期再见&＃xff01;&＃xff01;&＃xff01;

containerd&＃xff1a;https://github.com/containerd/containerd/blob/main/docs/getting-started.md
kubernetes&＃xff1a;https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/