通常 低级容器运行时 用来创建和运行容器，而高级容器运行时作为操作系统常驻进程用来管理这些容器，本文则通过实际案列来演示，高级容器运行时 Containerd 如何与各低级容器运行时(runc 、crun 、runsc 、kata-containers)集成，以及其中的一些关键配置项分享，希望大家喜欢
环境介绍
1、环境中各组件版本说明
系统版本 CentOS 7.9.2009( Kernel 5.4.188-1.el7.elrepo.x86_64)
containerd v1.6.6
runc version 1.1.3 , OCI specs: 1.0.2-dev
crun version 1.6 , OCI spec: 1.0.0
kata-runtime: 2.5.2 , OCI specs: 1.0.2-dev
runsc version release-20221010.0, OCI specs: 1.0.2-dev
Containerd
继 Kubernetes、Prometheus、Envoy 和 CoreDNS 之后, 2019 年 2 月 28 日，containerd 正式成为云原生计算基金会的毕业项目，是容器运行时的行业标准! Containerd 可作为 Linux 或 Windows 的守护进程运行，对其主机系统中所有镜像、容器的完整生命周期进行管理
1、二进制方式解压安装
# 部署 containerd + runc + cni plugins
wget https://github.com/containerd/containerd/releases/download/v1.6.6/cri-containerd-cni-1.6.6-linux-amd64.tar.gz
tar -zxvf cri-containerd-cni-1.6.6-linux-amd64.tar.gz -C /
# 主要部署目录说明
/etc/containerd/config.toml # containerd 配置文件
/etc/crictl.yaml # crictl 配置文件
/opt/cni/bin # containerd 需要使用到的 cni 插件
/usr/local/sbin/runc # runtime 类别，默认类型 runc
/usr/local/bin/containerd
/usr/local/bin/containerd-shim
/usr/local/bin/containerd-shim-runc-v1
/usr/local/bin/containerd-shim-runc-v2 # 实现 containerd 与 runc 的集成
/usr/local/bin/containerd-stress # stress test a containerd daemon
/usr/local/bin/critest
/usr/local/bin/ctd-decoder
# crictl 、ctr 用于与 containerd 交互的命令行工具
/usr/local/bin/crictl
/usr/local/bin/ctr
# containerd 不同二进制压缩包说明
containerd-1.6.8-linux-amd64.tar.gz # 仅包含 containerd 自身组件
cri-containerd-1.6.8-linux-amd64.tar.gz # 包含 containerd + crictl
cri-containerd-cni-1.6.8-linux-amd64.tar.gz # 包含 containerd + crictl + cni 插件；为方便、推荐这种方式
2、生成配置，修改 pause 镜像源
# 生成 containerd 默认配置
mkdir -p /etc/containerd && containerd config default > /etc/containerd/config.toml
# 修改镜像 pause:3.6 仓库地址
[plugins."io.containerd.grpc.v1.cri"]
- sandbox_image = "k8s.gcr.io/pause:3.6"
+ sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7"
# 开启 shim 的调试
[plugins."io.containerd.runtime.v1.linux"]
- shim_debug = false
+ shim_debug = true
3、CNI 插件 bridge 的默认网络配置
# cat /etc/cni/net.d/10-containerd-net.conflist
{
"cniVersion": "1.0.0",
"name": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"ranges": [
[{
"subnet": "10.88.0.0/16"
}],
[{
"subnet": "2001:4860:4860::/64"
}]
],
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "::/0" }
]
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true}
}
]
}
runC
runC 根据 OCI 规范生成和运行容器的 Runtime, 且作为 Containerd runtime 的默认配置类型， 使用 go 语言开发
# 替换 containerd 包默认的 runc ，因为该 runc 并未以静态编译方式包含其依赖程序 libseccomp
wget https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64
mv runc.amd64 /usr/local/sbin/runc
chmod +x /usr/local/sbin/runc
# Containerd runtime 的默认配置
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
Crun
Crun 是一个快速轻量级、低内存占用、全功能、兼容 OCI 规范、用于运行容器的 Runtime，使用 C 语言开发
# 1、二进制解压安装
wget https://github.com/containers/crun/releases/download/1.6/crun-1.6-linux-amd64
mv crun-1.6-linux-amd64 /usr/local/bin/crun && chmod +x /usr/local/bin/crun
# 2、修改 Containerd 配置，增加新的运行时 crun 支持
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
+ runtime_type = "io.containerd.runc.v2"
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
+ BinaryName = "/usr/local/bin/crun"
runSC
gVisor 是一个由 Google 开源、用 Go 语言编写的 Application 内核，其包含一个兼容 OCI 规范的 Runtime( runSC ), 该 runtime 在应用程序和主机内核之间提供隔离边界；runsc 也可以和主流的容器运行时(Docker、Containerd、CRI-O)、Kubernetes 集成，使运行沙盒(Sandbox)容器变得简单
主要特性：运行不受信任的工作负载、阻止容器逃逸、减少未经授权的主机访问，保护基础设施，增强 Kubernetes 安全！
# 部署 runsc containerd-shim-runsc-v1
wget https://storage.googleapis.com/gvisor/releases/release/latest/x86_64/runsc
wget https://storage.googleapis.com/gvisor/releases/release/latest/x86_64/containerd-shim-runsc-v1
chmod a+rx runsc containerd-shim-runsc-v1
mv runsc containerd-shim-runsc-v1 /usr/local/bin # 用于和 Containerd 集成的一个 shim
# 修改 Containerd 配置，增加新的运行时 runsc 支持
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
+ runtime_type = "io.containerd.runsc.v1"
Kata Containers
Kata Containers 是一个开源项目和社区，致力于构建轻量级虚拟机 (VM) 的标准实现；其主要组件 kata-runtime 用于运行根据 OCI 规范打包的容器
1、部署 kata, 并与 Containerd 集成
# 二进制方式部署
wget https://github.com/kata-containers/kata-containers/releases/download/2.5.2/kata-static-2.5.2-x86_64.tar.xz
tar xf kata-static-2.5.2-x86_64.tar.xz -C /
ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
export PATH=$PATH:/opt/kata/bin/:/opt/kata/libexec/ && source /etc/profile # 添加环境变量配置
# 修改 Containerd 配置，增加新运行时 kata 的支持
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
+ runtime_type = "io.containerd.kata.v2"
2、检查当前系统是否满足 kata 运行环境要求
# 若为如下输出、则表示满足环境要求
./kata-runtime check
# > WARN[0000] Not running network checks as super user arch=amd64 name=kata-runtime pid=5938 source=runtime
# > System is capable of running Kata Containers
# > System can currently create Kata Containers
# 若为如下输出、则表示不满足环境要求
./kata-runtime check
# > ERRO[0000] CPU property not found arch=amd64 description="Virtualization support" name=vmx pid=5938 source=runtime type=flag
# > ERRO[0000] kernel property kvm_intel not found arch=amd64 description="Intel KVM" name=kvm_intel pid=5938 source=runtime type=module
# > WARN[0000] modprobe insert module failed arch=amd64 error="exit status 1" module=kvm_intel name=kata-runtime output="modprobe: ERROR: could not insert 'kvm_intel': Operation not supported\n" pid=5938 source=runtime
# > ERRO[0000] ERROR: System is not capable of running Kata Containers arch=amd64 name=kata-runtime pid=5938 source=runtime ERROR: System is not capable of running Kata Containers
根据报错分析，其不满足环境要求的具体原因：首先是当前主机 CPU 不支持 vmx 虚拟化，其次是当前系统未加载内核模块 kvm_intel；解决方案如下:
编辑虚机 a7 配置 virsh edit a7，让虚机直接使用物理机 CPU 资源 (即虚机直通宿主机 CPU 资源)
<!-- 坑点注意: 首先关闭虚拟机，再修改配置，最后启动虚拟机，否则该配置不生效 !! -->
<!-- 将虚机 cpu 的默认配置 -->
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Broadwell</model>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='xsaveopt'/>
</cpu>
<!-- 更换为 -->
<cpu mode='host-passthrough'>
<topology sockets='2' cores='2' threads='2'/>
</cpu>
3、kata 配置及二进制文件介绍
/opt/kata/share/defaults/kata-containers/configuration.toml
/opt/kata/libexec/virtiofsd
/opt/kata/bin/cloud-hypervisor
/opt/kata/bin/containerd-shim-kata-v2
/opt/kata/bin/firecracker
/opt/kata/bin/jailer
/opt/kata/bin/kata-collect-data.sh
/opt/kata/bin/kata-monitor
/opt/kata/bin/kata-runtime
/opt/kata/bin/qemu-system-x86_64
环境验证
分别创建 runtime 为 runc、crun、runsc、kata 的沙箱及容器
1、测试镜像准备
crictl pull busybox:latest
crictl pull registry.aliyuncs.com/google_containers/pause:3.6
crictl images
# > docker.io/library/busybox latest ff4a8eb070e12 777kB
# > registry.aliyuncs.com/google_containers/pause 3.6 6270bb605e12e 302kB
2、沙箱配置 cat sandbox.json
{
"metadata": {
"name": "busybox-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "adishd83djaidwnduwk28bc1b"
},
"log_directory": "/tmp",
"linux": {
}
}
3、容器配置 cat container.json
{
"metadata": {
"name": "busybox"
},
"image":{
"image": "busybox"
},
"command": [
"top"
],
"log_path":"busybox.0.log",
"linux": {
}
}
3、创建沙箱、运行容器
crictl runp -r runc sandbox.json # 创建沙箱，可通过 -r 选项指定 runtime type ，取值有: runc(默认)、crun、runsv、kata
# > b39da7541afbd42c3e88703872d12287d9de4c7050c6aad91285e648de27daef
crictl create b39da7541afbd42c3e88703872d12287d9de4c7050c6aad91285e648de27daef container.json sandbox.json
# > d9b420869dc780c2d08fc89c0a0893fc82e53388eaf8077772e7731b29fad64c
crictl start d9b420869dc780c2d08fc89c0a0893fc82e53388eaf8077772e7731b29fad64c
# > d9b420869dc780c2d08fc89c0a0893fc82e53388eaf8077772e7731b29fad64c
crictl pods
# > POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
# > b39da7541afbd 2 minutes ago Ready busybox-sandbox default 1 (default)
crictl ps -a
# > CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
# > d9b420869dc78 busybox About a minute ago Running busybox 0 b39da7541afbd unknown
# 进入容器测试外网连通性
crictl exec -it d9b420869dc780c2d08fc89c0a0893fc82e53388eaf8077772e7731b29fad64c bash
# > / # ping 223.5.5.5
# > PING 223.5.5.5 (223.5.5.5): 56 data bytes
# > 64 bytes from 223.5.5.5: seq=0 ttl=115 time=56.582 ms
容器配置 sandbox.json 保持不变，通过变更沙箱配置 sandbox.json 中的 metadata.name 及 metadata.uid ，按如上流程分别创建 runtime 为 crun、runsc、kata 的沙箱及容器；最终运行情况如下
crictl pods
# > POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
# > d4cfc38d9b1b8 27 seconds ago Ready busybox-sandbox-kata default 1 kata
# > a9ff45df311d6 About a minute ago Ready busybox-sandbox-runsc default 1 runsc
# > e9341e0494b0e 2 minutes ago Ready busybox-sandbox-crun default 1 crun
# > 0b98eb53bac29 5 minutes ago Ready busybox-sandbox default 1 (default)
crictl ps -a
# > CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
# > b09d560292531 busybox 9 minutes ago Running busybox 0 d4cfc38d9b1b8 unknown
# > d52c1ebe2f7c7 busybox 11 minutes ago Running busybox 0 a9ff45df311d6 unknown
# > 728d21ddd444a busybox 12 minutes ago Running busybox 0 e9341e0494b0e unknown
# > 2cf63e63b2733 busybox 14 minutes ago Running busybox 0 0b98eb53bac29 unknown
重要提醒: 由于笔者时间、视野、认知有限，本文难免出现错误、疏漏等问题，期待各位读者朋友、业界大佬指正交流, 共同进步 !!
Q&A
1、runc: undefined symbol: seccomp_notify_respond
报错原因: runc 依赖于 [libseccomp](https://github.com/seccomp/libseccomp) ，而当前系统并未安装 libseccomp；有两种解决方式, 其一是系统环境安装该依赖，其二是 runc 采用静态编译方式(会包含依赖程序)
2、E1023 13:47:23.779289 8775 remote_runtime.go:421] “CreateContainer in sandbox from runtime service failed” err=“rpc error: code = Unknown desc = failed to resolve image "": ambiguous digest string” podSandboxID=“e930c50100f8e”
FATA[0000] creating container: rpc error: code = Unknown desc = failed to resolve image “”: ambiguous digest string
报错原因: 执行命令 crictl create 创建容器时，沙箱配置放在了容器配置的前面，而正确写法是: crictl create ee868498edcc1 container.json sandbox.json
