一 前言
之前分析了cni的插件原理，具体的CNI的实现是怎么样？偶然遇到一个cni实现，作者非常厉害，为了方便学习自己做了一些改造，代码见github。
二 节点环境
image
for i in {0..2}
do
nodename=$(kubectl get nodes -o jsonpath="{.items[${i}].metadata.name}")
nodeip=$(kubectl get nodes -o jsonpath="{.items[${i}].status.addresses[0].address}")
podcidr=$(kubectl get nodes -o jsonpath="{.items[${i}].spec.podCIDR}")
echo "nodeip: "$nodeip", podcidr: "$podcidr",nodename: "${nodename}
done
nodeip: 172.17.0.2, podcidr: 10.244.0.0/24,nodename: kind-control-plane
nodeip: 172.17.0.4, podcidr: 10.244.2.0/24,nodename: kind-worker
nodeip: 172.17.0.3, podcidr: 10.244.1.0/24,nodename: kind-worker2
三 NODE网络配置
kindnet使用Deamonset部署，每个节点都会部署。由于POD没有使用PID隔离，网络隔离，所以可以在POD中看到的网络设置是POD所在节点的网络设置。
Kindnet列表
[root@test ~]# kubectl get pod -A -n kube-system -l "app=kindnet" -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system kindnet-fwnl8 1/1 Running 0 6h36m 172.17.0.2 kind-control-plane <none> <none>
kube-system kindnet-vsh6r 1/1 Running 0 6h36m 172.17.0.3 kind-worker2 <none> <none>
kube-system kindnet-zxhcz 1/1 Running 0 6h36m 172.17.0.4 kind-worker <none> <none>
登陆Kindnet
选择 kindnet-zxhcz，它部署在节点kind-worker。
kubectl exec -it kindnet-zxhcz -n kube-system /bin/bash
网卡信息
kind-worker节点网卡cni0的地址为10.244.2.1，eth0地址为172.17.0.4，跟开始获取的网络环境一致，注意pod没有网络隔离。
[root@kind-worker app]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3e:1c:10:0f:dd:e9 brd ff:ff:ff:ff:ff:ff
inet 10.244.2.1/24 scope global cni0
valid_lft forever preferred_lft forever
4: veth_50d368@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether 66:93:d4:51:f3:86 brd ff:ff:ff:ff:ff:ff link-netnsid 1
6: veth_903ae5@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0 state UP group default qlen 1000
link/ether 3e:1c:10:0f:dd:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 2
324: eth0@if325: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.4/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
路由信息
1）访问10.244.2.0直接通过cni0接口，因为10.224.2.*就在本机
2）访问10.244.1.0 需要通过网关172.17.0.3，因为
[root@kind-worker app]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
10.244.0.0 172.17.0.2 255.255.255.0 UG 0 0 0 eth0
10.244.1.0 172.17.0.3 255.255.255.0 UG 0 0 0 eth0
10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
iptables
masq主要解决pod访问公网问题，如果pod不需要访问公网，可以不设置。
[root@kind-worker app]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- anywhere anywhere /* kubernetes postrouting rules */
masq all -- anywhere anywhere ADDRTYPE match dst-type !LOCAL
Chain masq (1 references)
target prot opt source destination
RETURN all -- anywhere 10.244.0.0/24
RETURN all -- anywhere 10.244.1.0/24
RETURN all -- anywhere 10.244.2.0/24
MASQUERADE all -- anywhere anywhere
CNI 插件设置
/etc/cni/net.d/设置插件信息，这里为my-cni,可执行程序放在/opt/cni/bin目录下。
[root@kind-worker app]# cat /etc/cni/net.d/cni-config.conf
{
"cniVersion": "0.3.0",
"name": "my-cni",
"type": "my-cni",
"podcidr": "10.244.2.0/24"
}
[root@kind-worker app]# ls /opt/cni/bin/my-cni
/opt/cni/bin/my-cni
四 容器网络测试
部署一个DS，主要是方便POD跨节点通信问题。
DS部署
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: demo
labels:
app: demo
spec:
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:
- name: demo
image: huiwq1990/centos
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- sleep 5000
EOF
[root@test ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo-t5495 1/1 Running 0 2m56s 10.244.2.7 kind-worker <none> <none>
demo-vz42s 1/1 Running 0 2m56s 10.244.1.8 kind-worker2 <none> <none>
PING POD
节点worker2上的POD访问节点worker上的POD，可以访问通。
[root@demo-vz42s /]# ping 10.244.2.7
PING 10.244.2.7 (10.244.2.7) 56(84) bytes of data.
64 bytes from 10.244.2.7: icmp_seq=1 ttl=62 time=0.111 ms
Worker节点抓包
注意这里是从worker节点抓包，看的是节点的网络包，不是pod里的。
[root@kind-worker app]# tcpdump -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:39:34.384281 IP 10.244.1.8 > 10.244.2.7: ICMP echo request, id 26, seq 1, length 64
15:39:34.384310 IP 10.244.2.7 > 10.244.1.8: ICMP echo reply, id 26, seq 1, length 64
15:39:35.384849 IP 10.244.1.8 > 10.244.2.7: ICMP echo request, id 26, seq 2, length 64
15:39:35.384875 IP 10.244.2.7 > 10.244.1.8: ICMP echo reply, id 26, seq 2, length 64
15:39:36.384841 IP 10.244.1.8 > 10.244.2.7: ICMP echo request, id 26, seq 3, length 64
15:39:36.384861 IP 10.244.2.7 > 10.244.1.8: ICMP echo reply, id 26, seq 3, length 64
15:39:37.384831 IP 10.244.1.8 > 10.244.2.7: ICMP echo request, id 26, seq 4, length 64
15:39:37.384851 IP 10.244.2.7 > 10.244.1.8: ICMP echo reply, id 26, seq 4, length 64
