linux运维知识体系

logo
linux运维知识体系

Flannel工作原理及故障案例

24/07/17
#Fanael #K8S
120
0

📊 Flannel工作原理

⚙️ 同节点各Pod实现数据通信原理

🔧 环境准备

[root@master231 flannel]# cat 01-deploy-one-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-xiuxian
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xiuxian
  template:
    metadata:
      labels:
        app: xiuxian
        version: v1
    spec:
      nodeName: worker232
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
[root@master231 flannel]# kubectl apply -f  01-deploy-one-worker.yaml
[root@master231 flannel]# kubectl get pods -o wide
[root@master231 flannel]# kubectl exec -it deploy-xiuxian-7d8fb489b7-gcbp9 -- sh
/ # ip a
/ # route -n
[root@master231 flannel]# kubectl exec -it deploy-xiuxian-7d8fb489b7-ltsnb  -- sh
/ # ip a
       valid_lft forever preferred_lft forever
/ # route -n

🔧 去worker节点测试验证

[root@worker232 ~]# ip a
……
6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:28:99:ca:1f:85 brd ff:ff:ff:ff:ff:ff
    inet 10.100.1.1/24 brd 10.100.1.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::3828:99ff:feca:1f85/64 scope link 
       valid_lft forever preferred_lft forever
……
8: vetha41e1689@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
    link/ether 1e:87:cc:8f:28:5d brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::1c87:ccff:fe8f:285d/64 scope link 
       valid_lft forever preferred_lft forever
9: veth05444754@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
    link/ether 86:67:6c:b7:d9:1c brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::a4e3:f4ff:fe4f:7fe7/64 scope link 
       valid_lft forever preferred_lft forever

🔧 验证桥接网卡cni0

[root@worker232 ~]# apt install bridge-utils
[root@worker232 ~]# brctl show cni0
[root@worker232 ~]# ifconfig cni0

🔧 抓包测试

🧩 ping测试

[root@master231 flannel]# kubectl exec -it deploy-xiuxian-7d8fb489b7-gcbp9 -- sh
/ # ping 10.100.1.91 -c 3

🧩 查看抓包结果

[root@worker232 ~]# tcpdump -i cni0 icmp

🔧 验证cni0网卡可以实现同节点不同pod数据通信

🧩 移除网卡

[root@worker232 ~]# ifconfig cni0
[root@worker232 ~]# ip link delete cni0
[root@worker232 ~]# ifconfig cni0

🧩 再次ping测试

[root@master231 flannel]# kubectl exec -it deploy-xiuxian-7d8fb489b7-6spql -- sh
/ # ping 1.100.1.101 -c 3

🔧 添加cni0网卡

🧩 添加cni0网卡

[root@worker233 ~]# ip link add cni0 type bridge
[root@worker233 ~]# ip link set dev cni0 up
[root@worker232 ~]# ip addr add 10.100.1.1/24 dev cni0
[root@worker232 ~]# ifconfig cni0

🧩 将网卡设备添加到cni0网桥

对应的是容器的对端设备，就是宿主机的虚拟网卡
[root@worker232 ~]# brctl show cni0	
[root@worker232 ~]# brctl addif cni0 veth34f2d3c5
[root@worker232 ~]# brctl addif cni0 vetha075523a
[root@worker232 ~]# brctl show cni0

🧩 再次ping测试

[root@master231 flannel]# kubectl exec -it deploy-xiuxian-7d8fb489b7-6spql -- sh
/ # ping 10.100.1.101 -c 3

⚙️ 不同节点各Pod实现数据通信

🔧 环境准备

[root@master231 flannel]# cat 02-ds-tow-worker.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ds-xiuxian
spec:
  selector:
    matchLabels:
      app: xiuxian
  template:
    metadata:
      labels:
        app: xiuxian
        version: v1
    spec:
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
[root@master231 flannel]# kubectl apply -f 02-ds-tow-worker.yaml 
[root@master231 flannel]# kubectl get pods -o wide
[root@master231 flannel]# kubectl exec -it ds-xiuxian-k49kr -- sh
/ # ip a
/ # route -n
[root@master231 flannel]# kubectl exec -it ds-xiuxian-pb9gz -- sh
/ # ip a
/ # route -n

🔧 ping测试

[root@master231 flannel]# kubectl exec -it ds-xiuxian-k49kr -- sh
/ # ping 10.100.1.106 -c 3
/ # ping 10.100.1.106 -c 3
/ # ping 10.100.1.106 -c 3
……

🔧 抓包测试

[root@worker232 ~]# tcpdump -i eth0 -nn |grep overlay
1.很明显，这里就有报文
[root@worker232 ~]# tcpdump -i flannel.1 -nn icmp
在etho网卡中根本就识别不了icmp的报文
[root@worker232 ~]# tcpdump -i eth0  -nn  icmp

📊 Flannel注意事项

  1. 如果Pod无法访问的情况，要检查个节点的宿主机网络路由，如果没有相应的路由，则可能会导致无法进行数据转发；
我的Pod网络是10.100.0.0/16
[root@worker232 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.100.0.0      10.100.0.0      255.255.255.0   UG    0      0        0 flannel.1  # 我们的Pod网络是10.100.0.0/16
10.100.1.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
10.100.2.0      10.100.2.0      255.255.255.0   UG    0      0        0 flannel.1
[root@worker232 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.1.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::c01d:20ff:fe77:6496  prefixlen 64  scopeid 0x20<link>
        ether c2:1d:20:77:64:96  txqueuelen 0  (Ethernet)
        RX packets 12  bytes 1008 (1.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 1008 (1.0 KB)
        TX errors 0  dropped 39 overruns 0  carrier 0  collisions 0
  1. Flannel的宿主机默认会占用一个c的地址
1. 注意观察，子网掩码是"255.255.255.0"，也就是当前节点的Pod的子网都是10.100.1.0/24
[root@worker232 ~]# ifconfig cni0 
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.1.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::3828:99ff:feca:1f85  prefixlen 64  scopeid 0x20<link>
        ether 3a:28:99:ca:1f:85  txqueuelen 1000  (Ethernet)
        RX packets 20  bytes 1232 (1.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1726  bytes 75966 (75.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
  1. 总结：
    1. 每个worker节点可用的IP地址是254个，减去一个网关和广播地址，就是一个节点最多有254个Pod；
    2. 优于每个节点会给分配一个c的网络，因此网络的数量是10.100.0.0/16，总共能分的256个IP子网，因此K8S集群的worker数量应该控制在256个；

📊 Flannel的工作模式切换

⚙️ Flannel的工作模式

🔧 udp

  • 早期支持的一种工作模式，由于性能差，目前官方已弃用。

🔧 vxlan

  • 将源数据报文进行封装为二层报文(需要借助物理网卡转发)，进行跨主机转发。

🔧 host-gw

  • 将容器网络的路由信息写到宿主机的路由表上。尽管效率高，但不支持跨网段。

🔧 directrouting

  • 将vxlan和host-gw工作模式工作。

⚙️ 切换Flannel的工作模式为"vxlan"

🔧 修改配置文件

[root@master231 flannel]# vim kube-flannel-v0.27.0.yml 
...
  net-conf.json: |
    {
      "Network": "10.100.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

🔧 重新创建资源

[root@master231 cni]# kubectl delete -f kube-flannel.yml 
[root@master231 cni]# kubectl apply -f kube-flannel.yml

🔧 检查网络

[root@worker232 ~]# route -n
[root@worker232 ~]# ip route
[root@worker232 ~]# ifconfig cni0

⚙️ 切换Flannel的工作模式为"host-gw"

🔧 修改配置文件

[root@master231 flannel]# vim kube-flannel-v0.27.0.yml 
...
  net-conf.json: |
    {
      "Network": "10.100.0.0/16",
      "Backend": {
        "Type": "host-gw"
      }
    }

🔧 重新创建资源

[root@master231 ~]# kubectl delete -f  kube-flannel-v0.27.0.yml
[root@master231 ~]# kubectl apply -f  kube-flannel-v0.27.0.yml

🔧 检查网络

[root@worker232 ~]# ip route
[root@worker232 ~]# route -n
[root@worker232 ~]# ifconfig cni0

⚙️ 切换Flannel的工作模式为"Directrouting"(推荐配置)

🔧 修改配置文件

[root@master231 cni]# vim kube-flannel.yml
...
  net-conf.json: |
    {
      "Network": "10.100.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "Directrouting": true
      }
    }

🔧 重新创建资源

[root@master231 ~]# kubectl delete -f  kube-flannel-v0.27.0.yml
[root@master231 ~]# kubectl apply -f  kube-flannel-v0.27.0.yml

🔧 检查网络

[root@worker232 ~]# route -n

📊 Flannel故障案例

⚙️ 案例一：未安装Flannel插件

🔧 环境复现

[root@master231 pods]# kubectl get pods -o wide
NAME   READY   STATUS              RESTARTS   AGE    IP       NODE        NOMINATED NODE   READINESS GATES
xixi   0/1     ContainerCreating   0          169m   <none>   worker233   <none>           <none>
[root@master231 pods]# kubectl describe pod xixi 
Name:         xixi
Namespace:    default
Priority:     0
Node:         worker233/10.0.0.233

...

Events:
  Type     Reason                  Age                      From     Message
  ----     ------                  ----                     ----     -------
  Warning  FailedCreatePodSandBox  9m43s (x3722 over 169m)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ae48b3c943557dafdc5f8a3b06897da299233021ed2fd907818cc5acf86c16eb" network for pod "xixi": networkPlugin cni failed to set up pod "xixi_default" network: loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          4m43s (x3850 over 169m)  kubelet  Pod sandbox changed, it will be killed and re-created.

🔧 总结

  1. 发现缺少'/run/flannel/subnet.env'这个文件，说明没有部署Flannel插件，安装插件即可。

⚙️ 案例二：缺少Flannel程序

🔧 环境复现

[root@master231 flannel]# kubectl get pods -o wide
NAME               READY   STATUS              RESTARTS   AGE   IP       NODE        NOMINATED NODE   READINESS GATES
ds-xiuxian-dcjsg   0/1     ContainerCreating   0          10s   <none>   worker232   <none>           <none>
ds-xiuxian-vjbnw   0/1     ContainerCreating   0          10s   <none>   worker233   <none>           <none>

[root@master231 flannel]# kubectl describe pod ds-xiuxian-dcjsg 
Name:           ds-xiuxian-dcjsg
Namespace:      default
Priority:       0
Node:           worker232/10.0.0.232
...
Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               15s               default-scheduler  Successfully assigned default/ds-xiuxian-dcjsg to worker232
  Warning  FailedCreatePodSandBox  15s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "1fcadf406345fda4f800feb9ec42ddce495da0ecef8ba4f5cb5ebbdad795e4fe" network for pod "ds-xiuxian-dcjsg": networkPlugin cni failed to set up pod "ds-xiuxian-dcjsg_default" network: failed to find plugin "flannel" in path [/opt/cni/bin], failed to clean up sandbox container "1fcadf406345fda4f800feb9ec42ddce495da0ecef8ba4f5cb5ebbdad795e4fe" network for pod "ds-xiuxian-dcjsg": networkPlugin cni failed to teardown pod "ds-xiuxian-dcjsg_default" network: failed to find plugin "flannel" in path [/opt/cni/bin]]
  Normal   SandboxChanged          4s (x2 over 14s)  kubelet            Pod sandbox changed, it will be killed and re-created.

🔧 问题原因

  1. 在"/opt/cni/bin"路径下找不到一个名为"flannel"的二进制文件。

🔧 解决方法

  1. 官方关于Flannel的初始化容器中存在Flannel设备，因此删除Pod后就能够自动生成该程序文件。
[root@master231 flannel]# kubectl get pods -o wide -n kube-flannel --show-labels
[root@master231 flannel]# kubectl -n kube-flannel delete pod -l k8s-app=flannel
[root@master231 flannel]# kubectl get pods -o wide -n kube-flannel

⚙️ 案例三：缺失程序包文件

🔧 故障复现

[root@worker232 ~]# ll /opt/cni/bin/
[root@worker232 ~]# mount -t tmpfs -o size=90M tmpfs /opt/
[root@worker232 ~]# df -h | grep opt
[root@worker232 ~]# ll /opt/

🔧 创建测试的Pod

[root@master231 flannel]# kubectl get pods -o wide
[root@master231 flannel]# kubectl describe pod ds-xiuxian-hhvn2
Name:           ds-xiuxian-hhvn2
Namespace:      default
Priority:       0
Node:           worker233/10.0.0.233
...
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               11s   default-scheduler  Successfully assigned default/ds-xiuxian-hhvn2 to worker233
  Warning  FailedCreatePodSandBox  11s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "52b723c545c56d611e265b25bbd930fd66c9f1b651c65550fd515ea9702cbec6" network for pod "ds-xiuxian-hhvn2": networkPlugin cni failed to set up pod "ds-xiuxian-hhvn2_default" network: failed to find plugin "loopback" in path [/opt/cni/bin], failed to clean up sandbox container "52b723c545c56d611e265b25bbd930fd66c9f1b651c65550fd515ea9702cbec6" network for pod "ds-xiuxian-hhvn2": networkPlugin cni failed to teardown pod "ds-xiuxian-hhvn2_default" network: failed to find plugin "portmap" in path [/opt/cni/bin]]
  Normal   SandboxChanged          11s   kubelet            Pod sandbox changed, it will be killed and re-created.

🔧 问题原因

  1. 在'/opt/cni/bin'目录下缺失'portmap'程序，该程序主要做端口映射相关的。
  2. 需要检查一下该目录是否有程序文件，也有可能是运维同事误操作挂载导致的问题。检查是否有挂载点冲突问题。

🔧 解决方法

  1. 取消挂载即可恢复
umount /opt/

⚙️ 案例四：误删除CNI程序包

🔧 故障原因：

  1. CNI程序包丢失

🔧 解决方法

参考链接：https://github.com/containernetworking/plugins

[root@worker232 ~]# wget https://github.com/containernetworking/plugins/releases/download/v1.8.0/cni-plugins-linux-amd64-v1.8.0.tgz
[root@worker232 ~]# tar xf cni-plugins-linux-amd64-v1.8.0.tgz  -C /opt/cni/bin/
[root@worker232 ~]# ll /opt/cni/bin/

⚙️ 案例五：flannel路由缺失

🔧 故障现象

  1. curl命令pod中的ip发现卡住

🔧 故障原因

  1. 本地路由没有10.100.0.0/16的路由
[root@master231 deployments]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.254      0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

🔧 解决方法

  • 删除kube-flannel中的pod，会重新分配路由
[root@master231 deployments]# kubectl -n kube-flannel delete pods --all
[root@master231 deployments]# kubectl get pods -n kube-flannel -o wide
[root@master231 deployments]# kubectl get pods -n kube-flannel -o wide
1.发现本地路由存在10.100.0.0/16的路由
[root@master231 deployments]# route -n