概要
minikubebのKubernetesでnetwork policyを使ってみたいと思い調べてみました
環境
- minikube version: 1.33.1
- VM Driver ：docker
- Runtime：containerd
- Ubuntu 22.04.4 LTS（WSL）
- Kubernetes v1.30.0
調べてみる
Kubernetesで Network Policies を使用するには、ネットワーク プラグインの
Container Network Interface (CNI) が必要になりますが、minikubeの標準値では未設定みたいですね
minikubebのKubernetesでCNIを使う方法を調べてみると主なパターンは以下みたいです
- Minikubeのオプションを使用してMinikube内蔵のCNI（calico）を使用する
- calico のoperator をインストールする
- calico をマニフェストでインストールする
Minikubeのオプション（--cni calico） について
minikubeでクラスター起動時にオプションとして「--cni calico」を使用することで
cniとなるCalico のインストールと必要な設定を自動で全部してくれるみたいです
Minikubeの公式サイトでは、これで実行するよう書かれているので、
現時点ではこれが推奨されている利用方法かもしれません
例
# calico 有りのClusterを起動する
minikube start --network-plugin=cni --cni=calicocalico の operator について
Calico を管理するOperatorをインストールすることでCalico を管理します。
minikubeでクラスター起動時にkubeletのオプションでCNIをセットし起動後に
Operator Podを起動することでCalico 環境を作成します。
例
#Clusterを起動
minikube start --cni=false --network-plugin=cni --extra-config=kubeadm.pod-network-cidr=192.168.0.0/16 --subnet=172.16.0.0/24
# Cluster起動後に以下でCalico Podを起動
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yamlcalico のマニフェスト について
ClusterにCalico を直接インストールします。
minikubeでクラスター起動時にkubeletのオプションでCNIをセットし起動後に
Calico Podを起動することでCalico 環境を作成します。
例
#Clusterを起動
minikube start --network-plugin=cni
# Cluster起動後に以下でCalico Podを起動
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yamlMinikube のcalico 設定の詳細については、以下を参照してください
-Network Policy
-Network Plugins
-Quickstart for Calico on minikube
※Quickstart for Calico on minikubeのドメインdocs.tigera.io は、
Tigera 社が提供する Calico および Calico Enterprise の公式ドキュメントサイト
minikubeでcalico を試してみる
今回は、一番簡単にCNIを使えるMinikubeオプション（--cni calico） を使って環境を作成してみます
- Minikube でオプションを使ってKubernetes Clusterを起動します
tech-0222@MSI:tmp$ minikube profile list
|----------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|----------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
| minikube | docker | docker | 192.168.58.2 | 8443 | v1.28.3 | Stopped | 1 | * | |
|----------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
tech-0222@MSI:tmp$
tech-0222@MSI:tmp$ minikube start --cni calico --container-runtime=containerd --bootstrapper=kubeadm -p test-1
😄 [test-1] minikube v1.33.1 on Ubuntu 22.04 (amd64)
✨ Automatically selected the docker driver. Other choices: kvm2, qemu2, ssh
📌 Using Docker driver with root privileges
👍 Starting "test-1" primary control-plane node in "test-1" cluster
🚜 Pulling base image v0.0.44 ...
💾 Downloading Kubernetes v1.30.0 preload ...
> preloaded-images-k8s-v18-v1...: 375.69 MiB / 375.69 MiB 100.00% 4.78 Mi
🔥 Creating docker container (CPUs=2, Memory=3900MB) ...
📦 Preparing Kubernetes v1.30.0 on containerd 1.6.31 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring Calico (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "test-1" cluster and "default" namespace by default
tech-0222@MSI:tmp$
tech-0222@MSI:tmp$ minikube profile list
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| minikube | docker | docker | 192.168.58.2 | 8443 | v1.28.3 | Stopped | 1 | * | |
| test-1 | docker | containerd | 192.168.49.2 | 8443 | v1.30.0 | Running | 1 | | * |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
tech-0222@MSI:tmp$
- PODの状態を確認すると一部のPodがなかな起動してこない。。
tech-0222@MSI:~$ kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-vnwkm 0/1 ContainerCreating 0 7m19s
calico-node-p5fcx 0/1 Init:0/3 0 7m20s
coredns-7db6d8ff4d-6nct2 0/1 ContainerCreating 0 7m19s
etcd-test-1 1/1 Running 0 7m35s
kube-apiserver-test-1 1/1 Running 0 7m34s
kube-controller-manager-test-1 1/1 Running 0 7m33s
kube-proxy-j2gwj 1/1 Running 0 7m20s
kube-scheduler-test-1 1/1 Running 0 7m33s
storage-provisioner 1/1 Running 1 (6m50s ago) 7m32s
tech-0222@MSI:~$- 起動していないPODのイベントログをみると、おそらくPodの「calico-node-p5fcx」が起動してこないので、関連したPodが起動できていない？
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod coredns-7db6d8ff4d-6nct2
Name: coredns-7db6d8ff4d-6nct2
～～～略～～～
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m52s default-scheduler Successfully assigned kube-system/coredns-7db6d8ff4d-6nct2 to test-1
Warning FailedCreatePodSandBox 8m52s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "854beb24f37d1f4476f82054935e6a51a80be9f676fea5968502684291edce48": failed to find network info for sandbox "854beb24f37d1f4476f82054935e6a51a80be9f676fea5968502684291edce48"
Warning FailedCreatePodSandBox 8m40s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3e3be41b7b218381aeb90bdab8c5c14413ea05b8606b0774b02496c2e790564b": failed to find network info for sandbox "3e3be41b7b218381aeb90bdab8c5c14413ea05b8606b0774b02496c2e790564b"
Warning FailedCreatePodSandBox 8m27s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de162f636de67dd33e86de92debb1f69fd8d0d31c647f00123ac9bdb2ba78ea2": failed to find network info for sandbox "de162f636de67dd33e86de92debb1f69fd8d0d31c647f00123ac9bdb2ba78ea2"
Warning FailedCreatePodSandBox 8m15s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c3c65d4f49b3aab5274c8514f1e3f1587873c80c38b24b91ec65daca8b75d728": failed to find network info for sandbox "c3c65d4f49b3aab5274c8514f1e3f1587873c80c38b24b91ec65daca8b75d728"
Warning FailedCreatePodSandBox 8m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "051059e3bdb1989004a7c2a55799f0aeda4e3ad16b7a09ce67d7fba1b8cb189c": failed to find network info for sandbox "051059e3bdb1989004a7c2a55799f0aeda4e3ad16b7a09ce67d7fba1b8cb189c"
Warning FailedCreatePodSandBox 7m47s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "943a05e0e166f8808db31bef6ba06f16761d01e37eae726395065bf17e3ca9ca": failed to find network info for sandbox "943a05e0e166f8808db31bef6ba06f16761d01e37eae726395065bf17e3ca9ca"
Warning FailedCreatePodSandBox 7m35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ecb153cd88f610a7b390089c1d4805be818804a56dcfb577f4bb4a654f3c7a0f": failed to find network info for sandbox "ecb153cd88f610a7b390089c1d4805be818804a56dcfb577f4bb4a654f3c7a0f"
Warning FailedCreatePodSandBox 7m21s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1cfcf6969b46af6bf331f6db9caf05e09d175ea545e7f39585bc4dd1fc73e70e": failed to find network info for sandbox "1cfcf6969b46af6bf331f6db9caf05e09d175ea545e7f39585bc4dd1fc73e70e"
Warning FailedCreatePodSandBox 7m7s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a3d054605375d1a43262a95cfa22c4d88c148ddfa5d0916a4c04276096ae5c98": failed to find network info for sandbox "a3d054605375d1a43262a95cfa22c4d88c148ddfa5d0916a4c04276096ae5c98"
Warning FailedCreatePodSandBox 3m30s (x17 over 6m52s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f1db310f022702b40a8d3731f31de3b865f3bccc5bc276ceed9b4102148a635c": failed to find network info for sandbox "f1db310f022702b40a8d3731f31de3b865f3bccc5bc276ceed9b4102148a635c"
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod calico-kube-controllers-ddf655445-vnwkm
Name: calico-kube-controllers-ddf655445-vnwkm
～～～略～～～
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m11s default-scheduler Successfully assigned kube-system/calico-kube-controllers-ddf655445-vnwkm to test-1
Warning FailedCreatePodSandBox 9m11s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7d6cd72496689667cbce1a861d04fbb9069fe8d28a94aa439bd92e198e1b023f": failed to find network info for sandbox "7d6cd72496689667cbce1a861d04fbb9069fe8d28a94aa439bd92e198e1b023f"
Warning FailedCreatePodSandBox 8m58s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "071b54b77ab835602a975d9ea41f00c5e7177389df5902cb6e871dc60c67c10b": failed to find network info for sandbox "071b54b77ab835602a975d9ea41f00c5e7177389df5902cb6e871dc60c67c10b"
Warning FailedCreatePodSandBox 8m46s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3069488fd559bf6c5e728a9874c6e8812da00618f2bc2d105cf406c7af22df9e": failed to find network info for sandbox "3069488fd559bf6c5e728a9874c6e8812da00618f2bc2d105cf406c7af22df9e"
Warning FailedCreatePodSandBox 8m33s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "48e9a5786d20f8afe4899c481faa2f676028a76121959c5944dd4dd80d0bf436": failed to find network info for sandbox "48e9a5786d20f8afe4899c481faa2f676028a76121959c5944dd4dd80d0bf436"
Warning FailedCreatePodSandBox 8m19s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "318717112bac7be1fb85098fa5190605010abcb9f3fc42b2853cc527cd9d9c21": failed to find network info for sandbox "318717112bac7be1fb85098fa5190605010abcb9f3fc42b2853cc527cd9d9c21"
Warning FailedCreatePodSandBox 8m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e66e3acaa149585e77a47fc20d1dd249c727aedc3ac2127a745d3beeeccff687": failed to find network info for sandbox "e66e3acaa149585e77a47fc20d1dd249c727aedc3ac2127a745d3beeeccff687"
Warning FailedCreatePodSandBox 7m54s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "14d04c3cf12e6b4b3dbbb1349a6a42edfdf9c709a830944e299287e011f58210": failed to find network info for sandbox "14d04c3cf12e6b4b3dbbb1349a6a42edfdf9c709a830944e299287e011f58210"
Warning FailedCreatePodSandBox 7m43s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a8e03a7ff5a71bacb90e654d1b6973eee3d442d0554fddfbf74c6fb6434e7e2e": failed to find network info for sandbox "a8e03a7ff5a71bacb90e654d1b6973eee3d442d0554fddfbf74c6fb6434e7e2e"
Warning FailedCreatePodSandBox 7m28s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a478819d4c4ef82105ce50c3a3b9d41ce9b5217c5a51e4b5096bdc602ffe8978": failed to find network info for sandbox "a478819d4c4ef82105ce50c3a3b9d41ce9b5217c5a51e4b5096bdc602ffe8978"
Warning FailedCreatePodSandBox 3m45s (x17 over 7m14s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f81884f498f1afc039f419284e9d5b7a665c5560a99820eba6c08450b5bc4094": failed to find network info for sandbox "f81884f498f1afc039f419284e9d5b7a665c5560a99820eba6c08450b5bc4094"
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod calico-node-p5fcx
Name: calico-node-p5fcx
～～～略～～～
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m26s default-scheduler Successfully assigned kube-system/calico-node-p5fcx to test-1
Normal Pulling 9m25s kubelet Pulling image "docker.io/calico/cni:v3.27.3"
tech-0222@MSI:~$- 15分ぐらいすると「calico-node-XX」PODのSTATUSが 「Init:2/3」になるが、それから何も変化がない
- ログを見るとイメージがダウンロードできていると思うが、これ以上先に進まない
tech-0222@MSI:~$ kubectl -n kube-system describe pod calico-node-p5fcx
～～略
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned kube-system/calico-node-p5fcx to test-1
Normal Pulling 16m kubelet Pulling image "docker.io/calico/cni:v3.27.3"
Normal Pulled 4m48s kubelet Successfully pulled image "docker.io/calico/cni:v3.27.3" in 12m0.864s (12m0.864s including waiting). Image size: 88418575 bytes.
Normal Created 4m48s kubelet Created container upgrade-ipam
Normal Started 4m48s kubelet Started container upgrade-ipam
Normal Pulled 4m47s kubelet Container image "docker.io/calico/cni:v3.27.3" already present on machine
Normal Created 4m47s kubelet Created container install-cni
Normal Started 4m47s kubelet Started container install-cni
Normal Pulling 4m46s kubelet Pulling image "docker.io/calico/node:v3.27.3"
tech-0222@MSI:~$
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system logs calico-node-p5fcx
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-p5fcx" is waiting to start: PodInitializing
tech-0222@MSI:~$
- Statusが変なPod「calico-node-XXX」「calico-kube-controllers-XXXXX」「coredns-XXXX」を削除すると再作成され正常に起動する
#削除直後
tech-0222@MSI:~$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-w5bpf 0/1 ContainerCreating 0 5m2s
calico-node-tmfnl 1/1 Running 0 4m12s
coredns-7db6d8ff4d-f85fh 1/1 Running 0 5m2s
etcd-test-1 1/1 Running 0 33m
kube-apiserver-test-1 1/1 Running 0 33m
kube-controller-manager-test-1 1/1 Running 0 33m
kube-proxy-j2gwj 1/1 Running 0 33m
kube-scheduler-test-1 1/1 Running 0 33m
storage-provisioner 1/1 Running 1 (32m ago) 33m
tech-0222@MSI:~$
#成功
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod coredns-7db6d8ff4d-f85fh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m43s default-scheduler Successfully assigned kube-system/coredns-7db6d8ff4d-f85fh to test-1
Warning FailedCreatePodSandBox 2m42s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "81aac3491b47b3a8b9ff3de45224f8a319b3a71c5e2f8affce9f61f9ecc3e500": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Normal SandboxChanged 111s (x5 over 2m42s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 111s kubelet Container image "registry.k8s.io/coredns/coredns:v1.11.1" already present on machine
Normal Created 111s kubelet Created container coredns
Normal Started 111s kubelet Started container coredns
Warning Unhealthy 108s (x2 over 109s) kubelet Readiness probe failed: Get "http://10.244.70.129:8181/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
tech-0222@MSI:~$
#全部起動した
tech-0222@MSI:~$ kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-w5bpf 1/1 Running 0 6m11s
calico-node-tmfnl 1/1 Running 0 5m21s
coredns-7db6d8ff4d-f85fh 1/1 Running 0 6m11s
etcd-test-1 1/1 Running 0 34m
kube-apiserver-test-1 1/1 Running 0 34m
kube-controller-manager-test-1 1/1 Running 0 34m
kube-proxy-j2gwj 1/1 Running 0 34m
kube-scheduler-test-1 1/1 Running 0 34m
storage-provisioner 1/1 Running 1 (34m ago) 34m
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod calico-kube-controllers-ddf655445-w5bpf
Name: calico-kube-controllers-ddf655445-w5bpf
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m37s default-scheduler Successfully assigned kube-system/calico-kube-controllers-ddf655445-w5bpf to test-1
Warning FailedCreatePodSandBox 6m36s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "62945083cdf6b22b237ba209d70e86e2cf351dea0f01aed81ae813bc4485fe63": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Normal SandboxChanged 5m44s (x5 over 6m36s) kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulling 5m44s kubelet Pulling image "docker.io/calico/kube-controllers:v3.27.3"
Normal Pulled 87s kubelet Successfully pulled image "docker.io/calico/kube-controllers:v3.27.3" in 4m17.326s (4m17.326s including waiting). Image size: 33422304 bytes.
Normal Created 87s kubelet Created container calico-kube-controllers
Normal Started 87s kubelet Started container calico-kube-controllers
tech-0222@MSI:~$
- なんとなく、削除して起動してしまったので、Clusterを再作成して再度チャレンジしてみます。
- 結論
- POD「calico-node-XX」のSTATUS が Init:2/3 となったら、STATUS が Running になっていないpodを全て削除すると正常に起動できるみたいですね
- 結論
#15分ぐらい経過すると calico-node-5qdnn が 2/3 まで起動してくる
tech-0222@MSI:~$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-nx4dx 0/1 ContainerCreating 0 15m
calico-node-5qdnn 0/1 Init:2/3 0 15m
coredns-7db6d8ff4d-7ph2b 0/1 ContainerCreating 0 15m
etcd-test-1 1/1 Running 0 16m
kube-apiserver-test-1 1/1 Running 0 16m
kube-controller-manager-test-1 1/1 Running 0 16m
kube-proxy-2kw22 1/1 Running 0 16m
kube-scheduler-test-1 1/1 Running 0 16m
storage-provisioner 1/1 Running 1 (16m ago) 16m
tech-0222@MSI:~$
tech-0222@MSI:~$ date
Sat Aug 31 14:34:55 JST 2024
tech-0222@MSI:~$
#PODのStatusが全然変わらない。。
tech-0222@MSI:~$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-nx4dx 0/1 ContainerCreating 0 23m
calico-node-5qdnn 0/1 Init:2/3 0 23m
coredns-7db6d8ff4d-7ph2b 0/1 ContainerCreating 0 23m
etcd-test-1 1/1 Running 0 25m
kube-apiserver-test-1 1/1 Running 0 25m
kube-controller-manager-test-1 1/1 Running 0 25m
kube-proxy-2kw22 1/1 Running 0 25m
kube-scheduler-test-1 1/1 Running 0 25m
storage-provisioner 1/1 Running 1 (24m ago) 25m
tech-0222@MSI:~$
#calico-node-5qdnn が Init:2/3 になったあとで以下をみるとメッセージが違う
tech-0222@MSI:~$ kubectl -n kube-system describe pod coredns-7db6d8ff4d-7ph2b
～～略～～
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned kube-system/coredns-7db6d8ff4d-7ph2b to test-1
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "87f13ef3ed2f95bcc662cccc5416967f16d1e2180e0802e7e0e1f6e1dcfeaeb9": failed to find network info for sandbox "87f13ef3ed2f95bcc662cccc5416967f16d1e2180e0802e7e0e1f6e1dcfeaeb9"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "54503ab80caafb428beb80d988ed7abefff935ad9986237cf84018e73695f935": failed to find network info for sandbox "54503ab80caafb428beb80d988ed7abefff935ad9986237cf84018e73695f935"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "306256206afd42898ecd2b9745d8f8f010db27d3a08576f8f54af58928fabdfa": failed to find network info for sandbox "306256206afd42898ecd2b9745d8f8f010db27d3a08576f8f54af58928fabdfa"
Warning FailedCreatePodSandBox 23m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7faf3778ddef0237b82771050f6eed0e14b0b81484a20bbe9c934fdc60836561": failed to find network info for sandbox "7faf3778ddef0237b82771050f6eed0e14b0b81484a20bbe9c934fdc60836561"
Warning FailedCreatePodSandBox 23m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b680e1dac6ad48928eb67a2a684579edbb284e3052e85ae4025c76b58535b279": failed to find network info for sandbox "b680e1dac6ad48928eb67a2a684579edbb284e3052e85ae4025c76b58535b279"
Warning FailedCreatePodSandBox 23m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "be28e6119e5cd81ed742e49d35f905b24f2cef4e47c45a5db12e2731ca050b09": failed to find network info for sandbox "be28e6119e5cd81ed742e49d35f905b24f2cef4e47c45a5db12e2731ca050b09"
Warning FailedCreatePodSandBox 23m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "55ef2a62e049a296865efdcd7c020809c464da8d382d203f51fa7b74326e56c0": failed to find network info for sandbox "55ef2a62e049a296865efdcd7c020809c464da8d382d203f51fa7b74326e56c0"
Warning FailedCreatePodSandBox 22m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8a4802b8417ec35a52c90d3c7750db3f84e49d4866712629cf0bc59fb7a3793e": failed to find network info for sandbox "8a4802b8417ec35a52c90d3c7750db3f84e49d4866712629cf0bc59fb7a3793e"
Warning FailedCreatePodSandBox 22m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aca8fc1bdfa043d8ed48b9b06b549fba09d0f29b1cebdc88d961e775f1e9eb0b": failed to find network info for sandbox "aca8fc1bdfa043d8ed48b9b06b549fba09d0f29b1cebdc88d961e775f1e9eb0b"
Warning FailedCreatePodSandBox 14m (x39 over 22m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4479f9a6528e0d1e92a82621d174613a2eba306b66440bf430b2b942bebb0d72": failed to find network info for sandbox "4479f9a6528e0d1e92a82621d174613a2eba306b66440bf430b2b942bebb0d72"
Normal SandboxChanged 4m24s (x34 over 11m) kubelet Pod sandbox changed, it will be killed and re-created.
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system describe pod calico-kube-controllers-ddf655445-nx4dx
～～略～～
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26m default-scheduler Successfully assigned kube-system/calico-kube-controllers-ddf655445-nx4dx to test-1
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8add1b1f6e86801916481635e11d4319f71ad3dcd96d405187778d7e867c75cb": failed to find network info for sandbox "8add1b1f6e86801916481635e11d4319f71ad3dcd96d405187778d7e867c75cb"
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6c8b707eff953ced1e8e3d89177e4fbd2e52b8047aa766e79e46963845f72388": failed to find network info for sandbox "6c8b707eff953ced1e8e3d89177e4fbd2e52b8047aa766e79e46963845f72388"
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d115bbd5e4c166055e30fe0bf67dc0efb25c916d625789cef46c12013afdcdcc": failed to find network info for sandbox "d115bbd5e4c166055e30fe0bf67dc0efb25c916d625789cef46c12013afdcdcc"
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9ca27656d85b3046d6c4fa9ae25fb6cfd9ea71aa842cc7d2bf37f322ccc57cea": failed to find network info for sandbox "9ca27656d85b3046d6c4fa9ae25fb6cfd9ea71aa842cc7d2bf37f322ccc57cea"
Warning FailedCreatePodSandBox 25m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "815305782df917886117641d47c9a164cb40c28e03eb92c20dd90069ab3ab0b4": failed to find network info for sandbox "815305782df917886117641d47c9a164cb40c28e03eb92c20dd90069ab3ab0b4"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1b9bad13273af7d9c40b1beab732907d8bf6fcbe4e5485a42aed55cef72752d3": failed to find network info for sandbox "1b9bad13273af7d9c40b1beab732907d8bf6fcbe4e5485a42aed55cef72752d3"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dacdf5d6df0458948d76a57b7768c69def07dd8be47911a91cbb1ed5fc136bee": failed to find network info for sandbox "dacdf5d6df0458948d76a57b7768c69def07dd8be47911a91cbb1ed5fc136bee"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c00e3bc4ed08fb348e3d5509c895561064c674018509c30ec98c7ab4fccf4c0f": failed to find network info for sandbox "c00e3bc4ed08fb348e3d5509c895561064c674018509c30ec98c7ab4fccf4c0f"
Warning FailedCreatePodSandBox 24m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dde0e81cc4e6901d8a541d8e1ec2e7cbca692192c9e3b740e8fd4b078efd3273": failed to find network info for sandbox "dde0e81cc4e6901d8a541d8e1ec2e7cbca692192c9e3b740e8fd4b078efd3273"
Warning FailedCreatePodSandBox 15m (x39 over 24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "63b8c96ad1fede038d7302289169e646c0fbdd2008fb7d123a65696724f98144": failed to find network info for sandbox "63b8c96ad1fede038d7302289169e646c0fbdd2008fb7d123a65696724f98144"
Normal SandboxChanged 57s (x58 over 13m) kubelet Pod sandbox changed, it will be killed and re-created.
tech-0222@MSI:~$
# StatusがRunning でないPodを全部削除するPODが起動してきますね
tech-0222@MSI:~$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-kzncj 0/1 ContainerCreating 0 5m28s
calico-node-lmvs9 0/1 Init:2/3 0 3s
coredns-7db6d8ff4d-h2qrm 1/1 Running 0 5m28s
etcd-test-1 1/1 Running 0 35m
kube-apiserver-test-1 1/1 Running 0 35m
kube-controller-manager-test-1 1/1 Running 0 35m
kube-proxy-2kw22 1/1 Running 0 34m
kube-scheduler-test-1 1/1 Running 0 35m
storage-provisioner 1/1 Running 1 (34m ago) 34m
tech-0222@MSI:~$
#少し経過すると全部起動してきました
tech-0222@MSI:~$ k get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-ddf655445-kzncj 1/1 Running 0 103m
kube-system calico-node-lmvs9 1/1 Running 0 97m
kube-system coredns-7db6d8ff4d-h2qrm 1/1 Running 0 103m
kube-system etcd-test-1 1/1 Running 0 132m
kube-system kube-apiserver-test-1 1/1 Running 0 132m
kube-system kube-controller-manager-test-1 1/1 Running 0 132m
kube-system kube-proxy-2kw22 1/1 Running 0 132m
kube-system kube-scheduler-test-1 1/1 Running 0 132m
kube-system storage-provisioner 1/1 Running 1 (131m ago) 132m
tech-0222@MSI:~$- スペック不足かと思いCPU、Memoryを2倍にしてClusterを再作成しましたが、同じ結果になりました
- 結論
- おそらくPod「calico-node-XX」のイメージサイズが大きくダウンロードに時間がかかる為、最初に起動してほしいPodが正常に起動しないみたいですね
- 私のネットワーク上の問題かもしれませんが、同じ事象になった際には、イメージがダウンロードされてからPodを再作成することを試してください
- 結論
tech-0222@MSI:~$ minikube profile list
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| minikube | docker | docker | 192.168.58.2 | 8443 | v1.28.3 | Running | 1 | * | |
| test-1 | docker | containerd | 192.168.49.2 | 8443 | v1.30.0 | Running | 1 | | * |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
tech-0222@MSI:~$
tech-0222@MSI:~$ minikube start --cni calico --container-runtime=containerd --bootstrapper=kubeadm --cpus=4 --memory=8192 -p test-2
😄 [test-2] minikube v1.33.1 on Ubuntu 22.04 (amd64)
✨ Automatically selected the docker driver. Other choices: kvm2, qemu2, ssh
📌 Using Docker driver with root privileges
👍 Starting "test-2" primary control-plane node in "test-2" cluster
🚜 Pulling base image v0.0.44 ...
🔥 Creating docker container (CPUs=4, Memory=8192MB) ...
📦 Preparing Kubernetes v1.30.0 on containerd 1.6.31 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring Calico (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "test-2" cluster and "default" namespace by default
tech-0222@MSI:~$
tech-0222@MSI:~$ minikube profile list
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
| minikube | docker | docker | 192.168.58.2 | 8443 | v1.28.3 | Running | 1 | * | |
| test-1 | docker | containerd | 192.168.49.2 | 8443 | v1.30.0 | Running | 1 | | |
| test-2 | docker | containerd | 192.168.67.2 | 8443 | v1.30.0 | Running | 1 | | * |
|----------|-----------|------------|--------------|------|---------|---------|-------|----------------|--------------------|
tech-0222@MSI:~$
#事象変わらず・・
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl -n kube-system get po -w
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-56sp7 0/1 ContainerCreating 0 48s
calico-node-j8mrw 0/1 Init:0/3 0 49s
coredns-7db6d8ff4d-bsfss 0/1 ContainerCreating 0 48s
etcd-test-2 1/1 Running 0 62s
kube-apiserver-test-2 1/1 Running 0 63s
kube-controller-manager-test-2 1/1 Running 0 62s
kube-proxy-dxnkh 1/1 Running 0 49s
kube-scheduler-test-2 1/1 Running 0 62s
storage-provisioner 1/1 Running 0 61s
tech-0222@MSI:~$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-ddf655445-56sp7 0/1 ContainerCreating 0 14m
calico-node-j8mrw 0/1 Init:2/3 0 14m
coredns-7db6d8ff4d-bsfss 0/1 ContainerCreating 0 14m
etcd-test-2 1/1 Running 0 15m
kube-apiserver-test-2 1/1 Running 0 15m
kube-controller-manager-test-2 1/1 Running 0 15m
kube-proxy-dxnkh 1/1 Running 0 14m
kube-scheduler-test-2 1/1 Running 0 15m
storage-provisioner 1/1 Running 0 15m
tech-0222@MSI:~$
#イメージファイルを調べるとかなり大きい
tech-0222@MSI:~$ eval $(minikube docker-env -p test-2)
❗ Using the docker-env command with the containerd runtime is a highly experimental feature, please provide feedback or contribute to make it better
Identity added: /home/tech-0222/.minikube/machines/test-2/id_rsa (/home/tech-0222/.minikube/machines/test-2/id_rsa)
Host added: /home/tech-0222/.ssh/known_hosts ([127.0.0.1]:32807)
tech-0222@MSI:~$
#ダウンロード前
tech-0222@MSI:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
kindest/kindnetd v20240202-8f1494ea 61f9956af801 4 months ago 68.6MB
gcr.io/k8s-minikube/storage-provisioner v5 18eb69d1418e 4 months ago 31.5MB
registry.k8s.io/coredns/coredns v1.11.1 1eeb4c7316ba 4 months ago 63.9MB
registry.k8s.io/etcd 3.5.12-0 44a8e24dcbba 4 months ago 153MB
registry.k8s.io/kube-apiserver v1.30.0 6b8e197b2d39 4 months ago 120MB
registry.k8s.io/kube-controller-manager v1.30.0 5f52f00f17d5 4 months ago 114MB
registry.k8s.io/kube-proxy v1.30.0 ec532ff47eaf 4 months ago 88.5MB
registry.k8s.io/kube-scheduler v1.30.0 2353c3a18032 4 months ago 65.2MB
registry.k8s.io/pause 3.9 7031c1b28338 4 months ago 750kB
tech-0222@MSI:~$
#ダウンロード後
tech-0222@MSI:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
calico/cni v3.27.3 1f2c6a13d436 2 minutes ago 196MB ★これ
kindest/kindnetd v20240202-8f1494ea 61f9956af801 4 months ago 68.6MB
gcr.io/k8s-minikube/storage-provisioner v5 18eb69d1418e 4 months ago 31.5MB
registry.k8s.io/coredns/coredns v1.11.1 1eeb4c7316ba 4 months ago 63.9MB
registry.k8s.io/etcd 3.5.12-0 44a8e24dcbba 4 months ago 153MB
registry.k8s.io/kube-apiserver v1.30.0 6b8e197b2d39 4 months ago 120MB
registry.k8s.io/kube-controller-manager v1.30.0 5f52f00f17d5 4 months ago 114MB
registry.k8s.io/kube-proxy v1.30.0 ec532ff47eaf 4 months ago 88.5MB
registry.k8s.io/kube-scheduler v1.30.0 2353c3a18032 4 months ago 65.2MB
registry.k8s.io/pause 3.9 7031c1b28338 4 months ago 750kB
tech-0222@MSI:~$minikubeでnetwork policyを試してみる
calicoが正常に動作していれば、network policy が動作するはずなので、
Kubernetesのドキュメント( Declare Network Policy)を参考に設定を試してみます
- nginx DeploymentとServiceを作成します
#deployment 作成
tech-0222@MSI:~$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
#Serviceを作成
tech-0222@MSI:~$ kubectl expose deployment nginx --port=80
service/nginx exposed
tech-0222@MSI:~$
#リソース確認
tech-0222@MSI:~$ kubectl get svc,pod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h25m
service/nginx ClusterIP 10.96.45.181 <none> 80/TCP 7m23s
NAME READY STATUS RESTARTS AGE
pod/nginx-bf5d5cf98-bd778 1/1 Running 0 7m28s
tech-0222@MSI:~$- nginx にアクセスを行うPodを作成しnginx にアクセスを試します
- 何も制限してないのでリクエストに対して応答がありますね
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl run busybox --rm -ti --image=busybox -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.96.45.181:80)
remote file exists
/ #
/ # exit
- nginx Serviceへのアクセスを制限するために、access: trueというラベルが付いたPodだけがクエリできるようにします
#マニフェストを作成
tech-0222@MSI:~$ cd /tmp/
tech-0222@MSI:tmp$ vi nginx-policy.yaml
tech-0222@MSI:tmp$ cat nginx-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
app: nginx
ingress:
- from:
- podSelector:
matchLabels:
access: "true"
tech-0222@MSI:tmp$
#デプロイ
tech-0222@MSI:tmp$ kubectl apply -f nginx-policy.yaml
networkpolicy.networking.k8s.io/access-nginx created
tech-0222@MSI:tmp$
#確認
tech-0222@MSI:tmp$ kubectl get networkpolicies.networking.k8s.io
NAME POD-SELECTOR AGE
access-nginx app=nginx 30s
tech-0222@MSI:tmp$
- accessラベルが定義されていない状態でServiceへのアクセスをテストする
- ポリシー通り、リクエストがタイムアウトしますね
tech-0222@MSI:~$ kubectl run busybox --rm -ti --image=busybox -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.96.45.181:80)
wget: download timed out
/ #
/ # exit
- nginx にアクセスを行うPodにaccessラベルを設定して作成してテストを試してみます
- ポリシー通り、ラベルが一致しているのでアクセスが成功しました
- これで問題なく、NetworkPolicy が正常に動作していることが確認できました！
tech-0222@MSI:~$
tech-0222@MSI:~$ kubectl run busybox --rm -ti --labels="access=true" --image=busybox -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.96.45.181:80)
remote file exists
/ #
tech-0222@MSI:tmp$ kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
busybox 1/1 Running 0 69s access=true
nginx-bf5d5cf98-bd778 1/1 Running 0 18m app=nginx,pod-template-hash=bf5d5cf98
tech-0222@MSI:tmp$
