I am a newbie to Kubernetes, and am facing some issues with the Multus plugin. I am using CentOS-7 and minikube.
$ minikube version
minikube version: v1.18.0
commit: ec61815d60f66a6e4f6353030a40b12362557caa-dirty
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
I am trying to integrate the multus plugin with K8s. To do this, I obtained the source code for multus from: https://github.com/k8snetworkplumbingwg/multus-cni
Compiled it with ./hack/build-go.sh
Then I have been following the details in https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md.
However, the pod never gets created; it gets stuck in ContainerCreating forever.
I create the network-attachment-definitions with the following in a yml file and kubectl apply -f:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens33",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.152.0/24",
"rangeStart": "192.168.152.200",
"rangeEnd": "192.168.152.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.152.1"
}
}'
This is the output of kubectl describe:
Name: macvlan-conf
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8s.cni.cncf.io/v1
Kind: NetworkAttachmentDefinition
Metadata:
Creation Timestamp: 2021-03-16T17:07:38Z
Generation: 1
Managed Fields:
API Version: k8s.cni.cncf.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:config:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-03-16T17:07:38Z
Resource Version: 102230
UID: 8473cdbf-1906-48b8-b180-e6f49502254d
Spec:
Config: { "cniVersion": "0.3.0", "type": "macvlan", "master": "ens33", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.152.0/24", "rangeStart": "192.168.152.200", "rangeEnd": "192.168.152.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.152.1" } }
Events: <none>
Then I create the pod using the following yml and kubectl apply -f :
apiVersion: v1
kind: Pod
metadata:
name: samplepod1
spec:
containers:
- name: samplepod1
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
The output of kubectl get pods is (samplepod1 is the relevant one):
NAME READY STATUS RESTARTS AGE
samplepod 0/1 Terminating 0 54m
samplepod1 0/1 ContainerCreating 0 35s
kubectl describe pods gives me:
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Tue, 16 Mar 2021 13:15:08 -0400
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
samplepod1:
Container ID:
Image: alpine
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/ash
-c
trap : TERM INT; sleep infinity & wait
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7vg94 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-7vg94:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7vg94
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 64s default-scheduler Successfully assigned default/samplepod1 to minikube
Warning FailedCreatePodSandBox 62s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "26bcf14c4ec6cf8b532233abeefd7f6d9b2493b62963692d6460215dbb86916d" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 59s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e478b9559dc0837e2f1e1a78f1ca4f40b521a5176b9680ffefcaa4b485981a06" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 56s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7749375a5a4598313bace4a83d14e146af73e686098f8d82ccf1700d49663a1b" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 53s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0d5c368e0b953c7463937d7c8f659b8554a467187a9f28462a348a7b64534689" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 50s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a93077c4f10db6c81d6ad23e353ebe024d32b5cbaf12e6618255c5286890ba34" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 47s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cc760d4034006f563bea6a4173e7fe0ae10e8bebe24327c9b28798121c9ca5d3" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "df255fbcd396c7f3bca09bce56220285aef8f063016ebc71f78428deb7ee24fb" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 41s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "62e3b999cb14cb116a2245d383fc05f819d0786da4c0ee4a24ce64fb239a9851" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07464566ca12539d54e73c3bf1ac73faebed6634703250954a8329e948954faf" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Normal SandboxChanged 29s (x12 over 61s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 26s (x4 over 36s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2c75fc7558d368e364a3c48793deb93dd77b6579b9a47eb8a82a7b4037ae344d" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
In this case, I have not even used "annotations" in the spec for the pod.
Could someone help me debug this?
Edited: I re-did the entire sequence. Point to note, is that I had to get a totally new and fresh RHEL8 VM installation because I'd destroyed the previous one totally. Now the results is still a failure, but slightly different as below.
The steps followed were :
$ sudo systemctl start docker
$ minikube start --driver=docker
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 24h v1.20.2
$ git clone https://github.com/intel/multus-cni.git && cd multus-cni
$ cat ./images/multus-daemonset.yml | kubectl apply -f -
At this point, its worth noting that there is nothing in the /opt/cni/bin directory (this was not even created), and the same situation with the /etc/cni/net.d/multus directory.
This was followed by:
$ kubectl get pods --all-namespaces | grep multus
kube-system kube-multus-ds-khd6k 1/1 Running 0 3m7s
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens160",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
EOF
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-conf created
$ kubectl describe network-attachment-definitions macvlan-conf
Name: macvlan-conf
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8s.cni.cncf.io/v1
Kind: NetworkAttachmentDefinition
Metadata:
Creation Timestamp: 2021-03-17T18:41:34Z
Generation: 1
Managed Fields:
API Version: k8s.cni.cncf.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:config:
Manager: kubectl-create
Operation: Update
Time: 2021-03-17T18:41:34Z
Resource Version: 1597
UID: 29e54d08-70ba-4ad6-8031-f22ad2a73382
Spec:
Config: { "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart": "192.168.1.200", "rangeEnd": "192.168.1.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.1.1" } }
Events: <none>
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
EOF
pod/samplepod created
$ kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:06 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.6/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
I was expecting a "net1" interface also as in the description on the web page: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md
The cni directories under /opt and /etc were never created.
Thanks for your help, in advance.
