sfgroups's questions

I have installed containerd 1.4.9 on CentOS steam 8 server.

based on this document https://containerd.io/docs/getting-started/. I have created default config file containerd config default > /etc/containerd/config.toml like this.

after restarting containerd, when I run crictl ps Its throwing below error

FATA[0000] listing containers failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

How to fix this error? after fix this I want to join this node to Kubernets cluster 1.21.3 using systemd cfgroup.

Thanks SR

I am trying to join new node to existing v1.21.3 cluster with Calico CNI. join command giving clusterCIDR warning.

How to fix this subnet warning message?

# kubeadm join master-vip:8443 --token xxx --discovery-token-ca-cert-hash sha256:xxxx
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0809 14:10:27.717696   75868 utils.go:69] The recommended value for "clusterCIDR" in "KubeProxyConfiguration" is: 10.201.0.0/16; the provided value is: 10.203.0.0/16
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

update:

I was using 10.201.0.0/16 during the cluster setup, later I changed to 10.203.0.0/16. not sure where its still getting 10.201.0.0/16 subnet value.

Here is the sub net value.

# sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr
    - --cluster-cidr=10.203.0.0/16

kubectl cluster-info dump | grep cluster-cidr
                            "--cluster-cidr=10.203.0.0/16",
                            "--cluster-cidr=10.203.0.0/16",
                            "--cluster-cidr=10.203.0.0/16",

step to update pod CIDR from 10.201.0.0/16 to 10.203.0.0/16

1. using this command updated the kubeadm-confg configmap kubectl -n kube-system edit cm kubeadm-config

podSubnet: 10.203.0.0/16

2. Updated kuber-controller-manger and restarted it.

sed -i 's/10.201.0.0/10.203.0.0/' /etc/kubernetes/manifests/kube-controller-manager.yaml

after updating the IP.

all config shows subnet as 10.203.0.0 but pods creating in `10.201.0.0' subnet.

# kubectl get cm kube-proxy -n kube-system -o yaml |grep -i clusterCIDR
    clusterCIDR: 10.203.0.0/16
# kubectl get no -o yaml |grep -i podcidr
    podCIDR: 10.203.0.0/24
    podCIDRs:
    podCIDR: 10.203.1.0/24
    podCIDRs:
    podCIDR: 10.203.2.0/24
    podCIDRs:
    podCIDR: 10.203.3.0/24
    podCIDRs:
    podCIDR: 10.203.5.0/24
    podCIDRs:
    podCIDR: 10.203.4.0/24
    podCIDRs:
    podCIDR: 10.203.6.0/24
    podCIDRs:

I have delete kube-proxy pod from my master nodes to get the new kube-proxy configmap values, when I reboot the node still kube-proxy is not coming up ?

How to start the kube-proxy?

# crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
19376bdbc55eb       c4d75af7e098e       17 minutes ago      Running             calico-node         2                   b4fb351577dd4
24253bf076500       6be0dc1302e30       17 minutes ago      Running             kube-scheduler      4                   182f9024a6957
57c5c9dec4ad2       3d174f00aa39e       17 minutes ago      Running             kube-apiserver      3                   e549a2bad6a02

Cluster information:

Kubernetes version: v1.21.3
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: bare-metal
Host OS: CentOS Stream 8
CNI and version: Calico
CRI and version: containerd

I have upgraded Kubernets cluster ( 3 master, 3 etcd servers database) from 1.18 to v1.19.13 and ETCD to 3.4.13. since than API servers are not stable, keep switching different etcd server, because of this cluster is not working properly. these cluster running on CentOS steam 8. This Cluster worked before the upgrade, after upgrade only I have seen this issue.

Any help to resolve this issue? Is there know issue with this version?

API server logs

I0731 00:54:39.498953       1 client.go:360] parsed scheme: "passthrough"
I0731 00:54:39.499025       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://0.0.0.02:2379  <nil> 0 <nil>}] <nil> <nil>}
I0731 00:54:39.499035       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0731 00:54:40.241615       1 client.go:360] parsed scheme: "passthrough"
I0731 00:54:40.241681       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://0.0.0.01:2379  <nil> 0 <nil>}] <nil> <nil>}
I0731 00:54:40.241691       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0731 00:54:45.348969       1 client.go:360] parsed scheme: "passthrough"
I0731 00:54:45.349030       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://0.0.0.03:2379  <nil> 0 <nil>}] <nil> <nil>}
I0731 00:54:45.349040       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0731 00:55:16.460379       1 client.go:360] parsed scheme: "passthrough"
I0731 00:55:16.460428       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://0.0.0.01:2379  <nil> 0 <nil>}] <nil> <nil>}
I0731 00:55:16.460439       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0731 00:55:17.461906       1 client.go:360] parsed scheme: "passthrough"

etcd looks healthy

# /opt/bin/etcdctl.sh   version
etcdctl version: 3.4.13
API version: 3.4

 /opt/bin/etcdctl.sh  endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 9.739533ms



# /opt/bin/etcdctl.sh  check perf
 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.042491s
PASS: Stddev is 0.001743s
PASS

# /opt/bin/etcdctl.sh  endpoint status --cluster -w table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://0.0.0.02:2379 | 15cd65a732ebd5d8 |  3.4.13 |   26 MB |     false |      false |      9305 |   19813854 |           19813854 |        |
| https://0.0.0.03:2379 | add66a254676e690 |  3.4.13 |   26 MB |      true |      false |      9305 |   19813854 |           19813854 |        |
| https://0.0.0.01:2379 | e2811ed02ce71623 |  3.4.13 |   26 MB |     false |      false |      9305 |   19813854 |           19813854 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

I have deployed 3 node external ETCD database (etcdctl version: 3.4.7) cluster for my Kubernetes v1.18.6 cluster using etcdadm tool. my certificate is expring in couple of months.

I believe kubeadm alpha certs renew all command will renew kubernetes certificates. May Know the correct steps to renew external ETCD database cluster certificate?

my cluster certificate details

# kubeadm alpha certs check-expiration 

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 20, 2021 14:13 UTC   152d                                    no
apiserver                  Jul 20, 2021 14:13 UTC   152d            ca                      no
apiserver-kubelet-client   Jul 20, 2021 14:13 UTC   152d            ca                      no
controller-manager.conf    Jul 20, 2021 14:13 UTC   152d                                    no
front-proxy-client         Jul 20, 2021 14:13 UTC   152d            front-proxy-ca          no
scheduler.conf             Jul 20, 2021 14:13 UTC   152d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 17, 2030 01:19 UTC   9y              no
front-proxy-ca          Apr 17, 2030 01:19 UTC   9y              no

Master node certificate details

/etc/kubernetes/pki/ca.crt,             Apr 17 01:19:52 2030 GMT
/etc/kubernetes/pki/apiserver.crt,             Jul 20 14:13:09 2021 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt,             Jul 20 14:13:10 2021 GMT
/etc/kubernetes/pki/front-proxy-ca.crt,             Apr 17 01:19:52 2030 GMT
/etc/kubernetes/pki/front-proxy-client.crt,             Jul 20 14:13:10 2021 GMT


/etc/etcd/pki/ca.crt,             Apr 17 01:19:35 2030 GMT
/etc/etcd/pki/server.crt,             Apr 19 01:19:36 2021 GMT
/etc/etcd/pki/peer.crt,             Apr 19 01:19:36 2021 GMT
/etc/etcd/pki/etcdctl-etcd-client.crt,             Apr 19 01:19:36 2021 GMT
/etc/etcd/pki/apiserver-etcd-client.crt,             Apr 19 01:19:36 2021 GMT

Thanks SR

I have removed the user from server local administrator group couple of days back. But its not updated in the SCCM database. How do I pull the local administrator group information to SCCM database?

Thanks SR

we have file in Linux and copied to AIX. same file smaller size in AIX.

Linux - ext3 file system AIX - jfs2 file system

both file systems in same block size 4k

size in Linux
     du -sk p14727347_112030_AIX64-5L.zip
    1160568 p14727347_112030_AIX64-5L.zip
Size in AIX    
      du -sk p14727347_112030_AIX64-5L.zip
    1159484 p14727347_112030_AIX64-5L.zip

Any idea why its different

Is there a way to get mount point permission. after filesystem is Mounted.

Thanks