rookie099's questions

I have a Pacemaker cluster that controls several resources of type ocf:heartbeat:IPaddr2 and one of type ocf:heartbeat:nginx. Since an upgrade to Debian 12 it can no longer start Nginx. What happens is that Pacemaker tries to bring up Nginx on one side, gives up after 40 seconds, then tries on the other side, and again gives up after 40 seconds. During both 40 second intervals Nginx appears to work normally. After these attempts the Nginx resource remains stopped and the error message reads "nginx start on ... could not be executed (Timed Out: Resource agent did not complete within 40s)".

I've gone through various log files but could not find a cause yet. How can I further diagnose and solve this issue? (As a workaround, I can start Nginx outside Pacemaker's control with systemctl start nginx.service. This is suboptimal compared to the cluster's normal operation, but does show that the configuration is intact.)

When attempting to run Apache inside a systemd container that was started with systemd-nspawn --private-users=pick ... (unlike --private-users=false in this solution) I encounter this error:

Permission denied: AH00072: make_sock: could not bind to address ...:999

What puzzles me is that the container has been granted capability CAP_NET_BIND_SERVICE (getpcaps 1 inside the container also confirms this), --capability=help indicates that the capability is supported, and netcat -l 999 -s ... (also inside the container) can apparently listen on the same port just fine.

What am I missing? Shouldn't the capability allow processes inside the container to open well-known ports on the host, no matter what their PIDs are?

UPDATE I made a mistake with the invocation of netcat. The correct command line is netcat -vl -p 999 -s ... and this now produces "Can't grab ...:999 with bind : Permission denied". So in fact neither Apache nor netcat can bind at this point, and this is not Apache-specific. Two more facts about the configuration: container is running as root (mapped to non-root pid on host), iptables on host are empty.

UPDATE So perhaps capability CAP_NET_BIND_SERVICE simply cannot transcend user namespaces.

I have a situation here an Nginx (nginx/1.14.2) reverse proxy acts as a load balancer for a couple of HTTP servers. Now and then an HTTP response does not get through to the client, as if the connection were cut off too early. This does not occur without the load balancer sitting between the clients and HTTP servers, so I guess it must have to do with the Nginx configuration. Why could this be the case and how can if further diagnose and ultimately fix the situation?

I have increased error_log to info and observe that only info (e.g. no warning) messages occur during the situation. I have also installed the NGINX Prometheus Exporter and it shows nginx_connections_accepted equal to nginx_connections_handled throughout the situation, so apparently it does not notice any failed connections (e.g. due to timeouts). nginx_connections_active goes up to 19 in my case, which should be fine since worker_processes * worker_connections = 2 * 768 >> 19. Among 100000 or so handled connections in total, only a few go wrong as described.

Typical entries from my Nginx configuration with respect to load balancing look as follows. I have chosen TCP load balancing (rather than specifically HTTP load balancing) on purpose.

stream {
  # ...
  server {
    listen 1.2.3.4:443;
    proxy_pass u_123;
  }
  upstream u_123 {
    server 1.2.3.5:2345 max_fails=0;
    server 1.2.3.6:2345 max_fails=0;
    server 1.2.3.7:2345 max_fails=0;
    server 1.2.3.8:2345 max_fails=0;
  }
  # ...
}

I am trying to get started with PodSecurityPolicy on a bare-metal Kubernetes 1.18.3 cluster with user management provided by Keycloak. psp/restricted should apply in namespace/restricted (both for a specific user user and the namespace's serviceaccount/default) , and psp/unrestricted should apply in namespace/unrestricted. I have the basics working (admission controller PodSecurityPolicy installed, etc.), and the following resources are in place:

apiVersion: v1
kind: List
items:
- kind: Role
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: user
    namespace: restricted
  rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
- kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: user
    namespace: restricted
  subjects:
  - kind: Group
    name: user
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: user
    apiGroup: rbac.authorization.k8s.io
- kind: PodSecurityPolicy
  apiVersion: policy/v1beta1
  kind: PodSecurityPolicy
  metadata:
    name: restricted
  spec:
    privileged: false
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    fsGroup:
      rule: RunAsAny
    volumes:
    - '*'
- kind: PodSecurityPolicy
  apiVersion: policy/v1beta1
  kind: PodSecurityPolicy
  metadata:
    name: unrestricted
  spec:
    privileged: true
    hostNetwork: true
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    fsGroup:
      rule: RunAsAny
    volumes:
    - '*'
- kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: restricted
  rules:
  - apiGroups: ["policy"]
    resources: ["podsecuritypolicies"]
    verbs: ["use"]
    resourceNames:
    - restricted
- kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: unrestricted
  rules:
  - apiGroups: ["policy"]
    resources: ["podsecuritypolicies"]
    verbs: ["use"]
    resourceNames:
    - unrestricted
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: restricted
  subjects:
  - kind: ServiceAccount
    name: default
    namespace: restricted
  - kind: User
    name: user
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: ClusterRole
    name: restricted
    apiGroup: rbac.authorization.k8s.io
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: unrestrictied
  subjects:
  - kind: Group
    name: system:nodes
    apiGroup: rbac.authorization.k8s.io
  - kind: ServiceAccount
    name: default
    namespace: unrestricted
  roleRef:
    kind: ClusterRole
    name: unrestricted
    apiGroup: rbac.authorization.k8s.io

In terms of use permission, everything looks as expected, e.g.:

kubectl auth can-i use podsecuritypolicy/restricted --as user --as-group=system:authenticated # yes
kubectl auth can-i use podsecuritypolicy/unrestricted --as user --as-group=system:authenticated # no

But what I observe is that while serviceaccount:restricrted:default cannot created privileged pods in namespace/restricted, user user apparently still can (at a time when that user is authenticated to the cluster):

kubectl create -f - <<EOF # succeeds (as expected)
apiVersion: v1
kind: Pod
metadata:
  name: unprivileged-test-pod
  namespace: restricted
spec:
  containers:
  - name:  pause
    image: k8s.gcr.io/pause
EOF
kubectl create -f - <<EOF # succeeds (unexpected)
apiVersion: v1
kind: Pod
metadata:
  name: privileged-test-pod
  namespace: restricted
spec:
  containers:
  - name:  pause
    image: k8s.gcr.io/pause
    securityContext:
      privileged: true
EOF

Both created containers bear an annotation kubernetes.io/psp: unrestricted, whereas I would have expected the creation of pod/unrestricted for when user user is authenticated in to fail. Things work as expected with kubectl create deployment (i.e. creation of the restricted unrestrictred deployments both indirectly by serviceaccount:default in namespace restricted succeed and fail respectively. Somehow the user (but not the service account appears to be bound to a too wide security policy.

What am I missing? How can I further diagnose the and solve the problem (i.e. prevent both serviceaccount/default in namespace/restricted and user user from creating privileged pods in namespace/restricted?

UPDATE I think I have now isolated to root cause, but do not know a good solution yet. It would appear that resources: ["*"], verbs: ["*"] in role/user also grants permission to use any (cluster-wide) resource psp. This is unintended: I want for role/user to permit user the "usual" activities inside namespace/restricted, not let it use each and any psp as well.

I have a simple Docker container for a purposeful sshd server whose image derives from alpine:latest. It currently uses /usr/sbin/sshd -D -e -ddd as its entrypoint. When I ssh into it for running a certain command remotely, that command executes alright, but then sshd and hence the container immediately shut down.

Elsewhere there is a recommendation titled "SSH login fix. Otherwise user is kicked off after login" in terms of enabling session optional pam_loginuid.so in /etc/pam.d/sshd. This sounds promising, but Alpine's sshd (from apk add openssh) apparently comes without PAM, plus I am not sure how this would function.

Can somebody please enlighten me as to how sshd inside an Alpine-based container can be made to persist across many SSH sessions in order to support normal operation?

UPDATE Here are the last few lines of sshd's log output before it shuts down:

Received disconnect from 1.2.3.4 port 29213:11: disconnected by user
Disconnected from user rts 1.2.3.4 port 29213
debug1: do_cleanup
debug3: mm_request_receive entering
debug1: do_cleanup

One of the nodes in my Kubernetes v1.16.3 cluster has ended up in a strange state. The node is running Debian 10.1 and Docker 19.03.1. When I try to kubectl exec sh into a container on that node I see these error messages in the node's journalctl -u docker (the 2nd is also reported by kubectl itself):

stream copy error: reading from a closed fifo
Error running exec ... in container: failed to create runc console socket: mkdir /tmp/pty...: no space left on device ..."

At this point I don't see which device may have become full. A simple df does not raise any alarms. How can I further diagnose and remedy the situation?

BTW, the problem so far has persisted across kubectl drain, systemctl restart docker, restarting kube-proxy and the CNI's pod, as well as several other measures.

I would like to prepare the environment for a Docker container such that when apt-get upgrade -s runs inside the container its result is the same as if the command were run outside on the host. This is in the context of and in preparation for adding a text collector apt.sh to an existing Prometheus node-exporter. I am already using a Kubernetes DaemonSet to control such containers (pods) in a way similar to an example from GitHub prometheus-community/node-exporter-textfile-collector-scripts.

Judging from its man page, apt-get works with files in /etc/apt, /var/cache/apt, and /var/lib/apt, and so my container mounts those read-only from the host. Currently my host and container run Debian 9 (Stretch) and Debian 10 (Buster) respectively. I account for this by running apt-get upgrade -s outside the container and apt-get upgrade -s -t stretch inside.

However the two runs still produce different results. What am I missing?

Pacemaker resource agents mention default values for their operations' timeouts, intervals, etc., in their metadata. For instance, crm_resource --show-metadata=ocf:heartbeat:IPaddr2 lists some:

<actions>
<action name="start" timeout="20s" />
<action name="stop" timeout="20s" />
<action name="status" depth="0" timeout="20s" interval="10s" />
<action name="monitor" depth="0" timeout="20s" interval="10s" />
<action name="meta-data" timeout="5s" />
<action name="validate-all" timeout="20s" />
</actions>

Will these be taken as default values when I do not specify explicit values when configuring a resource (e.g. with crm configure primitive ip_test IPaddr2 params ip=1.2.3.4 nic=eth0 cidr_netmask=32 iflabel=test) or will I explicitly have to set them when I want to employ such defaults (e.g. with crm configure primitive ip_test ... op monitor timeout=20s interval=10s)?

My current assumption is that the second is the case, and that all timeouts, intervals, etc., will normally default to 0 (not to any default values referred to in the meta-data) unless explicitly set otherwise. But I have not found any unambiguous statement in the documentation, nor a command that would show me all relevant values (crm resource show ip_nginx), so I am reaching out to somebody who may know better.

Can I configure a DNS zone such that both a.b.c and b.c can serve as host names? In other words, can b.c be both a subdomain and a hostname at the same time?

Is this perhaps just a matter of serving both A and NS (for delegation) records for b in c's zone?

The DRBD documentation (in section Integrating DRBD with Pacemaker clusters) recommends that DRBD should be disabled in a Pacemaker cluster:

If you are employing the DRBD OCF resource agent, it is recommended that you defer DRBD startup, shutdown, promotion, and demotion exclusively to the OCF resource agent. That means that you should disable the DRBD init script: chkconfig drbd off.

Under systemd this would amount to systemctl disable drbd.service.

Is there any harm in enabling DRBD despite of this recommendation? The idea would be to enable DRBD, but to disable Corosync and Pacemaker, so that after a cluster node fails and reboots it will continue receiving DRBD-synced data but will otherwise remain "passive". This should allow for the failed node to be analyzed before it can reenter the cluster, but live data still being saved on both cluster nodes in the meantime. What is the rationale behind the recommendation that would outweigh this rationale?

Does it make sense that apt-cache depends/rdepends are not always inverse to each other?

For instance, I have a host where apt-cache rdepends liblognorm1 lists these reverse dependencies: rsyslog liblognorm-dev, but apt-cache depends rsyslog does not include liblognorm1 (only liblognorm5) in its dependencies.

Is this a feature or a bug?

I have a situation where aptitude marks a package cgroup-bin as broken:

$ aptitude search '^cgroup-bin$'
pB  cgroup-bin ...

In its UI, aptitude provides the following explanation:

Some dependencies of cgroup-bin are not satisfied:
* cgroup-bin depends on cgroup-tools

Both dpkg -l cgroup-bin and dpkg -l cgroup-tools issue dpkg-query: no packages found matching ... because neither package is installed.

Both apt-cache policy cgroup-bin and apt-cache policy cgroup-tools identify version 0.41-6 from the Debian repository as candidate versions.

aptitude show cgroup-bin lists these (relevant) dependencies:

Depends: cgroup-tools

aptitude show cgroup-tools lists those:

Breaks: cgroup-bin (< 0.40~rc1-3~)
Replaces: cgroup-bin (< 0.40~rc1-3~)

I have another machine which has the same properties but where the package is not marked as broken.

What is going on here? Why is package cgroup-bin marked as broken (when from the dependencies it would seem that versions 0.41-6 can live in unison) and how can I remedy the situation?

I have a situation where a two-node Pacemaker cluster does not work any more after an upgrade. Package versions are pacemaker 1.1.16-1~bpo8+ and corosync 2.4.2-3~bpo8+1 under Debian Jessie.

Pacemaker is still able to start on one node. crm_node -l then lists that node as online, the second one as lost.

Pacemaker can no longer start on the second node. The following log messages in /var/log/corosync/logfile seem pertinent:

cib: info: validate_with_relaxng: Creating RNG parser context
pacemakerd: error: pcmk_child_exit: The cib process (1234) exited: Key has expired (127)
pacemakerd: notice: pcmk_process_exit: Respawning failed child process: cib
...
cib: info: validate_with_relaxng: Creating RNG parser context
pacemakerd: error: pcmk_child_exit: The cib process (1235) exited: Key has expired (127)
pacemakerd: notice: pcmk_process_exit: Respawning failed child process: cib
...
crmd: warning: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry
...
crmd: warning: do_cib_control: Couldn't complete CIB registration 16 times... pause and retry 
crmd: notice: crm_shutdown: Shutting down cluster resource manager | limit=1200000ms
pacemakerd: notice: pcmk_shutdown_worker: Shutdown complete

So it appears as if the second node attempts CIB registration and cancels the Pacemaker start after 16 failed attempts, and that the first node coniders the second as dead perhaps because it cannot register.

Who can one get out of a situation like this?

This piece of Debian documentation states that package ifenslave is necessary for bonding:

First install the ifenslave package, necessary to enable bonding

Yet I know of two Debian 7.10 (wheezy) hosts where dpkg --get-selections | grep ifenslave returns ifenslave-2.6 install (i.e. package is not installed) and there are multiple entries /proc/net/bonding/* with MII Status: up.

So is package ifenslave really necessary for bonding or is it not?

How can I find the correct firmware version number for a QLogic InfiniBand adapter on a Debian server?

ibstat displays an empty version number like so:

Firmware version: 
Hardware version: 2

lspci -vvv is also silent on the issue like so:

InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)
Subsystem: QLogic Corp. IBA7322 QDR InfiniBand HCA