Is that possible to delete a snapshot from a QCOW2 image without shutting down a virtual machine (QEMU+KVM)? Wouldn't running qemu-img snapshot -d ... break data consistency? If so, would that break consistency if one runs virsh pause ... before deleting a snapshot and then runs virsh resume ...? What do you think?
Thanks!
Is there a way to gather and show the disk usage stats of CephFS subvolumes?
Is there anything similar to rbd du for CephFS subvolumes?
The only idea I have is something really ugly:
filesystem="ceph-filesystem"
group_name="csi"
for subvolume in $(ceph fs subvolume ls "${filesystem}" --group-name "${group_name}" | jq -cr '.[].name?'); do
subvolume_info=$(ceph fs subvolume info "${filesystem}" "${subvolume}" "${group_name}")
bytes_quota="$(echo "${subvolume_info}" | jq '.bytes_quota')"
bytes_used=$(echo "${subvolume_info}" | jq '.bytes_used')
echo "${subvolume} ${bytes_quota} ${bytes_used}"
done
I believe, there should be something more elegant.
Thanks in advance.
Is that possible to tweak some settings that would make OpenLDAP always interpreting uid=username@domainname as uid=username,ou=domainname in authentication queries?
Of course, making the clients to do this job would be a much more ecological way, but this would be a less preferable option in my case.
I have a domain (let's call it dc=example,dc=org)
The domain has a branch (ou=users,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org).
There's a simpleSecurityObject in this domain (uid=admin,ou=managers,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org).
I need the uid=admin,*** user to have full (manage) access to the ou=users,*** branch, so I added the following olcAccess record:
to
dn.subtree="ou=users,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org"
by
dn.exact="uid=admin,ou=managers,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org"
It has added to the default set of rules:
dn: olcDatabase={1}mdb,cn=config
olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external
,cn=auth manage by * break
olcAccess: {1}to attrs=userPassword,shadowLastChange by self write by dn="cn=a
dmin,dc=example,dc=org" write by anonymous auth by * none
olcAccess: {2}to * by self read by dn="cn=admin,dc=example,dc=org" write by
* none
olcAccess: {3}to dn.subtree="ou=users,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=
infra,dc=example,dc=org" by dn.exact="uid=admin,ou=managers,ou=ftp,ou=servi
ces,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org" manage
But something seems to be wrong. When I run ldapsearch -D uid=admin,ou=managers,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org -W -b ou=users,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org, I get the following result:
# extended LDIF
#
# LDAPv3
# base <ou=users,ou=ftp,ou=services,dc=k9999,dc=z9999,dc=infra,dc=example,dc=org> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 2
result: 32 No such object
# numResponses: 1
command terminated with exit code 32
The object is available and cn=admin,dc=example,dc=org can see it without any difficulties, so it seems that my access rule is wrong. But what exactly it is?
It seems that the default rule #2 ({2} to * by self read by dn="cn=admin,dc=example,dc=org" write by * none) fires up earlier than the rule I added. Does that mean that I should always add my custom rules before it?
And why does this rule has by * none? Doesn't it contradict the OpenLDAP documentation?
The default access control policy is allow read by all clients
It seems that rule #2 (I mean this one:
to *
by self read
by dn="cn=admin,dc=example,dc=org" write
by * none
) isn't "default" at all. I've just found another instance of OpenLDAP (I deployed it about 4 years ago). This instance (2.4.40) doesn't have such rule at all. It seems that it has not been added by OpenLDAP itself, it rather has been added manually by someone or something else. Am I right? Doesn't OpenLDAP have such rule by default?
Is that possible to make OpenLDAP provide with different base DNs for different users?
Let me explain what exactly I want to acheive.
I have a domain (let's say, dc=example,dc=org).
I also have a phpLDAPadmin instance which purpose is to help me to manage this domain.
I also have a branch somewhere within this domain (let's say, ou=foo,ou=bar,dc=baz,dc=example,dc=org).
I also have a user (let's say, uid=admin,ou=bar,dc=baz,dc=example,dc=org).
I granted some permissions to this user to let them manage the branch I mentioned before (olcAccess: to dn.subtree="ou=foo,ou=bar,dc=baz,dc=example,dc=org" by dn.exact="uid=admin,ou=bar,dc=baz,dc=example,dc=org" manage).
The problem is that this user can't use phpLDAPadmin, because the OpenLDAP server propagates its base DN (by announcing the namingContext: dc=example,dc=org attribute, I guess), which the user doesn't have access to. So, phpLDAPadmin tries to show the user the contents of dc=example,dc=org, fails and laments that "This base cannot be created with PLA".
How to make phpLDAPadmin showing ou=foo,ou=bar,dc=baz,dc=example,dc=org as the base DN to this user instead of tryig to show them dc=example,dc=org which is not accessible at all?
Is there a way make OpenLDAP announcing with a different namingContext (ou=foo,ou=bar,dc=baz,dc=example,dc=org) to the user?
Or should I forget this idea and grant the user read-only access to each leave from the very top (dc=example,dc=org, dc=baz,dc=example,dc=org, ou=bar,dc=baz,dc=example,dc=org)?
Thank you.
It might be a very simple question, though somewhy I can't google the answer.
Let's assume we have a Helm chart with some default values such as the following:
parameters:
- parameterName: parameter1
value: value1
- parameterName: parameter2
value: value2
I need to add an element to this list in my extra values file. Of course, I can just duplicate the default values to my extra values file and add my extra parameters to them:
parameters:
- parameterName: parameter1
value: value1
- parameterName: parameter2
value: value2
- parameterName: parameterN
value: valueN
I don't like this idea, because when the maintainers of this Helm chart change their defaults, my defaults will override instead of following them.
What is the proper way to add my parameterN to the default values?
Thanks in advance?
I need to make a playbook that will take a template file, interpolate some variables and put the resulting file to another directory.
The first part of the problem is that the resulting file will be used as another Ansible playbook, so the source file contains other interpolations and these interpolations must be preserved as they are. As an example, let's assume that the source file contains something like that:
key1: {{ value1 }}
key2: {{ value2 }}
The value of key1 must be interpolated by my playbook (it shall become the values of the value1 variable), but at the same time value2 needs to be kept just as it is (it shall be {{ value2 }}.
The second part of the problem is that I can't just modify the source file and add backslashes before the curly brackets, because this file is being updated by other processes. Moreover, as the template file is being updated from time to time, I can't predict what variables needs to be skipped after the next update. I only know that I need to interpolate value1, but I don't know what are the other variables' names (today there are value2 and value3, tomorrow the developers renamed value2 to value2_deprecated and added value3 somewhere).
Can I ask Ansible that I need to interpolate value1 only?
Thank you.
I need to build a VPN connection between a network and a Kubernetes-cluster, so the applications hosted in this in this network could address to K8S-services via a secured tunnel.
So, I have a bunch of K8S-nodes in a self-hosted environment. I've added a separate server to this environment, this server works as a VPN gateway, it's connected to the same VLAN which the cluster nodes are connected to. The nodes have the following IP-addresses: 10.13.17.1/22, 10.13.17.2/22, 10.13.17.3/22 and so on. The VPN gateway has 10.13.16.253/22.
The Cluster IP CIDR is 10.233.0.0/18, the pod IP CIDR is 10.233.64.0/18.
The VPN-server supports an IPSec site-to-site connection with a remote network, 10.103.103.0/24. I use Calico as the networking manager, so I've set up my VPN server to keep BGP-sessions with all K8S-nodes. The VPN server's route table is full of prefixes announced by Calico nodes (10.233.0.0/18 is present too as well, of course), the cluster nodes have 10.103.103.0/24 and some other networks in their route tables, so BGP seems to be working fine. So far so good...
When I establish a connection to a service inside of the cluster from the VPN-server, everything is good too. The client (10.13.16.253) sends a SYN-packet to the service (10.233.10.101:1337), the worker receives this packet, changes it's destination IP-address to the IP-address of the pod (10.233.103.49:1337) and changes it's source IP-address to some IP-address (10.233.110.0) that will help the worker to receive the reply and give it back to the connection initiator. Here's what happens on the worker that receives this SYN-packet.
The SYN-packet comes to a worker:
22:04:25.866546 IP 10.13.16.253.56297 > 10.233.10.101.1337: Flags [S], seq 3575679444, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1385938010 ecr 0], length 0
The SYN-packed is being SNATed and DNATed and then it's being sent to the worker where the pod is running:
22:04:25.866656 IP 10.233.110.0.54430 > 10.233.103.49.1337: Flags [S], seq 3575679444, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1385938010 ecr 0], length 0
The reply has came:
22:04:25.867313 IP 10.233.103.49.1337 > 10.233.110.0.54430: Flags [S.], seq 2017844946, ack 3575679445, win 28960, options [mss 1460,sackOK,TS val 1201488363 ecr 1385938010,nop,wscale 7], length 0
The reply is being deSNATed and deDNATed to be sent to the connection initiator:
22:04:25.867533 IP 10.233.10.101.1337 > 10.13.16.253.56297: Flags [S.], seq 2017844946, ack 3575679445, win 28960, options [mss 1460,sackOK,TS val 1201488363 ecr 1385938010,nop,wscale 7], length 0
So, the connection is established and everyone is happy.
But when I try to connect to the same service from the external network (10.103.103.0/24) the worker who receives the SYN-packet does NOT change the source IP-address, it changes the destination IP-address only, so the packet's source IP-address is unchanged.
The SYN packet comes to a worker
21:56:05.794171 IP 10.103.103.1.52132 > 10.233.10.101.1337: Flags [S], seq 3759345254, win 29200, options [mss 1460,sackOK,TS val 195801472 ecr 0,nop,wscale 7], length 0
The SYN packet is being DNATed and being resent to the worker where the pod is running
21:56:05.794242 IP 10.103.103.1.52132 > 10.233.103.49.1337: Flags [S], seq 3759345254, win 29200, options [mss 1460,sackOK,TS val 195801472 ecr 0,nop,wscale 7], length 0
And nothing comes back in reply. :-(
So, I see that the destination IP-address is changed, so I can see these packets on the worker where the pod is running, but there are no replies to them:
21:56:05.794602 IP 10.103.103.1.52132 > 10.233.103.49.1337: Flags [S], seq 3759345254, win 29200, options [mss 1460,sackOK,TS val 195801472 ecr 0,nop,wscale 7], length 0
The external network (10.103.103.0/24) is being advertised by the VPN server via BGP, so all the workers know that this network is accessible via 10.13.16.253. When I run the ping-test from a host in the external network (10.103.103.1) to the IP-address of the service (10.233.10.101), the test passes, VPN works fine and routing tables seem to be correct.
So, why does the network "trust" to 10.13.16.253 and doesn't trust to 10.103.103.1? And why does the worker perform SNAT and DNAT for the packets from 10.13.16.253 and does not perform SNAT for the packets from 10.103.103.1? Should I add some policies to allow this traffic?
Thanks in advance for any clues!
I have a host that runs CentOS-6 with Linux 2.6.32-696.18.7.el6.x86_64 (yes, pretty old) and I ran out of disk space on the root filesystem.
The filesystem's size is 16G, du shows about 3G being used by files. Of course, I ran lsof +L1, found processes that held the big log-files that had been unlinked before (yes, there were a couple of such processes, so I killed them), but the filesystem is still full and anything can write to it (no, inodes are fine, there are no available blocks).
lsof +L1 still shows some files, but these files don't occupy too much size, so I'm pretty sure I can ignore that:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
init 1 root 9r REG 253,0 10406312 0 131692 /var/lib/sss/mc/initgroups (deleted)
pickup 2761 postfix 8r REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
dbus-daem 5232 dbus 4r REG 253,0 10406312 0 131692 /var/lib/sss/mc/initgroups (deleted)
hald 5271 haldaemon 19r REG 253,0 10406312 0 131692 /var/lib/sss/mc/initgroups (deleted)
ntpd 5528 ntp 4r REG 253,0 10406312 0 131730 /var/lib/sss/mc/initgroups (deleted)
qmgr 5631 postfix 8r REG 253,0 10406312 0 131730 /var/lib/sss/mc/initgroups (deleted)
crond 5655 root 8r REG 253,0 8406312 0 131792 /var/lib/sss/mc/passwd (deleted)
sssd 9042 root 16r REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
sssd_be 9045 root 18r REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm 9606 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 16799 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 17419 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
polkitd 28420 root 8r REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm 28802 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 30171 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 32110 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 32419 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 32563 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 34762 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 37823 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 39088 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
qemu-kvm 41613 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
nrpe 45821 nrpe 4r REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
sshd 52715 root 4w REG 253,0 10406312 0 134123 /var/lib/sss/mc/initgroups (deleted)
qemu-kvm 55329 root txt REG 253,0 4056888 0 397695 /usr/libexec/qemu-kvm (deleted)
What else should I try to do (I'm pretty sure that reboot will help, but I'd like to avoid doing that)?
Thanks!
Is there a way to visualize current CPU usage of a pod in a K8S cluster?
I tried different Prometheus metrics like namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate and other similar ones, but I always get average value for the last 5 minutes, so I have "stairs" on my graphs even if workload raises abruptly (please, see the screenshot).
At the same time, kubectl top pod shows more precise values, so when I raise workload to 100%, it starts showing 100% utilization in a few seconds.
Is there a way to provide these values to Grafana?
Thanks!
I'd like to set up horizontal auto-scaling for a deployment based on the metrics of the ingress-controller deployed in another namespace.
I have a deployment (petclinic) deployed in a certain namespace (petclinic).
I have an ingress controller (nginx-ingress) deployed in another namespace (nginx-ingress).
The ingress controller has been deployed with Helm and Tiller, so I have the following ServiceMonitor entity:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"creationTimestamp":"2019-08-19T10:48:00Z","generation":5,"labels":{"app":"nginx-ingress","chart":"nginx-ingress-1.12.1","component":"controller","heritage":"Tiller","release":"nginx-ingress"},"name":"nginx-ingress-controller","namespace":"nginx-ingress","resourceVersion":"7391237","selfLink":"/apis/monitoring.coreos.com/v1/namespaces/nginx-ingress/servicemonitors/nginx-ingress-controller","uid":"0217c466-5b78-4e38-885a-9ee65deb2dcd"},"spec":{"endpoints":[{"interval":"30s","port":"metrics"}],"namespaceSelector":{"matchNames":["nginx-ingress"]},"selector":{"matchLabels":{"app":"nginx-ingress","component":"controller","release":"nginx-ingress"}}}}
creationTimestamp: "2019-08-21T13:12:00Z"
generation: 1
labels:
app: nginx-ingress
chart: nginx-ingress-1.12.1
component: controller
heritage: Tiller
release: nginx-ingress
name: nginx-ingress-controller
namespace: nginx-ingress
resourceVersion: "7663160"
selfLink: /apis/monitoring.coreos.com/v1/namespaces/nginx-ingress/servicemonitors/nginx-ingress-controller
uid: 33421be7-108b-4b81-9673-05db140364ce
spec:
endpoints:
- interval: 30s
port: metrics
namespaceSelector:
matchNames:
- nginx-ingress
selector:
matchLabels:
app: nginx-ingress
component: controller
release: nginx-ingress
I also have the Prometheus Operaton instance, it has found this entity and it has updated the Prometheus' configuration with this stanza:
- job_name: nginx-ingress/nginx-ingress-controller/0
honor_labels: false
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- nginx-ingress
scrape_interval: 30s
relabel_configs:
- action: keep
source_labels:
- __meta_kubernetes_service_label_app
regex: nginx-ingress
- action: keep
source_labels:
- __meta_kubernetes_service_label_component
regex: controller
- action: keep
source_labels:
- __meta_kubernetes_service_label_release
regex: nginx-ingress
- action: keep
source_labels:
- __meta_kubernetes_endpoint_port_name
regex: metrics
- source_labels:
- __meta_kubernetes_endpoint_address_target_kind
- __meta_kubernetes_endpoint_address_target_name
separator: ;
regex: Node;(.*)
replacement: ${1}
target_label: node
- source_labels:
- __meta_kubernetes_endpoint_address_target_kind
- __meta_kubernetes_endpoint_address_target_name
separator: ;
regex: Pod;(.*)
replacement: ${1}
target_label: pod
- source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- source_labels:
- __meta_kubernetes_service_name
target_label: service
- source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- source_labels:
- __meta_kubernetes_service_name
target_label: job
replacement: ${1}
- target_label: endpoint
replacement: metrics
I also have a Prometheus-Adapter instance, so I have the custom.metrics.k8s.io API in the list of available APIs.
The metrics are being collected and exposed, so the following command:
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx-ingress/ingresses/petclinic/nginx_ingress_controller_requests" | jq .
gives the following result:
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx-ingress/ingresses/petclinic/nginx_ingress_controller_requests"
},
"items": [
{
"describedObject": {
"kind": "Ingress",
"namespace": "nginx-ingress",
"name": "petclinic",
"apiVersion": "extensions/v1beta1"
},
"metricName": "nginx_ingress_controller_requests",
"timestamp": "2019-08-20T12:56:50Z",
"value": "11"
}
]
}
So far so good, right?
And what I need is to set up the HPA entity for my deployment. Doing something like that:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: petclinic
namespace: petclinic
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: petclinic
minReplicas: 1
maxReplicas: 10
metrics:
- type: Object
object:
metricName: nginx_ingress_controller_requests
target:
apiVersion: extensions/v1beta1
kind: Ingress
name: petclinic
targetValue: 10k
Of course, this is incorrect, as the nginx_ingress_controller_requests is related to the nginx-ingress namespace, so it doesn't work (well, as it was expected):
annotations:
autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-08-19T18:43:42Z","reason":"SucceededGetScale","message":"the
HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2019-08-19T18:55:26Z","reason":"FailedGetObjectMetric","message":"the
HPA was unable to compute the replica count: unable to get metric nginx_ingress_controller_requests:
Ingress on petclinic petclinic/unable to fetch metrics
from custom metrics API: the server could not find the metric nginx_ingress_controller_requests
for ingresses.extensions petclinic"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2019-08-19T18:43:42Z","reason":"DesiredWithinRange","message":"the
desired count is within the acceptable range"}]'
autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":""},{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":1,"currentAverageValue":"10m"}}]'
autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Object","object":{"target":{"kind":"Ingress","name":"petclinic","apiVersion":"extensions/v1beta1"},"metricName":"nginx_ingress_controller_requests","targetValue":"10k"}}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"petclinic","namespace":"petclinic"},"spec":{"maxReplicas":10,"metrics":[{"object":{"metricName":"nginx_ingress_controller_requests","target":{"apiVersion":"extensions/v1beta1","kind":"Ingress","name":"petclinic"},"targetValue":"10k"},"type":"Object"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"petclinic"}}}
And here's what I see in the log-file of Prometheus-Adapter:
I0820 15:42:13.467236 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/petclinic/ingresses.extensions/petclinic/nginx_ingress_controller_requests: (6.124398ms) 404 [[kube-controller-manager/v1.15.1 (linux/amd64) kubernetes/4485c6f/system:serviceaccount:kube-system:horizontal-pod-autoscaler] 10.103.98.0:37940]
HPA looks for this metric in the deployment's namespace, but I need it to fetch it from the nginx-ingress namespace, just like so:
I0820 15:44:40.044797 1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/nginx-ingress/ingresses/petclinic/nginx_ingress_controller_requests: (2.210282ms) 200 [[kubectl/v1.15.2 (linux/amd64) kubernetes/f627830] 10.103.97.0:35142]
Alas, the autoscaling/v2beta1 API doesn't have the spec.metrics.object.target.namespace entity, so I can't "ask" it to fetch the value from another namespace. :-(
Would anyone be so kind as to help me to solve this puzzle? Is there a way to set up auto-scaling based on the custom metrics that belong to another namespace?
Maybe there's the way to make this metric available in the same namespace to which this ingress.extension belongs?
Thanks in advance for any clues and tips.
Is that possible to define an ingress (ingresses.networking.k8s.io) forwarding requests to a concrete static IP-address in a private network?
I have a service that isn't running on a K8S cluster, but it resides in a private network, and all K8S pods have access to this network. There are many auxiliary services in this network, I don't want to deploy these services in a K8S cluster due to certain reasons. And I'd like to have some of these services exposed via ingress-nginx, but at this time I don't see a way to define the backend as a static IP-address.
Is that possible?
Thanks!
I have a question on how to divide access to the same gluster for different K8S namespaces. Let's say, I have 3 different volumes (vol-a, vol-b, vol-c) in a gluster and I'd like to permit access to each of them for each of different namespaces (namespace-a - to vol-a, namespace-b - to vol-b, namespace-c - to vol-c).
Is that possible to implement such a scheme? Maybe there a way to authenticate a client of a gluster with some username and password? Is there a way to define these credentials in an endpoint configuration?
Thanks in advance!
I need to deploy an application which works as a CCM (cloud controller manager), so it needs to have access to the master servers.
I have a K8S cluster that has been set up by Kubespray, all my nodes are running kubelet that takes configuration from /etc/kubernetes/kubelet.conf. The kubelet.conf is shown below:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ***
server: https://localhost:6443
name: default-cluster
contexts:
- context:
cluster: default-cluster
namespace: default
user: default-auth
name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
This configuration file and the certificates are being provided to the CCM service, I added the following volumes and mountpoints to the deployment YAML:
containers:
- name: cloud-controller-manager
image: swisstxt/cloudstack-cloud-controller-manager:v0.0.1
# Command line arguments: https://kubernetes.io/docs/reference/command-line-tools-reference/cloud-controller-manager/
command:
- /root/cloudstack-ccm
- --cloud-provider=external-cloudstack
- --cloud-config=/config/cloud-config
- --kubeconfig=/var/lib/kubelet/kubelet.conf # Connection Params
- --v=4
volumeMounts:
- name: config-volume
mountPath: /config
- name: kubeconfig-config-file
mountPath: /var/lib/kubelet/kubelet.conf
- name: kubernetes-pki-volume
mountPath: /var/lib/kubelet/pki
- name: kubernetes-config-volume
mountPath: /var/lib/kubernetes
volumes:
- name: config-volume
configMap:
name: cloud-controller-manager-config
- name: kubeconfig-config-file
hostPath:
path: /etc/kubernetes/kubelet.conf
- name: kubernetes-pki-volume
hostPath:
path: /var/lib/kubelet/pki
- name: kubernetes-config-volume
hostPath:
path: /var/lib/kubernetes
So far, so good.
My problem is that my kubelet.conf is having the following sentence: .clusters.cluster.server: https://localhost:6443. So, kubelet is configured to interact with the master servers via a proxy-server that has been set up by Kubespray to distribute the connections between the master services.
So, when the CCM application inspect the kubelet.conf it understands that it should communicate with the master servers via https://localhost:6443, but inside of the pod of this application localhost:6443 is not being listened by this proxy server, so CCM can't use localhost:6443 to communicate with the master server, as localhost:6443 is accessible only from the node itself.
Here's the question: is there a way to make the node's localhost:6443 accessible from the pod? The only idea I have at this moment is to set up an SSH-tunnel between the pod and the node it's running at, but I don't like it, because (1) it requires to propagate some RSA-key on all the nodes and add it on every new node, (2) I have no idea on how to find out the IP-address of the node from behalf of a container.
Thanks for reading this rant. I'll be very grateful for all the ideas and clues.
What anomalies might happen if you try to join a bunch of vlans by bridges in a cluster of Linux servers?
We'll presume, you have a group of servers doing running VMs (qemu-kvm), these VMs have their virtual interfaces (vnet+) connected to, let's say, 4 VLAN interfaces (vlan1301, vlan1302, vlan1303, vlan1304), but you need to join these VLANs in one (as you need all 4 gateways be present on the same physical router's interface: 1.1.1.254/24, 2.2.2.254/24, 3.3.3.254/24, 4.4.4.254/24, all these addresses are aliases of the same interface accessible via vlan666.
Whot would I do? I'd create an interface to vlan666 on each host and have it bridged all together:
brctl addbr jjj
bectl addif jjj vlan666
brctl addif jjj vlan1301
brctl addif jjj vlan1302
brctl addif jjj vlan1303
brctl addif jjj vlan1304
What do you think, what are the dangers? Could something fail at some time?
Thanks for sharing your thoughts!
I've faced a quite odd issue, which I'd like to share with you. Maybe you'd help me with some ideas on what's going on.
There are 3 virtual machines on a KVM-powered host. Actually there are about 50 VMs, but they all are working fine, although these 3 VMs behavior is kind a bit unusual.
Here's a diagram illustrating these 3 VMs and 2 links between them:
When everything is OK, the TCP-session ("GET / HTTP/1.0" - "HTTP 200 OK") between these ones is looking like that:
00:58:43.885118 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [S], seq 926382744, win 14600, options [mss 1460,sackOK,TS val 277997 ecr 0,nop,wscale 7], length 0
00:58:43.885380 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [S.], seq 1849545379, ack 926382745, win 14480, options [mss 1460,sackOK,TS val 3702103 ecr 277997,nop,wscale 7], length 0
00:58:43.885957 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 277998 ecr 3702103], length 0
00:58:43.886000 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [P.], seq 1:213, ack 1, win 115, options [nop,nop,TS val 277998 ecr 3702103], length 212
00:58:43.886061 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [.], ack 213, win 122, options [nop,nop,TS val 3702104 ecr 277998], length 0
00:58:43.922286 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [P.], seq 1:372, ack 213, win 122, options [nop,nop,TS val 3702140 ecr 277998], length 371
00:58:43.922335 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [F.], seq 372, ack 213, win 122, options [nop,nop,TS val 3702140 ecr 277998], length 0
00:58:43.923150 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [.], ack 372, win 123, options [nop,nop,TS val 278035 ecr 3702140], length 0
00:58:43.923622 IP 192.168.111.2.55480 > 192.168.113.2.http: Flags [F.], seq 213, ack 373, win 123, options [nop,nop,TS val 278036 ecr 3702140], length 0
00:58:43.923671 IP 192.168.113.2.http > 192.168.111.2.55480: Flags [.], ack 214, win 122, options [nop,nop,TS val 3702142 ecr 278036], length 0
OK, everything is fine so far.
Then we save the pfSense configuration, destroy this VM, create a new one, install pfSense from scratch and restore its configuration from the backup file.
And here's what we see after that:
00:46:39.218193 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [S], seq 3622924060, win 14600, options [mss 1460,sackOK,TS val 674608862 ecr 0,nop,wscale 7], length 0
00:46:39.218316 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [S.], seq 152904245, ack 3622924061, win 14480, options [mss 1460,sackOK,TS va l 2977436 ecr 674608862,nop,wscale 7], length 0
00:46:39.218570 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 674608862 ecr 2977436], length 0
00:46:40.417623 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [S.], seq 152904245, ack 3622924061, win 14480, options [mss 1460,sackOK,TS val 2978636 ecr 674608862,nop,wscale 7], length 0
00:46:40.417947 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 674610062 ecr 2977436], length 0
00:46:43.158907 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674612803 ecr 2977436], length 16
00:46:43.360103 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674613004 ecr 2977436], length 16
00:46:43.761787 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674613406 ecr 2977436], length 16
00:46:44.565890 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674614210 ecr 2977436], length 16
00:46:46.174039 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674615818 ecr 2977436], length 16
00:46:49.389921 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674619034 ecr 2977436], length 16
00:46:51.753723 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 2989972 ecr 674560137], length 0
00:46:55.821824 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674625466 ecr 2977436], length 16
00:46:57.221625 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 2995440 ecr 674560137], length 0
00:47:08.157575 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3006376 ecr 674560137], length 0
00:47:08.685886 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674638330 ecr 2977436], length 16
00:47:30.029609 IP 192.168.113.2.http > 192.168.111.2.51672: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3028248 ecr 674560137], length 0
00:47:34.413785 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674664058 ecr 2977436], length 16
00:47:40.478757 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3038697 ecr 674610062], length 0
00:47:34.413785 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [P.], seq 1:17, ack 1, win 115, options [nop,nop,TS val 674664058 ecr 2977436], length 16
00:47:40.478757 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3038697 ecr 674610062], length 0
00:47:40.479216 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [FP.], seq 17:19, ack 2, win 115, options [nop,nop,TS val 674670123 ecr 3038697], length 2
00:47:45.946604 IP 192.168.113.2.http > 192.168.111.2.51674: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 3044165 ecr 674610062], length 0
00:47:45.946979 IP 192.168.111.2.51674 > 192.168.113.2.http: Flags [.], ack 2, win 115, options [nop,nop,TS val 674675591 ecr 3044165,nop,nop,sack 1 {1:2}], length 0
It looks like... I don't know, like they don't hear each other. They can ping each other, they even interact with each other, but it look like they just ignore some of the packets.
Both VMs are showing the same, so pfSense doesn't drop any of their packets. Although it seems that something wrong happens to there packets. Like they've been mangled by something.
This is the thing I can't understand, really. If you share any ideas with me, it will be fantastic.
Thanks in advance to everyone!
I had a file on an XFS filesystem which size was about 200 GB. It was a QCOW2 image containing a virtual disk of a KVM-driven virtual machine. Something went wrong (maybe it was some glitch of qemu-kvm, I'm not sure), the virtual machine had crashed and now I have a file that looks like that:
191090708 -rwxr--r--. 1 root root 737571587400425984 Oct 10 10:03 973d10e0-a5e3-4a59-9f98-4b9b9f072ade
So, it still occupies 191090708 blocks, but ls shows it as 656 petabytes.
Moreover, I have another file with the same pre-history, but on another filesystem (not XFS, but GFS2):
410855320 -rwxr--r--. 1 root root 7493992262336241664 Dec 13 2014 ac2cb28f-09ac-4ca0-bde1-471e0c7276a0
It occupies 410855320 blocks, but ls shows it as ~6.6 exabytes.
What do you think, is it safe to remove these files? Thank you!
P.S. It's so good to have snapshots taken on a regular basis! :) I don't know what I would do without them.
I would be really happy to learn how to make Postfix to "remember" that certain host is unreachable at the moment, so it would be good if we didn't perform any new connection attempts to this host for a while.
I know Postfix is following exponential backoff algorythm for recipients, but can the same idea be applied to recipients' hosts?
Thanks!
I faced a bit odd difficulty with policy-based routing on CentOS-6.6 server. I have a setup where I need to enroute all packets originated from 172.25.0.0/16 destinated to 172.24.0.0/16 and some XXX.155.106.50 via certain gateway (10.4.20.5) behind a VPN-tunnel. I'm just doing what I always did.
1) I defined zone2 table in my /etc/iproute2/rt_tables:
255 local
254 main
253 default
0 unspec
25 zone2
2) I added some rules, so here's what I get in reply to ip rule show:
0: from all lookup local
32762: from all to 172.25.0.0/16 lookup zone2
32763: from 172.25.0.0/16 lookup zone2
32766: from all lookup main
32767: from all lookup default
3) And, finally, I added some routes to the table, so in reply to ip route show table zone2 I get the following:
XXX.155.106.50 via 10.4.20.5 dev tun0
172.25.64.0/24 dev cloudbr0 scope link
172.25.65.0/24 dev cloudbr0 scope link
172.24.0.0/16 via 10.4.20.5 dev tun0
Everything seems quite legit, right? But, alas, these rules don't seem to be working, as tcpdump shows me all packets from 172.25.0.0/16 being sent not through my tunnel, but directly.
The most funny thing: when I do something like ping -I 172.25.65.254 XXX.155.106.50, everything works fine and packets are being routed through the tunnel. But if the packet comes from the network (by the way, the networking interface is bridged) - all these rules seems to be ignored and packets go directly. :(
Any clues & hints will be greatly appreciated!