Jeroen Jacobs's questions

This is weird, I'm having some instances running in a private subnet. Those subnets are not exposed to the Internet directly, they don't have a public IP and all outgoing traffic is routed through a NAT instance.

However, GuardDuty gives me warnings like this for all instances in this private subnet:

Recon:EC2/PortProbeUnprotectedPort
Action
Action type
PORT_PROBE


Blocked
false
First seen
12-21-2019 22:22:10 (a month ago)
Last seen
01-19-2020 11:18:12 (38 minutes ago)
Actor
IP address
159.65.11.106
Location
country:
Singapore
lat:
1.314
lon:
103.6839
Organization
asn:
14061
asnOrg:
DigitalOcean, LLC
isp:
Digital Ocean
org:
Digital Ocean

Additional information
Threat name
Scanner
Threat list name
ProofPoint
Local port
30539
Archived
false
Remote IP details
ipAddress:
5.101.0.209
location:
Moscow, Russia
organization:
PinSPB

So, multiple questions arise here:

• How is it even possible my instance gets scanned when it doesn't have a public IP adres?
• Why is the actor IP address different from the remote IP?

I have implemented SSH CA client signing on my servers. Sshd is configured on my servers with the following directive:

TrustedUserCAKeys /etc/ssh/trusted-users-ca.pem

I modified my local ssh config file so my cert is sent as well, when I connect to my servers:

Host *.internal.headincloud.be
        User centos
        IdentityFile ~/.ssh/datacenter-hic-deploy
        CertificateFile = ~/.ssh/datacenter-hic-deploy-cert.pub

This seems to work just fine, and I'm able to connect to my server without the need to deploy an authorized_keys file.

However, Ansible is unable to connect my servers:

TASK [Gathering Facts] *********************************************************************************************************************************************************************
fatal: [postgres-01]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host \"192.168.90.40\". Make sure this host can be reached over ssh", "unreachable": true}

Like I already mentioned, I'm able to connect via ssh just fine.

I suspect Ansible is not sending the certificate file along, and that's why I am unable to connect.

I tried modifying my ansible.cfg as follows:

ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -i ~/.ssh/datacenter-hic-deploy-cert.pub

or

ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -i /Users/jeroenjacobs/.ssh/datacenter-hic-deploy-cert.pub

Neither of those work.

I cannot a find a way to tell Ansible how to do this. Anyone an idea?

Imagine the following scenario:

• You run a kubernetes cluster in your datacenter, which was deployed with kubeadm.
• It consists of one masternode (running etcd as a static pod, as deployed by kubeadm) and 3 worker nodes
• the nodes as virtual machines running on vmware

Today, you open your e-mail and you are notified the datacenter will move to a new location. The physical servers will be turned off, moved to the new location and powered on again.

What is the correct shutdown procedure for your kubernetes cluster (without messing up your etcd data)?

This what I did:

• stopped the master server first (this includes etcd ofc), to prevent pods from being rescheduled to other nodes when I turn off the worker nodes.
• stopped each worker node

After the migration:

• powered on the worker nodes first
• powered on the master node next

After doing this, I ended up with one of two scenarios:

• etcd data is corrupt and the etcd pod exits with an error
• getting errors like this: "Operation cannot be fulfilled on nodes "worker-002": the object has been modified; please apply your changes to the latest version and try again". my logs are getting flooded with these messages.

How could this have been prevented? I don't think running etcd in HA mode would help here, as all etcd nodes would have to be shut down at once too, so you end up with a similar situation as a single node scenario. I get the impression that Etcd is quite... fragile, compared to other K/V stores like Consul.

Since Kubernetes 1.8, it seems I need to disable swap on my nodes (or set --fail-swap-on to false).

I cannot find the technical reason why Kubernetes insists on the swap being disabled. Is this for performance reasons? Security reasons? Why is the reason for this not documented?

I have an nginx loadbalancer which just proxies requests to a few upstream servers.

Now, what I would like to do is:

For one out of ten requests, I want to measure the response time of nginx (in milliseconds) and forward that value to an InfluxDB metric database.

I'm not sure how I can make this work. My initial idea was to leverage the "access_by_lua_block" functionality, but if I understand correctly, this is executed before the request is sent to the upstream server. Is there a way to execute lua-code after a response is received from the upstream server?

Just to be clear: I want to the response time of the nginx loadbalancer itself, not of the upstream application server. My final goal is to compare the response times of the loadbalancer with the response times of the upstream servers, to identify potential bottlenecks in the loadbalancer configuration.

I have the following keepalived.conf file:

vrrp_script chk_nginx {
  script "curl http://localhost/vrrp_healthcheck/"
  interval 2 # every 2 seconds
  weight 2 # add 2 points if OK
}

vrrp_instance VI_1 {
  interface ens160 # interface to monitor
  virtual_router_id 50
  priority 100
  virtual_ipaddress {
    192.168.120.25 label ens160:10
  }
  track_script {
    chk_nginx
  }
  notify /usr/local/bin/notify.sh
}

On the master node I see the following in the ifconfig output:

ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.120.20  netmask 255.255.255.224  broadcast 192.168.120.31
        ether 00:50:56:a5:c5:3a  txqueuelen 1000  (Ethernet)
        RX packets 138308  bytes 130425415 (124.3 MiB)
        RX errors 0  dropped 73  overruns 0  frame 0
        TX packets 122917  bytes 50788591 (48.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.120.25  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 00:50:56:a5:c5:3a  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 9730  bytes 812897 (793.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9730  bytes 812897 (793.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

What worries me, is the different broadcast and network address in the ens160:10 entry (which is the one created by keepalived).

Shouldn't those match the ones from the regular interface?

For the moment, my setup seems to work, but I want to make sure it is setup correctly.

I'm trying to figure out how I can use a single nfs share with k8s persistent volume claims.

For example, let's say I have a single nfs pv configured:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    path: /var/nfs_exports
    server: 10.9.0.205
    readOnly: false

Is it possible to create multiple volume claims that map to subdirectories within this single share?

For example again, let's say I create the following volume claims:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: influx-data
  namespace: kube-system
spec:
  storageClassName: nfs-storage
  accessModes:
  - ReadWriteMany
  resources:
     requests:
       storage: 5Gi
---

and:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: elasticsearch-data
  namespace: kube-system
spec:
  storageClassName: nfs-storage
  accessModes:
  - ReadWriteMany
  resources:
     requests:
       storage: 2Gi
---

I guess, that both claims will be bound to the pv, but there is no way to seperate the data of both elasticsearch and influxdb.

I hope you understand what I'm trying to do here (sorry, I find it difficult to explain). I just want to use a single nfs share that can be used by multiple pods, while still keeping their data seperate.

Okay, I'm trying to install ceph, using just "yum install ceph" from the epel repositories.

This is what I get:

--> Finished Dependency Resolution
Error: Package: 1:python-cephfs-0.80.7-0.8.el7.x86_64 (epel)
           Requires: python-rados = 1:0.80.7
           Available: 1:python-rados-0.94.5-1.el7.x86_64 (base)
               python-rados = 1:0.94.5-1.el7
Error: Package: 1:ceph-0.80.7-0.8.el7.x86_64 (epel)
           Requires: python-rados = 1:0.80.7
           Available: 1:python-rados-0.94.5-1.el7.x86_64 (base)
               python-rados = 1:0.94.5-1.el7
Error: Package: 1:ceph-common-0.80.7-0.8.el7.x86_64 (epel)
           Requires: python-rados = 1:0.80.7
           Available: 1:python-rados-0.94.5-1.el7.x86_64 (base)
               python-rados = 1:0.94.5-1.el7
Error: Package: 1:ceph-common-0.80.7-0.8.el7.x86_64 (epel)
           Requires: python-rbd = 1:0.80.7
           Available: 1:python-rbd-0.94.5-1.el7.x86_64 (base)
               python-rbd = 1:0.94.5-1.el7
Error: Package: 1:ceph-0.80.7-0.8.el7.x86_64 (epel)
           Requires: python-rbd = 1:0.80.7
           Available: 1:python-rbd-0.94.5-1.el7.x86_64 (base)
               python-rbd = 1:0.94.5-1.el7

Who should I report this too? To the Centos people, the epel people, or the ceph people?

And how can I fix this temporary? I need Ceph today.

I'm running the latest version of esxi (6.5) and vCenter. I created a VM template based on CentOS7, and used a LVM-based disk layout.

However, now vSphere customizations don't work, and I'm getting the following error:

Customization of the guest operating system 'rhel7_64Guest' is not supported in this configuration. Microsoft Vista (TM) and Linux guests with Logical Volume Manager are supported only for recent ESX host and VMware Tools versions. Refer to vCenter documentation for supported configurations.

The only thing I need to customize, are the ip address and hostname. I don't need to modify any disk sizes.

Is there any way to fix this? I'm already using the latest version of ESX and vSphere. My searches on the Internet reveal lots of out-dated information. Some mention the openvm-deploypkg tool, but It seems that the tool is now integrated with the native open-vm-tools package (source: https://github.com/vmware/open-vm-tools/issues/9).

I cannot understand why customizations can't deal with Linux LVM. I mean, LVM is not a new technology. it's a stable Linux component for many years now, and I'm fond of it. Did VMWare just turn back the clock more than 10 years for me?

I want to use kafka as a transport layer for collectd. I found that there is a write_kafka plugin for collectd, which sends all the gathered metrics to a kafka topic.

My intention was to have a few hosts as collectors (working as a kafka consumer groups) to get those metrics off the topic, and put them into a time-series database (influxdb or graphite).

collectd doesn't have a kafka consumer input plugin. graphite can't read kafka directly either (I think?). What piece of software am I missing here?

Just to make some things clear: I'm talking about the process logfile that contains the stdout and stderr messages.

This is my systemd unit file:

[Unit]
Description=Apache Kafka server
Documentation=http://kafka.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
PIDFile=/var/run/kafka.pid
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/jre"
Environment="KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/var/log/kafka"

ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target

Note that I added the KAFKA_LOG4J_OPTS environment variable.

However, that doesn't seem to do anything. This is the output, when I try to start my service:

Feb 11 00:55:30 kafka01 kafka[4047]: mkdir: cannot create directory ‘/opt/kafka/bin/../logs’: Permission denied
Feb 11 00:55:30 kafka01 kafka[4047]: OpenJDK 64-Bit Server VM warning: Cannot open file /opt/kafka/bin/../logs/kafkaServer-gc.log due to No such file or directory
Feb 11 00:55:30 kafka01 kafka[4047]: log4j:WARN No appenders could be found for logger (kafka.Kafka$).
Feb 11 00:55:30 kafka01 kafka[4047]: log4j:WARN Please initialize the log4j system properly.

So, can please someone tell me the correct way to change the location of the kafka log file? KAFKA_LOG4J_OPTS doesn't work, KAFKA_LOG_DIR doesn't work either (=I was hoping kafka implemented this similar to zookeeper). The kafka documentation doesn't tell me either.

I have my entire website behind a CloudFront distribution, and I used to have a Route53 zone, where the zone apex record for my website pointed to an ALIAS record for the CloudFront distribution.

However, now I'm switching to NS1 as a DNS provider. They also have a similar ALIAS record, however, there documentation states:

Geotargeting information is lost. Since it is the authoritative server for example.com that is issuing the queries for lb.example.net, then any intelligent routing functionality on the lb.example.net record will act upon the location of the authoritative server, not on your location. The EDNS0 edns-client-subnet option does not apply here. This means that you may be potentially mis-routed: for example, if you are in New York, and the authoritative server for example.com is in California, then lb.example.com will believe you to be in California and will return an answer that is distinctly sub-optimal for you in New York.

Basically, if I understand correctly, visitors of my website might be directed to a less than optimal CloudFront edge node.

Is this something I really need to worry about? For example: how big is the chance that a European visitor is directed to a CloudFront edge location in Asia?

I'm trying to integrate Kapacitor with our influxdb and collectd setup. However, It doesn't seem to work, and I don't understand why.

Collectd and Influxdb are running correctly, and I think Kapacitor is able to connect to influxdb. In the kapacitor log I see this:

[influxdb] 2016/04/22 09:46:42 I! started UDP listener for collectd_db default

This is the name of the influxdb database where collectd is recording metrics.

I created the following tick-file, and uploaded it to kapacitor and enabled it:

stream
    .from().measurement('cpu_value')
    .where(lambda: "type" == "percent")
    .where(lambda: "type_instance" == "idle")
    .alert()
        .crit(lambda: "value" <  100)
        // Whenever we get an alert write it to a file.
        .log('/tmp/alerts.log')

This was just a test script, which hopefully produces some output.

The script is enabled:

Name                          Type      Enabled   Executing Databases and Retention Policies
cpu_tick                      stream    true      true      ["collectd_db"."default"]

However, I don't see any recordings:

[centos@ip-xx-xx-xx-xx tmp]$ kapacitor list recordings
ID                                      Type    Size      Created      

"cpu_value" is a valid measurement in my database.

This is what I get in my error log:

[cpu_alert:stream1] 2016/04/28 13:00:51 E! error while evaluating WHERE expression: name "percent" is undefined. Names in scope: time,value,host,instance,type,type_instance

I have a simple question, with hopefully a simple solution.

I have a private Route53 zone, which is only available within a VPC. I want some kind of dynamic dns, so when servers are launched, they register themselves with the Route53 zone.

However, I can't come up with a clean solution. The hostname and domain are already set correctly by cloud-init and user-data, but I want a solution that accomplishes the following:

Register the hostname within the route53 zone: now, this is not too hard as the hostname is already correct in /etc/hostname. I just need to take this value and update the dns record. Not too fancy.

However, what I'm missing is how I can clean up Route53 when the server is turned off. I'm talking about the following scenario's:

• server is turned off correctly: What's the best place to put a script that only runs on shutdown?
• sudden IP change: Maybe there was a network interruption and we might have received a new IP address via DHCP. Is it possible to integrate my script with DHCP somehow?
• server lost power: maybe there was an issue at the region AZ, and we lost power. Maybe the instance is destroyed? I don't want the DNS records of those servers around in my DNS zone. Inactive servers should be removed from DNS after a few hours.

My greatest frustration is that DHCP + Dynamic DNS already solves most of those issues. However, dynamic updates via DHCP don't work with private Route53 zones.

Any suggestions?

I'm completely confused how I can add my servers to Foreman. I've setup a Foreman server, I pointed the /etc/puppet/puppet.conf "server" directive to my Foreman server, and I under "Provisioning=>Smart Proxy" I see the signature requests. I've signed them via the web-interface, but when I go to "Hosts=>All hosts", only the Foreman server is listed there.

What do I need to get my servers added to this list, so I can manage the servers via Puppet?

this is the puppet.conf file I use on my servers:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[agent]
server = foreman.srv.mydomain.com

I'm an absolute beginner at Puppet, and I'm having an issue when I try to install a package via the apt-module.

class newrelic {
        apt::source {
                'newrelic':location => 'http://apt.newrelic.com/debian/',
                repos => 'non-free',
                key => '548C16BF',
                key_source => 'https://download.newrelic.com/548C16BF.gpg',
                include_src => false,
                release => 'newrelic',
        }
        package {
                'newrelic-sysmond':ensure => 'present',
                notify => Service['newrelic-sysmond'],
                require => Class['apt::source'],
        }
        service {
                'newrelic-sysmond':ensure => 'running',
                enable => true,
                hasrestart => true,
                hasstatus => true,
                require => Exec['newrelic_config'],
        }
        exec {
                'newrelic_config':path => '/bin:/usr/bin',
                command => "/usr/sbin/nrsysmond-config --set license_key=xxxxxxx",
                user => 'root',
                group => 'root',
                require => Package['newrelic-sysmond'],
                notify => Service['newrelic-sysmond'],
        }
}

This is the error I receive:

Warning: Scope(Class[Apt::Update]): Could not look up qualified variable '::apt::always_apt_update'; class ::apt has not been evaluated
Warning: Scope(Class[Apt::Update]): Could not look up qualified variable 'apt::update_timeout'; class apt has not been evaluated
Warning: Scope(Class[Apt::Update]): Could not look up qualified variable 'apt::update_tries'; class apt has not been evaluated
Notice: Compiled catalog for host.domain.local in environment production in 0.33 seconds
Error: Could not find dependency Class[Apt::Source] for Package[newrelic-sysmond] at /home/jeroen/puppet/modules/newrelic/manifests/init.pp:16

Any idea what I'm doing wrong in my module?

We are putting a NetScaler device in front of our Domino iNotes servers, which will act as a reverse proxy.

However, we get "Error 400" errors in the iNotes webinterface, and the behaviour is consistent with this documented issue: https://www-304.ibm.com/support/docview.wss?uid=swg21453878

How do we prevent NetScaler from stripping off the "X-IBM-INOTES-NONCE" header?

I'm installing Ubuntu server on a disk with 12GB available. During the setup, I choose the default LVM-based partition layout.

However for some reason, Ubuntu decides that it only wants to use 4GB of this disk. How do I reclaim the remaining space of the hard disk? "lvextent" doesn't work btw...

output of df -h:

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu-root  4.3G  3.4G  754M  82% /
udev                     3.9G  4.0K  3.9G   1% /dev
tmpfs                    1.6G  756K  1.6G   1% /run
none                     5.0M     0  5.0M   0% /run/lock
none                     3.9G     0  3.9G   0% /run/shm
/dev/sda1                228M   25M  192M  12% /boot

output of pvdisplay:

  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               ubuntu
  PV Size               12.32 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              3154
  Free PE               8
  Allocated PE          3146
  PV UUID               dD06RZ-kGcL-1tTX-Ruds-XIDG-ssMd-FIUkzZ

my partitions:

Device       Boot         Start         End              Blocks        Id   System
/dev/sda1     *            2048      499711             248832         83   Linux
/dev/sda2                501758    26343423           12920833          5   Extended
/dev/sda5                501760    26343423           12920832         8e   Linux LVM

when I try lvextent, it says there is not enough diskspace.

I'm using the free version of VMWare's ESXi 5.1.0.

I'm logged in as root in the vSphere client, and I want to create some scheduled tasks. According to the documentation, I should go to "Home"=>"Management"=>"Scheduled tasks".

However, when I go to "Home", I only have "Inventory" and "Administration". There is no "Management" section.

Is this a limitation of the free version (I doubt it, but still), or am I missing the obvious here?

I'm trying to connect a NFS share to XenCenter. The NFS server is a ZFSGuru distro (uses FreeBSD).

The zfs volume was exported like this:

/sbin/zfs set sharenfs="on" temppool/share

According to "showmount", it's available:

showmount -e
/temppool/share   Everyone

However, when I try to connect to it with XenServer (so it can be used as storage for VHD), I get the following error:

Internal error:Failure("Storage_access failed with: SR_BACKEND_FAILURE_73: [; NFS mount error[opterr=mount failed with return code 32]; ]")

Anyone got an idea?

Update:

This is from the log on the NFS server:

Sep 3 16:23:10 zfsguru mountd[962]: mount request from 192.168.10.217 for non existent path /temppool/share/7c8d3f2f-e0e0-5263-ccad-1cd32a4139cf

Sep 3 16:23:10 zfsguru mountd[962]: mount request denied from 192.168.10.217 for /temppool/share/7c8d3f2f-e0e0-5263-ccad-1cd32a4139cf

Sep 3 16:23:11 zfsguru mountd[962]: mount request from 192.168.10.217 for non existent path /temppool/share/7c8d3f2f-e0e0-5263-ccad-1cd32a4139cf

Sep 3 16:23:11 zfsguru mountd[962]: mount request denied from 192.168.10.217 for /temppool/share/7c8d3f2f-e0e0-5263-ccad-1cd32a4139cf

Sep 3 16:28:20 zfsguru mountd[962]: mount request denied from 192.168.10.217 for /temppool/share/17922178-0dfb-edf3-0037-2eddd79b9d02

Sep 3 16:28:43 zfsguru last message repeated 5 times

Sep 3 16:35:00 zfsguru mountd[962]: mount request denied from 192.168.10.217 for /temppool/share/b5735ccf-1997-8d77-83a0-2f34e37dda8d

Sep 3 16:35:33 zfsguru last message repeated 4 times

Sep 3 16:35:34 zfsguru mountd[962]: mount request denied from 192.168.10.217 for /temppool/share/b5735ccf-1997-8d77-83a0-2f34e37dda8d

It seems XenServer is able to create the directories, but is enable to mount them afterwards.