Tom Klino's questions

I'm running minikube with the vbox driver. I noticed that when I get back to my workstation after it's been in sleep mode, I can't do kubectl get pods (or any other kubectl command).

Some digging and I found that the the HostOnly network adapter of the minikube machine is not working. I logged in to the machine with minikube ssh and tried to restart the interface but I'm just getting an error.

$ ifdown eth1 
ifdown: can't open '/etc/network/interfaces': No such file or directory

the only thing currently working is to minikube stop and minikube start again. But that takes long and I just want to get back to work.

any ideas on how to fix the network quicker after it's been in sleep mode? or prevent it losing the network altogether?

So my manager have left the company a few months back, leaving me to manage things on my own, and now, I'm looking through cloud trail events, and found that in one of our accounts, I have an explicit deny from viewing them.

Here is some background. We have a root account, from which we assume the admin role in other accounts, when I assume the role of admin in the production account, I get an explicit deny error message when trying to view cloud trail events.

I've been looking through all the policies that have the "Deny" effect on them anywhere, but nothing looks related. Also, the admin role I'm assuming is not linked to any policy other than the allow all policy. And yet, I get the same error message of explicit deny.

Any magic tool that I can use to understand where the explicit deny is coming from?

Note: I am aware that there are many similar questions, already looked at them, didn't help.

I am running a kafka benchmark test in a closed environment, I gave each broker a filesystem of 40GB for log persistence, and I'm quickly getting to a situation in which it crashes and does not get back up, because that file-system is being filled.

So, in order to avoid such a catastrophic failure in production, I tried setting log.retention.bytes=10737418240 (10GB) and tested it to see if kafka deletes logs before it reaches a situation in which it crashes

as shown in the graph, kafka didn't delete anything after passing 10GB (in other tests, it also got all the way to 40 and crashed again)

Here is my entire server.properties file:

# server settings
controlled.shutdown.enable=true
log.retention.bytes=10737418240
log.cleanup.policy=delete
log.segment.delete.delay.ms=10000
log.retention.check.interval.ms=1000
listeners=PLAINTEXT://:9092

zookeeper.connect=kafka-zookeeper:2181
zookeeper.session.timeout.ms=6000

# this must correlate to kafka's volume claim templates
log.dirs=/var/lib/kafka

# default settings for topics
auto.create.topics.enable=true
delete.topic.enable=true
num.partitions=10
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2
default.replication.factor=2
# 6291456 / 1024 / 1024 = 6Mb
replica.fetch.max.bytes=6291456
# 5242880 / 1024 / 1024 = 5Mb
message.max.bytes=5242880
group.initial.rebalance.delay.ms=3000

What am I missing?

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out.

However, when I'm logging in to one the brokers (using kubectl exec if relevant) and try to get some information about its state using one of the scripts in bin, I get an exception Address already in use:

root@kafka-0:/kafka# bin/kafka-consumer-groups.sh  --list --bootstrap-server kafka:9092
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.net.BindException: Address already in use
    at java.base/sun.nio.ch.Net.bind0(Native Method)
    at java.base/sun.nio.ch.Net.bind(Net.java:461)
    at java.base/sun.nio.ch.Net.bind(Net.java:453)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
    at jdk.httpserver/sun.net.httpserver.ServerImpl.bind(ServerImpl.java:134)
    at jdk.httpserver/sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
    at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
    at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:54)
    ... 6 more
FATAL ERROR in native method: processing of -javaagent failed
Aborted (core dumped)

A quick look using strace showed that it is trying to bind to port 80. I have no idea why that is even necessary for a script that is meant only to obtain data.

The same error happens on all scripts in the kafka bin directory.

Any ideas regarding on how to fix will be a great help

I've set up a new server in digital ocean, ubuntu 16.04. installed docker from the ubuntu repos and set up kuberenetes according to this guide:

https://kubernetes.io/docs/setup/independent/install-kubeadm/

I've tried several times, to kubeadm init following by a kubeadm reset before trying again.

the output log suggests to check that kubelet is running and it does:

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2018-09-05 13:34:35 UTC; 24min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 2615 (kubelet)
    Tasks: 15
   Memory: 44.8M
      CPU: 3min 18.521s
   CGroup: /system.slice/kubelet.service
           └─2615 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin

Sep 05 13:59:15 kubernetes-master kubelet[2615]: E0905 13:59:15.538370    2615 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin
Sep 05 13:59:17 kubernetes-master kubelet[2615]: E0905 13:59:17.518350    2615 kubelet_node_status.go:103] Unable to register node "kubernetes-master" with API server: Post https://48.88.103.0:6443/api/v1/nodes
Sep 05 13:59:18 kubernetes-master kubelet[2615]: E0905 13:59:18.552653    2615 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://48.88.103.0:6443/api/v1/ser
Sep 05 13:59:19 kubernetes-master kubelet[2615]: E0905 13:59:19.659689    2615 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://48.88.103.0:6443/api/v1/nodes?
Sep 05 13:59:20 kubernetes-master kubelet[2615]: W0905 13:59:20.556075    2615 cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d
Sep 05 13:59:20 kubernetes-master kubelet[2615]: E0905 13:59:20.576691    2615 kubelet.go:2110] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin
Sep 05 13:59:21 kubernetes-master kubelet[2615]: E0905 13:59:21.952965    2615 certificate_manager.go:299] Failed while requesting a signed certificate from the master: cannot create certificate signing request:
Sep 05 13:59:23 kubernetes-master kubelet[2615]: E0905 13:59:23.817566    2615 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "kubernetes-master" not fo
Sep 05 13:59:24 kubernetes-master kubelet[2615]: I0905 13:59:24.524068    2615 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Sep 05 13:59:24 kubernetes-master kubelet[2615]: I0905 13:59:24.594325    2615 kubelet_node_status.go:79] Attempting to register node kubernetes-master

It also suggest to check network connectivity and that the images are able to download. I verified with pinging outside the network and checking that kubeadm config images pull executes successfully.

I do see that I get a TLS handshake timeout a lot in the journalctl log when it tries to reach the server's IP address.

I have seen a lot of discussion about this issue online but none of the existing solution has helped me this far.

What else can I test for/check/try?

I have a recipe that sets up a docker instance with a specific mysql image I made for my app. As part of that recipe I would like to have it so, that after the container comes up with mysql running, a procedure for restoring data from backup will begin automatically.

For that I need to have the IP of the container (or a way to get it from DNS). However, Since I bring the container up automatically from a recipe, I have no way of knowing what that IP will be, so I'm trying to have docker set it with a hostname that it can later resolve.

Oddly, if I link a container to another, I will be able to resolve the host of the other container from within the one linked to it. But, I have found no way resolving that hostname from the host.

Here is the part of the recipe (chef) I use to bring up the container:

docker_container 'imhere-mysql' do
  hostname mysqlHost
  repo 'lutraman/imhere'
  tag 'mysql'
  env ["MYSQL_ROOT_PASSWORD=#{mysql_password}"]
  volumes [ '/var/imhere/mysql:/var/lib/mysql' ]
  action :run
end

The command I would like to use to restore data is something like this (constructed from the recipe):

"gunzip -c #{parent_data_dir}/#{db_restore_filename} | mysql -h #{mysqlHost} -u root -p'#{mysql_password}'"

only that running from the host, the name in mysqlHost (in the case at hand - imhere-mysql) is not resolved from the host.

There is a note at the bottom of this article, saying there should be a docker dns server running on 127.0.0.11 - but I fear this is internal to containers.

What I'm hoping I'll be able to do, is query docker (in the ideal situation, as a DNS, but any way will work) for the ip of the container named 'imhere-mysql'. What's the best way to do this in a way that will hopefully hold for future versions on docker and regardless of the host OS?

For testing purposes, I have brought up 2 Debian jessie VMs (using vagrant and virtualbox) and had them both running a mini web server and configures with keepalived.

The problem is that the BACKUP server transitions a few seconds after service keepalived restart to MASTER for no apparent reason.

I am also seeing VRRP multicast packets arriving on both servers and they look as expected. (which seems to mean that the solution suggested in Keepalived unwanted transition to master is not related)

The configurations files are as follows:

MASTER server:

global_defs {
        lvs_id tom_lvs
}

vrrp_instance tom_lvs {
        state MASTER
        interface eth1
    virtual_router_id 1
    lvs_sync_daemon_interface eth1
    priority 100
    preempt
        authentication {
                auth_type PASS
                auth_pass 1234
        }

    advert_int 1
    virtual_ipaddress {
        172.28.128.10/24
    }

        virtual_server 172.28.128.10 3000 {
                delay_loop 10
                lb_algo wlc
                lb_kind DR
                protocol TCP
                persistence_timeout 1800
                sorry_server 172.28.128.3 3000
                real_server 172.28.128.4 3000 {
                        weight 5
                        HTTP_GET {
                                url {
                                        path /index.html
                                }
                        }
                }
        }
}

BACKUP server:

global_defs {
        lvs_id tom_lvs
}

vrrp_instance tom_lvs {
        state BACKUP
        interface eth1
    virtual_router_id 2
        lvs_sync_daemon_interface eth1
    priority 96
    preempt
        authentication {
                auth_type PASS
                auth_pass 1234
        }

    advert_int 1
    virtual_ipaddress {
        172.28.128.10/24
    }

        virtual_server 172.28.128.10 3000 {
                delay_loop 10
                lb_algo wlc
                lb_kind DR
                protocol TCP
                persistence_timeout 1800
                sorry_server 172.28.128.3 3000
                real_server 172.28.128.3 3000 {
                        weight 5
                        HTTP_GET {
                                url {
                                        path /index.html
                                }
                        }
                }
        }
}

logs on the backup server that shows the transition from BACKUP to MASTER:

Feb 14 13:22:44 jessie Keepalived_vrrp[1145]: VRRP_Instance(tom_lvs) sending 0 priority
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Registering Kernel netlink reflector
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Registering Kernel netlink command channel
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Registering gratuitous ARP shared channel
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Opening file '/etc/keepalived/keepalived.conf'.
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Registering Kernel netlink reflector
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Configuration is using : 62104 Bytes
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: Using LinkWatch kernel netlink reflector...
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Registering Kernel netlink command channel
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Opening file '/etc/keepalived/keepalived.conf'.
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Configuration is using : 12103 Bytes
Feb 14 13:22:44 jessie Keepalived_vrrp[1176]: VRRP_Instance(tom_lvs) Entering BACKUP STATE
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Using LinkWatch kernel netlink reflector...
Feb 14 13:22:44 jessie Keepalived_healthcheckers[1175]: Activating healthchecker for service [172.28.128.3]:3000
Feb 14 13:22:48 jessie Keepalived_vrrp[1176]: VRRP_Instance(tom_lvs) Transition to MASTER STATE
Feb 14 13:22:49 jessie Keepalived_vrrp[1176]: VRRP_Instance(tom_lvs) Entering MASTER STATE

and this is what I see on both machines in tshark

Capturing on 'eth1'
  1   0.000000 172.28.128.4 -> 224.0.0.18   VRRP 60 Announcement (v2)
  2   0.001404 172.28.128.3 -> 224.0.0.18   VRRP 54 Announcement (v2)
  3   1.002283 172.28.128.4 -> 224.0.0.18   VRRP 60 Announcement (v2)
  4   1.004130 172.28.128.3 -> 224.0.0.18   VRRP 54 Announcement (v2)
  5   2.004466 172.28.128.4 -> 224.0.0.18   VRRP 60 Announcement (v2)
  6   2.006260 172.28.128.3 -> 224.0.0.18   VRRP 54 Announcement (v2)

I'm trying to get keepalived working between 2 debian jessie VMs.

I've defined the master machine to bind the VIP address to 172.128.28.6 as such:

global_defs {
        lvs_id tom_lvs
}

vrrp_instace tom_lvs {
        state MASTER
        interface eth1
    virtual_router_id 1
    priority 100
        authentication {
                auth_type PASS
                auth_pass 1234
        }

    advert_int 1
    virtual_ipaddress {
        172.28.128.6
    }

        virtual_server 172.28.128.6 3000 {
                delay_loop 10
                lb_algo wlc
                lb_kind DR
                protocol TCP
                persistence_timeout 1800
                sorry_server 172.28.128.3 3000
                real_server 172.28.128.4 3000 {
                        weight 1
                        HTTP_GET {
                                url {
                                        path /index.html
                                }
                        }
                }
        }
}

however, when I run ip addr I am not seeing a second IP attached to eth3:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:f6:86:bf brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fef6:86bf/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:05:a5:a1 brd ff:ff:ff:ff:ff:ff
    inet 172.28.128.4/24 brd 172.28.128.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe05:a5a1/64 scope link 
       valid_lft forever preferred_lft forever

I am obviously missing something here, but I've been at it for 2 days now and still didn't find it.

If at all relevant, the machines are running using vagrant and virtual-box, both defined to get their addresses from dhcp, and their IPs are 172.128.28.4 and 172.128.28.3 on the /24 subnet.

here is the output of grep Keepalived /var/log/messages:

Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Registering Kernel netlink reflector
Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Registering Kernel netlink command channel
Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Registering gratuitous ARP shared channel
Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Opening file '/etc/keepalived/keepalived.conf'.
Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Configuration is using : 58175 Bytes
Feb 13 11:39:51 jessie Keepalived_vrrp[1890]: Using LinkWatch kernel netlink reflector...
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Registering Kernel netlink reflector
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Registering Kernel netlink command channel
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Opening file '/etc/keepalived/keepalived.conf'.
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Configuration is using : 12031 Bytes
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Using LinkWatch kernel netlink reflector...
Feb 13 11:39:51 jessie Keepalived_healthcheckers[1889]: Activating healthchecker for service [172.28.128.4]:3000

I want to have a condition to do something in a chef recipe only if there is no other recipe in the run list, and another that will be true only if a specific recipe is not in the run list. Is there a way to do this?

Also, what does the best practices say about this kind of logic? Should this be avoided from some reason?

I want to create a docker image on top of the mysql one that already contains the necessary scheme for my app.

I tried adding lines to the Dockerfile that will import my scheme as a sql file. I did so as such (my Dockerfile):

FROM mysql

ENV MYSQL_ROOT_PASSWORD="bagabu"
ENV MYSQL_DATABASE="imhere"

ADD imhere.sql /tmp/imhere.sql

RUN "mysql -u root --password="bagabu" imhere < /tmp/imhere.sql"

To my understanding, that didn't work because the mysql docker image does not contain a mysql client (best practices state "don't add things just because they will be nice to have") (am I wrong about this?)

what might be a good way to do this? I have had a few things in mind, but they all seem like messy workarounds.

1. install the mysql client, do what I have to do with it, then remove/purge it.
2. copy the mysql client binary to the image, do what I have to do, then remove it.
3. Create the schema in another sql server and copy the db file themselves directly (this seems very messy and sounds to me like a contaminated pool of problems)

Any suggestions? Hopefully in a way that will be easy to maintain later and maybe conform with the best practices as well?

I have a VM (on oracle vbox) running Fedora17. I've installed asterisk 11 on it from sources. I've followed the wiki for installation (https://wiki.asterisk.org/wiki/display/AST/Creating+SIP+Accounts) to the letter.

The ip on the VM machine running fedora is 192.168.1.7

and I can ping it from the host machine (Ubuntu 12.04), which is at 192.168.1.2

I've tried registering with ekiga with the following settings:

user: [email protected]. Password: verysecretpassword registar: 192.168.1.7 but I'm getting an error "transport fail".

Also, while trying to register I'm logged in to the asterisk CLI with verbose level 3 and debug level 4 and nothing appears.

some more relevant data:

I've added the following code to the end of my sip.conf.sample file:

[demo-alice]
type=friend
host=dynamic
secret=verysecretpassword
context=users
deny=0.0.0.0/0
permit=192.168.1.0/255.255.255.0

[demo-bob]
type=friend
host=dynamic
secret=othersecretpassword
context=users
deny=0.0.0.0/0
permit=192.168.1.0/255.255.255.0

After I changed the sip.conf.sample file, I've created a copy of it and named it sip.conf. then I logged in to the asterisk CLI and typed sip reload.

Then I'm trying to register and ekiga client from my host machine at 192.168.1.2 but it doesn't work and nothing appears on the asterisk CLI while in verbose mode level 3.

BTW, If there is missing information about my question, please don't close it. comment about what you need to know and I'll edit it in to the question. tnx.