Marc Hughes's questions

I have a single node cluster in container engine that sits at ~40% cpu when idle as seen by the monitoring dashboard.

When I click through the monitoring, all the pods are at 0% cpu.

When I ssh into the instance, I can see that docker, kubelet, and heapster are the main culprits, but I don't understand what work they're doing.

If I look in the logs, I see lines like the following, repeated over and over many times per second. I'm guessing this is related.

gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="-job containers() = OK (0)"
gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="+job containers()"
gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="GET /containers/json"

What should be my next step to figure out why this is happening?

I have a single node kubernetes cluster in google container engine to play around with.

Twice now, a small personal website I host in it has gone offline for a couple minutes. When I view the logs of the container, I see the normal startup sequence recently completed, so I assume a container died (or was killed?) and restarted.

How can I figure out the how & why of this happening?

Is there a way to get an alert whenever a container starts/stops unexpectedly?

Occasionally, I get an apache process running at 100% CPU. I've tried letting it run it's course, but I've seen it go more than half an hour without giving up. My running hypothesis is there is an infinite loop in some application code somewhere.

I've used mod_status, but the PID doesn't show up there.

I've attached GDB and gotten this backtrace:

#0  0x00895424 in __kernel_vsyscall ()
#1  0x00574cde in poll () from /lib/i686/nosegneg/libc.so.6
#2  0x0012d520 in apr_poll () from /usr/lib/libapr-1.so.0
#3  0x00707902 in ?? () from /etc/httpd/modules/mod_wsgi.so
#4  0x0070e7b1 in ?? () from /etc/httpd/modules/mod_wsgi.so
#5  0x00d1e81a in ap_run_post_config ()
#6  0x00d09812 in main ()

So it looks like it's a mod_wsgi process causing me troubles. Is there a way for me to figure out what request goes with that process?

Alternatively, is there a way for me to attach a python profiler or debugger to see what code is running?

We've got a web app hosted on Amazon Web services. Our database is a multi-az RDS MySQL server running 5.1.57 and 3-4 app servers talk to it.

Today, we started seeing a lot of errors along the lines of "Lock wait timeout exceeded; try restarting transaction" - almost 1% of POST requests are seeing this.

There have been no modifications to the code running on the site. There have been no schema changes. We haven't had a big spike in traffic. I've been looking at the processes running, and none seem out of control.

I tried scaling our RDS instance from a small to a large, with no effect.

Two days ago, Amazon had some outages. As part of the recovery from that, our RDS server, and our app servers ended up in different availability zones, but all within the same region. But yesterday, everything was fine so I'm not convinced that's related.

The lock timeouts are in different types of requests and occur in different InnoDB tables.

I have noticed the number of open connections jumped when we started seeing problems, but they may be a symptom and not a cause.

What are my next steps in debugging this?