Manwe's questions

I have a problem where dns entry for a external domain broke. The nature of the problem at the time is unknown.

That domain got queried from kubernetes cluster pod in the Google Kubernetes Engine while the entry was broken. The problem persists (incident happened over 2 months ago) when querying that domain from the cluster.

The cluster dns resolver uses metadata.google.internal for dns resolving and from the cluster these queries with dig will:

dig problematic.external.domain @169.254.169.254
# does not resolve and takes over 2 seconds
dig problematic.external.domain  @1.1.1.1
# resolves correctly under 200ms

Creating a new vm in the same project and zone resolves the problematic domain correctly. This is affects only the active cluster metadata server dns resolver.

Is there a way to flush dns caches or any other suggestions?

In general I'm trying to avoid editing in-cluster dns settings and would prefer some other means to fix it.

Edit more info: NodeLocal DNSCache is already active in the cluster and referencing that documentation https://cloud.google.com/kubernetes-engine/docs/how-to/nodelocal-dns-cache the problem is the metadata dns server. This excerpt from the benefits list:

DNS queries for external URLs (URLs that don't refer to cluster resources) are forwarded directly to the local Cloud DNS metadata server, bypassing kube-dns.

Which is the ip 169.254.169.254

Problem using blktap with xen-4.1 running Ubuntu Precise stock kernel with dom0 xen-4.1.

I get:

[    5.580106] XENBUS: Waiting for devices to initialise: 295s...290s.
...
[  300.580288] XENBUS: Timeout connecting to device: device/vbd/51713 (local state 3, remote state 1)

And some syslog lines:

May 17 13:07:30 localhost logger: /etc/xen/scripts/blktap: add XENBUS_PATH=backend/tap/10/51713
May 17 13:07:31 localhost logger: /etc/xen/scripts/blktap: Writing backend/tap/10/51713/hotplug-status connected to xenstore.

with tap:aio: disk lines. file:/ works.

disk = [
    'tap:aio:/data/root.img,xvda1,w',
]

Problem exists with lucid and precises domU kernels and both guests work in Ubuntu hardy dom0 Host 64bit 2.6.24-28-xen xen-3.3

 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description:    Ubuntu 12.04 LTS
Release:        12.04
Codename:       precise