After a minimal debian stretch installation using (debian-9.8.0-amd64-netinst.iso), where hostname is left at the default debian and domain was left empty, installing postfix fails because the detected fully qualified hostname contains a trailing . character.

I think it is dhclient that saves the resolv.conf file on a minimal install without Network Manager. It has trailing . characters in its search clause even though the DHCP server doesn't send any (see below).

If hostname --fqdn is empty, postfix looks in resolv.conf with sed and finds and tries to use the FQDN with the trailing . which causes postfix installation to fail (see below).

Where is the bug? In postfix.postinst, in dhclient, with my understanding or somewhere else?

In reality this fails when installing postfix from a Dockerfile, but the same behavior is seen with a minimal debian installation, so it isn't really Docker related, apart from the fact that you can't control the FQDN of the "host"/container during docker build so we run into this problem. The workaround is of course to patch resolv.conf before installing postfix, but that is very much a hack. (Unless you have a better idea?)

Details:

/etc/hosts:

127.0.0.1   localhost
127.0.1.1   debian

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

/etc/resolv.conf

domain capmon.lan
search capmon.lan. capmon.
nameserver 10.10.10.251

Now running apt-get install postfix:

<snip>
setting myhostname: debian.capmon.lan.
setting alias maps
setting alias database
mailname is not a fully qualified domain name.  Not changing /etc/mailname.
setting destinations: $myhostname, debian, localhost.localdomain, , localhost
setting relayhost: 
setting mynetworks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
setting mailbox_size_limit: 0
setting recipient_delimiter: +
setting inet_interfaces: all
setting inet_protocols: all

Postfix (main.cf) is now set up with a default configuration.  If you need to 
make changes, edit /etc/postfix/main.cf (and others) as needed.  To view 
Postfix configuration values, see postconf(1).

After modifying main.cf, be sure to run 'service postfix reload'.

Running newaliases
newaliases: warning: valid_hostname: misplaced delimiter: debian.capmon.lan.
newaliases: fatal: file /etc/postfix/main.cf: parameter myhostname: bad parameter value: debian.capmon.lan.
dpkg: error processing package postfix (--configure):
 subprocess installed post-installation script returned error exit status 75
Processing triggers for systemd (232-25+deb9u12) ...
Processing triggers for rsyslog (8.24.0-1) ...
Errors were encountered while processing:
 postfix
E: Sub-process /usr/bin/dpkg returned an error code (1)

And that is because postfix's myhostname has a trailing .:

# grep ^myhostname /etc/postfix/main.cf
myhostname = debian.capmon.lan.

And that is because /var/lib/dpkg/info/postfix.postinst has:

# The resolver uses the last one found, and ignores the rest
mydom=$(sed -n 's/^search[[:space:]]*\([^[:space:]]*\).*/\1/p;s/^domain[[:space:]]*\([^[:space:]]*\).*/\1/p' /etc/resolv.conf | tail -1)
myhostname="$myhostname${mydom:+.$mydom}"

and sure enough:

# sed -n 's/^search[[:space:]]*\([^[:space:]]*\).*/\1/p' /etc/resolv.conf
capmon.lan.

I've checked (also in the transmitted bytes), and the DHCP server does not send trailing . characters:

With puppet I realize I can copy folders recursively and set permissions on individual files recursively. But I have a source folder with (many, many) files each with carefully configured permissions. How do I do the puppet equivalent of:

cp -a $source $dest

?

This:

file { '/some/dest':
    ensure => directory,
    source => "/some/src",
    recurse => true,
}

Does not preserve mode.

Also, Best Practices: Recursive File Resources says to only use recursive files:

When the total number of recursive files to manage is small (such as ten or fewer)

Ok, so for the sake of argument, lets say /some/src has 1000s of files with complicated modes.

According to that link, my other options are

• recursive_file_permissions defined type. Here I can only set all files to the same permissions, not set them to have the same permissions as the source == won't work for us.
• puppet/archive module
• build a package for the files, and use a Package resource to manage it.

Both of the two last approaches seem to involve an awful lot of work to achieve cp -a $source $dest. Unless puppet has primitives to create an archive or debian package from $source - I'm thinking not...

So: Is it really not possible to cp -a $source $dest using puppet?

I'm trying to get davfs2 to mount a WebDAV folder automatically on debian stretch at boot.

It seems that systemd attempts to mount the folder with mount.davfs before the DHCP client has an IP address (which of course fails), even with the _netdev mount option in /etc/fstab. (Without _netdev, mounting also fails but so does the entire boot process and I'm dropped into a root shell during boot.)

What magic can I give davfs so that it only tries to mount after we've gotten an IP address with DHCP? (And why isn't this the default...)

As can be seen from the journalctl output below, davfs2 attempts to mount before the DHCPACK and NFS mounts successfully after the DHCPACK. So NFS behaves, but davfs2 fails.

> grep davfs /etc/fstab
https://example.com/remote.php/webdav/ /home/peter/pmorch davfs user,rw,auto,_netdev 0 0

> sudo journalctl --since 13:20:48 | grep -P 'davfs|pmorch|DHCP|ISO'
Jan 18 13:20:49 snowden dhclient[669]: Internet Systems Consortium DHCP Client 4.3.5
Jan 18 13:20:49 snowden sh[657]: Internet Systems Consortium DHCP Client 4.3.5
Jan 18 13:20:49 snowden sh[657]: DHCPDISCOVER on ens18 to 255.255.255.255 port 67 interval 3
Jan 18 13:20:49 snowden dhclient[669]: DHCPDISCOVER on ens18 to 255.255.255.255 port 67 interval 3
Jan 18 13:20:49 snowden systemd[1]: Mounting /mnt/ISO-Dev...
Jan 18 13:20:49 snowden systemd[1]: Mounting /home/peter/pmorch...
Jan 18 13:20:49 snowden systemd[1]: Mounting /mnt/ISO...
Jan 18 13:20:49 snowden mount.davfs[751]: davfs2 1.5.4
Jan 18 13:20:49 snowden systemd[1]: home-peter-pmorch.mount: Mount process exited, code=exited status=1
Jan 18 13:20:49 snowden systemd[1]: Failed to mount /home/peter/pmorch.
Jan 18 13:20:49 snowden systemd[1]: home-peter-pmorch.mount: Unit entered failed state.
Jan 18 13:20:50 snowden dhclient[669]: DHCPREQUEST of 172.22.216.172 on ens18 to 255.255.255.255 port 67
Jan 18 13:20:50 snowden sh[657]: DHCPREQUEST of 172.22.216.172 on ens18 to 255.255.255.255 port 67
Jan 18 13:20:50 snowden sh[657]: DHCPOFFER of 172.22.216.172 from 172.22.216.251
Jan 18 13:20:50 snowden dhclient[669]: DHCPOFFER of 172.22.216.172 from 172.22.216.251
Jan 18 13:20:50 snowden dhclient[669]: DHCPACK of 172.22.216.172 from 172.22.216.251
Jan 18 13:20:50 snowden sh[657]: DHCPACK of 172.22.216.172 from 172.22.216.251
Jan 18 13:20:52 snowden systemd[1]: Mounted /mnt/ISO-Dev.
Jan 18 13:20:52 snowden systemd[1]: Mounted /mnt/ISO.

I'm trying to build the newest snmp packages on debian jessie.

However, https://packages.debian.org/jessie/snmp seems to show conflicting information. The page heading reads:

Package: snmp (5.7.2.1+dfsg-1+deb8u1 and others) [security]

And the link to the source package, net-snmp_5.7.2.1+dfsg-1+deb8u1.dsc has the exact same version string.

But towards the bottom, under "Download snmp", one can download version 5.7.2.1+dfsg-1+deb8u1+b1. Where does the +b1-suffix in the downloadable version come from?

http://security.debian.org/dists/jessie/updates/main/binary-amd64/Packages.gz

seems to agree that the downloadable version contains +b1 but the source package version doesn't:

> curl http://security.debian.org/dists/jessie/updates/main/binary-amd64/Packages.gz 2>/dev/null | zgrep -A 2 '^Package: snmp$' 
Package: snmp
Source: net-snmp (5.7.2.1+dfsg-1+deb8u1)
Version: 5.7.2.1+dfsg-1+deb8u1+b1

When I try to build the packages from source with dget http://security.debian.org/debian-security/pool/updates/main/n/net-snmp/net-snmp_5.7.2.1+dfsg-1+deb8u1.dsc and cowbuilder I get packages built with file names like this: snmp_5.7.2.1+dfsg-1+deb8u1_amd64.deb (without the +b1 in the version), just as I would expect. I've grepped fro b1 in the unpacked net-snmp-5.7.2.1+dfsg folder generated by dget and don't find anything that should append +b1 to the package name.

I've also tried to clone git://anonscm.debian.org/pkg-net-snmp/pkg-net-snmp.git which is mentioned by net-snmp-5.7.2.1+dfsg/debian/control but there are no tags for either version. The most relevant tag seems to be debian/5.7.2.1+dfsg-1 without even the +deb8u1. Never mind.

So how do I build the 5.7.2.1+dfsg-1+deb8u1+b1 version of the snmp packages that apt wants to install? Do I have the right version of the source package? If not, where should I get it?

Often, an installation of our on-site, debian-stable based application runs in a virtual machine - typically in VMware ESXi. In the general case we do not have visibility into or influence over their virtualization environment and do not have access to e.g. the VMware vCenter client or equivalent. I focus on VMware here, because that by far is the most common we see.

We'd like to:

• Tell a customer's VMware admin: You can run our application in e.g. your VMware ESX environment, as long as it meets performance criteria X, Y and Z.
• Be able to determine if criteria X, Y and Z are in fact met continuously (e.g. also right now), even on a running system (we cannot stop our application and run benchmarks, and an initial benchmark won't suffice, since performance in virtual environments changes over time).
• Have confidence that if criteria X, Y and Z are met, we will have adequate virtual HW resources to run our application with satisfactory performance.

Now what are X, Y and Z?

We have seen time and again, that when there are performance problems, the problem isn't with our application, but with the virtualization environment. E.g. another virtual machine uses tons of CPU, memory or the SAN on which the disks are actually stored get heavy use by something other than our application. We currently have no way to prove or disprove that.

Theoretically it could also be possible that sometimes our application is slow... ;-)

How does one determine the root cause of our performance problems: Virtual environment or our application?

There are typically 3 areas for performance problems CPU, Memory and DISK I/O.

CPU

In e.g. VMware the administrator can specify Reservation and Limit, expressed in MHz, but is e.g. 512MHz on one ESX host exactly the same as 512MHz on another ESX host, possibly in a completely different ESX cluster?

And how does one measure whether we actually get that? While our application is running, we can perhaps see that we are at 212% CPU utilization on 4 CPUs. Is that because our application is doing a lot or because another VM on the same host is running a CPU intensive task and using all the CPU?

Memory (Ballooning?)

If we ask for e.g. 16GB RAM, that is often configured, but because of ballooning, we actually only get 4GB, and surprise, our application performs poorly.

One can ask the VMware tools about the current ballooning, but we've find that it often lies (or is inaccurate at least). We've seen examples where the OS thinks there is 16GB total RAM, the sum of the resident memory (RSS) of all processes is 4GB RAM, but there is only 2GB RAM free, even when VMware tools tells us there is 0 ballooning :-(

Also, just adding RSS together isn't valid, as there could easily be shared RAM, e.g. copy-on-write memory so 512MB + 512MB doesn't necessarily mean 1GB but could mean something less. So one can't simply subtract RSS from all processes to get a measure for how much RAM should be free and thereby detect ballooning reliably. One can detect some cases of ballooning, but there are other cases where ballooning is in effect, but not detectable by this method.

Disk I/O

I guess we could graph over time the number of disk reads and writes, the number of bytes read and written, and the IO wait %. But will that give us an accurate picture of disk I/O? I imagine that if there is a bitcoin miner running in another VM using all the CPU, our IO wait % will go up, even if the underlying SAN gives exactly the same performance, simply because our CPU resources go down, and hence IO wait (which is measured in %) goes up.

So in summary, what language can we use to describe to e.g. a VMware admin, what performance we need, in a portable and measurable way?

We have an SNMP client talking to an SNMP agent through NAT. That works most of the time. We send an SNMP request e.g. from 1.2.3.4:12345 to 10.0.0.1:161 and get a reply from 10.0.0.1:161 to 1.2.3.4:12345.

However, some agents have multiple IP addresses, and for these when for such a server an SNMP request from 1.2.3.4:12345 to 10.0.0.1:161 we may get a reply from a totally different IP address, e.g. from 10.0.1.1:161 to 1.2.3.4:161. I'm presuming this is because SNMP uses UDP and therefore doesn't have a connect, and the agent's default gateway uses 10.0.1.1:161.

This works fine on a "real" directly-connected client without NAT, but when NAT is involved the client's response is not sent through NAT to the client.

We're using standard iptables NAT.

• Is there any way to get iptables to only look at port numbers for UDP NAT so it'll send these replies from other IP addresses through?
• Are there better NAT implementations for our purposes that we haven't tried?

Changing the IP address we ask is unfortunately not an option...

I'm trying to setup a simple ppp tunnel over ssh. It works on several machines just fine. But on one machine, pppd gets "stuck":

> pgrep pppd | xargs ps up
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      4178  0.0  0.1   3020  1088 pts/1    Ds+  05:28   0:00 /usr/sbin/pppd

Any attempt to kill it (even sudo kill -9 4178) has no effect that I can see. strace -p 4178 also hangs similarly. After it has been started for a while, I start getting messages in dmesg like shown below.

It is started like so from another machine:

ssh -t root@server /usr/sbin/pppd passive noauth

When I do this to one of the machines that work, the remote end's pppd spits out garbage/binary data to the console (as expected). When I do it to the one that fails, I get no output from pppd, but the ssh session eventually times out. If I instead ssh to the machine, and then run /usr/sbin/pppd passive noauth in a separate step I also get the expected binary output.

I now have a couple of questions:

• What could be up with the one machine where pppd fails? I don't even know where to start looking...
• What could be the difference between ssh -t root@server /usr/sbin/pppd passive noauth in a single step and ssh root@server and /usr/sbin/pppd passive noauth in two steps?
• How can it be that I can't kill the process even with sudo kill -9? The only way I know is to reboot.

(I've tried searching for something similar but didn't get anywhere so I'm sorry I don't have any more leads)

Any ideas?

The problem machine runs in debian on VMware "hardware" (as do the ones that work) and it exhibits the problem when cloned and on both debian lenny (original) and squeeze (after upgrade)

dmesg entries:

[ 1198.727248] INFO: task pppd:4178 blocked for more than 120 seconds.
[ 1198.727507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1198.727904] pppd            D ece2dc9c     0  4178   4174 0x00000004
[ 1198.727908]  00000098 00000082 f2503520 ece2dc9c 0000b1e7 00000000 c148d1c0 c148d1c0
[ 1198.727913]  f2a06100 f6e071c0 00000000 ece2dc18 f5cd07e0 00000000 ece2d400 ece2dc9d
[ 1198.727918]  00c52300 ece2dcbc f67bfef8 ec98e480 f291cec0 00000000 c10cf5b0 c10dfd21
[ 1198.727923] Call Trace:
[ 1198.727926]  [<c10cf5b0>] ? nameidata_to_filp+0x37/0x41
[ 1198.727929]  [<c10dfd21>] ? dput+0x21/0xb7
[ 1198.727932]  [<c11cfecc>] ? tty_ldisc_ref_wait+0x5f/0x76
[ 1198.727935]  [<c104de7a>] ? wake_up_bit+0x5c/0x5c
[ 1198.727938]  [<c11cb91b>] ? tty_ioctl+0x85f/0x8ba
[ 1198.727941]  [<c10fec18>] ? do_lock_file_wait+0x3d/0xd9
[ 1198.727944]  [<c1162c97>] ? _copy_from_user+0x2b/0x102
[ 1198.727946]  [<c11cb0bc>] ? tty_check_change+0xb9/0xb9
[ 1198.727949]  [<c10dbeb7>] ? do_vfs_ioctl+0x485/0x4c7
[ 1198.727952]  [<c10db59a>] ? do_fcntl+0x24f/0x3a2
[ 1198.727954]  [<c10dbf3a>] ? sys_ioctl+0x41/0x58
[ 1198.727957]  [<c12c6a1f>] ? sysenter_do_call+0x12/0x28
[ 1318.457225] INFO: task sshd:4174 blocked for more than 120 seconds.
[ 1318.457500] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1318.457896] sshd            D f25024cc     0  4174   2393 0x00000000
[ 1318.457901]  00000098 00000086 f2a06940 f25024cc 0000b246 00000000 c148d1c0 c148d1c0
[ 1318.457906]  f2503520 f6e071c0 00000000 3f056585 0000000f ece2d4bc 3f056585 f2503520
[ 1318.457911]  ec98bb38 ec98bbdc 00000000 00000000 00000000 c12c09b5 f2503520 c10327cb
[ 1318.457916] Call Trace:
[ 1318.457926]  [<c12c09b5>] ? schedule_hrtimeout_range_clock+0x3c/0xd9
[ 1318.457931]  [<c10327cb>] ? try_to_wake_up+0x13f/0x13f
[ 1318.457935]  [<c11cfecc>] ? tty_ldisc_ref_wait+0x5f/0x76
[ 1318.457940]  [<c104de7a>] ? wake_up_bit+0x5c/0x5c
[ 1318.457943]  [<c11c9ad3>] ? tty_poll+0x32/0x5e
[ 1318.457947]  [<c10dd4d5>] ? do_select+0x2a1/0x42e
[ 1318.457950]  [<c10dcb83>] ? poll_freewait+0x69/0x69
[ 1318.457953]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457955]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457958]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457960]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457963]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457965]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457968]  [<c10dcc25>] ? __pollwait+0xa2/0xa2
[ 1318.457971]  [<c10429c2>] ? lock_timer_base+0x19/0x35
[ 1318.457974]  [<c1042eb5>] ? __mod_timer+0x10c/0x116
[ 1318.457977]  [<c1042f89>] ? mod_timer+0x69/0x6e
[ 1318.457981]  [<c121325d>] ? sk_reset_timer+0xc/0x16
[ 1318.457984]  [<c1252f57>] ? tcp_event_new_data_sent+0x66/0x6b
[ 1318.457987]  [<c1255b85>] ? tcp_write_xmit+0x7a7/0x86a
[ 1318.457990]  [<c121760d>] ? __alloc_skb+0x50/0xfd
[ 1318.457994]  [<c12c12bc>] ? _raw_spin_lock_bh+0x8/0x1e
[ 1318.457996]  [<c1212e98>] ? release_sock+0x10/0xc4
[ 1318.457999]  [<c124b543>] ? tcp_sendmsg+0x6dd/0x7b7
[ 1318.458003]  [<c1162c97>] ? _copy_from_user+0x2b/0x102
[ 1318.458006]  [<c10dd7a0>] ? core_sys_select+0x13e/0x1c3
[ 1318.458009]  [<c12102a3>] ? sock_aio_write+0xc0/0xd4
[ 1318.458012]  [<c10d0655>] ? do_sync_write+0xa0/0xe4
[ 1318.458016]  [<c10b141c>] ? handle_mm_fault+0x222/0x238
[ 1318.458019]  [<c10f6096>] ? fsnotify+0x1de/0x1f9
[ 1318.458022]  [<c10dd9e8>] ? sys_select+0x6e/0x8f
[ 1318.458024]  [<c10d105e>] ? sys_write+0x3c/0x63
[ 1318.458028]  [<c12c6a1f>] ? sysenter_do_call+0x12/0x28

I telnet to an HP switch from my (xterm|gnome-terminal). Exit telnet. That breaks the behaviour of the terminal. (If you don't have a HP switch to telnet to, one can cat this file in a terminal instead).

Now type a very long command (more than one line, and see that the second line doesn't wrap down to the next line, but "stays in the current line". Which makes it impossible to edit prior commands or use the terminal for anything longer than one line.

See that cat filewithlonglines.txt or git log -p only shows the first 80 chars of lines longer than 80 chars.

reset doesn't work. stty -g > stty.works, run telnet, stty $(stty.works) restores changed stty settings, but behaviour is still broken.

I have to close the terminal and start a new one. :-(

What can I do besides reset and stty save/restore to fix my broken terminals after a telnet?

EDIT: I've discovered that if I use screen like in one of these commands:

screen bash -c "cat hptelnet.escape ; bash"

or

screen telnet hpswitch

Then my terminal still works properly when screen exits. So a workaround is to put screen in front of the command that messes up my terminal, but I'd still like to know how to recover a terminal that already has been messed up...

How does one display/visualize a trace of a session with ANSI Escape codes embedded in it?

I have a tshark trace of a telnet session to a HP switch. From that, I Follow TCP Stream and get a pseudo-text representation of the telnet session (with embedded escape chars).

less hptelnet.escape

can be used to display it, but that breaks down quickly as the file gets just slightly bigger.

I've concocted a quick script that makes it sort of readable but it is far from finished. The ANSI escape sequences are just not simple.

Does anybody have any better ideas as to how one can view the "contents" of such a telnet trace?