Nils's questions

I have been updating some servers to the latest patchlevels and wanted to check the security state regarding Meltdown, Spectre and Spectre-NG.

Using the current V 0.39+ of the spectre-meltdown-checker I was astonished about my findings in a VM running CentOS 7 with VMWare ESXi 6.5 (all patched up to date).

Here I found "vulnerable for Spectre 2".

Strange. That was patched months ago by using retpoline.

OK - the new check states that retpoline is not enough. But CentOS 7 is lacking RSB. So the only other remaining option is to enable IBRS.

Strangely it is not activated automatically at boot-time.

Since I do not want to write a init-script

echo 1 >/sys/kernel/debug/x86/ibrs_enabled

Which will turn on IBRS an will get rid of Spectre v2 (according to the test-results of the script).

The question is: What is the best way to enable IBRS permanently on boot?

I was not able to find an according kernel-option.

In this article there is a nice recipe for how to use a RAM-disk as cache-device for a classical LVM volume.

Assumed you have an elder disk, lots of RAM and no SSD, you can boost disk performance to native RAM-throughput using this technique.

So I did this on an LVM which is used for my virtual VM running Windows 10. Voila: Disk-throughput was 4 times faster within the VM (average throughput, best used while patching Windows).

All was well - until I shutdown my linux-system (CentOS 7).

Data Loss!

The shutdown will not disassemble that cache. The same would be true in a power-failure-situation (yes - there will be data loss).

However - there has to be a way to recover what is left. But LVM will not let you operate on a VG with missing disks.

So - is there a receipe for this case out there?

Like

• recover missing LVM cache disk with a new disk
• force clean state
• access cached LV again

In the last step one would make filesystem repairs and would recover missing/corrupted files from backup (using rsync).

With CentOS/RHEL 7 there are a couple of changes (compared to CO/RHEL 6). One of it is the use of grub2 instead of grub.

Per default the OS seems to use a UUID to "find" the boot-device.

Is there an easy to use receipe to get back to device-names (like /dev/sda1) instead?

Background of the question: I am intending to clone additional VMs from a template. Base is a new (virtual) disk device with a different UUID.

If I can not revert to sda1 I will need to change the UUID of the clone in the grub.cfg to the new UUID - which is plan "B".

Update 2017-10-26

The kernel-parameter for root= will be changed to the disk - see the answer from Thomas below.

There remains a problem with this section, generated by grub2-mkconfig:

    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  716433ab-9e30-42a7-a272-6c66243499d2
    else
      search --no-floppy --fs-uuid --set=root 716433ab-9e30-42a7-a272-6c66243499d2
    fi

This still contains the search for the UUID. If it can't be found, the boot-process will go to error "not found" or something like that. After pressing ENTER the system will boot up ok.

The remaining Q is how to inactivate that section (I did not find the place to disable the feature_platform_search_hint)?

I am currently in the process of migrating from SLES11 SP3 as Dom0 - running the "old" xend/xm-stack to CentOS6 XEN4 running the "new" xl-stack.

I stumbled across a strange problem: After pygrub starts up I can see the correct boot line. But: then I get this:

xenconsole: Could not open tty `': No such file or directory
  libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console child [0] exited with error status 2

That message is repeated another time, leaving me on the console of the Dom0 with a DomU in the state "running" doing nothing. I have to kill the DomU with "xl destroy" to get back into a defined state.

On the web I found nothing useful on the topic. The CentOS6 XEN4 Wiki does only seem to cover xend/xm.

When doing the same procedure with a PV CentOS 5 DomU the DomU starts up without modification.

Since more than five years I am using XEN in a productive environment.

My constellation: SLES11 (included: OpenSource XEN 4.x), DRBD 8.3 in primary/secondary mode.

Now I would have loved to go away from SLES to RedHat or CentOS, but sadly there was no xen-Dom0-support in the major-release 6 any more.

This has changed with a CentOS6-branch that developed an alternative repository (XEN4) with a version 3 kernel and XEN 4.x. Now the kernel-module included with that kernel is a drbd-module 8.4 (no extra rpm needed!).

Many of my scripts work with DRBD 8.3 and need to be re-written for 8.4. since the CLI has changed a lot between these versions.

In this constellation the drbd-user-space rpm (drbd-utils) comes from elrepo in flavours for 8.3 and 8.4.

My idea: Use the drbd83-utils (so I still have the old CLI syntax) with the 8.4 kernel module. Are there any major drawbacks to be expected?

OR: Alternative: Replace 8.4 with 8.3 by using an older kmod-drbd83?

iSCSI with two DRBD primary nodes is a bad idea to use if the two paths get concurrent write requests. But I am thinking about using this idea as backend storage for an ESXi 5.5U2 host.

I already did test this with primary/secondary configurations and a classical failover-cluster.

What ESXi does at this point is that it detects a multipath und uses only one path actively. So in this constellation the concurrent write io-problem does not seem to arise.

Now the problem in both cases (primary/secondary or primary/primary) is: How do I shutdown an iSCSI server (iSCSI target provider in iSCSI terms) that has active open connections to an iSCSI client (iSCSI initiator in iSCSI terms)?

I currently use CentOS 5 on the target servers.

CO5 uses tgtd to provide the targets. To my astonishment the normal stop method fails, if there are connected clients. Instead the forcedstop seems to be what I need in this case.

I want to shutdown one server cleanly (I have to stop access to the target, so I can switch drbd to secondary) and the other server should then automatically become active (nothing to do there in this constellation IMHO).

Questions in that context: Is the following ok, or am I missing something?

1. forced stop of tgtd (will first offline the targets)
2. tear down IP into the direction of the initiator (different line than that used for drbd-replication)
3. shutdown drbd (making it secondary first)
4. reboot or shutdown server

For I new setup I bought a Essentials Plus package for ESXi 5.5U2.

This contains (according to the activation details):

• Three ESXi licenses for 2-socket-servers
• One vCenter license

My two-socket-servers are not ready to deploy yet, so I started building a test-environment with an old 4-socket-machine using the evaluation license (i.e. no licence at all).

I want to keep that test environmen up after acivation of my two production servers are active, too.

Now my question: What will happen if I install my 2-socket-licence on my 4-socket-server?

1. ESXi will refuse to install that key
2. ESXi will swith to RO mode
3. ESXi will stop working
4. ESXi will halv the number of processors available
5. something else?

I could live with 4. - in cases 1 to 3 I will have to pull out two of my processors and will have to pyhsically re-distribute my RAM to the remaining two processors.

In this case I do not think this Q is off-topic. I just need a real-world answer and think others might have the same "problem" if using VMware in a professional way.

I suspect that one of our server applications has hit its max open file limit.

The application is running in user-space with its own account. The init-script starts a large number of processes which in turn start a number of sub-processes and a large number of threads.

According to the book I set in /etc/security/limits.conf:

USERNAME - nofile 2048

I suspect that the application has hit the limit - by looking at the temporary file directory I found more than 2000 files there.

After raising the limit to 4096 and doing an application restart I found more than 2100 files there.

Now the question: If the application hit the limit 2048 - why was this not logged in /var/log/messages?

syslog-ng is the current syslog-daemon in use.

/etc/syslog-ng/syslog-ng.conf

options { long_hostnames(off); sync(0); perm(0640); stats(3600); };
source src {
    internal();
    unix-dgram("/dev/log");
    unix-dgram("/var/lib/ntp/dev/log");
};
filter f_iptables   { facility(kern) and match("IN=") and match("OUT="); };
filter f_console    { level(warn) and facility(kern) and not filter(f_iptables)
                      or level(err) and not facility(authpriv); };
filter f_newsnotice { level(notice) and facility(news); };
filter f_newscrit   { level(crit)   and facility(news); };
filter f_newserr    { level(err)    and facility(news); };
filter f_news       { facility(news); };
filter f_mailinfo   { level(info)      and facility(mail); };
filter f_mailwarn   { level(warn)      and facility(mail); };
filter f_mailerr    { level(err, crit) and facility(mail); };
filter f_mail       { facility(mail); };
filter f_cron       { facility(cron); };
filter f_local      { facility(local0, local1, local2, local3,
                           local4, local5, local6, local7); };
filter f_messages   { not facility(news, mail, cron, authpriv, auth) and not filter(f_iptables); };
filter f_warn       { level(warn, err, crit) and not filter(f_iptables); };
filter f_alert      { level(alert); };
filter f_auth       { facility(authpriv, auth); };
destination console  { pipe("/dev/tty10"    group(tty) perm(0620)); };
  log { source(src); filter(f_console); destination(console); };
destination xconsole { pipe("/dev/xconsole" group(tty) perm(0400)); };
  log { source(src); filter(f_console); destination(xconsole); };
destination auth { file("/var/log/auth"); };
  log { source(src); filter(f_auth); destination(auth); };
destination newscrit { file("/var/log/news/news.crit"); };
  log { source(src); filter(f_newscrit); destination(newscrit); };
destination newserr { file("/var/log/news/news.err"); };
  log { source(src); filter(f_newserr); destination(newserr); };
destination newsnotice { file("/var/log/news/news.notice"); };
  log { source(src); filter(f_newsnotice); destination(newserr); };
destination mailinfo { file("/var/log/mail.info"); };
  log { source(src); filter(f_mailinfo); destination(mailinfo); };
destination mailwarn { file("/var/log/mail.warn"); };
   log { source(src); filter(f_mailwarn); destination(mailwarn); };
destination mailerr  { file("/var/log/mail.err" fsync(yes)); };
  log { source(src); filter(f_mailerr);  destination(mailerr); };
destination mail { file("/var/log/mail"); };
  log { source(src); filter(f_mail); destination(mail); };
destination cron { file("/var/log/cron"); };
  log { source(src); filter(f_cron); destination(cron); };
destination localmessages { file("/var/log/localmessages"); };
  log { source(src); filter(f_local); destination(localmessages); };
destination messages { file("/var/log/messages"); };
  log { source(src); filter(f_messages); destination(messages); };
destination firewall { file("/var/log/firewall"); };
  log { source(src); filter(f_iptables); destination(firewall); };
destination warn { file("/var/log/warn" fsync(yes)); };
  log { source(src); filter(f_warn); destination(warn); };

I set up a CentOS6-XEN4-Server according to these instructions.

Now I want to set up bridges - first manually:

brctl addbr test - but this fails with:

"add Bridge failed: Package not installed"

Googling about that I tried to manually modprobe bridge which triggers an error that is caused-according to dmesg by:

bridge: Unknown symbol ipv6_dev_get_saddr (err 0)

modinfo bridge reveals that the bridge-modules depends on stp,llc and ipv6.

Great. I do not want IPv6, I do not need IPv6 and have blacklisted IPv6.

1. Is there any way to get brctl running without enabling IPv6?

2. Why is that depency there at all?

Update 2014-05-14

I found this to be a general CentOS 6/RH 6 problem. It also applies to the bonding module.

The accepted answer describes the root-cause - so this problem will also arise on comparable kernel-versions and is not restricted to CentOS/RedHat/Scentific Linux 6.

I wanted to attach a comment to a system that is being monitored with Nagios. I prefer using check_mk as GUI. Now I stumbled across this: I can set a comment as sticky and/or persistent.

So I asked our Nagios-admin what the difference between sticky and persistent would be.

It turned out, that he did not know about "sticky" - this has to be something check_mk-specific.

After a Google and a review of check_mk documents I could not find anything about that topic.

So: What is the difference between sticky and persistent for Nagios-service-comments?

Update: Here is a screenshot - check_mk quicksearch for a specific server, then select the hamer-symbol. Then this will show up:

The question is about the Acknowledge-Box: sticky vs. persistent

I am currently in the progress of migrating our kickstart-files from CentOS 5 to CentOS 6.

In CentOS 5 there was a nice anaconda/kickstart-statement that allowed the inclusion of other kickstart-statements via http. Example:

%ksappend http://myinstallserver.intranet.domain/ks/fslayout_phys.include

When I try the same statement in CentOS 6.4 I get an error during installation:

The following problem occured on line 19 of the kickstart file:

Unable to open %%ksappend file

Does anyone out there know what the issue is there?

The official RH-documentation does not say anything about ksappend any more.

When virtualisation was new, we tried to virtualise everything, but then we noticed use cases where the our virtual machines were much slower than a bare metal.

For us, we use the following rules when deciding not to virtualise:

• Network IO intensive applications (i.e. with many interrupts/packets)
• Disk IO intensive applications (if not on SAN storage)
• RAM intensive (this is the most precious resource)

We have had these experiences with Xen and DRDB as well has Hyper-V's shared-nothing with DAS. Is this the case with all hypervisors?

What (other) metrics should I look for when deciding to virtualise an application/server or not?

I am about to develop a little script to gather information for a chroot-jail.

In my case this looks (at the first glance) pretty simple: The application has a clean rpm-install and did install almost all files into a sub-directory of /opt.

My idea is:

• Do a find of all binaries
• Check their library-dependencies
• Record the results into a list
• Do a rsync of that list into the chroot-target-directory before startup of the application

Now I wonder - ist there any script around that already does such a job (perl/bash/python)?

So far I found only specialized solutions for single applications (like sftp-chroot).

Although is does not matter (imho) - OS is CentOS 5 x86_64 current minor release and patch-level.

rpm -ql is IMHO not generic enough, since it will only cover rpm-based distributions. The mention of the "clean install" above was just to mention that the files of the software are not distributed across the whole file-system. So my starting point is - at the moment - a find /opt/directory/... that should work on almost any system (even not Linux).

We have training rooms where normally Windows XP is installed (via PXE). The "normal" DNS/DHCP infrastructure are Windows-Servers. The training room has its own VLAN (different from the Windows servers), so there is most propably an IP helper for DHCP requests active on the Cisco router where all PCs from that room are connected to.

Now we wanted to convert some of the PCs to Linux instead. The idea was: Put our own Laptop with a DHCP server into the VLAN of the room and override the "normal" DHCP response. The idea was that this should work, since a directly attached DHCP server in that VLAN should have a faster response-time than the "normal" DHCP server located some hops away from that VLAN.

It turned out that this did not work. We had to manually release the lease on the original DHCP server to get it working.

On the Laptop we did see the client requesting the IP and "our" dhcp was sending NACKs to the Windows IP request, before that we did offer our own response.

Old Question: Why did this not work out as expected? What is making the PC regain its old lease?

Update 2012-08-08:

The regain-issue has been explained in the DHCP-RFC. Now this explains why the PC regains its old lease.

Now we do release the IP from the Windows-DHCP-server before giving it another try.

Again - the Windows-DHCP-server wins.

I suspect that there is some algorithm for the dhcp-client which determines the "best" dhcp-answer for the client. The new question is:

How does the client choose the "best" answer?

I have set up a DNS-server on SLES10 (currently bind 9.6) on a multi-homed server. This server can be queried from all internal networks and delivers answers for all internal networks. We have two separate DNS "master" zones. Each of these zones is being served by a number of authoritative Windows-DNS-servers.

Now my linux-server is a secondary DNS server for one of these zones (private internal zone) and acting as forwarder for the other zone (public internal zone).

Until recently this setup worked without problems. Now I get - upon querying the public internal zone (e.g. by the host command on a linux client) the error-message

;; Truncated, retrying in TCP mode

a wireshark-dump revealed the cause of this: The first query goes out in UDP mode, the answer does not fit into UDP (due to the longish list of authoritative NS), then it is retried in TCP mode, delivering the right answer.

Now the question: Can I configure my bind to query the forwarders in TCP mode without trying UDP first?

Update: Trying my hand on ASCII-art...

+--------------+   +--------------+   +-----------------+
| W2K8R2 DNS   |   | SLES 10 DNS  |   | W2K8R2 DNS      |
| Zone private +---+ All internal +---+ Zone public     |
| internal 2x  |   |   Zones      |   | internal 30+ x  |
+--------------+   +-+----------+-+   +-----------------+
                     |          |
                  +--+---+   +--+---+
                  |Client|   |Client|
                  +------+   +------+

I want to set up a simple unencrypted tunnel between a linux-server (running SLES9) and a client (running XP SP3). The reason for this is that there are many network devices in between and the client does build up many many IP connections. I want to test, if the process of building up these connections is the core of massive latency problems with this application. (There is a WAN between server and client).

Now the question: There are a lot of howtos on how to set up GRE tunnels (which seems appropriate here) on linux and cisco.

But how do I set up a GRE-tunnel on the XP-client?

I found that in XEN I can use vmware-disk images. But can I actually create such a disk with pure XEN-means?

I want to create a server with XEN and move the corresponding disk later to VMWARE (in a simple case this is just a data disk).

I currently run XEN that comes with SLES10 SP4.

I directly connected two PowerEdge 6950 crossover (using straight lines) on two different PCIe-adapters.

I get a gigabit link on each of these lines (1000 MBit, full duplex, flow contol in both directions).

Now I am trying to bond these interfaces into bond0 using the rr-algorithm on both sides (I want to get 2000 MBit for a single IP session).

When I tested the throughput by transferring /dev/zero to /dev/null using dd bs=1M and netcat in tcp mode I get a throughput of 70 MB/s - not - as expected more than 150MB/s.

When I use the single lines I get about 98 MB/s on each line, if I used a different direction for each line. When I use the single lines I get 70 MB/s and 90 MB/s on the line, if traffic goes into the "same" direction.

After reading through the bonding-readme (/usr/src/linux/Documentation/networking/bonding.txt) I found the following section to be useful: (13.1.1 MT Bonding Mode Selection for Single Switch Topology)

balance-rr: This mode is the only mode that will permit a single TCP/IP connection to stripe traffic across multiple interfaces. It is therefore the only mode that will allow a single TCP/IP stream to utilize more than one interface's worth of throughput. This comes at a cost, however: the striping often results in peer systems receiving packets out of order, causing TCP/IP's congestion control system to kick in, often by retransmitting segments.

    It is possible to adjust TCP/IP's congestion limits by
    altering the net.ipv4.tcp_reordering sysctl parameter. The
    usual default value is 3, and the maximum useful value is 127.
    For a four interface balance-rr bond, expect that a single
    TCP/IP stream will utilize no more than approximately 2.3
    interface's worth of throughput, even after adjusting
    tcp_reordering.

    Note that this out of order delivery occurs when both the
    sending and receiving systems are utilizing a multiple
    interface bond.  Consider a configuration in which a
    balance-rr bond feeds into a single higher capacity network
    channel (e.g., multiple 100Mb/sec ethernets feeding a single
    gigabit ethernet via an etherchannel capable switch).  In this
    configuration, traffic sent from the multiple 100Mb devices to
    a destination connected to the gigabit device will not see
    packets out of order.  However, traffic sent from the gigabit
    device to the multiple 100Mb devices may or may not see
    traffic out of order, depending upon the balance policy of the
    switch.  Many switches do not support any modes that stripe
    traffic (instead choosing a port based upon IP or MAC level
    addresses); for those devices, traffic flowing from the
    gigabit device to the many 100Mb devices will only utilize one
    interface.

Now I changed that parameter on both connected servers on all lines (4) from 3 to 127.

After bonding again I get about 100 MB/s but still not more than that.

Any ideas why?

Update: Hardware details from lspci -v:

24:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
        Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 24
        Memory at dfe80000 (32-bit, non-prefetchable) [size=128K]
        Memory at dfea0000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at dcc0 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Mask- 64bit+ Count=1/1 Enable-
        Capabilities: [e0] Express Endpoint, MSI 00
        Kernel driver in use: e1000
        Kernel modules: e1000

Update final results:

8589934592 bytes (8.6 GB) copied, 35.8489 seconds, 240 MB/s

I changed a lot of tcp/ip and low-level-driver options. This includes enlargement of the network buffers. This is why dd now shows numbers greater than 200 MB/s: dd terminates while there is still output waiting to be transferred (in send buffers).

Update 2011-08-05: Settings that were changed to achive the goal (/etc/sysctl.conf):

# See http://www-didc.lbl.gov/TCP-tuning/linux.html
# raise TCP max buffer size to 16 MB. default: 131071
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# raise autotuninmg TCP buffer limits
# min, default and max number of bytes to use
# Defaults:
#net.ipv4.tcp_rmem = 4096 87380 174760
#net.ipv4.tcp_wmem = 4096 16384 131072
# Tuning:
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# Default: Backlog 300
net.core.netdev_max_backlog = 2500
#
# Oracle-DB settings:
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
kernel.shmmax = 2147659776
kernel.sem = 1250 256000 100 1024
net.core.rmem_default = 262144
net.core.wmem_default = 262144
#
# Tuning for network-bonding according to bonding.txt:
net.ipv4.tcp_reordering=127

Special settings for the bond-device (SLES: /etc/sysconfig/network/ifcfg-bond0):

MTU='9216'
LINK_OPTIONS='txqueuelen 10000'

Note that setting the biggest possible MTU was the key to the solution.

Tuning of the rx/tx buffers of the involved network cards:

/usr/sbin/ethtool -G eth2 rx 2048 tx 2048
/usr/sbin/ethtool -G eth4 rx 2048 tx 2048

I want to set up a maillist consisting of several e-mail-addresses using an alias in /etc/aliases.

Is there a known limitation here with regards to number of lines/chars/entries per alias?

From what I read in the manpage I can include a file containing the addresses for that alias.

I expect some hundred addresses. If this detail is important: I am using postfix on CentOS 5 x86_64.

For a certain job I sometimes need more memory in a DomU (CentOS 5). The trigger for the start of the job are some conditions that I can only check within that DomU: The DomU is waiting for an upload. Once it is finished it needs to process the received data - best suited would be a ram-disk.

For this scenario I have set up the DomU with a memory setting of 1 GB min and 16 GB max.

Now I am trying to find an elegant way to resize the memory to max/min from within the DomU.

The first solution for this would be using ssh from DomU to all possible Dom0s and then trigger the "xm mem-set" commands there (with sudo).

I've read some things about the xen-store. Triggers were mentioned... So this makes me think that there should be a better way. It turned out that memory balloning would be a better way.

This should work within the DomU:

echo $((4096*1024*1024)) >/proc/xen/balloon

Should resize the memory to 4 GB.

But: cat /proc/xen/ballon says:

Current allocation:  2165536 kB
Requested target:    4192256 kB
Low-mem balloon:    14611680 kB
High-mem balloon:          0 kB
Driver pages:              0 kB
Xen hard limit:      2165536 kB

Now where does that xen hard limit come from?

Result is now that my DomU has 2 GB RAM. xm list in Dom0 states that the DomU is still at 1 GB RAM...

What's going wrong here? And yes - since the DomU-version is below 3.0.4 I added "mem=16G" as kernel-boot-parameter to my DomU.

With Dom0 SLES11 SP1 (XEN 4.0.1) and CentOS 5.6 DomU (still XEN 3.0.3?) the echo 4G >/proc/xen/ballon did not do anything first - but after I did some successful xm mem-sets from the Dom0 (up to 16 GB - which worked), the /proc/xen/ballon did work within the DomU, too.