pgoetz's questions

For both machines:

• Ubuntu 20.04
• Mellanox MT28908 ConnectX-6 cards in ethernet mode
• Kernel module: mlx5_core (from MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu20.04-x86_64)

The ConnectX-6 cards are directly connected to each other using an Nvidia DAC using RFC1918 addresses; i.e. a little 2-node network. The machines are server grade rackmount systems, with one having an Intel Xeon Gold 5318Y CPU, and the other 2 x Intel Xeon Platinum 8358 CPUs. These cards are supposed to provide 100Gb/s connectivity. However

root@zakynthos:~# iperf3 -c 192.168.101.1
Connecting to host 192.168.101.1, port 5201
[  5] local 192.168.101.2 port 45974 connected to 192.168.101.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.17 GBytes  27.2 Gbits/sec  227    912 KBytes       
[  5]   1.00-2.00   sec  2.58 GBytes  22.1 Gbits/sec  1382    755 KBytes       
[  5]   2.00-3.00   sec  2.36 GBytes  20.3 Gbits/sec  2459   1.08 MBytes       
[  5]   3.00-4.00   sec  3.07 GBytes  26.4 Gbits/sec  3433    723 KBytes       
[  5]   4.00-5.00   sec  3.36 GBytes  28.9 Gbits/sec  3004   1.05 MBytes       
[  5]   5.00-6.00   sec  2.83 GBytes  24.3 Gbits/sec  1692    942 KBytes       
[  5]   6.00-7.00   sec  2.47 GBytes  21.2 Gbits/sec  1819   1022 KBytes       
[  5]   7.00-8.00   sec  3.03 GBytes  26.0 Gbits/sec  1578    841 KBytes       
[  5]   8.00-9.00   sec  3.37 GBytes  28.9 Gbits/sec  1920   1024 KBytes       
[  5]   9.00-10.00  sec  2.91 GBytes  25.0 Gbits/sec  1239   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  29.1 GBytes  25.0 Gbits/sec  18753             sender
[  5]   0.00-10.04  sec  29.1 GBytes  24.9 Gbits/sec                  receiver

I'm used to seeing iperf3 deliver around 90% of the rated bandwidth for a card, so to me this looks terrible and unacceptable. Anyone have any insights as to why I'm not seeing at least 80-90Gb/s performance here?

We've been experimenting with integrating linux systems into what is largely a Windows domain, with a Windows-based Active Directory server. After considerable trial and error, we have successfully configured sssd (on RHEL, CentOS, and Ubuntu) to allow AD user logins bound by AD security groups. We solved the problem of allowing AD logins long ago; it was a bit trickier to restrict access to only particular AD domain users, as was necessary in our use case. The last hurdle involves the experiment of trying to "automount" home directories from a Windows storage server, with something like pam mkhomedir creating these home directories if they don't exist already. I say experiment because we can always fall back to using a linux storage server. The optimal system would have users with some level of linux home directory Windows user profile unification (similar to what Samba gives you with NT domain profiles), but at the moment we can't get this working for linux clients at all. Using a command like this

mount.cifs //cnsdisk/Home/pgoetz /home/pgoetz -o username=pgoetz,sec=krb5,vers=3.0,uid=pgoetz,cruid=pgoetz

Will work for exactly one user that already has a tgt kerberos ticket to mount a windows share, but a second user is then locked out. My thought is if we can't even get manual mounts to work, there's no hope of constructing an automated solution.

I'm not expecting anyone to be able to answer this (a RHEL knowledge base article professed to solve this; didn't work for us); just frustrated that these kinds of problems haven't already been thoroughly worked out and documented.

Edit: I'm pretty sure the multiuser option provided above is obsolete, but RHEL tech support included it in their solution, so I left it in. Don't think that helps, though.

I have an Arch linux ZFS backup system which wasn't updated for a long time because it was working perfectly and there was no reason to bother. After a recent upgrade, though, my ZFS datasets are no longer automatically mounted when I reboot. Unfortunately, the services don't indicate any error messages, and I can run

# zfs mount -a

after the system is booted with no problem. Some particulars:

[root@elephant etc]# uname -a
Linux elephant 4.16.11-1-ARCH #1 SMP PREEMPT Tue May 22 21:40:27 UTC 2018 x86_64 GNU/Linux
[root@elephant etc]# pacman -Q | grep spl
spl-dkms 0.7.9-1
spl-utils 0.7.9-1
[root@elephant etc]# pacman -Q | grep zfs
zfs-dkms 0.7.9-1
zfs-utils 0.7.9-1

root@elephant etc]# zfs get mountpoint backup/www
NAME                              PROPERTY    VALUE                              SOURCE
backup/www  mountpoint  /backup/www  default
[root@elephant etc]# zfs get mountpoint backup/data
NAME                               PROPERTY    VALUE                               SOURCE
backup/data  mountpoint  /backup/data  default
[root@elephant etc]# zfs get mountpoint backup/metadata
NAME             PROPERTY    VALUE             SOURCE
backup/metadata  mountpoint  /backup/metadata  default

Notice in particular that the zfs-mount service seems to be perfectly happy:

[root@elephant ~]# systemctl -l status zfs*
● zfs.target - ZFS startup target
   Loaded: loaded (/usr/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Mon 2018-05-28 15:30:18 CDT; 1min 32s ago

May 28 15:30:18 elephant systemd[1]: Reached target ZFS startup target.

● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2018-05-28 15:30:18 CDT; 1min 32s ago
  Process: 659 ExecStart=/usr/bin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
  Process: 656 ExecStartPre=/sbin/modprobe zfs (code=exited, status=0/SUCCESS)
 Main PID: 659 (code=exited, status=0/SUCCESS)

May 28 15:30:12 elephant systemd[1]: Starting Import ZFS pools by cache file...
May 28 15:30:18 elephant systemd[1]: Started Import ZFS pools by cache file.

● zfs-zed.service - ZFS Event Daemon (zed)
   Loaded: loaded (/usr/lib/systemd/system/zfs-zed.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-05-28 15:30:18 CDT; 1min 32s ago
     Docs: man:zed(8)
 Main PID: 1586 (zed)
    Tasks: 3 (limit: 4915)
   Memory: 5.6M
   CGroup: /system.slice/zfs-zed.service
           └─1586 /usr/bin/zed -F

May 28 15:30:18 elephant systemd[1]: Started ZFS Event Daemon (zed).
May 28 15:30:18 elephant zed[1586]: ZFS Event Daemon 0.7.9-1 (PID 1586)
May 28 15:30:18 elephant zed[1586]: Processing events since eid=0
May 28 15:30:18 elephant zed[1591]: eid=1 class=history_event pool_guid=0x7314E37F1A1C0088
May 28 15:30:18 elephant zed[1593]: eid=2 class=config_sync pool_guid=0x7314E37F1A1C0088
May 28 15:30:18 elephant zed[1595]: eid=3 class=pool_import pool_guid=0x7314E37F1A1C0088
May 28 15:30:18 elephant zed[1618]: eid=5 class=config_sync pool_guid=0x7314E37F1A1C0088

● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/usr/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2018-05-28 15:30:12 CDT; 1min 38s ago
  Process: 657 ExecStart=/usr/bin/zfs mount -a (code=exited, status=0/SUCCESS)
 Main PID: 657 (code=exited, status=0/SUCCESS)

May 28 15:30:12 elephant systemd[1]: Starting Mount ZFS filesystems...
May 28 15:30:12 elephant systemd[1]: Started Mount ZFS filesystems.
[root@elephant ~]#

However:

[root@elephant ~]# zfs mount
[root@elephant ~]# 
[root@elephant ~]# zfs mount -a
[root@elephant ~]# zfs mount
backup                          /backup
backup/data  /backup/data
backup/metadata                 /backup/metadata
backup/www  /backup/www

I've run out of time available to deal with this and am just going to switch all the datasets over to legacy mount, but I'm still curious why this worked for zfs 0.6.x but isn't working for 0.7.9 -- possibly some kind of systemd incompatibility?

This problem is basically driving me insane, at this point. I have an Ubuntu 16.04 NFS server that was working fine with this configuration:

/etc/fstab:
UUID=b6bd34a3-f5af-4463-a515-be0b0b583f98  /data2  xfs  rw,relatime  0  0
/data2  /srv/nfs/cryodata    none    defaults,bind    0  0
/usr/local       /srv/nfs/local    none    defaults,bind    0  0

and

/etc/exports
/srv/nfs  192.168.159.31(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryodata  192.168.159.31(rw,sync,no_subtree_check)
/srv/nfs/local      192.168.159.31(rw,sync,no_subtree_check)

This has all been working fine for months on the one nfs client using this configuration so far using these client side /etc/fstab entries:

kraken.bio.univ.edu:/local  /usr/local  nfs4  _netdev,auto  0  0
kraken.bio.univ.edu:/cryodata  /cryodata  nfs4  _netdev,auto  0  0

However, since this is a very large storage server, it was decided that it needs to accommodate several labs. So, I moved all the stuff that had been scattered across the /data2 partition into a /data2/cryodata subdirectory, and updated /etc/fstab on the server and /etc/exports as follows:

/etc/fstab:
...
/data2/cryodata  /srv/nfs/cryodata    none    defaults,bind    0  0
/data2/xray      /srv/nfs/xray    none    defaults,bind    0  0
/data2/EM        /srv/nfs/EM    none    defaults,bind    0  0
/usr/local       /srv/nfs/local    none    defaults,bind    0  0

and

/etc/exports
/srv/nfs  192.168.159.31(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryodata  192.168.159.31(rw,sync,no_subtree_check)
/srv/nfs/EM  192.168.159.31(rw,sync,no_subtree_check)
/srv/nfs/xray  192.168.159.31(rw,sync,no_subtree_check)
/srv/nfs/local  192.168.159.31(rw,sync,no_subtree_check)

This simply does not work! When I try to mount the new mount on the client using the same client /etc/fstab entry:

{nfs client} /etc/fstab:
...
kraken.bio.univ.edu:/local  /usr/local  nfs4  _netdev,auto  0  0
kraken.bio.univ.edu:/cryodata  /cryodata  nfs4  _netdev,auto  0  0

.

# mount -v /cryodata
mount.nfs4: timeout set for Sat Feb 24 09:24:38 2018
mount.nfs4: trying text-based options 'addr=192.168.41.171,clientaddr=192.168.159.31'
mount.nfs4: mount(2): Stale file handle
mount.nfs4: trying text-based options 'addr=192.168.41.171,clientaddr=192.168.159.31'
mount.nfs4: mount(2): Stale file handle
mount.nfs4: trying text-based options 'addr=128.83.41.171,clientaddr=129.116.159.31'
...

The /usr/local continues to mount without problems. The first time I tried this I did forget to unexport/export the filesystems using exportfs -var before making changes, but since then I've switched back and forth, being careful to unexport and umount everything, with several server reboots in between. The original mount of a bind mount of the entire partition always works, and the bind mount of a subdirectory fails with the stale nfs handle message every time. I've tried enabling other nfs clients that have never mounted these partitions and get exactly the same error message: in this case it is definitely a server side problem. I've checked /var/lib/nfs/etab to make sure it's cleared out between mount attempts, etc.

I thought the technique of bind mounting into an nfs server root directory resolved all these kinds of issues, but apparently not? The odd thing is /usr/local is a subdirectory of another partition, and it always mounts fine. It is on an ext3 md raid 1, although I can't imagine this matters.

I've spent hours on this and have almost broken google looking for a solution to no avail.

I am trying to install the RDS role on an extremely generic Windows 2012R2 server. No matter how I do it (say using the Add Roles and Features Wizard), the install fails with the following error message:

Session Collection

    Unable to create the session collection.

A screen shot can be found here: http://imgur.com/RXyLhNQ

This machine is a collocated Dell Poweredge R730 with 256GB RAM and 2 Xeon E5-2680 v3 processors (16 cores total).

After the install fails, Server Manager shows that the Remote Desktop Services role has been installed by providing a panel for it, but if you click on the panel and select overview you get this message:

A Remote Desktop Services deployment does not exist in the server pool.
To create a deployment, run the Add Roles and Features Wizard and select
the Remote Desktop Services installation option.

If I try the same install, using exactly the same Windows 2012 Server ISO file on a cobbled together i5 desktop in my office and following exactly the same installation steps the install succeeds without a problem.

I'm clueless as to why this is happening. The server has 185GB of RAID 1 SSD disk (should be enough, the desktop has a 225GB disk). I've tried adding the role before applying patches, after applying patches, using PowerShell from a remote Windows 2012 server, etc., and nothing seems to work. I'm installing all the RDS roles on a single machine as per a dozen tutorials on the web, so am pretty sure I'm following the installation steps properly; in any case, the install works on the desktop in my office.

Also, I can't find anything of use in the Windows Event log files.

I'm starting to wonder if Microsoft has some kind of hook which prevents the RDS role from being installed on a machine with too much RAM or too man cores.

My apache 2.4 vhost configuration looks like this:

<Directory />
   Require all denied
</Directory>

<Directory /http/mystuff.org/html>
   Options FollowSymLinks Indexes Includes
   XBitHack on
   Require all granted
</Directory>

<Directory /http/mystuff.org/html/secretstuff>
    Options Indexes FollowSymLinks
    AuthType Digest
    AuthName "archives"
    AuthUserFile /etc/httpd/private/secretstuff.htaccess
    Require valid-user
</Directory>

The Require valid-user directive is simply ignored (i.e. anyone can get to the secretstuff folder without having to authenticate) and I can't figure out how to make it work. I've tried adding Require all denied to the beginning of the , tried using <Location> instead of <Directory> (someone on stackexchange suggested this), but nothing seems to work. I even tried

<RequireAll>
   Require valid-user
</RequireAll>

thinking this might be needed because of the Require all allowed in the parent directory, but this didn't work, either. I've read through the apache 2.4 documentation a half a dozen times now and haven't a clue as to why my directives to require a valid user are being ignore. The same configuration worked fine in apache 2.2.

Finally, I believe I have all the necessary modules loaded:

LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authz_user_module modules/mod_authz_user.so

I've been reading everything I can find on PAM (e.g. http://wpollock.com/AUnix2/PAM-Help.htm), however I'm still confused about how precisely the advanced options are supposed to work. For example, every reference says this:

Each of the four control-flag keywords (required, requisite, sufficient, and
optional) have an equivalent expression in terms of the [...] syntax:

required
    [success=ok new_authtok_reqd=ok ignore=ignore default=bad]

My understanding is that modules can return a variety of tokens, and that the action associated with each token is described in the configuration file. Based on that understanding, what does

new_authtok_reqd=ok

have to do with the required control flag? What's the meaning/purpose of

ignore=ignore ?

This alone:

success=ok

is what I would think matches the required behavior, but does

default=bad

mean that if the module returns any other action token, the module fails? Is the action=value token success=ok overruled by default=bad, or vice versa? Which takes precendence? It's not clear from anything I've read.

More generally, suppose I have something like

[success=done default=die]

What happens if the module returns success and one other token?

Finally, I can't find the answer to this question, either: can every value ok, done, bad, die, ignore, reset, N be associated with any action? What would it even mean to say

[default=done] ?

I run a small network where a linux web/mail server also provides NAT for a collection of windows boxes. One of these windows machines is apparently misbehaving (ZeroAccess botnet even though I can't find any problems using Norton PowerEraser and have only been able to find 3 outgoing port 16465 packets in /var/syslog in weeks of logging). For some reason setting a firewall rule to drop all outgoing 16465 packets doesn't solve the problem, either, but that's an unrelated issue.

This issue is causing the mail server to be blacklisted with, for example, spamhaus.org. Since I can't find the infected windows host, I hit upon the idea of using IP aliasing to send the NAT traffic out through a different IP address.

External interface:

eth0:   216.82.212.230
eth0:1  72.48.103.182

Internal interface:

eth1:   172.18.90.1

In my firewall rules I then changed

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 216.82.212.230

to

iptables -t nat -A POSTROUTING -s 172.18.90.0/24 -o eth0 -j SNAT --to 72.48.103.182

The problem is now everything appears to be going out as 72.48.103.182, including traffic from the mail server. If I ssh from the firewall host to another external machine, the connection is identified as coming from 72.48.103.182.

This doesn't make sense to me as I'm specifically specifying the source IP's which are supposed to be NATed. Originally I tried the line above without "-s 172.18.90.0/24" and got exactly the same result.

Any thoughts on what is going on? I'm not an iptables expert by stretch of the imagination, but have made an effort to try and research this before posting to serverfault.

=======================

root@www:etc# ip ro
216.82.212.0/24 dev eth0  proto kernel  scope link  src 216.82.212.230
72.48.103.0/24 dev eth0  proto kernel  scope link  src 72.48.103.182
172.18.90.0/24 dev eth1  proto kernel  scope link  src 172.18.90.1
default via 216.82.212.254 dev eth0  src 72.48.103.182  metric 100
default via 216.82.212.254 dev eth0  metric 100

I tried installing the 64-bit 13.04 Ubuntu server on a fairly generic superMicro box with dual OCZ SSD's I plan to software RAID. The install failed with a "Unable to install busybox-initramfs" message. This appears to be a bug documented on launchpad. (I got the same error message even when ignoring the second SSD.)

I checked console 4 for details:

apt-install or in-target is already running, so you cannot run either of them again until the other instance finishes.
You may be able to use `chroot /target ...` instead.

in-target: Unexpected error; command not executed:
`sh -c debconf-apt-progress --no-progress --logstderr -- apt-get -q -y --no-remove install busybox-initramfs`

I'm not sure what to do with this error message without sifting through the installer code, but the person who reported the bug originally on launchpad suggested that he found a workaround by "preseeding the installation with linux-server". Does anyone have any idea what this means / how to do this?