datakid's questions

I was hardening my webserver - CentOS 7.5.1804, httpd 2.4.6_80 - and followed the myriad of examples that suggested disabling mod_autoindex. (see Commenting out LoadModule autoindex_module modules/mod_autoindex.so)

httpd doesn't restart because /etc/httpd/conf.d/autoindex.conf has entries that httpd no longer understands. Fine, removed autoindex.conf.

Then I updated the server, and found that autoindex.conf had been replaced in the update, again preventing the httpd from starting.

Does anyone know why autoindex.conf is

1. independant of httpd.conf
2. not wrapped in <IfModule mod_autoindex></IfModule> markers?

Subsequently, does anyone know how to turn mod_autoindex off and keep it off without threatening to prevent httpd from starting?

The issue we are seeing is that some of our NFS mounted directories are disappearing. Literally just disappearing. They continue to exist at the server end.

At the client (CentOS 7.4, using el-repo's kernel-ml at version 4.16.6, nfs-utils-1:1.3.0-0.48.el7) end the drive is non-responsive. Calls to ls /mountpoint just hang. The OS still thinks the drive is mounted - rebooting a server requires a hard reset because the shutdown process hangs on unmounting the drive.

In the logs we are seeing very little - at the server end we see nothing. Some, but not every, time we see something like this in /var/log/messages on the client:

May 29 16:55:22 papr-res-compute06 kernel: INFO: task STAR:1370 blocked for more than 120 seconds. May 29 16:55:22 papr-res-compute06 kernel: Not tainted 4.16.6-1.el7.elrepo.x86_64 #1 May 29 16:55:22 papr-res-compute06 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 29 16:55:22 papr-res-compute06 kernel: STAR D 0 1370 1351 0x00000080 May 29 16:55:22 papr-res-compute06 kernel: Call Trace: May 29 16:55:22 papr-res-compute06 kernel: __schedule+0x290/0x890 May 29 16:55:22 papr-res-compute06 kernel: ? out_of_line_wait_on_atomic_t+0x110/0x110 May 29 16:55:22 papr-res-compute06 kernel: schedule+0x36/0x80 May 29 16:55:22 papr-res-compute06 kernel: io_schedule+0x16/0x40 May 29 16:55:22 papr-res-compute06 kernel: bit_wait_io+0x11/0x50 May 29 16:55:22 papr-res-compute06 kernel: __wait_on_bit+0x66/0x90 May 29 16:55:22 papr-res-compute06 kernel: out_of_line_wait_on_bit+0x91/0xb0 May 29 16:55:22 papr-res-compute06 kernel: ? bit_waitqueue+0x40/0x40 May 29 16:55:22 papr-res-compute06 kernel: nfs_wait_on_request+0x4b/0x60 [nfs] May 29 16:55:22 papr-res-compute06 kernel: nfs_lock_and_join_requests+0x12a/0x540 [nfs] May 29 16:55:22 papr-res-compute06 kernel: ? radix_tree_lookup_slot+0x22/0x50 May 29 16:55:22 papr-res-compute06 kernel: nfs_updatepage+0x120/0x9f0 [nfs] May 29 16:55:22 papr-res-compute06 kernel: ? nfs_flush_incompatible+0xc5/0x1c0 [nfs] May 29 16:55:22 papr-res-compute06 kernel: nfs_write_end+0xe2/0x3c0 [nfs] May 29 16:55:22 papr-res-compute06 kernel: generic_perform_write+0x10b/0x1c0 May 29 16:55:22 papr-res-compute06 kernel: ? _cond_resched+0x19/0x30 May 29 16:55:22 papr-res-compute06 kernel: ? _cond_resched+0x19/0x30 May 29 16:55:22 papr-res-compute06 kernel: nfs_file_write+0xd4/0x250 [nfs] May 29 16:55:22 papr-res-compute06 kernel: do_iter_readv_writev+0x109/0x170 May 29 16:55:22 papr-res-compute06 kernel: do_iter_write+0x7f/0x190 May 29 16:55:22 papr-res-compute06 kernel: vfs_writev+0x84/0xf0 May 29 16:55:22 papr-res-compute06 kernel: ? handle_mm_fault+0x102/0x220 May 29 16:55:22 papr-res-compute06 kernel: ? _cond_resched+0x19/0x30 May 29 16:55:22 papr-res-compute06 kernel: do_writev+0x61/0xf0 May 29 16:55:22 papr-res-compute06 kernel: SyS_writev+0x10/0x20 May 29 16:55:22 papr-res-compute06 kernel: do_syscall_64+0x79/0x1b0 May 29 16:55:22 papr-res-compute06 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2 May 29 16:55:22 papr-res-compute06 kernel: RIP: 0033:0x7f0cc1563230 May 29 16:55:22 papr-res-compute06 kernel: RSP: 002b:00007fff6f75f2c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000014 May 29 16:55:22 papr-res-compute06 kernel: RAX: ffffffffffffffda RBX: 0000000000002025 RCX: 00007f0cc1563230 May 29 16:55:22 papr-res-compute06 kernel: RDX: 0000000000000002 RSI: 00007fff6f75f300 RDI: 000000000000000a May 29 16:55:22 papr-res-compute06 kernel: RBP: 000000001a7b0860 R08: 0000000000000000 R09: 00007f0cc15b8a00 May 29 16:55:22 papr-res-compute06 kernel: R10: cccccccccccccccd R11: 0000000000000293 R12: 000000000000000a May 29 16:55:22 papr-res-compute06 kernel: R13: 00007fff6f75f300 R14: 0000000000000028 R15: 0000000000001ffd

We cannot reproduce this error reliably. There is very little in the way of commonalty between instances of the drives dropping - this is on a HPC with around 40 servers and 100 users. Usually only affects a single server, but has happened across hardware (Cisco UCSB-B200-M4 w 368GB and UCSME-142-M4 w 32GB). The one thing that is common is that the drives in question hold biological data which can be very large files (sometimes over half a TB).

So, we would like to monitor that the NFS drives are up because SLURM keeps assigning jobs to servers that have this problem, and those jobs will never complete.

My colleague has whipped up a couple of shell scripts that ping this and ls that and write to file and email when necessary. It's clever and would have been fun to write (awk!, cut!), but is absolutely the (in my mind) wrong way to do it.

A quick google shows that nfsiostat might be useful but there is little in the way of good instruction, what there is doesn't seem to map to our OS (in particular "count" is necessary despite what the man page says) and most importantly, we don't want stats on usage. We want existence information. Literally "are you there nfsdrive, it's me root?" I admit I need to read more on nfsiostat.

Another solution I've seen is to move from nfs mounts listed in fstab to using autofs. The CentOS docs finish two versions ago but do state:

This is not a problem with one or two mounts, but when the system is maintaining mounts to many systems at one time, overall system performance can be affected.

Given that we have around 10-12 mount points - does this count as many? My colleague suggested that many was more in the realm of 100s. Regardless, this information is now old. Looking into the Redhat docs, we see that it is a copy/paste.

I've seen these two errata/bugs

• https://access.redhat.com/errata/RHBA-2018:0981
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/bug_fixes_file_systems

but we are unable to upgrade immediately because the servers are production and in a clinical setting. Plus, there is no indication that this is explicitly our problem, and given that we can't reproduce the problem at will, we can't test on our dev machines - although I have been trying to reproduce on our dev machines. There are also more recent El Repo kernel-ml versions, but the same problem exists - can't just update overnight - these servers are often at 100% 24/7.

So, I guess I have two questions:

• what is the best method people know of for monitoring the existence of a drive (ie - grepping mount is an unacceptable answer)
• should we migrate from /etc/fstab to autofs (which, if we were to do, would probably include a kernel and OS update as well)

We have Foreman 1.12.4 w Katello 3.1 installed on Centos 7.2

After a small error with pip in which yum-controlled software was updated and broke our Katello install, I rolled back as best I could and set off a fresh sync on all products.

Some products worked fine, but about half return this error

RPM1004: Error retrieving metadata: Not found

I can't see anything online that would suggest a solution to this error. There have been some suggested solutions - for eg "install the errata", which would be possible except the Katello product is one that won't syncronise.

Does anyone know how I might go about syncing against the repos? Here is one of the repos that isn't syncing - it looks like there is metadata in there (by which I mean, there is a repodata dir with files in it).

https://fedorapeople.org/groups/katello/releases/yum/3.1/pulp/el7/x86_64/

To be honest I'm a bit lost - I understand the error message (can't find the metadata) but I can't find a solution - it is as simple as finding the metadata and putting it in the right place on the filesystem somewhere, or is it more complex than that?

We tried very hard to install and love Spacewalk without success. So we moved to Satellite 6, which on Centos 7 is the Foreman/Katello/Pulp/Candlepin combo.

It's all set up fine, and we are using it to deploy, update and install packages.

One thing I can't find is how to get a list of hosts which have package X installed. (in this case, postgresql-server)

Is it possible and I'm just not finding it, or am I asking too much?

Where can I find information about CentOS package management?

I installed Spacewalk recently and noticed that FreeIPA was listed as ready for an update. The versions are differently named, and the FreeIPA mailing list said it was a CentOS issue. I can't find who/where to ask what the difference in the versions is.

The versions are:

• Installed package ipa-server-4.2.0-15.el7.centos.3.x86_64
• Update candidate ipa-server-4.2.0-15.el7_2.6.x86_64

Is there any way to pin rpm packages to versions in SpaceWalk?

We have kernels running on rare hardware, compiling the kernel is a pain and we don't want or need to do it any more often than major security. There are a number of associated libraries that fall into this category too.

Very similar to this question but three years later and I'm using SpaceWalk on Centos rather than Satellite. Also, given the time that's passed ... surely this would have been implemented by now?

Further, this solution doesn't work for us, as we are not seeing any errata in SpaceWalk. There seem to be a couple of solutions (eg by Steve Meier and liedekef) but I am nervous to go down this route.

In a Windows Server 2008 R2 AD, is it possible to get a list of OUs managed by a User from the User object or does one needs to parse the list of OUs?

(ie, is there representation on the User object of OU management or is it only a var on the OU)

I have BackupExec 2012 attempting to backup a Windows Server 2003 VMWare instance.

I often get unable to quiesce error: "The virtual machine may be too busy to quiesce to take the snapshot."

I've tried looking through the event logs but it's painstaking, and I've not found anything of note at 5am. But not being a Windows native, I'm concerned that there is something I'm missing.

The backup is set to run at 5am - how can I get a good idea of what the server is doing at 5am?

I asked this question yesterday when I (seemingly) couldn't get traffic across my router.

I've spent a lot of time since then with packet capturing, the ASA 5505 cli/ASDM and wireshark.

I've finally realised that my issue is that there are two entries to the LAN.

The LAN is 10.10.5.0/24, with a PDC (DHCP/DNS/etc) at 10.10.5.5, Watchguard Firewall at 10.10.5.1

One entry point has a Watchguard Firewall set up and it's purpose is to manage almost all of the traffic in our organisation. In fact, it does all the traffic except for one specific set.

Which is the other entry point, for a single client. That client's server is at 10.10.5.30 and the ASA 5505 router is 10.10.5.2 (bridged across a modem).

We run services on a number of our servers, on UDP 3000. For all the other servers, all internet, ftp and etc traffic we want it to go out the main Watchguard gateway. And it does.

But for that one server, I want just the port 3000-3002 UDP to go out through the ASA 5505. All UDP data from other servers (including port 3000-3002) is to go through the WG.

I've discovered that while the traffic is coming in through 10.10.5.2 (the ASA5505), the correct server, 10.10.5.30, is receiving the data, BUT then 10.10.5.30 is trying to send the ACK requests back across the 10.10.5.1 firewall.

The symptoms are that it looks like the ASA has no traffic going across it, and we aren't getting any flow. How can I force certain protocols and ports for this server across the ASA rather than the WG?

The ASA 5505 has NAT rules to forward all UDP straight to .30 and to drop everything else.

Important Notes: other servers are using the same protocol and port through the WatchGuard at 10.10.5.1, so I can't just push ALL the LAN's UDP port 3000 data out the one connection - I need to split it.

Update Essentially I need to create a route on the server 10.10.5.30 explicitly for UDP ports 3000-3002. I think. It is a Windows 2003 Server SP2

I have inherited my first Cisco router and am having trouble understanding how to do NAT port forwarding correctly.

ASA: 8.0.2 ASDM: 6.1(1)

I have a single external ip address on the outside (via bridged modem) and a LAN on the inside. The LAN has another connection to the net and a PDC providing DHCP/DNS, and shouldn't be using this connection.

This connection should be doing very few things things:

• Forwarding all incoming traffic on UDP ports 3000-3002 to same ports on 10.10.5.30
• Forwarding all incoming traffic on TCP port 3005 to same port on 10.10.5.30
• allowing traffic back out on same ports and protocols
• Blocking everything else.

I think the problem we are having is that the traffic isn't going back out - the implicit deny on the inside keeps getting flagged when I do a Packet Trace.

Also, since I've started my small fiddlings to fix this issue, the inside, implicit any to any-less-secure rule has now disappeared (ooops!) - any ideas on how to address that issue?

I am comfortable on the command line with instruction (potentially your answer), less so when experimenting and investigating (my current situation).

Sanitized Conf:

: Saved
:
ASA Version 8.0(2) 
!
hostname BLAH-ASA
domain-name BLAH.local
enable password BLAH encrypted
names
name 15.8.15.3 BLAH_ASA_Outside
name 192.168.6.254 vpn_local
name 10.10.5.5 FS1 description BLAH File Server
name 10.10.5.30 W_BASE description W Base Station
!
interface Vlan1
 nameif inside
 security-level 100
 ip address 10.10.5.2 255.255.255.0 
!
interface Vlan2
 nameif outside
 security-level 0
 pppoe client vpdn group global
 ip address pppoe 
!
interface Ethernet0/0
 switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
passwd BLAH encrypted
boot system disk0:/asa802-k8.bin
ftp mode passive
clock timezone EST 10
clock summer-time EDT recurring last Sun Oct 2:00 last Sun Mar 3:00
dns server-group DefaultDNS
 domain-name BLAH.local
object-group service W_TCP tcp
 description TCP 3005
 port-object eq 3005
object-group service W_UDP udp
 description UDP 3000 - 3002
 port-object eq 3000
 port-object eq 3001
 port-object eq 3002
access-list outside_1_cryptomap extended permit ip 10.10.5.0 255.255.255.0 host vpn_local 
access-list outside_access_in extended permit tcp any host BLAH_ASA_Outside eq pptp 
access-list outside_access_in remark W UDP Ports 3000 - 3002
access-list outside_access_in extended permit udp any host BLAH_ASA_Outside object-group W_UDP 
access-list outside_access_in remark W TCP Port 3005
access-list outside_access_in extended permit tcp any host BLAH_ASA_Outside object-group W_TCP 
access-list outside_access_in remark VPN ACL
access-list outside_access_in extended permit ip host vpn_local 10.10.5.0 255.255.255.0 
access-list outside_access_in extended permit gre any any 
access-list inside_nat0_outbound extended permit ip 10.10.5.0 255.255.255.0 host vpn_local 
pager lines 24
logging enable
logging asdm informational
mtu inside 1500
mtu outside 1500
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-611.bin
no asdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
static (inside,outside) tcp interface pptp FS1 pptp netmask 255.255.255.255 
static (inside,outside) udp interface 3000 W_BASE 3000 netmask 255.255.255.255 
static (inside,outside) udp interface 3001 W_BASE 3001 netmask 255.255.255.255 
static (inside,outside) udp interface 3002 W_BASE 3002 netmask 255.255.255.255 
static (inside,outside) tcp interface 3005 W_BASE 3005 netmask 255.255.255.255 
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 165.228.0.1 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout uauth 0:05:00 absolute
dynamic-access-policy-record DfltAccessPolicy
http server enable
http 10.10.5.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto map outside_map 1 match address outside_1_cryptomap
crypto map outside_map 1 set pfs 
crypto map outside_map 1 set peer 132.3.9.254 
crypto map outside_map 1 set transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
telnet 10.10.5.0 255.255.255.0 inside
telnet timeout 5
ssh timeout 5
console timeout 0
vpdn group global request dialout pppoe
vpdn group global localname USERNAME
vpdn group global ppp authentication chap
vpdn username USERNAME password ********* 
dhcpd auto_config outside
!
dhcpd address 10.10.5.3-10.10.5.33 inside
!

threat-detection basic-threat
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect pptp 
!
service-policy global_policy global
ntp server 8.8.8.8 source outside prefer
username ROOT password PASSWORD encrypted privilege 15
username ROOT2 password PASSWORD encrypted privilege 15
tunnel-group 132.3.9.254  type ipsec-l2l
tunnel-group 132.3.9.254  ipsec-attributes
 pre-shared-key *
prompt hostname context 
Cryptochecksum:20
: end

I have inherited a DHCP server set up with a very small address pool and a number of reserved static addresses.

Originally a small network, set up by non professionals, we are now running out of address space. The address pool is quite small (x.x.x.110-200) and is over 85% full. Because of the bountiful riches of address space, the static reservations are liberally spread across x.x.x.1-100 with large gaps in between.

My question is "in server 2003 R2 are address pools and reservations required to be mutually exclusive or can they overlap and the reservations are given precedence/priority/excluded from the pool?"

IE, can I expand the address pool to x.x.x.50-250 and not be concerned about the half dozen machines scattered between .50 and .110 or do I need to move those reservations down to the mostly empty .30-50 range and then expand the address pool?