Dan Pritts's questions

I'm trying to set up sshd to run under xinetd on an RHEL7 server. I run sshd on an alternate port, using xinetd to limit what IPs can connect.

This works fine on RHEL6, and also on RHEL7 if I disable SELinux. However, the targeted SELinux policy on RHEL7 is preventing it.

Unfortunately, it's not logging much useful in /var/log/audit when it fails. My connection attempt yields two lines of successful CRYPTO_KEY_USER operations, and then this (a single line, which I've wrapped):

type=USER_LOGIN msg=audit(1485378997.248:18523): pid=6812 uid=0 
auid=4294967295 ses=4294967295 
subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 
msg='op=login acct="(unknown)" exe="/usr/sbin/sshd" 
hostname=? addr=A.B.C.D terminal=ssh res=failed'

audit2why doesn't even acknowledge that a problem exists. I'd guess it is looking for DENY events or something like that, not failures.

I added debug logging to sshd and found this (not wrapped this time):

debug1: SELinux support enabled [preauth]
debug3: ssh_selinux_change_context: setting context from 'system_u:system_r:sshd_t:s0-s0:c0.c1023' to 'system_u:system_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
debug3: privsep user:group 74:74 [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types: ssh-rsa [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
Write failed: Permission denied [preauth]

I presume this is my problem - the transition from sshd_t to sshd_net_t. However, without any useful information from the audit logs, I've reached the limit of my SElinux debugging skills.

A colleague suggested I review the audit logs from a successful connection in permissive mode. Unfortunately, there's no mention of a successful action as sshd_net_t. Here are the relevant log entries: http://pastebin.com/raw/9sSVpgLq

I did see some relevant information on the redhat bug tracker, but it didn't get me very far toward a solution. https://bugzilla.redhat.com/show_bug.cgi?id=1008580

I feel like this must be a faq but I've not been able to figure out the right terminology to find the answer.

I've got a large active directory group-of-groups that contains 101% of the users I would like to give access to a certain resource.

I'd like to somehow explicitly remove a few users from this group, but leave them in their original group (which is a member of the top-level group).

e.g.:

G1 = A,B,C
G2 = D,E,F
G3 = G1 + G2 - F = A,B,C,D,E

I think in some situations I could explicitly remove privileges like this via a group policy. That won't work in this case, I'm working with an Isilon NAS. It looks at group membership, nothing else.

The AD domain is managed by central IT; I believe it's running on windows 2012r2 now, but I'm not 100% sure of this.

On my RHEL5 systems, sendmail stopped working after I installed an update. The new version is sendmail-8.13.8-10.el5_11. I got errors like this in the logs:

NOQUEUE: SYSERR(nobody): can not write to queue directory /var/spool/clientmqueue/

and like this when I tried to mail from the command line:

WARNING: RunAsUser for MSP ignored, check group ids (egid=53, want=51)
can not write to queue directory /var/spool/clientmqueue/ (RunAsGid=0, required=53): Permission denied

It worked (for years) until the update.

I'm looking to use iptables hashlimit to limit abusive web crawlers, much like this question is trying to limit ssh bruteforce scans.

Every once in a while they hit an inefficient code path on our site. This brings us to our knees because they parallelize so heavily, and come in so fast (e.g., 3-5 incoming connections per second). End users don't hit this too often, and when they do, it's not 10x or 20x in parallel.

I know I'll have some tuning to do, to ensure the burst size is adequate for real users on browsers, and to make sure that my per-ip checking doesn't hurt a couple users behind NAT. All this seems doable, though. Tuning it on our live site shouldn't be too big a deal, I'll just log instead of dropping for the first couple weeks.

That said, I'm a little concerned about the memory usage of hashlimit. Mostly, I want to ensure that the site doesn't go down because this iptables rule doesn't have enough memory.

The Fine Manual for iptables-extensions says:

--hashlimit-htable-size buckets
The number of buckets of the hash table
--hashlimit-htable-max entries
Maximum entries in the hash.

But it's not entirely clear what are the buckets and what are the entries.

Also, what happens when the hash table fills up (maximum entries or buckets)? Hopefully the rule fails and iptables moves on to the next rule, but it doesn't really say.

Here's the rule I'm considering. It works as designed in limited testing, but load-testing with thousands of remote IPs is a little tricky.

iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW \
  -m hashlimit --hashlimit-name=WWW --hashlimit-above 1/sec --hashlimit-burst 50 \
  --hashlimit-mode srcip -j LOGACCEPT

I've recently converted some java apps to run with linux manually-configured hugepages, as described here. I point out "manually-configured" because they are not transparent hugepages, which gave us some performance issues.

So now, I've got about 10 tomcats running on a system and I am interested in knowing how much memory each one is using.

I can get summary information out of /proc/meminfo as described in Linux Huge Pages Usage Accounting.

But I can't find any tools that tell me about the actual per-process hugepage usage.

I poked around in /proc/pid/numa_stat and found some interesting information that led me to this grossity:

function pshugepage () {
    HUGEPAGECOUNT=0
    for num in `grep 'anon_hugepage.*dirty=' /proc/$@/numa_maps | awk '{print $6}' | sed 's/dirty=//'` ; do
        HUGEPAGECOUNT=$((HUGEPAGECOUNT+num))
    done
    echo process $@ using $HUGEPAGECOUNT huge pages
}

or this, in perl:

sub counthugepages {
    my $pid=$_[0];
    open (NUMAMAPS, "/proc/$pid/numa_maps") || die "can't open numa_maps";
    my $HUGEPAGECOUNT=0;
    while (my $line=<NUMAMAPS>) {
        next unless ($line =~ m{ huge }) ;
        next unless ($line =~ m{dirty=});
        chomp $line;
        $line =~ s{.*dirty=}{};
        $line =~ s{\s.*$}{};
        $HUGEPAGECOUNT+=$line;
    }
    close NUMAMAPS;
    # we want megabytes out, but we counted 2-megabyte hugepages
    return ($HUGEPAGECOUNT*2);
}

The numbers it gives me are plausible, but i'm far from confident this method is correct.

Environment is a quad-CPU dell, 64GB ram, RHEL6.3, oracle jdk 1.7.x (current as of 20130728)

[EDIT]

Bad question, mostly - see my answer

[/EDIT]

We've got a system running RHEL6, x64. We are using a local installation of apache 2.2.22 from source. we serve primarily:

1. mod_perl applications (with a local installation of perl 5.16.0)
2. tomcat applications proxied with mod_jk

Here is some context; the main question is below.

All of this talks to an Oracle backend.

We are having issues with Oracle becoming unresponsive. We think this is because we're hitting the maximum process limit in oracle. We've upped the process limit, but now we are hitting memory pressure on the oracle server. We have tons of oracle sessions sitting idle. I can trace a bunch of them back to the httpd processes.

We have mod_perl's Apache::DBI start up a new connection to the database with each httpd child that's spawned. We are concerned that these are not always getting closed out properly when the httpd's exit...and the httpd's are exiting very frequently. I know that it would be good to modify the mod_perl applications to use some better form of db connection pooling; we plan to pursue that but would like to solve our immediate problem sooner.

So here's the main question.

We are using the prefork MPM.

The apache child processes are lasting at most a few minutes. Log analysis shows that each one is serving fewer than 50 clients before exiting; the last request each child serves is OPTIONS * HTTP/1.0 on some sort of internal connection; I'm under the impression that this is a "ping" from the master process.

I've adjusted the MPM config as follows. I didn't want to raise MinSpareServers too high, because, after all, i'm trying to minimize the number of sessions to oracle.

MinSpareServers       5
MaxSpareServers       30
MaxClients            150
MaxRequestsPerChild   10000

Right now we're serving 250-300 requests per minute.

We've got 21 httpd's running, the eldest (other than the master, owned by root) being 3 minutes old.

This rate of reaping of the apache children really seems excessive. What could be causing it?

Apache was built with:

  $ ./configure --prefix=/opt/apache --with-ssl=/usr/lib --enable-expires --enable-ext-filter --enable-info --enable-mime-magic --enable-rewrite --enable-so --enable-speling --enable-ssl --enable-usertrack --enable-proxy --enable-headers --enable-log-forensic

Apache config info:

% /opt/apache/bin/httpd -V
Server version: Apache/2.2.22 (Unix)
Server built:   Jul 23 2012 22:30:13
Server's Module Magic Number: 20051115:30
Server loaded:  APR 1.4.5, APR-Util 1.4.1
Compiled using: APR 1.4.5, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/apache"
 -D SUEXEC_BIN="/opt/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

modules are compiled into apache rather than shared libs:

% /opt/apache/bin/httpd -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_ext_filter.c
  mod_include.c
  mod_filter.c
  mod_log_config.c
  mod_log_forensic.c
  mod_env.c
  mod_mime_magic.c
  mod_expires.c
  mod_headers.c
  mod_usertrack.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_scgi.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_ssl.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_info.c
  mod_cgi.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_speling.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

One final note - the red hat httpd, apr, and perl packages are all installed, but ldd shows that none of those libraries are linked with the running httpd.