hobodave's questions

Background

I have been struggling to get my SIP phones to register behind a brand new router and switch in our brand new office. Our PBX is hosted offsite. I have worked with our provider to attempt several different approaches. We have tried regular NAT to connect to their NAT-aware session border controller. We have tried using siproxd (the pfSense package) to intercept the SIP registration requests and register on the phones behalf. Finally, we have tried configuring the phones manually to register with the siproxd daemon on my local network.

Throughout testing we have seen the phones do all of the following successfully:

• Contact the hosted FTP server by IP address
• Download the configuration from said server
• Perform DNS queries to resolve the IP address of the NTP server
• Query the NTP server to set the time
• Perform DNS queries to resolve the IP address of the SIP server(s)

Symptoms

After the phones have done all of the pre-registration tasks successfully, we never see the registration attempt hit the pfSense box, or the provider's PBX. I have enabled the highest level of debugging in siproxd on my end and have seen nary a TCP connection or UDP packet. However, a simple telnet to port 5060 from a workstation will generate expected log messages. Performing a packet capture on the pfSense box showed absolutely no SIP traffic attempts.

What the heck?

My final troubleshooting step that thoroughly stumped me and brought me to ask this question was as follows. I first mirrored the switch port that a phone was plugged into to my workstation switch port. I performed a packet capture of all traffic on the interface. To my surprise I saw SIP registration packets coming from the phone. Here is an example:

Clearly the phone is trying to register with the PBXs (those are the correct IP addresses as well).

My next step was to mirror the switch port that feeds into the LAN side of the pfSense router. I saw all of the FTP, NTP, and DNS traffic from the 172.200.22.102 phone coming out of the switch, but not a trace of the SIP packets. This is completely baffling to me! What is causing only the SIP traffic to vanish within the switch?

Environment

• Hardware
  • Router/Firewall: Netgate m1n1wall 2D2
  • Switch: HP 1810G-24
  • Phones: Polycom SoundPoint IP 501
• Software: pfSense 2.0-RC3

Switch Configuration

The phone with IP Address 172.22.200.102 is in port 4 of this switch, the router LAN link is in port 22.

I can share any more settings that may be needed.

I had one of our Apache httpd servers "die" in a weird way overnight. I have looked at all relevant logs and cannot find anything out of the ordinary.

This happened in our development environment thankfully, but I am rather concerned because our dev environment nearly identically mirrors our production environment.

Can anyone tell me what could have happened here?

Symptoms & Troubleshooting Steps

• Apache httpd was completely unresponsive
• Load average of 0.0 across the board
• No log anomalies
• ps aux | grep httpd appeared normal
• iptables -L showed port 80 open
• SELinux is enabled and properly configured: no AVCs logged
• No configuration changes were made to any system components, no matter how insignificant
• Apache httpd was reloaded at 17:11:25 server time to clear APC (php) cache
  • Successful requests were made after this reload through 17:20 after which I left for the day
  • Nothing remarkable happened after reload
• The following morning the httpd processes were unresponsive
• Telnet to port 80 connected successfully
  • Manual GET, POST, OPTIONS, and HEAD requests received no response whatsoever; nothing logged

Logs

access_log

172.22.44.41 - - [03/Mar/2011:17:20:18 -0600] "GET /admin.php?type=notifications HTTP/1.0" 200 44326 "http://umc.dev.example.com/admin.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"
172.22.44.41 - - [04/Mar/2011:14:45:16 -0600] "GET / HTTP/1.0" 200 62921 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"

error_log

[Thu Mar 03 17:11:25 2011] [notice] SIGHUP received.  Attempting to restart
[Thu Mar 03 17:11:25 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Mar 03 17:11:25 2011] [notice] Digest: done
[Thu Mar 03 17:11:25 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Mar 04 14:45:05 2011] [notice] caught SIGTERM, shutting down
[Fri Mar 04 14:45:05 2011] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Fri Mar 04 14:45:05 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 04 14:45:05 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 04 14:45:05 2011] [notice] Digest: done
[Fri Mar 04 14:45:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

ps aux | grep httpd

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      8477  0.0  1.9 408748  9876 ?        Ss   Feb24   0:03 /usr/sbin/httpd
apache   19933  0.4 14.6 436436 74692 ?        S    Mar03   0:01 /usr/sbin/httpd
apache   19936  0.4  8.4 419708 43148 ?        S    Mar03   0:01 /usr/sbin/httpd

/var/log/messages

Feb 27 04:02:01 webdev1 syslogd 1.4.1: restart.
Mar  4 14:43:35 webdev1 yum: Installed: strace-4.5.18-5.el5_5.5.x86_64

Versions

OS & Kernel

• CentOS release 5.5 (Final)
• Linux webdev1.int.example.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Apache httpd

Server version: Apache/2.2.3
Server built:   Aug 30 2010 12:28:40
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

PHP

• php53u-5.3.5-3.ius.el5

Repositories used

CentOS Base & Updates for all packages with the following exception: php53u from IUS Community repository.

We're planning to migrate our production network from a VLAN-less configuration to a tagged VLAN (802.1q) configuration. This diagram summarizes the planned configuration:

One significant detail is that a large portion of these hosts will actually be VMs on a single bare-metal machine. In fact, the only physical machines will be DB01, DB02, the firewalls and the switches. All other machines will be virtualized on a single host.

One concern that has been is that this approach is complicated (overcomplicated implied), and that the VLANs are only providing an illusion of security, because "VLAN hopping is easy".

Is this a valid concern, given that multiple VLANs will be used for a single physical switch port due to virtualization? How would I setup my VLANs appropriately to prevent this risk?

Also, I've heard that VMWare ESX has something called "virtual switches". Is this unique to the VMWare hypervisor? If not, is it available with KVM (my planned hypervisor of choice)?. How does that come into play?

My primary question is in the title, and can be answered as simply as that. However, I think some background information may be relevant.

I'm doing some research into upgrading one of our three year old Dell PE 1950's to be a heavyweight (for a 1U) DB server. The CPUs are currently two entry-level Xeon 5110s (1.6 GHz, Dual Core, "Woodcrest" family).

I've been researching and pricing OEM parts in the Xeon 5400 (2.0 to 3.4 GHz, Quad Core, "Harpertown" family).

In the course of this research I've learned that there are two steppings: C0 and E0, however I can't find many plain-English specifics as to what is different, besides their release dates of Nov. '07 and Aug. '08 respectively. All I can see is "XSAVE/XRSTOR instructions were added", but what does that mean for my needs?

Normally I'd just not care and get the E0 to play it safe because it's the latest & greatest for this family of CPUs. However, I have found some rather significant pricing differences between the C0 and E0 steppings as shown here:

Model   Speed (GHz)  E0 USD x 2  C0 USD x 2
E5405   2.0                 570         560
E5410   2.33                740         520
E5420   2.5                 860        1020
E5430   2.66               1020        1010
E5440   2.83                  ?           ?
E5450   3.0                1560        1580
X5450   3.0                   ?           ?
X5460   3.16                N/A        1020
X5462   2.8                1160        1120 
X5470   3.33               1420         N/A
E5472   3.0                   ?        1430 
X5472   3.0                   ?         740 **
X5482   3.2                   ?        1620

_{? = Could not find price from "reputable" vendor, yet.
Prices are for two CPUs}

Note the cost of the C0 X5472, $740 -- the equivaent of the E0 E5410.

Now, until I priced the X5472 I was leaning towards the E5410 or E5420. However I'd be foolish to ignore the C0 price anomaly.

I'm not going to ask for a recommendation, but I would like to know if that is a misleadingly low price.

• Am I going to see a difference between a C0 and E0 stepping in a MySQL DB server (32 GB RAM if it matters)?
• Am I missing out on any critical functionality? Will it compromise the reliability of my production environment?
• Is this totally worth it? (yes, subjective, but just this! :)

First, yes I have seen this question:

Find (and kill) old processes

The answers there are incorrect and do not work. I have voted and commented accordingly.

The processes I want to kill look like this when listed with ps aux | grep page.py:

apache     424  0.0  0.1   6996  4564 ?        S    07:02   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache    2686  0.0  0.1   7000  3460 ?        S    Sep10   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache    2926  0.0  0.0   6996  1404 ?        S    Sep02   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache    7398  0.0  0.0   6996  1400 ?        S    Sep01   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache    9423  0.0  0.1   6996  3824 ?        S    Sep10   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   11022  0.0  0.0   7004  1400 ?        S    Sep01   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   15343  0.0  0.1   7004  3788 ?        S    Sep09   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   15364  0.0  0.1   7004  3792 ?        S    Sep09   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   15397  0.0  0.1   6996  3788 ?        S    Sep09   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   16817  0.0  0.1   7000  3788 ?        S    Sep09   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   17590  0.0  0.0   7000  1432 ?        S    Sep07   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   24448  0.0  0.0   7000  1432 ?        S    Sep07   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py
apache   30361  0.0  0.1   6996  3776 ?        S    Sep09   0:00 /usr/bin/python2.6 /u/apps/pysnpp/current/bin/page.py

I'm looking to setup a simple daily cron that will find and kill any page.py processes older than an hour.

The accepted answer on the aforementioned question does not work, as it doesn't match a range of times, it simply matches processes that have been running from 7 days to 7 days 23 hours 59 minutes and 59 seconds. I don't want to kill processes that have been running from 1-2 hours, but rather anything greater than 1 hour.

The other answer to the aforementioned question using find does not work, at least not on Gentoo or CentOS 5.4, it either spits out a warning, or returns nothing if the advice of said warning is followed.