Cory J's questions

Issue

I run an IRC server for 20-50 users. We sometimes have issues with messages not arriving in a timely fashion or at all. After some packet captures we determined that messages sit in the server's "Send-Q". When a message doesn't arrive I'll look at "netstat -ct" output and see something like this:

Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 1756 ubuntu:ircd 10.8.1.7:63602 ESTABLISHED

Sometimes if I wait for a couple of minutes, the Send-Q will go to 0 and the message will be delivered, other times the client times out. My question is, why doesn't it just deliver the messages? What causes them to sit in Send-Q so long?

sshd also exhibits similar behavior, my ssh sessions freeze sometimes they come back, sometimes they time out.

Background

Not sure if the infrastructure here could be related to the issue, so here's what it looks like: these clients are on Windows 7 connecting with OpenVPN. OpenVPN server is on PFSense, the IRC server is on a local (NAT'd) LAN connected to PFSense. I have a firewall rule in place to allow clients to talk to 6667 on the server.

Investigating...

Latency/loss - looks decent enough. Not the best link ever but I would think this would be fine for IRC and SSH. Here is a ping from my client to the server, this is while my IRC and SSH are intermittantly hanging:

Ping statistics for 10.8.5.2:
    Packets: Sent = 4478, Received = 4460, Lost = 18 (0% loss)

Approximate round trip times in milli-seconds: Minimum = 17.2 ms, Maximum = 273.4 ms, Average = 32.3 ms

MSS/MTU issues - MTU appears to be fine. OpenVPN mtu-test on my client says:

Thu Dec 03 12:41:21 2015 NOTE: Empirical MTU test completed [Tried,Actual] local->remote=[1589,1589] remote->local=[1589,1589]

...and here's my manual test:

> ping -f -l 1472 10.8.5.2

Pinging 10.8.5.2 with 1472 bytes of data:
Reply from 10.8.5.2: bytes=1472 time=23ms TTL=63

> ping -f -l 1473 10.8.5.2

Pinging 10.8.5.2 with 1473 bytes of data:
Packet needs to be fragmented but DF set.

Bandwidth/throughput - did some iperf tests to make sure there wasn't a throughput issue. Again, looks decent enough:

iperf -c 10.8.5.2
------------------------------------------------------------
Client connecting to 10.8.5.2, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[  3] local 10.8.0.23 port 18587 connected with 10.8.5.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  26.0 MBytes  21.8 Mbits/sec

Thanks, any help understanding "Send-Q" or more specific ideas about this issue would be much appreciated. Let me know if I can provide any more info here.

Update

Found out that I actually had massive packet loss. Pings from client->VPN didn't show this, but it was very apparent when using fping from VPN->client. I noticed it was only the Windows clients, and reinstalling the newest OpenVPN client seems to have fixed the loss. It might have been related to the OpenVPN TAP adapter being installed via disk imaging. Installing it manually per-machine seems to fix the problem.

Background: I'm working on replacing my single DHCP server with 2 in a failover configuration. When I do a DHCP request I see the lease is created in both dhcpd.leases files on each server. However, I also use a script that add, deletes, and modifies host definitions via OMAPI. I notice when I make a modification on one server, it is not syncronized to the peer.

Question: Should changes caused by OMAPI operations be syncronzed between 2 DHCP servers in a failover configuration the way leases are? If not, should I deal with this by simply making the same changes on both servers? I don't want to do anything that might corrupt my leases file(s), etc.

I am trying to create a firewall rule on an Ubuntu 10.04 server running isc-dhcpd. I only want dhcp to be accessible by a single relay host (172.1.1.1). I have iptables set up like so:

# iptables -vnL
Chain INPUT (policy ACCEPT 5325 packets, 523K bytes)
pkts bytes target     prot opt in     out     source               destination         
0     0 ACCEPT     udp  --  *      *       172.1.1.1            0.0.0.0/0           udp dpt:67 
1497  533K DROP    udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:67 

This is my attempt to drop all UDP traffic destined to port 67 unless it's coming from 172.1.1.1.

However, with this rule in place, when I start the dhcpd I immediately see DHCP requests start to come in via broadcast (the log says via eth0).

Do I need another firewall rule to catch the broadcast traffic? I've tried adding a rule like

iptables -A INPUT -p udp -d 255.255.255.255 -j DROP

which should drop all UDP broadcast traffic. But again when I start dhcpd I see tons of requests via broadcast.

EDIT: According to this page:

For most operations, DHCP software interfaces to the Linux IP stack at a level below Netfilter. Hence, Netfilter (and therefore Shorewall) cannot be used effectively to police DHCP.

So is there any way to accomplish what I want here?

I've been able to get some of my linux servers authenticating users against my LDAP directory server, but I've been having some trouble trying to do this with nss_ldap and pam_ldap in FreeBSD.

Going from FreeBSD official docs here: http://www.freebsd.org/doc/en/articles/ldap-auth/client.html

I install the 2 packages, and create a config file /usr/local/etc/ldap.conf, and also a symlink to this file in the same directory, nss_ldap.conf. According to the docs they can both use the same config file. I'm keeping it very simple until I can get it working:

ldap.conf/nss_ldap.conf:

base    dc=corp,dc=example,dc=org
host    192.168.0.100
ldap_version 3
binddn cn=admin,dc=corp,dc=example,dc=org
bindpw secret

NSS works as far as I can tell. A "getent passwd" shows information from the LDAP directory as well as local stuff.

Now I want to authenticate, so I add a line to /etc/pam.d/sshd:

# auth
auth        sufficient  pam_opie.so     no_warn no_fake_prompts
auth        requisite   pam_opieaccess.so   no_warn allow_local
#auth       sufficient  pam_krb5.so     no_warn try_first_pass
#auth       sufficient  pam_ssh.so      no_warn try_first_pass
auth        sufficient  /usr/local/lib/pam_ldap.so no_warn try_first_pass debug
auth        required    pam_unix.so     no_warn try_first_pass

I restart ssh (not sure if this is necessary), and then try and log in with an LDAP user that doesn't exist locally (coryj). It fails silently and logs show:

Sep  9 13:13:54 freebsd-testbox sshd[12684]: pam_ldap: error trying to bind as user "uid=coryj,ou=Users,dc=corp,dc=example,dc=org" (Invalid credentials)

Why is it trying to bind with the user I'm trying to authenticate when I specified a binddn/bindpw? I also tried rootbinddn with a .secret file with the same result. On linux binddn seems to work, where here it seems to be ignored.

I know my ldap.conf and pam files will need some more work, just trying to convince the thing to bind as admin when authenticating at this point.

I am attempting to set up a VERY simple DHCP server. I am using Ubuntu 10.04, and I installed the "dhcp3-server" package. The server has one interface (eth0) statically set to 10.0.0.10(/21). I want to hand out IPs from the network 10.255.224.0/19.

Doing what I've been told by various online guides, I edit /etc/default/dhcp3-server to include the line "INTERFACES="eth0".

Then I write up my dhcpd.conf. I had to put an empty subnet or I get the error:

No subnet declaration for eth0 (10.0.0.10).
** Ignoring requests on eth0.  If this is not what
you want, please write a subnet declaration
in your dhcpd.conf file for the network segment
to which interface eth0 is attached. **

Once I add this empty subnet the daemon will start. I try another machine acting as a DHCP client to test it. This is what I see in the dhcpd logs:

dhcpd: DHCPDISCOVER from 00:1d:09:b1:f5:dc via eth0: network 10.0.0/21: no free leases

That "network" definitly doesn't look right, but I can't find where the problem is that is preventing my test client from getting an IP. Here is my dhcpd.conf:

ddns-update-style none;
option domain-name "example.org";
option domain-name-servers 10.10.10.83, 10.10.10.84;
default-lease-time 600;
max-lease-time 7200;
authoritative;
log-facility local7;

#empty subnet to prevent error on start
subnet 10.0.0.0 netmask 255.255.248.0 { 
}

subnet 10.255.224.0 netmask 255.255.224.0 {
    option subnet-mask 255.255.224.0;
    option broadcast-address 10.255.255.255;
    option routers 10.0.0.1;
    range 10.255.224.2 10.255.255.254;
}

I've worked with ISC-DHCPd some in the past, but this is the first server that I've tried to build and deploy myself. Any hints would be super!

I am using Ubuntu 10.04 Desktop, which comes by default with the "Network Manager" (which I am highly suspicious of). I am having an issue with resolving DNS zones on 2 different nameservers.

resolv.conf:

$ cat /etc/resolv.conf
#Generated by Network Manager
search example.org
nameserver 10.0.0.20 #public nameserver, contains public.example.org
nameserver 10.0.0.30 #private nameserver, contains private.example.org

And the problem. Whichever name server is listed first will resolve :

$ ping host.public.example.org
PING host.public.example.org (10.0.0.50) 56(84) bytes of data.

The one listed second will not:

$ ping host.private.example.org
ping: unknown host host.private.example.org

But it IS there:

$ dig @10.0.0.30 host.private.example.org
...
;; ANSWER SECTION:
host.private.example.org.   3600    IN  A   10.0.0.60
...

If I reverse the order of nameservers in /etc/resolv.conf, host.private will then be accessible and host.public will not. Why? From resolv.conf man page:

If there are multiple servers, the resolver library queries them in the order listed.

It isn't just ping that has problems, I noticed this when Thunderbird couldn't get mail because the mail server's DNS record is in the 2nd server listed. Shouldn't what I'm trying to do here work?

I have recently set up a server with FreeRADIUS to authenticate and keep track of admin logins on some of my devices.

I'm looking at getting my Linux and FreeBSD machines authenticating against RADIUS using the pam_auth_radius module. I am currently working with an Ubuntu 10.04 virtual machine for testing. Also I am currently testing with RADIUS authenticating only remote SSH users, I would like ALL authentication to be done this way at some point.

As per instructions from all around the web, I install pam_auth_radius and add a line referencing the module in /etc/pam.d/sshd and added my server address and shared secret to /etc/pam_radius_auth.conf.

After doing this I can authenticate against RADIUS...however, only users that I have added to the test machine (using adduser) can authenticate. For example, user "cory" is a valid RADIUS user, for which the RADIUS returns "Accept", but I keep getting "Permission Denied" while trying to log in via ssh until I "adduser cory".

Do I need to add each and every admin user to each and every server? I realize I can make un-passworded users, so it isn't really a security issue, but it just seems to defeat the point of having a central authentication server if I still have to do account management on each device.

Is there a way to create a temporary user/directory/etc dynamically? Or can users authenticating via RADIUS be mapped to an existing account to remove the need to create each user?

I have a /24 network that is subnetted into a bunch of small chunks. I have recently gone into each router on the network (mostly Cisco) in order to document how this network had been divided. Now looking at a ping sweep output from:

nmap -sP 192.168.1.*

I see that some but not all reserved "network" and "broadcast" IPs respond to pings. For example, the network 192.168.1.80/29 has the network of 192.168.1.80 and a broadcast of 192.168.1.87. On this particular subnet, both of these IPs give me a ping response from the external interface of the router (192.168.5.20).

Many of the other subnets behave in a similar manor. However others do not. Looking at the router configs, nothing really jumps out at me that looks like it would cause this behavior.

Does anyone know the reason for this behavior? Do I want those addresses to respond or not? Slightly unrelated: should I have reverse DNS entries for the network and broadcast IPs?

In the man page for FreeBSD ping:

 -s packetsize
         Specify the number of data bytes to be sent.  The default is 56,
         which translates into 64 ICMP data bytes when combined with the 8
         bytes of ICMP header data.  Only the super-user may specify val-
         ues more than default.  This option cannot be used with ping
         sweeps.

Does anyone know how I can let non-superusers send pings larger then 56? Can I increase the default size, or somehow allow -s to be used by non-root users? I am aware of the security implications, that this could allow users to start a ping flood, so if I can restrict this to allowing only a single (shell-less) user, all the better.

I'm looking for a way to identify as many consumer VoIP users on my ISP network as possible using packet analysis.

My setup is like this:

On my core switch, all traffic going in and out of gigabit1 is SPAN'd to gigabit2, where I have a linux server connected.

Here's what I've been trying:

I ran tshark looking filtering for my networks and the standard SIP ports. Something like:

tshark -i eth0 -f "(net 1.2.3.4/24 or net 4.5.6.7/24) and (port 5060 or port 5061)" -w ./outfile

Ran this for over a week, then searched through the output file for a list of unique IPs.

I see a few SIP users this way, but not nearly as many as I should. Is there something wrong with this approach? It was my understanding that most major consumer VoIP products, such as Vonage, used SIP for signalling.

What (I think) I want:

I want to actually classify protocols, looking for SIP, RTP etc. If I install L7 Filter I can use iptables to mark the traffic I am interested in, but I don't know where I would go from there to get a list of unique IPs.

I am open to any suggestions.

On Linux systems I can

watch -n1 tail /var/log/whatever.log

or

watch -n1 grep somestuff /var/log/whatever.log

To show updates to a log every 1 seconds. On FreeBSD however, the watch command does something else entirely. Who knows a good FreeBSD command for what I'm trying to do? =)

Trying to adjust a cron job to run every 90 min. It was previously running every 20 min, which was a simple cron job:

*/20 * * * * whatever

To change it to every 90, it seems like I need to split it into 2 jobs, I've done this:

0 0,3,6,9 * * * whatever
30 1,4,7,10 * * * whatever

Is this right? The job doesn't seem to kick off.

I am attempting to use ntop to monitor traffic over a Cisco Catalyst switch. I was assuming that in order to see any of the traffic, I'd have to use monitor, as described here: http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml.

Howver, before I did anything on the switch, I simply plugged my ntop server in and fired up ntop. To my suprise, I instantly see 3+ pages of hosts, and thousands of packets. How is ntop seeing this?

I have verified that no monitoring exists on the switch (run as en):

cs1.pvdc#show monitor
  No SPAN configuration is present in the system.

My ntop server is Ubuntu 8.04, I haven't done ANY configuration, I just installed the ntop package. This is also a fresh Ubuntu install.

Is there anything else on my switch besides "monitor" that might cause my switch to mirror all its traffic like this? I've tried plugging ntop into different ports with the same results.

UPDATE: It appears to be more then just broadcast traffic showing up in ntop, for example, I can see when my IPs have talked to the DNS server or generated HTTP traffic. If my switch is misconfigured, can anyone point me in the right direction towards rectify this? Not a Cisco expert.

I am replacing my BIND authoritative nameserver with PowerDNS w/ gmysql backend. I'd like to have my primary domain example.com and its associated hosts like www.example.com to be available to other internet nameservers. However, I want certain subdomains (such as in.example.com to resolve to internal IPs, and therefore I don't want them available to other nameservers. For example, host.in.example.com might point to 192.168.x.x.

I accomplished this before in BIND by specifying an ACL for the IP range of my internal network, then adding an allow-query line to the in.example.com zone.

Is this approach bogus? Is there some alternative to ACLs in PDNS? Do these internal names need to be delegated to another server? It would be convenient to have in one place for administration.

I am currently working on a new authoritative nameserver using powerdns on Ubuntu 8.04 LTS. I'd like to have AppArmor protecting this service like it did with bind, but when I look in /etc/apparmor.d/, there was no AppArmor profile for this service installed by default.

Any experienced pdns admins know what all files pdns accesses, so I can define a profile? Or better yet, does anyone HAVE a profile for pdns?

Many thanks for any suggestions.