Julien Vehent's questions

We have a simple multi-homed setup with two routers that advertise our AS to two ISP. The second ISP (ISP B) is only used for backup when ISP A goes down, so we prepended our AS 3 times on this route.

Note: These are example ASN, not the real ones.

I spend a couple of hours this morning poking at looking glass routers all over the internet, and none of them list our backup route with the prepended path. I checked the south african internet exchange, the london internet exchange, oregon internet exchange and a couple dozen ISPs. All of them have multiples routes through ISP A, often with 3 or 4 hops. The route through ISP B should, at least, appear somewhere and have 5 or 6 hops. But I couldn't find it anywhere. (I checked the full bgp tables on the looking glass routers, using show ip bgp 65000)

My questions are:

• Is there a limit to the size of a route after which most routers will simply discard the route ?
• Is our backup route even going to work when ISP A goes down, if no router knows about it ?

Our two routers are connected on iBGP. Would it be possible that the route through ISPB is not announced because the iBGP session prefers the route through ISPA ? This is what non-exist-map and advertise-map are supposed to do, but none of those are used in either routers.

A user located on the other side of the planet has troubles connecting to our PPTP VPN. The VPN is a regular windows server 2003 VPN and works fine for most users from the US, but for this specific person, it doesn't connect at all.

The login window hangs on "verifying username/password" and dies after ~30 seconds. I took a network trace on the server side, and this is what I see. 10.250.10.10 is the internal IP of the server, 121.X.X.255 is the public IP of the user.

Apparently there is no new network equipment on the user side, and definitely nothing has changed on the server side.

Any idea on how to diagnose this ?

I have Oracle10g running on Solaris 10. The configuration of the OS is standard, and there is a packet filter firewall running on the server.

When I run a long query using sqldeveloper, the server reset the connection after 15 minutes of inactivity.

The timeline is:

1. client establishes the connection to server port 1521, no ssl
2. server accept and receives the query from the client. the last packet exchanged is a TCP ACK from the server to the client
3. ... 15 minutes (to the second) pass
4. server send a RST packet to the client, and nothing else is exchanged between the hosts afterward

I couldn't find any timeout parameter for TNS listener, so I'm guessing it might be a solaris or packet filter issue ?

My enterprise manager instance on Oracle 10.2.0.4 throws the following alert on a regular basis:

Name=db4
Type=Host
Host=db4
Metric=Disk Utilization (%)
Disk Device=sd1
Timestamp=Oct 4, 2011 2:15:50 PM GMT
Severity=Critical
Message=Disk Utilization for sd1 is 96.12%
Rule Name=Host Availability and Critical States
Rule Owner=SYSMAN

The DB is hosted on Solaris 10. The datafiles are stored on /home and the binaries on /opt. And there is free space on both volumes...

-bash-3.00$ df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c0t0d0s0 9.9G 4.3G 5.4G 45% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 38G 968K 38G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1
9.9G 4.3G 5.4G 45% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/dsk/c0t0d0s3 20G 355M 19G 2% /var
swap 38G 152K 38G 1% /tmp
swap 38G 28K 38G 1% /var/run
/dev/dsk/c0t0d0s4 20G 13G 6.8G 66% /opt
/dev/dsk/c0t1d0s0 825G 611G 205G 75% /home

Is this alert from EM a false positive ? A bug that I should patch ? The really weird thing is that I have another database running on the same server, with EM installed, and that secondary EM doesn't throw the error.

My company is launching a new website with potentially large waves of visitors in very short windows (estimate is around 14k visitors in a 2 minutes window).

So, I'm reviewing our configuration, and my biggest problem right now is our single node HTTP frontend that uses keep-alive. The frontend is running lighttpd 1.4 on CentOS 5.4.

Some assumptions:

• a browser usually opens 6 parallels TCP connections to keep-alive
• the browser will keep the connection open until the timeout is reached, even if the tab is closed (observed in FF, might not be true on every browser)
• on the server side, each connection will consume ~150K of memory in the kernel (I use conntrack and want to keep it, is that estimation correct ?)
• all of our servers are hosted on the east coast. the RTT from a server in las vegas is around 80ms.
• The home page with keep-alive uses ~25 TCP connections and 1500 packets. Without keep-alive, this number rises to ~210 TCP conenctions and over 3200 packets.

So, 6*14000 = 84,000 TCP connections. 84,000 * 150KB ~= 12GB of memory. Here is the problem: 1. I don't have that amount of memory available on the front end. 2. lighttpd 1.4 is not very comfortable with that amount of connections to manage. it hurts the hits/s a lot.

But on the other end, I'm concerned about the 80ms RTT if I deactivate keepalive.

I am going to mitigate some of these issues with a CDN and a secondary www record with a secondary lighttpd. but the debate concerns the keep-alive feature. I'd like to turn it off, but I'm worried that the impact on page opening time is going to be high (high RTT, and double the amount of packets).

Once of the content retrieval done, we have a lot of ajax requests for browsing the site that usually fit in a single tcp connection. But I'm not certain that the browser will free the other connections and just keep one open.

I know there have been a number of discussion about keep-alive consuming to much resources. I kind of agree with that, but given the assumptions and the situation (a RTT between 80ms and 100ms for half our users), do you think it's wise to deactivate it ?

As a side question: do you know where I can find the information regarding connection size and conntrack size in the kernel ? (other than printf size_of(sk_buff) ).

--- edit: some test results I configured conntrack to accept 500k connections (given the memory footprint, it shouldn't exceed 200MB) and launch an ab test.

ab -n 20000 -c 20000 -k http://website.com/banner.jpg

From what I saw in tcpdump, ab establishes all connections before doing the GET. So I get an idea of how much memory is consumed by those 20k connections.

slabtop returns

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
 40586  40586 100%    0.30K   3122       13     12488K ip_conntrack

and top

 PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP CODE DATA COMMAND
 15   0  862m 786m  780 S 22.9  3.3   1:44.86  76m  172 786m lighttpd

12MB for ip_conntrack and 786MB for lighttpd are OK for my setup. I can easily manage 4x that.

So, keepalive that is, with a idle timeout set to 5 seconds.

I'm looking for advices regarding VoIP products. I need to build an in-house VoIP PBX for one of my company's office, and while I'm fairly sure we will go with Asterisk (still leaving the door open to OpenSIPS), I'm not certain I want to build it myself, or buy an appliance.

We want around 30 parallel lines, and do the convertion VoIP to analog before it leaves the office (no hosted PBX or anything like that, we don't trust the network bandwidth enough, it's just a basic DSL there).

One of the requirement is to be able to customize the system. We want to plug it into our Active Directory and CRM system, extract statistics, conversation and voicemail to audio files, etc...

I've looked at Digium, they have interesting prices (cost is an issue, of course). But does it leave the door open to customization ? Do you have access to the database, the logs, the system in general ?

What else is there on the market ?

It's kind of difficult to select an answer, since I would asking for advice and not to solve a specific problem. So pardon me if I wait a little more before I award the answer to one of you.

I'm wondering one thing. I'm playing with an EC2 instance and would like to use some local subnet in the 192.168.1.0/29 range. So I've setup aliases on eth0 as follow:

[root@server jvehent]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 12:25:8B:0a:B1:EB  
          inet addr:10.28.118.95  Bcast:10.28.118.255  Mask:255.255.254.0
[...]
eth0:1    Link encap:Ethernet  HWaddr 12:25:8B:0a:B1:EB    
          inet addr:192.168.1.1  Bcast:192.168.1.7  Mask:255.255.255.248
[...]
eth0:2    Link encap:Ethernet  HWaddr 12:25:8B:0a:B1:EB    
          inet addr:192.168.1.2  Bcast:192.168.1.7  Mask:255.255.255.248
[...]
eth0:3    Link encap:Ethernet  HWaddr 12:25:8B:0a:B1:EB    
          inet addr:192.168.1.3  Bcast:192.168.1.7  Mask:255.255.255.248
[...]
eth0:4    Link encap:Ethernet  HWaddr 12:25:8B:0a:B1:EB  
          inet addr:192.168.1.4  Bcast:192.168.1.7  Mask:255.255.255.248
[...]
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0

Very well, so I get the following routing table

[root@server jvehent]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0    0.0.0.0         255.255.255.248 U     0      0        0 eth0
10.28.118.0     0.0.0.0         255.255.254.0   U     0      0        0 eth0
0.0.0.0         10.28.118.1     0.0.0.0         UG    0      0        0 eth0

Now, If I want two services to talk to each other using the local aliases, I can set netcat as follow:

listening netcat:

nc -l 192.168.1.4 1664

connecting netcat:

nc -s 192.168.1.1 192.168.1.4 1664

And, from what I see, that traffic goes through the localhost interface. I'm checking with tcpdump -i lo and I see packets going through. Nothing on eth0.

My basic knowledge of the kernel tell me that since the kernel owns all the ip, it does all the operations internally and copy the packets from a socket to another without passing it to the NIC. correct ?

Is there a way to force the kernel to send those packets on the network ?

I'm considering using LVM striping over several EBS volumes. So let's imagine the following case: Volume group DATA is created with 3 physical disks sdf1, sdg1 and sdh1. I create a logical volume "store" inside DATA and enable striping on this, with 3 stripes and a stripesize at 128KB so I split my blocks across the 3 physical volumes.

Now, I want to add a 4th physical volume to grow the size of the volume group and logical volume. I add my physical volume to DATA, increase the size of "store" and change the number of stripe from 3 to 4.

What happens to my "store" volume ? Do the previously existing blocks get re-organized ? Otherwise, if my 3 previous physical volumes were full, only the 4th one will be used to store the new blocks, and there would be no striping.

I am migrating a bunch of mailboxes to google apps. Each user owns several mailboxes each serving different domains (john has [email protected], [email protected], and so on...)

Currently, those accounts are hosted on (edit:NOT an exchange server) an old SMTP/POP server we want to replace, and I need to edit their outlook 2007 configuration to change the pop, smtp and password parameters.

The hard way to do it is to connect to each outlook session and edit the parameters manually. I want to avoid that. Because that represents over 700 accounts spread between 40 users... :'(

How can I automate this configuration change ? In the active directory ? Using a PRF file ?

note: I'm a linux sysadmin with very little knowledge of windows's black magic.

One of my hosted server has some XFS problems. after the last crash, some of my RRD folders got corrupted.

example (sorry, it's in french):

# rm *
rm: impossible de supprimer « create_rrd.sh »: La structure a besoin d'un nettoyage
rm: impossible de supprimer « old »: est un dossier
rm: impossible de supprimer « tcgraph.log »: La structure a besoin d'un nettoyage
rm: impossible de supprimer « tcgraph.rrd »: La structure a besoin d'un nettoyage

The normal move would be to restart the system in single user, or using a live cd, and run a xfs_repair on /dev/sda. Unfortunately (it would be too easy), the hosting company provides an option to restart on a live cd, that doesn't work. And visiting the datacenter is not an option.

It seems that I can't touch the inodes at all, I get that "structure needs a cleanup" message everytime.

So, question is: does anyone know a way to repair/fix that XFS filesystem by hand ? Any low-level XFS manipulation tool that could help ?

I'm running into a CSS problem. This is a setup based on Debian Squeeze (nginx/0.7.67, php5/cgi) on which I installed the latest Roundcube 0.5 beta.

PHP is properly processed, login works fine but the CSS files are not loaded and Firefox is throwing the following errors:

Error: The stylesheet https://webmail.example.net:10443/roundcube/skins/default/common.css?s=1290600165

was not loaded because its MIME type, "text/html", is not "text/css". Source File: https://webmail.example.net:10443/roundcube/?_task=login Line: 0

Error: The stylesheet https://webmail.example.net:10443/roundcube/skins/default/mail.css?s=1290156319

was not loaded because its MIME type, "text/html", is not "text/css". Source File: https://webmail.example.net:10443/roundcube/?_task=login Line: 0

As far as I understand, nginx doesn't see the .css extension (because ofthe ?s= argument) and thus set the mime type with the default value, being text/html.

Should I fix this in nginx (and how ?) or is it roundcube's related ?

Edit: It seems that it's nginx related. The content-type isn't set for any other type than text/html. I had to include manually the following declarations to force CSS and JS content-types. That's ugly, and I never had the problem before... any idea ?

location ~ \.css {
    add_header  Content-Type    text/css;
}
location ~ \.js {
    add_header  Content-Type    application/x-javascript;
}