Chris Phillips's questions

We've been trying to get a grip on a really weird problem where we can wget a page from Apache 2.2.19 on solaris 10 and some permutations of requests reliably take various fixed lengths of time to respond.

It looks to be based around the closing of the TCP socket, from a tcpdump on the client, we typically see a pause between the server filling the TCP window with its response for the final time, and the final lump of data along with the FIN from the server.

So on the wire it just hangs mid transfer of the HTTP response, but a netstat on the server shows a socket in FIN_WAIT_1. we can't do a tcpdump on the server to clarify but it looks to us like the the OS has handed off the TCP conversation to the hardware, so believes it's started the 4way close, yet the NIC never puts that packet, nor the outstanding data packets (maybe 1 or 2 @ 1500 bytes and a straggler of 400 or whatever).

That's about as clean a picture as we can paint, apart from one recent test we did which looked amazing... a file of 64076 bytes is served by apache - 432ms delay for the last packet. we add ONE char to the file, and that caused an additional packet to be created, 53 bytes which also included the FIN, as opposed to the empty 52 byte FIN we see on the slightly smaller file, the one with the delay. The existence of this extra packet was reliably changing how the FIN conversation happened, and made the conversation go from almost half a second to a few milliseconds.

Most typically over the life of out investigations, this delay has been 4.6 seconds, again we see the window size fluctuate, SACK's going back to Apache when needed, but the LAST time that window gets full, it hangs for 46. seconds, and then back comes a final 2 or 3kb of data, and the FIN that Solaris thinks it sent out ages ago.

Our tcpdumps are on a F5 BigIP, so there is an ASIC for the traffic to traverse, as well as a Cisco 6509 (L2 only), however we do see the same user experience when doing a wget on a neighboring Solaris box, so don't believe it's any black magic that the BigIP is doing.

But this is all mixed up with confusion about window sizes and MSS's and the likes, but if this is sounding familiar to anyone, we're all ears!

I've a syslog-ng instance on 2.0.9, which is old, but... this is enterprise IT and upgrading the versions is... fun... running on Solaris 10. I have this strange problem where some clients stop being able to stay connected to the server on TCP.

When a client is working I can start syslog-ng on the client, it connects and sends data, and STAYS connected...

12:20:13.200547 IP (tos 0x0, ttl  64, id 13064, offset 0, flags [DF], proto: TCP (6), length: 60) 10.37.128.185.35765 > 10.37.141.31.shell: S, cksum 0xade4 (correct), 1572869826:1572869826(0) win 5840 <mss 1460,sackOK,timestamp 958735818 0,nop,wscale 7>
12:20:13.202279 IP (tos 0x0, ttl  63, id 27707, offset 0, flags [DF], proto: TCP (6), length: 64) 10.37.141.31.shell > 10.37.128.185.35765: S, cksum 0x434d (correct), 3180100791:3180100791(0) ack 1572869827 win 32942 <nop,nop,timestamp 2210148518 958735818,mss 1460,nop,wscale 2,nop,nop,sackOK>
12:20:13.202327 IP (tos 0x0, ttl  64, id 13065, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.128.185.35765 > 10.37.141.31.shell: ., cksum 0x0499 (correct), ack 1 win 46 <nop,nop,timestamp 958735820 2210148518>
12:20:13.202823 IP (tos 0x0, ttl  64, id 13066, offset 0, flags [DF], proto: TCP (6), length: 140) 10.37.128.185.35765 > 10.37.141.31.shell: P, cksum 0x179d (correct), 1:89(88) ack 1 win 46 <nop,nop,timestamp 958735820 2210148518>
12:20:13.204061 IP (tos 0x0, ttl  63, id 27708, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.141.31.shell > 10.37.128.185.35765: ., cksum 0x83d6 (correct), ack 89 win 32920 <nop,nop,timestamp 2210148518 958735820>
12:20:13.205558 IP (tos 0x0, ttl  64, id 13067, offset 0, flags [DF], proto: TCP (6), length: 124) 10.37.128.185.35765 > 10.37.141.31.shell: P, cksum 0xc071 (correct), 89:161(72) ack 1 win 46 <nop,nop,timestamp 958735823 2210148518>
12:20:13.206247 IP (tos 0x0, ttl  63, id 27709, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.141.31.shell > 10.37.128.185.35765: ., cksum 0x839d (correct), ack 161 win 32902 <nop,nop,timestamp 2210148518 958735823>

When a client is failing to stay connected, I see the server instantly disconnecting with a FIN...

12:20:02.441949 IP (tos 0x10, ttl  64, id 8231, offset 0, flags [DF], proto: TCP (6), length: 60) 10.37.128.185.46121 > 10.37.141.31.shell: S, cksum 0xeb7e (correct), 1553390564:1553390564(0) win 5840 <mss 1460,sackOK,timestamp 958725059 0,nop,wscale 7>
12:20:02.443817 IP (tos 0x0, ttl  63, id 27678, offset 0, flags [DF], proto: TCP (6), length: 64) 10.37.141.31.shell > 10.37.128.185.46121: S, cksum 0xe379 (correct), 3007391908:3007391908(0) ack 1553390565 win 32942 <nop,nop,timestamp 2210147442 958725059,mss 1460,nop,wscale 2,nop,nop,sackOK>
12:20:02.443840 IP (tos 0x10, ttl  64, id 8232, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.128.185.46121 > 10.37.141.31.shell: ., cksum 0xa4c5 (correct), ack 1 win 46 <nop,nop,timestamp 958725061 2210147442>
12:20:02.445689 IP (tos 0x0, ttl  63, id 27679, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.141.31.shell > 10.37.128.185.46121: F, cksum 0x2444 (correct), 1:1(0) ack 1 win 32942 <nop,nop,timestamp 2210147442 958725061>
12:20:02.445737 IP (tos 0x10, ttl  64, id 8233, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.128.185.46121 > 10.37.141.31.shell: F, cksum 0xa4c1 (correct), 1:1(0) ack 2 win 46 <nop,nop,timestamp 958725063 2210147442>
12:20:02.447244 IP (tos 0x0, ttl  63, id 27680, offset 0, flags [DF], proto: TCP (6), length: 52) 10.37.141.31.shell > 10.37.128.185.46121: ., cksum 0x2441 (correct), ack 2 win 32942 <nop,nop,timestamp 2210147442 958725063>

Now this issue was originally seen to be on different clients, but in this case it's the ery same box. I generated the successful messages by restarting the client syslog-ng service and the unsuccesful ones from a telnet to the server port.

I've also started a new instance of the syslog-ng server on a different port, and on localhost a telnet to 514 connects and disconnects...

 $ telnet localhost 514
 Trying 127.0.0.1...
 Connected to localhost.
 Escape character is '^]'.
 Connection to localhost closed by foreign host

but on the other port, on a new process we get a connection left open nicely...

 $ telnet localhost 1140
 Trying 127.0.0.1...
 Connected to localhost.
 Escape character is '^]'.
 ^]
 telnet
 quit
 Connection to localhost closed.

So something in syslog-ng or in Solaris 10 appears to take a disliking to SOME of these connections after an undefined period of time of the process running. This is linked against tcpwrappers, with "syslog-ng: ALL" defined in hosts.allow, and the behaviour i'm seeing looks similar to that which would happen if tcpwrappers was preventing the connection I think, but I don't think that that IS the part of the system at fault, as it seems to generic.

the "localhost to new process" behaviour looks to be the same as the remote connections, it doesn't look like a firewall getting in the way doing strange things or anything. And I'm lost.

Any guesses, pointers appreciated!

If I have a client trying to connect to a TCP server, and the client sends a couple of retransmits trying to establish a connection, is there ANY difference in the SYNs it will send? If the server SYN-ACKs on the 3rd packet, does the client have any indication as to which copy of the SYN packet the reply is for?

I've been advised by a vendor that an appliance we have will abort a connection attempt if it recieves a SYN-ACK for the original packet after it has retransmitted it, and this sounds impossible to me. This same guy also told be a retransmitted packet would have a different source port to the original, so I have pretty low confidence in anything he tells me!

I've a large F5 LTM load balancer environment I'm looking after which we are configuring to rate limit requests at a URI level. For human web users we are 302ing them to a busy page, but we also run web services and need to politely tell the soap client to go forth and procreate as well. Not knowing a whole heap about SOAP/XML responses I'm not clear on what we should tell the client. Will a very basic 500 reponse suffice or do we need to craft an legit SOAP response as well?

I've implemented a pair of openldap proxies running a few meta databases to merge and filter an AD DC cluster. I'm having a few issues around the clients failing over between boxes when one service is not functioning. One example was when a central syslog server was down and slapd couldn't send logs to it, so whilst the TCP socket opened, the process would not respond as it was waiting to clear a backlog of syslog messages first. Here I'd have expected the clients to fail to the second server, but they didn't, despite reporting that they were connecting to it:

2011-10-10T11:45:01.220367+01:00 gibsvlin-mkt-product worker_nscd: nss_ldap: could not search LDAP server - Server is unavailable
2011-10-10T11:45:01.231725+01:00 gibsvlin-mkt-product worker_nscd: nss_ldap: could not search LDAP server - Server is unavailable
2011-10-10T11:45:01.235354+01:00 gibsvlin-mkt-product worker_nscd: nss_ldap: could not search LDAP server - Server is unavailable
2011-10-10T11:45:01.242156+01:00 gibsvlin-mkt-product worker_nscd: nss_ldap: reconnected to LDAP server ldap://10.5.10.117:389/
2011-10-10T11:45:01.248505+01:00 gibsvlin-mkt-product worker_nscd: nss_ldap: could not search LDAP server - Server is unavailable

So we see the "reconnected to" message for the backup server, yet it doesn't seem to actually ever get anything out of it. If I suspend down the VM that is fudged up, so no TCP connectivity is possible, then everything fails over nicely.

It feels like there is a subtly in the failure logic that would sort this out, but I'm unable to tune it nicely, assuming that is the solution. The matching client ldap.conf reads:

scope sub
ldap_version 3
nss_base_passwd          dc=domain,dc=local?sub?&(uidNumber=*)
nss_base_group          dc=domain,dc=local?sub?&(gidNumber=*)
nss_initgroups_ignoreusers root,ldap,dbus,xfs,haldaemon,nscd,nocpulse
bind_timelimit 1
timelimit 5
idle_timelimit 5
nss_reconnect_tries 3
nss_reconnect_sleeptime 1
nss_reconnect_maxconntries 3
bind_policy soft
uri ldap://10.3.110.117:389/ ldap://10.5.10.117:389/
base dc=bwinparty,dc=local
nss_initgroups backlink
pam_login_attribute uid
ssl no

It seems strange that whilst nscd etc are connecting to the 2nd server, we can not log in. Previously the bind_timelimit and timelimit were both 5, which to me suggested that if it ran out of time binding, then there was no time left to do anything else within the over all timelimit window. No improvement was noticed on that tweak though.

Has anyone come across the use of this config for passwd and groups config in nsswitch.conf? Where I'm working I've been told it's been shown to help situations where a group exists both locally and in ldap which was causing issues for group memberships etc. However this config seems to totally mess up nscd which will be aware of the groups and all their members but will not flip the data around to say the user is a member of all it's remote groups.

Initially it seems, given a fully available environment, to be exactly the same as [FOUND=return] which is an implict default between stages anyway. However apparently a lengthy ticket with Redhat resulted in the recommended use of that configuration.

Is something different about the requirements of GPG key formats in RHEL6 that is stopping my standard GPG key being imported? it installs fine on RHEL4 and RHEL5 but not RHEL6, citing the key import failed. I've also noticed that some repos like EPEL have a specific EPEL6 key, suggesting again something is different, but I can't find reference to what that is.

We have a j2ee webapp on tomcat which is using ehcache with multicast discovery. Except it's not discovering anything. There appears to to be no multicast traffic visible on the network but we're a little unclear how to really troubleshoot it. We have a few positive things so far...

if I tcpdump for the relevant multicast IP address on one device and ping it from the other then I see the echo requests coming in.

if we run a simple java multicast listener tool that our web developers provided then it can see the multicast requests from the machine that it is on.

But the negatives also...

tcpdumping we see no other traffic on that address at all, even whilst seeing this java tool printing periodic multicast summaries "tcpdump -vn -i eth3 ether multicast" still shows nothing, where eth3 is the nic on which the route for 224.0.0.0/4 lives. There is no routing / IGMP involved, all traffic is on a single vlan, upon which unicast traffic is flowing fine.

we see no other traffic on the java tool, only the local machine.

It may be relevant that these are CentOS VM's running on a series of ESX 4.1 systems. We intend, come the morning, to install systems on the same ESX machine and see if, when using the vSwitch alone, we see communications there, but as the levels of multicast specific knowledge is really pretty low compared to normal unicast knowledge, we're kinda stumped. There appears to be nothing relevant to do on either the cisco switches or the esx level networking from what we are aware of, yet the multicast discovery is just not happening.

If anyone can give pointers to useful tools for shooting this, if not outright pointers for resolution, it'd be really appreciated.

I have a situation where I need to mount a few NFS shares on a series of clients on replicated VM networks. So their source IP addresses are not unique. In between the client and the server there is a routing Linux VM, which can do all sorts of things as deemed appropriate. I'm really unsure how best to approach making these shares reachable.

Possible solutions:

1. Fully route traffic end to end, with an iptables masquerade on the "router" to Source NAT all client traffic.
2. Do Masquerading AND Destination NAT on the router, making the internal machines think they are going to the "router" for NFS. This makes routing tables much simpler as everything then appears to be local. Maybe using a secondary IP on the "router" for clarity with all the NFS ports in use.
3. Mount and then re-export NFS on the "router" VM. This feels really bad, but would give extra control to only allow certain mounts to be mounted by certain clients. As above, any NAT solution means all end clients look to be the same under NFSv3, which could cause security concerns. This also makes the routing tables a non issue. I think I saw comment that performance would significantly drop this way.

Any other options / thoughts gratefully appreciated.

Is it possible to use vSwitches on ESX 4 to allow multiple identical private subnets to be natted out to an external subnet?

I'm used to the logic of natting a single private subnet out to a routable subnet on a physical network, but can I configure something to take, say 5 port groups, all in their own vSwitches and all using the same /24 subnet and end up with all appropriate outbound traffic from ALL of these subnets down to a single IP representative of the physical ESX machine they are coming from?

I'd assume that as a NAT is occurring there is a suitable managed state table, meaning that a response packet from a physical remote machine will be put back on the right originating subnet even though that IP might exist on 4 other VM subnets.

Thanks

Chris

I'm looking at implementing a systems using a number of tagged and a native vlan connected to a server over a a/p bonded interface. The untagged vlan is for physical machine access, the tagged vlans are connected to bridges and then to QEMU VM's inside the machine. Hopefully this plan is fine, but I'm trying to implement a crippled version of this in a dev environment due to a lack of underlying network config in this location where I just have the same single vlan delivered to the machine on a tag AND plain. I'm nto clear if this is going to work (and that I should just be confident that it will work using different vlans) as I'm seeing odd things like a vm is arping out over the vlan out to the core switch, but the arp reply is coming back on the untagged interface. Now an ARP reply is unicast right? So it's a deliberate thing to send the ARP response on the untagged interface, and not a case that a broadcast response isn't being passed on the tagged side... i.e. there's some underlying logic pushing it that way. Something about the MACs somehow? This is on a CentOS 5.5 machine, vlan's from vconfig. (I've seen reference to the Linux mac-vlan project work, but that's not available here by default.)

so 1) Should having the SAME vlan tagged and untagged work?

2) Will different tagged vlans to the untagged interface work nice and easily?

I'm trying to get a vlan interface on a bonded nic (Centos 5.5) and whilst the interface has been very happily created with vconfig I'm seeing no traffic on it at all. Running tcpdump and tshark on the underlying eth0 I see no sign at all of vlan tags in the traffic, and I'm wondering if there's somethign I'm missing on the server side as the network dept say they are sending me the tagged data. I've got the 8021q module loaded, however under lsmod it shows it's only being used by the cxgb3 module, for an unused onboard iSCSI card, whereas my nics (on an HP DL380 G7) are driven by bnx2 and e1000e modules. Should these modules be listing 8021q as used module? should I have something conrete in /etc/modprobe.conf?

Thanks

Chris

I've a KVM system upon which I'm running a network bridge directly between all VM's and a bond0 (eth0, eth1) on the host OS. As such, all machines are presented on the same subnet, available outside of the box. The bond is doing mode 1 active / passive, with an arp_ip_target set to the default gateway, which has caused some issues in itself, but I can't see the bond configs mattering here myself.

I'm seeing odd things most times when I stop and start a guest on the platform, in that on the host I lose network connectivity (icmp, ssh) for about 30 seconds. I don't lose connectivity on the other already running VM's though... they can always ping the default GW, but the host can't. I say "about 30 seconds" but from some tests it actually seems to be 28 seconds usually (or at least, I lose 28 pings...) and I'm wondering if this somehow relates to the bridge config.

I'm not running STP on the bridge at all, and the forwarding delay is set to 1 second, path cost on the bond0 lowered to 10 and port priority of bond0 also lowered to 1. As such I don't think that the bridge should ever be able to think that bond0 is not connected just fine (as continued guest connectivity implies) yet the IP of the host, which is on the bridge device (... could that matter?? ) becomes unreachable.

I'm fairly sure it's about the bridged networking, but at the same time as this happens when a VM is started there are clearly loads of other things also happening so maybe I'm way off the mark.

Lack of connectivity:

# ping 10.20.11.254                                          
PING 10.20.11.254 (10.20.11.254) 56(84) bytes of data.                          
64 bytes from 10.20.11.254: icmp_seq=1 ttl=255 time=0.921 ms                    
64 bytes from 10.20.11.254: icmp_seq=2 ttl=255 time=0.541 ms                    
type=1700 audit(1293462808.589:325): dev=vnet6 prom=256 old_prom=0 auid=42949672
95 ses=4294967295                                                               
type=1700 audit(1293462808.604:326): dev=vnet7 prom=256 old_prom=0 auid=42949672
95 ses=4294967295                                                               
type=1700 audit(1293462808.618:327): dev=vnet8 prom=256 old_prom=0 auid=42949672
95 ses=4294967295                                                               
kvm: 14116: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079               
kvm: 14116: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffdd694a              
kvm: 14116: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079               
64 bytes from 10.20.11.254: icmp_seq=30 ttl=255 time=0.514 ms                   
64 bytes from 10.20.11.254: icmp_seq=31 ttl=255 time=0.551 ms                   
64 bytes from 10.20.11.254: icmp_seq=32 ttl=255 time=0.437 ms                   
64 bytes from 10.20.11.254: icmp_seq=33 ttl=255 time=0.392 ms 

brctl output of relevant bridge:

# brctl showstp brdev
brdev
 bridge id      8000.b2e1378d1396
 designated root    8000.b2e1378d1396
 root port         0            path cost          0
 max age          19.99         bridge max age        19.99
 hello time        1.99         bridge hello time      1.99
 forward delay         0.99         bridge forward delay       0.99
 ageing time         299.95
 hello timer           0.50         tcn timer          0.00
 topology change timer     0.00         gc timer           0.04
 flags          


vnet5 (3)
 port id        8003            state            forwarding
 designated root    8000.b2e1378d1396   path cost        100
 designated bridge  8000.b2e1378d1396   message age timer      0.00
 designated port    8003            forward delay timer    0.00
 designated cost       0            hold timer         0.00
 flags          

vnet0 (2)
 port id        8002            state            forwarding
 designated root    8000.b2e1378d1396   path cost        100
 designated bridge  8000.b2e1378d1396   message age timer      0.00
 designated port    8002            forward delay timer    0.00
 designated cost       0            hold timer         0.00
 flags          

bond0 (1)
 port id        0001            state            forwarding
 designated root    8000.b2e1378d1396   path cost         10
 designated bridge  8000.b2e1378d1396   message age timer      0.00
 designated port    0001            forward delay timer    0.00
 designated cost       0            hold timer         0.00
 flags          

I do see the new port listed as learning, but in line with the forward delay, only for 1 or 2 seconds when polling the brctl output on a loop.

ifconfig without sample VM:

bond0     Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          inet6 addr: fe80::d685:64ff:fe65:fa4e/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:21168629 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9280285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8777768179 (8.1 GiB)  TX bytes:2671736365 (2.4 GiB)

bradSP1   Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1656 (1.6 KiB)  TX bytes:6592 (6.4 KiB)

brawSP1   Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4996 (4.8 KiB)  TX bytes:6592 (6.4 KiB)

brdev     Link encap:Ethernet  HWaddr B2:E1:37:8D:13:96  
          inet addr:10.20.11.129  Bcast:10.20.11.255  Mask:255.255.255.0
          inet6 addr: fe80::d685:64ff:fe65:fa4e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16663718 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8800468 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3268513274 (3.0 GiB)  TX bytes:2587834869 (2.4 GiB)

brmgtSP1  Link encap:Ethernet  HWaddr 1A:CA:AE:08:1C:42  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:699322 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:928721301 (885.6 MiB)  TX bytes:6706 (6.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:20412120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9280285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8720799421 (8.1 GiB)  TX bytes:2671736365 (2.4 GiB)
          Interrupt:169 Memory:f4000000-f4012800 

eth1      Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:756509 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:56968758 (54.3 MiB)  TX bytes:0 (0.0 b)
          Interrupt:186 Memory:f2000000-f2012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3937 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3937 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6641553 (6.3 MiB)  TX bytes:6641553 (6.3 MiB)

vnet0     Link encap:Ethernet  HWaddr B2:E1:37:8D:13:96  
          inet6 addr: fe80::b0e1:37ff:fe8d:1396/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:59861 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5924530 errors:0 dropped:0 overruns:2 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:6405635 (6.1 MiB)  TX bytes:1987480170 (1.8 GiB)

vnet1     Link encap:Ethernet  HWaddr 1A:CA:AE:08:1C:42  
          inet6 addr: fe80::18ca:aeff:fe08:1c42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:541798 errors:0 dropped:0 overruns:0 frame:0
          TX packets:61998 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:802746110 (765.5 MiB)  TX bytes:6498514 (6.1 MiB)

ifconfig with sample VM:

bond0     Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          inet6 addr: fe80::d685:64ff:fe65:fa4e/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:21285120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9291457 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8948482155 (8.3 GiB)  TX bytes:2673235824 (2.4 GiB)

bradSP1   Link encap:Ethernet  HWaddr 2A:18:E1:2D:1A:EC  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1656 (1.6 KiB)  TX bytes:6592 (6.4 KiB)

brawSP1   Link encap:Ethernet  HWaddr 96:55:AA:14:67:07  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:109 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4996 (4.8 KiB)  TX bytes:6592 (6.4 KiB)

brdev     Link encap:Ethernet  HWaddr 16:5C:BC:E5:90:11  
          inet addr:10.20.11.129  Bcast:10.20.11.255  Mask:255.255.255.0
          inet6 addr: fe80::d685:64ff:fe65:fa4e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16673094 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8801611 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3279365967 (3.0 GiB)  TX bytes:2587927761 (2.4 GiB)

brmgtSP1  Link encap:Ethernet  HWaddr 1A:CA:AE:08:1C:42  
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:699342 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:928723605 (885.6 MiB)  TX bytes:6706 (6.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:20528382 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9291457 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8891497316 (8.2 GiB)  TX bytes:2673235824 (2.4 GiB)
          Interrupt:169 Memory:f4000000-f4012800 

eth1      Link encap:Ethernet  HWaddr D4:85:64:65:FA:4E  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:756738 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:56984839 (54.3 MiB)  TX bytes:0 (0.0 b)
          Interrupt:186 Memory:f2000000-f2012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3937 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3937 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6641553 (6.3 MiB)  TX bytes:6641553 (6.3 MiB)

vnet0     Link encap:Ethernet  HWaddr B2:E1:37:8D:13:96  
          inet6 addr: fe80::b0e1:37ff:fe8d:1396/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:69818 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6034715 errors:0 dropped:0 overruns:2 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:7763947 (7.4 MiB)  TX bytes:2149238089 (2.0 GiB)

vnet1     Link encap:Ethernet  HWaddr 1A:CA:AE:08:1C:42  
          inet6 addr: fe80::18ca:aeff:fe08:1c42/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:650557 errors:0 dropped:0 overruns:0 frame:0
          TX packets:72519 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:964153780 (919.4 MiB)  TX bytes:7896728 (7.5 MiB)

vnet2     Link encap:Ethernet  HWaddr AA:4B:22:76:D2:EC  
          inet6 addr: fe80::a84b:22ff:fe76:d2ec/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10521 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108765 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:1398214 (1.3 MiB)  TX bytes:161408138 (153.9 MiB)

vnet3     Link encap:Ethernet  HWaddr 96:55:AA:14:67:07  
          inet6 addr: fe80::9455:aaff:fe14:6707/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

vnet4     Link encap:Ethernet  HWaddr 2A:18:E1:2D:1A:EC  
          inet6 addr: fe80::2818:e1ff:fe2d:1aec/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

vnet5     Link encap:Ethernet  HWaddr 16:5C:BC:E5:90:11  
          inet6 addr: fe80::145c:bcff:fee5:9011/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:241 errors:0 dropped:0 overruns:1 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 b)  TX bytes:47167 (46.0 KiB)

All pointers, tips or stabs in the dark appreciated.

I'm working in an environment that is using a subversion server in a central location, which needs to feed its contents out to 4 different environments (prod, test etc...) Within these environments we do not need subversion functionality at all, merely to be able to access the contents of all /tags/ held within the repo by http and local filesystem.

To do this, we have two possible approaches:

1) subversion mirrors in all environments, updated using svnsync and then made available within each location via svn -> web_dav_svn -> davfs (FS access here) -> apache (HTTP access here)

2) svn export of /tags in a central location and then rsyncing the files out to the other locations, possibly after having hardlinked the previously exported file structure.

Whilst option 1 seems "slicker" in many ways (although going through apache twice???) it pushes lots of the risk down towards a live running environment, rather than keeping it back in the central location, which is essentially "offline" insofar as it never being used in the running of these environments. I'd also be concerned about putting a pretty much disposable and ignored fuse project like davfs so deep into the project. Rsync is boring and stable, and seeing as all we want is file access under our own terms, we can get that with a really basic and reliable toolset.

Any thoughts really appreciated.

Is anyone aware of a way to access all revisions of a subversion repository within a file system? I would really benefit from being able to reference a file locally as, for example, "/var/svn/repo/rev4235/project/trunk/file.c" or such like.

I can fairly straightforwardly write some scripts to manually stage different versions of files out of subversion in a manually created directory tree or something, but directly reaching into svn would be much more preferable.

[aside: I'm aware of svnfs but this only presents the current revision, and whilst I've been able to create the exact functionality I need on apache, I can't see any way to maybe mount that web server e.g. httpfs (needs file range access which I don't think is possible infront of a mod_dav_svn service) or mod_dav & davfs etc... (can I provide read only mod_dav access infront of mod rewrite infront of mod_dav_svn??!?!)]

Thanks

Chris

I'm looking for a tool, preferably a minimal Linux build, which can, or fairly simply be modified to, boot as an ISO image and automatically download a disk image over a network and apply it to a blank disk.

In actual fact, I'm fine dealing with the actual scripting to make this happen, but I need at least a base image which will have network access, curl, dd etc, to be of use to me. I'm not looking for any flexibility of choice here, I need it to be totally hands free and dumb, but need to achieve this tasks with as little dev work as I can manage. I think I could possibly just make my own initrd or such, but I've no idea how that works with having network access etc.

[background]This is actually for use with KVM virtual machines, which will eventually be running W2K3, but to deliver the disk image using a wider build framework of cobbler and koan I would rather boot some installation media which will autonomously install a system and reboot when done, allowing a similar higher level flow as installing Redhat with a kickstart script (but without the external resources etc...), so I don't want to pre deliver an image to the host etc, as that's done for a kickstart[/background]

Thanks

Chris

EDIT: So whilst the solutiosn given are valid in their own right, I really needed to NOT have a self contained solution, and needed only a few pieces to integrate into cobbler / koan etc...

What I've currently done is to use a modified kickstart script to initially boot as if to install CentOS but then in the %pre section is just do a wget to download a disk image and smear it across the virtual disk. on reboot, one windows box! Works much better than I thought and also keeps the deployment flow and wrapper scripting totally standardized.

I've webapp on tomcat6 which is connecting to an M$ PlayReady DRM instance on IIS6.0 The performance is seen to be best when we bench mark (using ab) the DRM service with 25 concurrent connections, which gives about 250 requests per second, which is ace. higher concurrent connections results in TCP/IP timeouts and other lower level mess. But there is no way to control how the tomcat app connects to the service - it's not internally managing a pool of connections etc, they are all isolated http connections to the server. Ideally I'd like a situation where we can have 25 http 1.1 connections being kept alive permanently from tomcat and requesting the licenses through this static pool of connections, which I think would the best performance. But this is not in the code, so was looking for a way to possibly simulate this at the Linux level. I was possibly thinking that iptables connlimit might be able to gracefully handle these connections, but whilst it could limit, it'd probably still annoy the app. What about a proxy? nginx (or possibly squid) seems potentially appealing to run on the tomcat server and hit on localhost as we might want to add additional DRM servers to use under load balance anyway. Could this take 100 incoming connections from tomcat, accept them all and proxy over the the IIS server in a more respectful manner? Any other angles?

EDIT - looking over mod_proxy for apache, which we are already using for conventional use on an apache instance in front of this tomcat instance, might be ideal. I can set a max value on the proxy_pass to only allow 25 connections, and keep them alive permanently. Is that my answer?

Many thanks,

Chris

Howdy cowboys and cowgirls,

If I have a VM (either KVM or ESXi) serving static content on Apache and a video streaming webapp on Tomcat, is there a logic in running multiple instances on the same VM on the same piece of kit and load balancing them? To me it seems conceptually pointless as when taking the same incoming web requests they will merely share the resources that would have otherwise been dedicated to a single instance, however I can imagine scenarios whereby a higher capacity can be leveraged by using 2, 3 or more identical VM's, maybe along the lines of threading performance within tomcat or such like, however any reason I think of tends to imply bad coding and workaround territory not best practise design. Example hardware here is a fairly capable box like a HP DL380 with 8 or 12 cores and 64GB of RAM serving around 4000 concurrent media connections, one way or another.

Update: In terms of other benefits like redundancy and patching, these are not issues as this scenario is likely to be replicated on upto 100 physical machines, all load balanced.

Update2: I also have concerns in my head about the ability to load balance multiple identical services from an external LB. if you are monitoring connection latency and such from the LB then it should be the case that 2 vm's - 1 with 5 connections, one with 500 should actually appear to be functioning identically as they are both pulling resource from the same pool (without VM CPU pinning etc.). Hammering one box would also cripple the other quiet one, so make the distribution of connections really abnormal and confusing.

Thanks

Chris

I'm working on a design for a Linux based appliance and looking to use KVM for the virtualisation layer. The system has 3 VM's interconnected in very specific ways. The fag packet diagram I've been given looks like this...

       www
        |
+------ | ------------+
|       |             |
|      |A|            |
|       |             |
|       |             |
|      |B|-----|C|    |
|       |             |
+------ | ------------+
        |
       dbs

This basically mirrors the data flow - a web server at the front with a back end server behind it, and a subsidiary service on the side. A and C must be on "separate" networks due to internet proximity etc. This is fine for the bulk of the system, but this means that all management traffic for vm's A and C would need to pass through vm B to reach an external syslog server or such. Similarily for ssh access in, we'd need something like port forwarding on B. Whilst the volume of this traffic is minimal compared to the application traffic, it feels pretty horrible, especially as the automated building of these vm's using kickstart or some such, mean that we can't start building A or C until B is finished.

So I'm thinking about making it all a lot more confusing with something like this:

     www
      |
+---- | ----------------------+
|     |                       |
|     |       +-----------+   |
|     |       |           |   |
|    |A|-----|B|-----|C|  |   |
|     |       |       |   |   |
|     |       b       |   |   |
|     +-----a x c-----+   |   |
|             d           |   |
|             |           |   |
+------------ | --------- | --+
              |           |
             mgt         dbs

So the data flow between A B and C for the app is largely identical (but drawn flat here) and we add in a second management nic to each VM connected to another bridge which also has an interface on the host (a, b and c). iptables and forwarding is then used between these virtual host interfaces and the outside world (d) to ensure that whilst all machines can directly reach external services lower down in the environment, they can't reach each other in a way that means compromise of vm A does not reduce the security to any larger extent. I could probably use ebtables and a single bridge in my additions but I think it'd look too subtle with A and C being on the same subnet still.

It's a fair cop to note that there's a significant angle on what things look like on a diagram for management as well as the actual technical credibility of things (hence "separate" vs separate)! Simplicity is also a huge motivator, but at the cost of chucking all additional traffic through a 2nd VM feels real uncomfortable, as well as adding maybe 5 to 10 minutes for the build in a time sensitive environment

So essentially, am I nuts?

Thanks

Chris