Kyle's questions

After reading the fine advice and accepted answer at Stunnel too many clients , I'm finding that this looks like our problem, but I'm having trouble applying the solution.

First of all, ulimit is a shell thing that impacts processes spawned from that shell (from what I understand).

So I'm reviewing /etc/security/limits.conf and added this line:

# stunnel resets 2011-03-03 kdh
*       hard    nofile  65536

And issued a /etc/init.d/stunnel4 restart, but I'm still seeing the dreaded too many connections message:

[admin@p2378442 ~]$ sudo tail -f /var/log/stunnel.log |grep "too many"
2011.03.03 13:43:07 LOG4[8461:3086272208]: Connection rejected: too many clients >=500)
2011.03.03 13:43:08 LOG4[8461:3086272208]: Connection rejected: too many clients >=500)
2011.03.03 13:43:08 LOG4[8461:3086272208]: Connection rejected: too many clients >=500)

Have I properly applied the new open file limit? Do I need to actually reboot?

We have a load balancer where if we refresh the stats page over and over, we will notice that the Current sessions value will stay stable for a time, say 45-50 sessions per server more or less, then suddenly we will refresh the page and one server will have 0 and the other server will have 2. Then, on the next reload of the stats page, we're right back to 45-50. We will be refreshing the page every 1-2 seconds

My question is, what would cause the stats module to suddenly show that there are only two sessions open, then go right back to the normal session count within a one to two second period?

Edit: I've also found that all stats look like they reset during this period, not just current sessions. For instance, here's a before, during and after on bytes in - it jumps down and back up too - not just current sessions... though I believe Bytes In is a long term "only grows higher " stat, not a "snapshot" stat.

bytes in before: 231,766,635
bytes in during: 7,704,962
bytes in after: 233,395,640

Edit 2: Thanks Willy, looks like you are right. The comment in the config file was misleading there I have to say.

[admin@p2378442 ~]$ ps ax |grep hapro 
9833 pts/0 S+ 0:00 grep hapro 
18375 ? Ss 0:02 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
18376 ? Ss 0:25 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid 

I've automated a deployment, and used 7-zip to compress a directory as part of the automation. Whenever 7-zip comes to compress the directory, however, it seems to take a long time (over 60 seconds) at the start when it just says "Scanning" ... it doesn't seem to be adding files to the archive at this time. Also, its not using significant CPU time.

It's not a big directory; maybe 8 MB, and not an overly large number of files - less than 300.

What's it doing? Should I be concerned? Can I speed it up?

So I want to do asynchronous, disconnected, across-the-Internet messaging with WCF. Probably due to my background in Apache ActiveMQ, I'm looking at MSMQ for this.

In ActiveMQ, it was a simple thing to encrypt a queue connection; you simply chose an SSL based connection to the broker, and you had a basic SSL layer to protect your communication between the broker and the clients. (Should I consider using WCF clients with an ActiveMQ broker?)

When I hit Google for MSMQ encryption and MSMQ SSL I'm not finding anything similar. Here's hoping I'm just missing something?

I have found some semi-convoluted looking stuff about encrypting messages, things that seem off the mark about using HTTPS, and things that require an Active Directory - but I'm looking to protect all of the communication, not just the messages, and we prefer not to use HTTPS as a binding as we require disconnected operation, and we will have no Active Directory to work with.

What's the best way to do this?

After having just spent months setting up a fairly complex VPN, I'm beginning to look at alternatives for the future. Some of my network providers use MPLS to connect to us, and I suppose it works fairly well. I know many ATM (automated teller machine) networks use MPLS, which I suppose it a vote of confidence for its security properties.

http://en.wikipedia.org/wiki/MPLS_VPN is rather succinct:

"MPLS VPN is a family of methods for harnessing the power of Multiprotocol Label Switching (MPLS) to create Virtual Private Networks (VPNs). MPLS is well suited to the task as it provides traffic isolation and differentiation without substantial overhead.[citation needed]

Layer 3 MPLS VPN

A layer 3 MPLS VPN, also known as L3VPN, combines enhanced BGP signaling, MPLS traffic isolation and router support for VRFs (Virtual Routing/Forwarding) to create an IP based VPN. Compared to other types of VPN such as IPSec VPN or ATM, MPLS L3VPN is more cost efficient and can provide more services to customers."

My question is : how cumbersome / expensive is it to set up an MPLS network? Is it the kind of thing where you can buy the hardware and DIY, or do you really need to go to a service provider? I can get "managed" VPN's for $100/month right now (which I have no idea if this is good or bad), my five partner IPSEC "hairpin" topology thereby costs me 6,000 a year. Would that be better invested in MPLS?

I have a partner who would like us to use a commercial product called Connect Direct where we would normally use ssh based scp or sftp. Besides reading their website, which is of course not quite impartial, is there any material (or do any of you have any opinions or info) on why this software is better than cron (or at) and scp scripting?

So I'm building out a shell script to check out all of our relevant svn repositories for analysis in svnstat. I've gotten all of this to work manually, now I'm writing up a bash script in cygwin on my Vista laptop, as I intend to move this to a Linux server at some point.

Edit: I gave up on this and wrote a simple .bat script. I'll figure out the Linux deployment some other way.

Edit: added the sleep 30 and svn log commands. I can tell now, with the svn log command, that it's not getting to the svn log ... this time, it did Applications, and ran the log, and then check out Database, and froze. I'll put the sleep 30 before and after the log this time.

co2.sh

#!/bin/bash
 function checkout {
        mkdir $1
        svn checkout svn://dev-server/$1 $1
        svn log --verbose --xml >> svn.log $1
        sleep 30
}


cd /cygdrive/c/Users/My\ User/Documents/Repos/wc

checkout Applications
checkout Database
checkout WebServer/www.mysite.com
checkout WebServer/anotherhost.mysite.com
checkout WebServer/AnotherApp
checkout WebServer/thirdhost.mysite.com
checkout WebServer/fourthhost.mysite.com
checkout WebServer/WebServices

It works, for the most part - but for some reason it has a tendency to stop working after a few repositories, usually right after finishing a repository before going to the next one. When it fails, it will not recover on its own. I've tried commenting out the svn line, it goes in and creates all the directories just fine when I do that - so its not that.

I'm looking for direction as well as direct advice. Cygwin has been very stable for me, but I did start using the native rxvt instead of "bash in a cmd.exe window" recently. I don't think that's the problem, as I've left top on remote systems running all night and rxvt didn't seem to mind. Also I haven't done any bash scripting in cygwin so I suppose this might not be recommended; though I can't see why not. I don't want all of WebServer, hence me only checking out certain folders like that.

What I suspect is that something is hanging up the svn checkout. Any ideas here?

Edit: this time when I hit ctrl+z to cancel out, I forgot I was on Windows and typed ps to see if the job was still running; and as you can see there are lots of svn processes hanging around... strange.

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ 

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ jobs
[1]-  Stopped                 bash co2.sh
[2]+  Stopped                 ./co2.sh

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ kill %1

[1]-  Stopped                 bash co2.sh

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ 
[1]-  Terminated              bash co2.sh

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ 

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ 

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ ps
      PID    PPID    PGID     WINPID  TTY  UID    STIME COMMAND
     7872       1    7872       2340    0 1000   Jun 29 /usr/bin/svn
     7752       1    6140       7828    1 1000   Jun 29 /usr/bin/svn
     6192       1    5044       2192    1 1000   Jun 30 /usr/bin/svn
     7292       1    7452       1796    1 1000   Jun 30 /usr/bin/svn
     6236       1    7304       7468    2 1000   Jul  2 /usr/bin/svn
     1564       1    5032       7144    2 1000   Jul  2 /usr/bin/svn
     9072       1    3960       6276    3 1000   Jul  3 /usr/bin/svn
     5876       1    5876       5876  con 1000 11:22:10 /usr/bin/rxvt
      924    5876     924      10192    4 1000 11:22:10 /usr/bin/bash
     7212       1    7332       5584    4 1000 13:17:54 /usr/bin/svn
     9412       1    5480       8840    4 1000 15:38:16 /usr/bin/svn
S    8128     924    8128       9452    4 1000 17:38:05 /usr/bin/bash
     9132    8128    8128       8172    4 1000 17:43:25 /usr/bin/svn
     3512       1    3512       3512  con 1000 17:43:50 /usr/bin/rxvt
I   10200    3512   10200       6616    5 1000 17:43:51 /usr/bin/bash
     9732       1    9732       9732  con 1000 17:45:55 /usr/bin/rxvt
     3148    9732    3148       8976    6 1000 17:45:55 /usr/bin/bash
     5856    3148    5856        876    6 1000 17:51:00 /usr/bin/vim
     7736     924    7736       8036    4 1000 17:53:26 /usr/bin/ps

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ jobs
[2]+  Stopped                 ./co2.sh

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ 

Here's an strace on the PID of the hung svn program, it's been like this for hours. Looks like its just doing nothing. I keep suspecting that some interruption on the server is causing this; does svn have a locking mechanism I'm not aware of?

Kyle Hodgson@KyleHodgson-PC ~/winUser/Documents/Repos
$ strace -p 7304
**********************************************
Program name: C:\cygwin\bin\svn.exe (pid 7304, ppid 6408)
App version:  1005.25, api: 0.156
DLL version:  1005.25, api: 0.156
DLL build:    2008-06-12 19:34
OS version:   Windows NT-6.0
Heap size:    402653184
Date/Time:    2009-07-06 18:20:11
**********************************************

I have an older HP NetServer LPr with what is apparently a Symbios SCSI card connecting to a Quantum SuperLoader 3 that is DLT based. From time to time, we seem to lose the connection to the autoloader. It's usually due to flaky power, but not totally sure why; sometimes when this happens the Autoloader's LED's are orange and it needs to be power cycled. The annoying workaround currently is to reboot the machine. As it is our production VPN and DNS server in addition to being our backup server, this is less than optimal.

In Debian (Sarge) is there not some command one can type to get the card to notice that it has the autoloader connected again?

dcr1:/proc# grep -i symbios /proc/pci
    SCSI storage controller: LSI Logic / Symbios Logic 53c895 (rev 1).
dcr1:/proc# uname -a
Linux dcr1 2.4.27-3-686 #1 Tue Dec 5 21:03:54 UTC 2006 i686 GNU/Linux
dcr1:/proc# mt status
mt: /dev/tape: No such device
dcr1:/proc# ls -l /dev/tape
lrwxrwxrwx 1 root root 8 2007-02-07 16:01 /dev/tape -> /dev/st0
dcr1:/proc#

That mt status command will show the actual st0 status when things are working correctly. The No such device message is usually the second clue that we need to reboot - the first clue is usually that the backups didn't run.

When logging in to a Debian Sarge install from cygwin, if I run top the output is "double spaced" for lack of a better word. The end result is that I can see the last few lines for the last few processes, but its not really usable:

28849 root       9   0  1896 1884 1392 S  4.6  0.5   0:01.36 dump



28861 root      12   0  1104 1104  848 R  1.6  0.3   0:00.14 top

    5 root      10   0     0    0    0 D  0.3  0.0   0:00.01 bdflush

    1 root       8   0   444  444  396 S  0.0  0.1   0:15.48 init

    2 root       8   0     0    0    0 S  0.0  0.0   0:00.01 keventd

    3 root      19  19     0    0    0 S  0.0  0.0   0:00.12 ksoftirqd_CPU0

I'm pretty sure this might be a termcap issue. Am I missing something simple?