gareth_bowles's questions

I've started seeing a situation on our internal Atlassian Confluence instance where we get snappy performance most of the time, but a couple of times an hour requests (both read and write operations) just hang for up to a few minutes (longest I've seen is 5 minutes).

I could plough through http://confluence.atlassian.com/display/CONF34/Performance+Tuning but just wanted to check if anyone could point me to anything obvious.

Our setup uses Confluence standalone fronted by an Apache 2.2 instance on the same host, which uses mod_proxy to forward requests to Confluence. Apache is set up to use HTTPS for all requests. We have a MySQL DB on a separate server.

There's no obvious spikes in CPU or memory usage on the Confluence or DB servers, so I don't think garbage collection is the problem, and I've tried increasing the MaxThreads in Apache to no avail.

Thanks in advance ...

I just started to get a Nagios warning from our build server, stating that the number of processes has exceeded the limit. Looking at our Munin graphs, I can see that the number of processes has increased steadily from 280 in December to the current value of 430.

I'm wondering how I can go about identifying the causes of the increased number of processes, so that I can restart services or adjust their configuration as necessary.

Server details: CentOS 5.1, the main things running are our Hudson build server which runs under Tomcat, and an Apache httpd server which is mainly just a proxy for Hudson. I've tried restarting httpd and Tomcat, but the number of processes stayed the same. "top" says that only one of the processes is active; the rest are sleeping.

Managing EC2 access keys and X.509 certificates can become challenging when you start to deal with large numbers of instances. Do any EC2 users here have good policies and/or tools for:

• rotating EC2 access keys and X.509 certificates
• preventing copies of keys / certs from proliferating onto instances and AMIs
• keeping the keys in a centralized location with the appropriate access ?

Here's my situation: I'm setting up a test harness that will, from a central client, launch a number of virtual machine instances and then execute commands on them via ssh. The virtual machines will have previously unused hostnames and IP addresses, so they won't be in the ~/.ssh/known_hosts file on the central client.

The problem I'm having is that the first ssh command run against a new virtual instance always comes up with an interactive prompt:

The authenticity of host '[hostname] ([IP address])' can't be established.
RSA key fingerprint is [key fingerprint].
Are you sure you want to continue connecting (yes/no)?

Is there a way that I can bypass this and get the new host to be already known to the client machine, maybe by using a public key that's already baked into the virtual machine image ? I'd really like to avoid having to use Expect or whatever to answer the interactive prompt if I can.

My shop has been using the FreeNX server on Fedora 11 for a while now and mostly getting good results, especially with performance, but we have some annoying problems with client connections. There are two main issues:

1. Client sessions sometimes freeze after a long time (seems to be at least 2 hours of having the session active)
2. We often have to make multiple attempts to start a new client session, especially if a previous session was suspended rather than terminated. In qwuite a few cases, we've had to restart the NX server to get around this.

Our NX server configuration is the default except that we've enabled logging level 7 to /var/log/nxserver.log, and set the font server to "unix:/7100" so that it uses xfs.

Does anyone have any ideas for making things more stable ?

I needed to start a new EC2 instance today and decided to try out the new spot instances, where you can reduce your instance cost by bidding on the maximum per-hour price you're prepared to pay. Since today's spot price was only 3.5c / hour, compared with 8.5c / hour for an on-demand instance, I was wondering: if I just bid a really high price, say 10c / hour, can I effectively be sure of getting a much cheaper long-running instance than an on-demand instance (since the spot instances are only charged by the current spot price) ?

I suppose it's theoretically possible for the spot price to go over the on-demand price, but as far as I can tell from the data on the AWS site, the spot price has always been well below that.

UPDATE: I've been monitoring the small instance spot price for a few weeks and it's been very consistent, only varying between 2.9c and 3.1c. I wrote up a more detailed blog post on spot instances, including a link to a Nagios plugin I built to monitor the spot price.

I'm debugging a Linux application that allows remote jobs to be submitted, logging the output from each job in a new file. The log file paths conform to:

/joblogs/job_*/JOB.LOG

where the wild card represents the unique job number.

I want to be able to tail each job log, including new logs that are created after I issue the tail (or whatever) command. I thought I'd be able to do this using multitail, but I can't figure out the right set of parameters to use. For example,

multitail -q 1 "/joblogs/job_*/JOB.LOG"

seems to create a window for each new log file just the way I want, but it doesn't show any output in the file window.

Anyone know how to do this, either with multitail or another Linux tool ?

Hopefully the question says it all; materials I've found on the web suggest I should be able to copy my VM from OSX to Linux. I copied all the VM files from OSX to Linux, but when I try to open my VM using "Open existing virtual machine", it seems to kill the VMware server web console. After restarting the console, the import job for the new VM has disappeared.

Here are my version details:

VMware Fusion 2.0.6 on OSX 10.6 (Snow Leopard 64-bit)

VMware Server 2.0 on Fedora 11 (64-bit)

Has anyone else had success with this ?

I just started using the nifty rsnapshot utility for backups to our NAS. I really like the tool (which uses rsync and links unchanged files to keep the backup size down), but I'm running into problems copying certain links, e.g:

sudo /bin/cp -al /NAS/Backups/rsnapshot/hourly.0 /NAS/Backups/rsnapshot/hourly.1
/bin/cp: cannot create link `/NAS/Backups/rsnapshot/hourly.1/hourly.0/percival/home/stuffs/Workspaces/AppscioWork/MPF.bad/mpf-core/gtk-doc.make': No such file or directory

I'm not sure what is going on here, as the source file exists and has read permissions:

ls -l /NAS/Backups/rsnapshot/hourly.0/percival/home/stuffs/Workspaces/AppscioWork/MPF.bad/mpf-core/m4/gtk-doc.m4
lrwxrwxrwx. 1 nobody nobody 29 2009-12-30 22:38 /NAS/Backups/rsnapshot/hourly.0/percival/home/stuffs/Workspaces/AppscioWork/MPF.bad/mpf-core/m4/gtk-doc.m4 -> /usr/share/aclocal/gtk-doc.m4

ls -l /usr/share/aclocal/gtk-doc.m4
-rw-r--r--. 1 root root 1324 2009-02-24 17:50 /usr/share/aclocal/gtk-doc.m4

Is there a way to either fix this error, or have the cp command ignore the error and keep going ?

What post-install steps do you find yourself taking every time you install a new Linux server or VM ? I'm interested in things that could apply regardless of what use the server is to be put to, especially those that can save time down the line, and ways to automate each step.

Here is my set of initial tasks; note that I nearly always use Fedora.

1. Enable network startup at boot time
2. Add myself to the sudoers file
3. Add my SSH keypair to ~/.ssh/authorized_keys to avoid having to log in with a password
4. Add the new server to ~/.ssh/config on my client machine(s) so I can log in with "ssh newserver"
5. Check out my customized .bashrc and .bash_profile from source control
6. If on the internal network, set selinux enforcing to permissive
7. If on the external network, install / configure blockhosts to stop the script kiddies
8. Install / configure NRPE daemon for Nagios monitoring
9. Install / configure NX server for remote GUI access

I'm running on a couple of different Linux distros (Fedora 11, CentOS 5 and SuSE 10.2) and often run into a problem where a new script that I want to run as a cron job works fine when called direct from the command line, but fails when called from cron due to slight variations in the PATH or other required environment variables.

Is there a way I can test run an individual script as if it's being run from cron, without having to run the entire crontab or use run-parts to run the entire cron.daily/ .hourly etc. directory ? I guess I could temporarily edit my crontab to run the script in the next couple of minutes, but I'd rather have a more standalone solution so that I don't risk messing up the crontab.

I'm hoping one of the many network gurus here can help me with a problem that just started to happen on our office network. We have a Linksys RV082 router; short description of the problem is that clients on the internal office network are intermittently unable to resolve host names via DNS.

Here's the troubleshooting info I've gathered so far:

• External connections to servers inside our office are fine.
• Outbound connections from the office network are fine if I use the IP address (e.g. browse to http://myhostname.com fails with "host not found", but browse to http:[IP address of myhostname.com] works). This is why I'm assuming it's a DNS problem.
• Problem happens from multiple machines on the office network (it doesn't seem to be restricted to any particular machine).
• Restarting the router fixed the problem for a short time, but the problem came back after about 15 minutes.
• Changing the DNS servers defined in the router from our ISP's DNS servers to OpenDNS made things better for a couple of hours, but now we're back to the same problem again. (Note; we don't have a local DNS proxy, the router is set up to go direct to a pair of external DNS servers).

Thanks in advance for your help !

Does anyone know of a tool that will analyze an Apache access log and show the location of each client that accessed the site, ideally showing city, state and country on a map in the same way that Google Analytics does ?

I've looked at Webalizer and AWStats, but they only seem to go down to country level, they only show tables of data rather than maps, and they also seem to infer the country based on the domain name of the client rather than doing a location lookup on the IP address. For example, my Webalizer setup shows the country of a .com domain as just ".com", but it will show the country of a .co.uk domain as the UK.

I love Amazon AWS, but I'm tearing my hair out over the obscure usage data they give you. All they seem to have is either an extremely high level summary that just shows you your total monthly cost for each service, or a very hard to read spreadsheet that gives you very detailed usage data but leaves out really obvious things like the cost of each line item (it only shows you the amount of usage, so you have to go and find the rate for each service separately and then multiply rate x usage to get the cost).

Does anybody know of any tools or services that give you better insight into AWS usage data, e.g. total bandwidth cost in and out of AWS per day, or EC2 instance cost per day per instance ? Ideally I'd like to be able to pull the data programatically, so that I don't have to download a spreadsheet and massage it manually.

I've had good results from using Google Analytics to visualize visitors to our web sites. However, GA won't work for RSS feeds since it requires some JavaScript to be embedded into each Web page. Have any of you guys found a Google-like solution for monitoring RSS feed traffic that shows things such as vistor locations displayed on a map ?

I'm working on a Web service which is hosted on EC2 and needs to have a varying number of instances running, depending on load. We have the basic service up and running, but one of the things we're struggling with is the time it takes to provision and launch a Windows instance (we are using some third prty tools that only run on Windows). I've seen this take anywhere from 10 minutes up to a pretty staggering 45 minutes.

Does anyone have any tips on how to speed up the launch of an EC2 instance ? Since the AMIs for Windows servers are large compared to Linux AMIs, for example, I'm wondering if one thing might be to make sure that the S3 bucket containing the AMI is located in the same zone where the instance is launched, which would presumably make provisioning the new instance faster.

I've been running Linux servers on Amazon EC2 for a while now; the experience has been great. I've recently needed to bring up a Windows server to run some Windows-only software that our product needs to use, and am running into a problem figuring out how to install the software, which is only available on DVD.

With Linux I can just install packages from a Web-based repository and take advantage of EC2's fast network throughput, but so far on the Windows instance I've had to upload my ISO images to EC2 and mount them from the Windows EC2 instance. For some reason I'm getting really slow upload speeds to EC2, even though the regular upload speed from our office is pretty good (around 7Mbps).

I've also tried mounting the DVD drive on my machine as a local drive on the EC2 instance via Remote Desktop, and then running the software install from the local drive, but I run into the same slow upload speed issue.

Does anyone have a better way to install software from physical media onto an EC2 instance ?

I have a scheduled script that does an hourly svnsync backup of our Subversion repositories. I was running it from an entry in the root crontab without problems, but decided I'd like to run it from /etc/cron.hourly instead for extra visibility (and because one of our engineers accidentally deleted the crontab because he thought "crontab -r" meant "read the crontab ;-))

The svnsync commands in the cron.hourly script all fail with a message saying that the SSL certificate for the SVN repository needs to be accepted (this is the message you get interactively the first time that user accesses the SVN repository, but once the certificate I accepted the message doesn't come up again).

So it seems to me that the script is being executed under a different user environment when run from cron.hourly than when it's run via the root crontab. Can anyone explain the difference ?

UPDATE: I should have mentioned my distro, I'm using anacron on CentOS 5.1.

UPDATE 2: Thanks for the suggestions so far; I think this is turning into more of a Subversion question. I always try to encapsulate my environment into my scripts, but the problem here is that I'm not sure what it is in (or lacking in) the environment that makes SVN ask for the SSL certificate to be accepted when I run my script from cron.hourly. I'm guessing it's something to do with the way that the run-parts script is executed.

When I'm ssh'ed into one of our office servers (which run Fedora 10) from home, my session times out after a fairly short period of activity (5 minutes or so). I've tried using TcpKeepAlive on the client side, to no effect.

The thing I don't understand is that if I'm in the office on the company LAN, I can leave a session inactive all day without it timing out, so the behaviour seems to be dependent on my location.

Any ideas why this is happening and how to prevent timeouts when I'm not on the LAN ? I'm using the Terminal client on Mac OSX if that helps.

UPDATE - Dave Drager's suggestion of using the ServerAliveInterval set to non-zero with TcpKeepAlive=no worked for me. Regarding some of the other answers, the ClientAlive... settings aren't accepted by the Mac OSX SSH client.

My company has a software product that's written in C for a Linux platform, built with autotools and distributed via binary packages. To make the binaries, we first produce a source RPM and then compile the source from the SRPM.

Currently we only provide RPM packages for 64-bit Fedora 10, but we want to start providing packages for multiple Linux distributions - 32-bit as well as 64-bit - and possibly different versions of each distribution as well (e.g. Fedora 11 as well as Fedora 10).

I've heard that the best way to produce builds for multiple Linux flavours is to have a single build server and use a different chrooted environment for each set of packages that you want to build. Does anyone have a good resource that explains this in more detail, maybe with examples of well known projects that use this build mechanism, or have a better alternative to achieve the same goal ?

NOTE - I'm fairly familiar with how to use tools like checkinstall to produce packages in the right format for different distros. What I'm less familiar with is how to compile the same code multiple times on the same machine and end up with the correct binary files for each distro that I want to package for.