James Green's questions

I have a mailserver (running Exim as MTA/MDA, and Dovecot for IMAP access) with around 50 users, and around 100GB total data (including some huge accounts, some defunct ones, etc). The mail is all stored in Maildirs. We have a sudden need to pull out all mails where the headers (To, From, Cc, etc) contain one of a handful of domains, to satisfy a request from our lawyers.

Now, I can hack together an inefficient solution (grep -R through the mail archive for the domains in question, a touch of cut, sort and uniq to get just the distinct filenames, copy all those files to a new Maildir and take it from there, perhaps) but this is going to take a hell of a long time to run on the hardware available. Is there a tool out there that will take away the pain of this process for me?

Platform isn't a huge issue - the server in question runs Ubuntu 12.04, but I have a sufficiently recent snapshot of the data I can mount on a machine running anything reasonable - and there's no requirement that the solution be FOSS, although the indicative software budget is in the hundreds not thousands of pounds.

I suspect there's a really obvious answer to this that Google isn't showing me, probably because I have the wrong search terms - anyone have an experience of this?

Thanks!

I'm quite surprised that I can't see an answer to this anywhere on the site already, nor in the MySQL documentation (section 5.2 seems to have logging otherwise well covered!)

If I enable binlogs, I see a small performance hit (subjectively), which is to be expected with a little extra IO -- but when I enable a general query log, I see an enormous performance hit (double the time to run queries, or worse), way in excess of what I see with binlogs. Of course I'm now logging every SELECT as well as every UPDATE/INSERT, but, other daemons record their every request (Apache, Exim) without grinding to a halt.

Am I just seeing the effects of being close to a performance "tipping point" when it comes to IO, or is there something fundamentally difficult about logging queries that causes this to happen? I'd love to be able to log all queries to make development easier, but I can't justify the kind of hardware it feels like we'd need to get performance back up with general query logging on.

I do, of course, log slow queries, and there's negligible improvement in general usage if I disable this.

(All of this is on Ubuntu 10.04 LTS, MySQLd 5.1.49, but research suggests this is a fairly universal issue)

Our company is growing, and growing fast. And everything we do is online -- e.g. our internal business applications are web-apps, hosted in a datacentre (we have multiple office locations), and we scan all our mail into a DMS hosted in the same datacentre.

Up until now, we've had an ADSL line (or two, at sites that have inter-site VOIP set up -- we have a dedicated ADSL line for this to avoid contention issues) per site, and this has been fine, but we're starting to run into speed issues -- not helped by the fact at our primary site we're so far from a telephone exchange, our ADSL syncs at ~2M down, ~0.5M up, on a good day, with a following wind.

Clearly, I need more bandwidth, and I've been asked to present proposals to management, with costs. No problem -- I need to go away and research pricing for EFM, leased lines, and all the usual jazz.

When we looked at phone lines, we were advised by a Telco to allow 1 line per 3 typical office users, as an industry norm, and this has largely been borne out -- I guess in our telephony usage, we're quite typical. But I've seen no such recommendation for Internet bandwidth, and searching for "bandwidth per user" and similar just finds discussion of quota or throughput management, not how much capacity to allow.

Is this information out there? What do those of you in larger organisations allow, when you're setting up sites and connections? Is there some typical number of Mbps per user that you'd be comfortable above and unhappy below?

I appreciate my piece of string might not be the same length as anyone else's, but I think as we scale up, we'll approach a "typical" figure that other organisations have also reached...

Following an outage, I've ended up with 2 mailboxes for each of my users, on separate machines. I've shut the interim machine down now, and put the mailboxes on the real server in /srv/scratch/$username/Maildir/..., and need to integrate the messages from them, ideally preserving read/seen state, into the users' canonical mailboxes, /srv/mail/$username/Maildir/.... I have no preference for whether this is as a sub-folder, or in their main inbox, however I am confident none of the temporary mailboxes contain sub-folders themselves. (If they do, more fool the users, who were explicitly told not to do this on the temporary accounts!)

Is there a good way to do this which minimises risk of loss of mail, and means their clients will automatically pick up the new folder/messages (whichever it needs to be) when they reconnect? Alternatively, what do I need to do to tell Dovecot about the new messages after dumping them into the Maildirs?

Both servers were built using Exim4 for delivery and Dovecot for IMAP.

I have a Xen domU provided by a third party, running Ubuntu (10.04, server edition, stock -server kernel). This server runs Dovecot and Exim4, with mail stored in Maildirs, and runs a fairly typical LAMP stack with most applications in Perl, and all data stored either in a directory tree full of TIFF files, or a MySQL DB. This server has been operation for around 3 months for LAMP stuff, and a month serving mail. All filesystems (except swap) are Ext3.

A couple of weeks ago we suddenly found a whole bunch of TIFF files which were no longer accessible, as noted by our backup script (using rsync). rsync on the remote host reported the following errors:

rsync: readlink_stat("/srv/data/documents/archive/pdf/2007/Aug/06/085717/00000002.TIF") failed: Input/output error (5)
rsync: readlink_stat("/srv/data/documents/archive/pdf/2007/Aug/06/085717/00000001.TIF") failed: Input/output error (5)
rsync: readlink_stat("/srv/data/documents/archive/pdf/2011/Jan/04/125227/XSMDESC.DAT") failed: Input/output error (5)
rsync: readlink_stat("/srv/data/documents/archive/pdf/2011/Jan/04/125227/DOC010.XST") failed: Input/output error (5)
rsync: readlink_stat("/srv/data/documents/archive/pdf/2011/Jan/04/125227/00000001.TIF") failed: Input/output error (5)

...and so on. The files will have been created either late December, or on the date given in the path, whichever is the later, as we migrated our data to this machine late last year. No process to my knowledge will have written to the files since -- only read from them.

Throughout that day, we noted the list of affected files increasing, so that night we unmounted that filesystem (a Xen Virtual Block Device) and ran a fsck, which found and fixed many, many errors. The affected files were now gone. However, the corruption stopped spreading once the fsck was complete and the filesystem remounted.

(As an aside, to illustrate the kind of luck we've had here -- the single disk holding our only backup of this data died catastrophically the same afternoon. Yes, really. Our only other backup was from Dec 10th 2010...)

It may or may not be relevant that the vast majority of the files affected were created on Jan 4th or 5th of this year -- however some were documents from 2006/7, and some were newer.

With the fsck complete and the machine now apparently stable, we were worried -- the hosting provider could find no root cause, and nor could we -- and we'd lost data, but at least the corruption had stopped.

Skip forward several days, and a routine mysqldump refuses to dump 3 tables because they are marked as crashed. mysqlcheck confirms this, and REPAIR TABLE [foo] fixes all 3, in 2 cases reporting fewer rows found after the event than before. Again the vendor can see no root cause, there has been no interruption to power, disk access or mysqld. The problem appears unrelated but -- in 3 months hosting on this server, we've already lost more data than in several years of running these applications on a variety of different (but never virtual!) platforms.

Finally, this week we have found 3 files on the FS which appear to have turned to binary gunk -- more specifically, all 1s (or all \0xFF if you prefer). All 3 files (2 small text config files, 1 100-ish line perl script) were part of our web application, and would be frequently read but written to only when we deployed a new version, which works by updating a local "working" copy, exporting that working copy to get a clean fresh install, and pointing a symlink at that fresh install. The files were broken in the working copy and propagated from there, and the modification times on all files were consistent with them not having been changed for many weeks (in which time there had been several deployments, all of which went fine!) so the content clearly changed without the mtime being updated.

Any one of these events would have me restoring from backups, scratching my head and carrying on with my life, but 3 in a fortnight has me waiting for the next thing to happen.

My questions is simple: is it even possible that these 3 events are connected, and if so, where should I be looking for a root cause?

(Answers regarding solutions are also welcome, however we are already in the process of setting up a parallel platform running CentOS, on VMware, with the same vendor, to try rule out distribution, kernel, hypervisor and virtual block device related issues. It would be great to know which of those was the issue, but if we don't have a diagnosis, and replacing that whole stack works, that'll help me sleep at night ... eventually.)

As always if any extra information would help, please comment and I will update accordingly!

I am migrating a bunch of changes from our dev database, consisting of some extra tables, the data from a subset of these tables, and some new functions. Using phpmyadmin I have exported the structure of the tables as an SQL file, and the data of the subset where I need it. However phpmyadmin offers no way to export the functions en-masse that I can see.

Am I missing a trick, or should I just be manually creating a .sql file containing each of the function definitions?

(I appreciate that in the time it's taken to type this question and wait for answers I can probably have done the latter, but it feels inelegant given how straightforward the first 2 SQL files were to create automatically!)

I have a Xen PV guest, running Ubuntu 10.04. I do not run the underlying host. Kernel is the stock one provided by Ubuntu:

Linux nephos 2.6.32-21-server #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC 2010 x86_64 GNU/Linux

The machine servers as a LAMP web/DB server with a bunch of perl web applications we've developed in-house. Since we went live and let our users loose on the machine on Monday morning, reliably once a day it has gone into a state where we can't reboot it from the command line, CGI scripts become unresponsive, ping times shoot up, and even commands like ls fail in certain directories (possibly ones to which writes are pending).

top shows a number of processes in state D, mostly named fleet.cgi or doc.pl, which are our applications. Attempts to kill or kill -9 these processes silently fail. sudo reboot returns, claiming the machine is about to go down, but never sends the broadcast message to other shell users that it is about to do so, nor does it reboot the machine.

When the machine begins to lock up, lines like the following appear in syslog:

Dec 14 12:05:45 nephos kernel: [71040.150212] INFO: task mysqld:2708 blocked for more than 120 seconds.
Dec 14 12:05:45 nephos kernel: [71040.150234] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 14 12:05:45 nephos kernel: [71040.150247] mysqld        D ffff880002d4dbc0     0  2708      1 0x00000000
Dec 14 12:05:45 nephos kernel: [71040.150256]  ffff8800fa5e9918 0000000000000286 0000000000015bc0 0000000000015bc0
Dec 14 12:05:45 nephos kernel: [71040.150264]  ffff8800ec5883c0 ffff8800fa5e9fd8 0000000000015bc0 ffff8800ec588000
Dec 14 12:05:45 nephos kernel: [71040.150272]  0000000000015bc0 ffff8800fa5e9fd8 0000000000015bc0 ffff8800ec5883c0
Dec 14 12:05:45 nephos kernel: [71040.150280] Call Trace:
Dec 14 12:05:45 nephos kernel: [71040.150309]  [<ffffffff8116d1d0>] ? sync_buffer+0x0/0x50
Dec 14 12:05:45 nephos kernel: [71040.150320]  [<ffffffff815555c7>] io_schedule+0x47/0x70
Dec 14 12:05:45 nephos kernel: [71040.150325]  [<ffffffff8116d215>] sync_buffer+0x45/0x50
Dec 14 12:05:45 nephos kernel: [71040.150330]  [<ffffffff81555a9a>] __wait_on_bit_lock+0x5a/0xc0
Dec 14 12:05:45 nephos kernel: [71040.150334]  [<ffffffff8116d1d0>] ? sync_buffer+0x0/0x50
Dec 14 12:05:45 nephos kernel: [71040.150339]  [<ffffffff81555b78>] out_of_line_wait_on_bit_lock+0x78/0x90
Dec 14 12:05:45 nephos kernel: [71040.150347]  [<ffffffff81084fe0>] ? wake_bit_function+0x0/0x40
Dec 14 12:05:45 nephos kernel: [71040.150353]  [<ffffffff8116c1e7>] ? __find_get_block_slow+0xb7/0x130
Dec 14 12:05:45 nephos kernel: [71040.150357]  [<ffffffff8116d396>] __lock_buffer+0x36/0x40
Dec 14 12:05:45 nephos kernel: [71040.150365]  [<ffffffff81212164>] do_get_write_access+0x554/0x5d0
Dec 14 12:05:45 nephos kernel: [71040.150369]  [<ffffffff8116cb57>] ? __getblk+0x27/0x50
Dec 14 12:05:45 nephos kernel: [71040.150374]  [<ffffffff81212371>] journal_get_write_access+0x31/0x50
Dec 14 12:05:45 nephos kernel: [71040.150381]  [<ffffffff811c5f9d>] __ext3_journal_get_write_access+0x2d/0x60
Dec 14 12:05:45 nephos kernel: [71040.150386]  [<ffffffff811b7c7b>] ext3_reserve_inode_write+0x7b/0xa0
Dec 14 12:05:45 nephos kernel: [71040.150392]  [<ffffffff8155748e>] ? _spin_unlock_irqrestore+0x1e/0x30
Dec 14 12:05:45 nephos kernel: [71040.150396]  [<ffffffff811b7ccb>] ext3_mark_inode_dirty+0x2b/0x50
Dec 14 12:05:45 nephos kernel: [71040.150401]  [<ffffffff811b7e71>] ext3_dirty_inode+0x61/0xa0
Dec 14 12:05:45 nephos kernel: [71040.150406]  [<ffffffff81165c22>] __mark_inode_dirty+0x42/0x1e0
Dec 14 12:05:45 nephos kernel: [71040.150412]  [<ffffffff81159f8b>] file_update_time+0xfb/0x180
Dec 14 12:05:45 nephos kernel: [71040.150422]  [<ffffffff810f5300>] __generic_file_aio_write+0x210/0x470
Dec 14 12:05:45 nephos kernel: [71040.150430]  [<ffffffff8114f49d>] ? __link_path_walk+0xad/0xf80
Dec 14 12:05:45 nephos kernel: [71040.150435]  [<ffffffff810f55cf>] generic_file_aio_write+0x6f/0xe0
Dec 14 12:05:45 nephos kernel: [71040.150441]  [<ffffffff8114311a>] do_sync_write+0xfa/0x140
Dec 14 12:05:45 nephos kernel: [71040.150446]  [<ffffffff81084fa0>] ?  autoremove_wake_function+0x0/0x40
Dec 14 12:05:45 nephos kernel: [71040.150453]  [<ffffffff8100f392>] ? check_events+0x12/0x20
Dec 14 12:05:45 nephos kernel: [71040.150461]  [<ffffffff81250946>] ? security_file_permission+0x16/0x20
Dec 14 12:05:45 nephos kernel: [71040.150466]  [<ffffffff81143418>] vfs_write+0xb8/0x1a0
Dec 14 12:05:45 nephos kernel: [71040.150470]  [<ffffffff81143db2>] sys_pwrite64+0x82/0xa0
Dec 14 12:05:45 nephos kernel: [71040.150477]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b

I've installed the Ubuntu package linux-virtual to ensure I have a suitable kernel installed, but its dependencies have been satisfied by the kernel I currently have. I'm at a bit of a loss where else to look here really. Under normal load nothing untoward appears in iotop, but equally I'm not aware of any sudden increase in usage that triggers the fault -- that said, the machine has been running these applications with just 1/2 test users for weeks, and is only failing now a dozen people are hitting them all day.

Do I just need a machine with better IO capabilities (or to reduce my apps' need for same) or is this something I can approach with a bit of tuning?

Updated 15/12/2010 23:41: If it is helpful (and I suspect it's a crucial detail) the guest is running under paravirtualisation.

I'm not sure this is the right forum for this question -- although I'm confident I'll be told if not! -- but I've read the fine manual (at least, such a manual as I have), I've googled and I cannot get any insight into where to even start solving this problem.

I have a bunch of Mitel 5312 handsets, talking to a 3300 ICP controller. Some handsets are at a remote location, get an address from my DHCP server over there, and use the Mitel "Teleworker" extension to connect in over the Internet. The remaining handsets were set up with static IPs by a BT-supplied engineer, on the same subnet as the ICP itself. So far, so good.

I have one remaining teleworker licence, and need to move a handset from the home location to the remote. I've managed to boot it and configure teleworker, but I cannot for the life of me see where I tell it to forget its static IP, and make a DHCP request.

Any ideas? Should I be looking on the controller, or holding magic combinations of buttons on the handset itself?

EDIT: Following some advice from Robert, below, I've broken out a spare device and reassigned the profile for this user's extension to the MAC of the new phone, and a new profile to the old MAC. Unfortunately this still doesn't get me anywhere -- the new handset now asks for the teleworker install password.

I suspect I'm going to have to get a Mitel engineer involved here, since I've never been given that password... Unless anyone has any great ideas?

I have a very small network of maybe two dozen machines, to which I've just added 7 IP telephones, talking to a Mitel controller on another site. I also have 2 ADSL links, using consumer ADSL routers. The 7 phones are on the same physical network as the PCs.

The 2 routers currently have internal IPs of 192.168.0.253 and 192.168.0.254: my existing machines all get addresses via DHCP and a default gateway of 192.168.0.254, but I would like the phones to have 192.168.0.253 as their gateway. DHCP service is provided by a Windows Server 2003 box, currently set up with a single scope with a fairly basic set of options.

I've tried creating a different scope on a different subnet for the phones, but presumably because the Windows server isn't on that subnet DHCP just times out on the phones. I've also tried creating "Reservations" for the phones, but the DHCP MMC will not let me change the Router option for a reservation.

What's the trick I'm missing here?

Our business is taking over some new premises (currently housing multiple companies) and for at least the first year will be operating out of a pair of portacabins, around 50 metres apart.

Users in both cabins will need to be on our internal LAN, and our phone system. What's the most cost-effective way to link these buildings?

Stringing fibre between poles has been discussed but seems expensive; a trench/duct for underground fibre even more so (~£100/metre) -- so it seems like WiFi is my best option, but does this have reliability issues over this sort of range -- given there will be commercial vehicles and heavy plant crossing the direct LoS between the buildings, do I need to get the WiFi kit high off the ground, or can I just use high powered and/or directional wifi equipment?

Similarly, should I be looking at cordless handsets for one of the buildings, or should I be looking at just getting a data connection and using IPT for internal telephony?

Does anyone have a good complete strategy for backing up a bunch of virtual machines running under VirtualBox?

I intend to run a handful of virtual machines on a single hardware platform and back them up nightly to external disks, which will be taken off site every weekend in rotation.

(Just to clarify, I don't intend to shut the machines down during this process, if I can avoid it)

Am I right in thinking I can just write a script which, each night, creates a snapshot, copies the 'main' disk image to the external media, then removes the snapshot again?

Better ideas are much appreciated!