timday's questions

On my local machine (i7 9200) running Debian Wheezy amd64 I can get some significant speedups on some "big data"/HPC type stuff by:

• Following the instructions here to reserve some RAM for huge pages and setting up hugetlbfs.
• Running my application using libhugetlbfs' (2.17) nifty HUGETLB_MORECORE=yes to redirect its mallocs to 2M pages.

The application also runs reasonably well on Debian Wheezy on EC2 (I'm using latest wheezy AMI) with normal 4k pages (some scalabilty testing tried on c3.2xlarge, c3.4xlarge and c3.8xlarge instances). But I'm curious to also see whether I see similar benefits using huge pages on EC2, if it's possible.

I fired up a c3.3xlarge instance and set up huge pages as usual. And after that /proc/meminfo does indeed report

HugePages_Total:    4096
HugePages_Free:     4095

However after compiling libhugetlbfs, it's make func self-testing triggers some kernel errors. The system soon after seems to lock up, but not before I'd had time to inspect dmesg and see a bunch of call stacks with various xen_ and hugetlb_fault symbols in them. Once it became unresponsive, system needed a forcible stop from the AWS console to get it to halt.

I did try booting up again and just running my app with HUGETLB_MORECORE=yes anyway (in case the make func testing was breaking on something obscure I didn't actually need), but much the same thing happened again.

Any success stories with libhugetlbfs on EC2 (preferably with Debian), or recipes for getting it working correctly ?

Research: there's scant googleable info about huge pages on EC2 (or Xen) out there. I did find this, which seems to report the same problem: /proc/meminfo reports hugepages available, but attempting to use them kernel panics. Article predates the new c3 instances but suggests a cc2.8xlarge might be worth a go due to it using HVM instead of PVM.

Update: couldn't find an up-to-date Debian AMI for HVM, but tried an Ubuntu one (13.04 "raring") on a cc2.8xlarge and libhugetlbfs and HUGETLB_MORECORE=yes does seem to work fine on that. The only thing is, it actually slows my application down a little!

I've been playing with EC2 spot instances, with the ultimate aim of migrating a standard instance hosting a website to a spot instance (yes, I'm well aware this really isn't what spot instances are for, but in this case I'm quite prepared to trade price for availability).

However, I was a little surprised to discover that when the price threshold is crossed, the spot instance is terminated, not stopped, and indeed that it's simply not possible to stop a spot instance, only to terminate it.

So: is there some (recommended) way of getting the instances spawned by spot instance requests to simply associate themselves with a particular EBS volume on creation, so that the full machine state can simply be rolled forward across a series of spot instances ? Or am I fundamentally misunderstanding what spot instances are all about ?

I have a regular EC2 instance with an elastic IP. The few times the instance has been stopped, I've just manually re-associated the IP with it via the AWS Management console. I'm now thinking about migrating it to a spot instance, which means I want a way of automatically re-associating it the IP on startup if/when the machine is shutdown due to being priced out of the market. How to achieve this is well described in numerous places (e.g here).

My question is:

Like many people with this problem, I'm wary of putting my AWS credentials file on the instance itself. I have a vague memory of seeing something about it being possible to create additional keys with more limited permissions, but am having trouble finding any concrete details. So: is it possible to create a key which I can happily put on the machine knowing it can be used for little more than an ec2-associate-instance (but not login to other instances, or generally running riot with my AWS account), and how would I actually achieve this ?

I have an EC2 "micro instance" running Canonical's Ubuntu 10.04 LTS. Has been running for 6-9 months now, infrequently rebooted (once every few weeks at the most).

I just did what I thought was a routine aptitude update, aptitude full-upgrade. On noticing there seemed to have been some new -ec2 linux images installed, I rebooted the system. While it seemed to reboot and go back to "running" status on the console, it didn't come back with its usual ssh and http services. I've tried stopping and starting it, re-associating it's elastic IP... no joy.

The strange thing is, "Get System Log" (AWS console) returns a completely blank log. Empty. Nothing. Not one character. (At least it's empty after the first start-stop; before the stop it just contained a final line about restarting).

I've tried a few stop-start cycles but no improvement.

Any advice what to try next to get my instance back to life ?

A regular filesystem check on a Debian Lenny system triggered an fsck, and that nuked a handful of links in the /etc/rc?.d hierarchy (unfortunately I didn't keep a list).

The system seems to boot and run normally, but I'm worried its storing up trouble for the future.

Is there an easy (fairly automatic) way of rebuilding this piece of the system ? As I understand it, the links are generally manipulated by package postinst scripts using update-rc.d (and I haven't made any changes from the installed defaults).

Without any better ideas, my plan is one of:

• Diff a listing with another similar system to identify which packages need their links repairing.
• Wait until the system is upgraded to Squeeze (hopefully not too long :^) and assume the mass package upgrade will restore all the missing links.

I have a failed 750GByte SATA Seagate drive which has failed within the first year of it's 3 year warranty.

The Seagate warranty site stresses the importance of testing the drive with "SeaTools" (and obtaining some diagnostic error code) before returning it.

If I attempt to boot up a machine (Dell T5400, XP64) with the drive connected by a SATA cable, the machine doesn't get past POST (waited 5+ mins to see if there was a timeout... maybe should have waited longer?). If I disconnect the drive, disable that SATA channel and reboot it still doesn't post. If I disconnect the drive I can boot, but if I hotplug the drive once past POST SeaTools doesn't detect it.

Is the drive just very dead (too dead for SeaTools to even notice it) or is there some way I can get it to be visible to SeaTools ?

I've tried the drive in 2 machines. The other one does eventaully POST after a couple of minutes (and there's no hint of the drive in the BIOS's view of drives), but is a Linux box (so no SeaTools).

I'm looking at a system with Debian/Lenny on a couple of old IDE PATA drives (several RAID-1 partitions including /boot) and a relatively recently added SATA drive. The system works fine but I've been poking around the GRUB setup in anticipation of upgrading the old drives...

If I go straight to the grub command line when the boot menu appears and inspect the drives there (using

grub> root (<TAB>...)

) then I see hd0 and hd1 as the PATAs and hd2 as the SATA.

However, after the system boots up if I do the same thing then I see hd0 is now the SATA and hd1 and hd2 are the PATAs. (Also, the system maps /dev/sda,c,b to SATA,PATA,PATA.)

This makes me a bit nervous about making changes from the grub> prompt once the machine has booted. So:

Why does the device order seen by grub change between these 2 cases ?

Also:

Correct functioning of some GRUB commands seems to rely on device.map being correct. Currently it contains a single line:

(hd0)   /dev/md0

(Which is the RAID1 /boot) but I've no idea where that came from (this system goes back to woody I think)! grub-mkdevicemap wants to regenerate this as hd[0,1,2]->sd[a,b,c](but I haven't let it). update-grub tells me: warning: grub-probe can't find drive for /dev/sdb1. Any suggestions as to what a healthy device.map should look like, in light of the changing enumeration order ? (It's unclear to me whether it should contain the pre- or post- boot device enumeration order).

I have a Server 2008 install with the Terminal Server role enabled, and Remote Desktop enabled.

If I try and connect to it with Remote Desktop Connection, how can I choose which sort of desktop I'm going to get (ie choose between the "real" server console and a faux Terminal Server one) ?

From what I'm seeing at the moment I'm always connecting to the console, so maybe I just haven't got TS set up right (but I'm curious to know how the choice would be exposed once it is working right, or what rules determine which sort of connection is established).

I have a couple of machines in my server room normally powered off which it's useful to be able to power up remotely.

One of them is an old dual-P3 IBM serverWorks Intel STL2 machine on which wake-on-LAN has performed flawlessly for years. I hit its MAC address with etherwake, and hey-presto there it is a minute or so later (unless it decided it was due an fsck).

The new system I'm having problems with is built on an Asus P5E3. In the BIOS' Power/APM settings wake is enabled for modem/PCI/PCIe devices. That alone doesn't let me wake it using etherwake, but if I use ethtool -s eth0 wol g to enable magic-packet waking, then I can... once. Following any boot (whether via WOL or power-on) the state seems to always revert back to 'd' (disabled). How can I get the state set by ethtool to "stick" indefinitely ?

All systems involved are Debian Lenny, 2.6.26 kernel. Motherboard network interfaces being used, no add-in cards involved.

Thanks for any help