Home / server / Questions / 152733
Accepted
UsAaR33
Asked: 2010-06-19 22:19:02 +0800 CST

Creating swap files faster

  • 772

I'm using Amazon EC2 and wish to be able to quickly generate large swapfiles (~10+GB) on instance startup. Unfortunately, I/O speed on my instances (c1.xlarge) is slow enough (20 MB/s) that this operation takes 10+ minutes, which is unacceptable for my usage.

I am aware that the swapfiles must be pre-alocated to use, so that I can't use sparse files.

However, is there some command to allocate blocks w/out spending the large amount of time zero-ing out the blocks? Also, if this command exists, am I correct in assuming that a page in the swapfile is zero'd out before a user process has access to it (mitigating security concerns)?

linux filesystems amazon-ec2 swap

4 Answers

  1. I have success using 'fallocate' command. It takes less than 1 second.

    sudo fallocate -l 10G /swapfile

    Afterwards, you can run mkswap. For more info, you can use the great Digital Ocean tutorial for ubuntu here, that may work without modification for other linux flavors, at https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04.

    • 3
  2. Best Answer

    You didn't indicate what method you were trying to avoid.

    Traditionally, you would issue a dd command that would in turn pump out a zero'd file of the appropriate size, then run mkswap, add the entry to /etc/fstab, and then swapon to activate it. I've attached a rather hastily-written example shell script that I'm sure has errors (it's late where I'm at and the fstab entry is far from perfect)

    #!/bin/bash
# --- allocate 10Gbyte of swap space as 10 separate 1Gbyte files
# --- that are brought online sequentially during processing
for swpidx in 01 02 03 04 05 06 07 08 09 10
do
  dd if=/dev/zero of=/swapfile.$swpidx bs=16738 count=65536
  mkswap /swapfile.$swpidx
  echo "/swapfile.$swpidx    swap    swap    default    0 0" >> /etc/fstab
  swapon -a
done
swapon -s

    However, it sounds like you are trying to avoid this method. The fastest solution that I could provide would be to use a swap partition, which does not require the zero-out process, and can be brought online in minutes. If your instance is running LVM and you have an existing volume group that you could carve a partition out of, that would work just as well, and the allocation could be completed in just a few minutes.

    I think I should mention that carving out a swap space of this size is a bit unusual, even for a server; and I only say that because most servers have several gigs of RAM attached when dealing with programs/data of that size. Not to pry or anything, but are you really needing that much swap space?

    Another thing you may wish to consider is re-tuning your workload, rather than trying to dynamically allocate swap space. While it's great to have that much "on-demand", as you yourself pointed out, it will quickly become a bottleneck due to the slow I/O throughput on your server instance. By the time you exhaust your memory and you're essentially "living in swap", you'll find that the 20Mbyte/sec transfer rate turns your instance into a 386SX.

    • 2

  3. Turns out you can use a loopback device to put swap on sparse file: http://www.option-c.com/xwiki/Loop_Mounting_Swap_Partition

    • 0

  4. Your c1.xlarge instance should have, or be able to have, some substantial local instance store (ephemeral, won't persist across launch). You may have to be sure the space is allocated when launching. I've not done this so I can't give you details. I would definitely not use a swap file.

    • 0