Caching/preloading files on Linux into RAM

vmtouch seems like a good tool for the job.

Highlights:

• query how much of a directory is cached
• query how much of a file is cached (also which pages, graphical representation)
• load file into cache
• remove file from cache
• lock files in cache
• run as daemon

vmtouch manual

EDIT: Usage as asked in the question is listed in example 5 on vmtouch Hompage

Example 5

Daemonise and lock all files in a directory into physical memory:

vmtouch -dl /var/www/htdocs/critical/

EDIT2: As noted in the comments, there is now a git repository available.

This is also possible using the vmtouch Virtual Memory Toucher utility.

The tool allows you to control the filesystem cache on a Linux system. You can force or lock a specific file or directory in the VM cache subsystem, or use it to check to see what portions of a file/directory are contained within VM.

How much of the /bin/ directory is currently in cache?

$ vmtouch /bin/
           Files: 92
     Directories: 1
  Resident Pages: 348/1307  1M/5M  26.6%
         Elapsed: 0.003426 seconds

Or...

Let's bring the rest of big-dataset.txt into memory...

$ vmtouch -vt big-dataset.txt
big-dataset.txt
[OOo                                                 oOOOOOOO] 6887/42116
[OOOOOOOOo                                           oOOOOOOO] 10631/42116
[OOOOOOOOOOOOOOo                                     oOOOOOOO] 15351/42116
[OOOOOOOOOOOOOOOOOOOOOo                              oOOOOOOO] 19719/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOo                        oOOOOOOO] 24183/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo                  oOOOOOOO] 28615/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo              oOOOOOOO] 31415/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo      oOOOOOOO] 36775/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo  oOOOOOOO] 39431/42116
[OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 42116/42116

           Files: 1
     Directories: 0
   Touched Pages: 42116 (164M)
         Elapsed: 12.107 seconds

A poor man's trick for getting stuff into the filesystem cache is to simply cat it and redirect that to /dev/null.

Linux will cache as much disk IO in memory as it can. This is what the cache and buffer memory stats are. It'll probably do a better job than you will at storing the right things.

However, if you insist in storing your data in memory, you can create a ram drive using either tmpfs or ramfs. The difference is that ramfs will allocate all the memory you ask for, were as tmpfs will only use the memory that your block device is using. My memory is a little rusty, but you should be able to do:

 # mount -t ramfs ram /mnt/ram 

or

 # mount -t tmpfs tmp /mnt/tmp

and then copy your data to the directory. Obviously, when you turn the machine off or unmount that partition, your data will be lost.

After some extensive reading on the 2.6 kernel swapping and page-caching features I found 'fcoretools'. Which consists of two tools;

• fincore: Will reveal how many pages the application has stored in core memory
• fadvise: Allows you to manipulate the core memory (page-cache).

(In case someone else finds this interesting I'm posting this here)

There are two kernel settings that can help considerably even without using other tools:

swappiness

tells linux kernel how aggressively it should use swap. Quoting the Wikipedia article:

Swappiness is a property for the Linux kernel that changes the balance between swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value means the kernel will try to avoid swapping as much as possible where a higher value instead will make the kernel aggressively try to use swap space. The default value is 60, and for most desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may improve interactivity (decreasing response latency.)

vfs_cache_pressure

Quoting from vm.txt:

Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. ...

By setting swappiness high (like 100), the kernel moves everything it doesn't need to swap, freeing RAM for caching files. And by setting vfs_cache_pressure lower (let's say to 50, not to 0!), it will favor caching files instead of keeping application data in RAM.

(I work on a large Java project and every time I run it, it took a lot of RAM and flushed the disk cache, so the next time I compiled the project everything was read from disk again. By adjusting these two settings, I manage to keep the sources and compiled output cached in RAM, which speeds the process considerably.)

You may be able to have a program that just mmaps your files then stays running.

If you have plenty of memory you can simply read in the files you want to cache with cat or similar. Linux will then do a good job of keeping it around.

I very much doubt that it is actually serving files from the disk with 3 GB RAM free. Linux file caching is very good.

If you are seeing disk IO, I would look into your logging configurations. Many logs get set as unbuffered, in order to guarantee that the latest log information is available in the event of a crash. In systems that have to be fast regardless, use buffered log IO or use a remote log server.

Desktop computers (eg. ubuntu) already uses preloading files (at least, popular shared libraries) to memory on boot. It is used to speed up booting and startup time of different bloarware like FF, OO, KDE and GNOME (with evolution bloat-mailer).

The tool is named readahead http://packages.ubuntu.com/dapper/admin/readahead

There is also corresponding syscall: readahead(2) http://linux.die.net/man/2/readahead

There is also project of preloading daemon: http://linux.die.net/man/8/preload