Quick and dirty summary: I want something like a write cache that drains only when the system isn't busy.
I want something along the lines of this question RAM disk and physical RAID with a slight twist.
For a particular build I'm doing often, I would like to keep build output in a ramdisk that eventually gets written out to disk. One particular target I'm building is really a bunch of cp, tar, gzip and such so I'm disk bound. In a tight debug cycle I want it to be fast, and don't care so much if the build output is destroyed by power-off. However, it would be nice if the ramdisk was synced out to disk when the system wasn't busy. I can imagine doing this with a a cron job, but I'm asking you all on the off chance there's a more coherent solution that combines the characteristics of the other question (ramdisk that overflows to disk) with this new twist (given enough idle disk time, the contents of the ramdisk make it out to disk too). Idealy, the whole thing looks like a single mountpoint, where I set the total size and the amount of ram to use.
--- updated ---
I don't think the page cache does what I want because I really want pretty fast write performance.
What you're asking for is a deferred write mechanism, i.e. writes to the ramdisk receive priority over writes to permanent storage, but all data eventually writes to disk, correct?
As funny as it sounds, you might be able to get away with using an LVM mirror to complete this process.
Make the ramdisk a member of a volume group with a physical drive.
Mirror the ramdisk to your hard drive. Note that LVM mirrors are direction-specific, i.e. data flows from one PV to another unidirectionally.
Mount the LVM volume somewhere as a unified filesystem.
Writes are spooled up and written to the LVM-based ramdisk (and by virtue of the mirror, the physical drive as well). Unlike a RAID-1 where the writes are synchronous and parallel (both drives write out at once), an LVM mirror is asynchronous and sequential (the primary drive receives the write, then LVM pushes the write to the mirror). This comes close (but not quite 100%) to the behaviour you're looking for. Keep in mind that LVM does put pressure on pending writes to clear to disk, so any "idling" you see will be measured in seconds at best, partial seconds at worst.
The flip side of this arrangement is that you now have a very nice persistence mechanism. When you start up, create your ramdisk and mirror the existing drive to it; once the mirror is complete, break the mirror, and reverse the direction (ramdisk -> hdd). This means every restart will result in your data being put into the ramdisk, and just before shutdown, written from ramdisk back to a hard drive. It could probably be scripted in perl or a shell script.
I'm sure that there are other ways to do this, but this is the quick'n'dirty version. I'll think about it a bit more and see what I can figure out.
What you describe is exactly what your operating system's disk cache is supposed to do. Modern OS are very good at this if they have enough memory at their disposal.
In my opinion: Give your machine enough RAM and let the OS do the hard work.
Couple of wacky ideas. First, Puppy Linux kinda does this; it syncs your working space to permanent every so often. It seems to do this with smart scripts and simple copy commands.
Second ... what if you scheduled an rsync between ramdisk->real disk every so often?