What are the advantages/disadvantages of different SSD to HDD cacheing options (dm-cache, flashcashe...)?

I really don’t know where to start, since all this is excellent information. I will start with some info about SSD’s, then a description of all the different caching methods, and just go from there. I hope that you

Advantages/Disadvantages

• Price: SSDs are somehow expensive
• Maximum and Common Capacity: High capacity SSD’s are very rare and expensive
• Speed: This is where SSDs gets the edge
• Durability: An SSD has no moving parts

Best to have a hybrid system, to have the best of both worlds (capacity, reliability, speed, etc.)

The Linux 3.9 kernel (made available on April 28, 2013) introduces SSD caching. The kernel's Device mapper now includes a cache target called dm-cache that enables SSDs or other storage device to be used as a cache for a hard drive. It essentially speeds up data writes and reads as it allows the faster SSD to first cache data and then transfer it to the slower hard drive.

^Source:Iwn

Flashcache is a module originally written and released by Facebook(Mohan Srinivasan, Paul Saab and Vadim Tkachenko) in April of 2010. It is a kernel module that allows Writethrough caching of a drive on another drive. This is most often used for caching a rotational drive on a smaller solid-state drive for performance reasons. This gives you the speed of an SSD and the size of a standard rotational drive for recently cached files. FlashCache is a general purpose writeback block cache for Linux.

^{Source:ArchLinux}

Bcache is a Linux kernel block layer cache. It allows one or more fast disk drives such as flash-based solid state drives (SSDs) to act as a cache for one or more slower hard disk drives.

Hard drives are cheap and big, SSDs are fast but small and expensive. Wouldn't it be nice if you could transparently get the advantages of both? With Bcache, you can have your cake and eat it too.

Bcache patches for the Linux kernel allow one to use SSDs to cache other block devices. It's analogous to L2Arc for ZFS, but Bcache also does writeback caching (besides just write through caching), and it's filesystem agnostic. It's designed to be switched on with a minimum of effort, and to work well without configuration on any setup. By default it won't cache sequential IO, just the random reads and writes that SSDs excel at. It's meant to be suitable for desktops, servers, high end storage arrays, and perhaps even embedded.

The design goal is to be just as fast as the SSD and cached device (depending on cache hit vs. miss, and writethrough vs. writeback writes) to within the margin of error. It's not quite there yet, mostly for sequential reads. But testing has shown that it is emphatically possible, and even in some cases to do better - primarily random writes.

^{Source:Bcache}

Bcache has a big disadvantage, and that it takes away memory from the system to implement the cache.

EnhanceIO is a solution that runs beneath the application layer, enabling applications to utilize the performance benefits of SSDs without major IT infrastructure changes. An SSD cache can yield most of the benefits of switching from HDDs to SSDs at a fraction of the cost of an all-SSD system. A cached system typically operates on less power than an HDD-based system of similar performance, and that creates a side benefit by reducing cooling requirements.

An SSD cache can also extend the useful life of an existing system by improving performance to meet growing demands through an incremental investment, rather than through a wholesale upgrade/replacement of the existing system.

Caching also enables faster access to the data without the extra storage administration overhead to acquire and install new disk shelves, configuring new LUNs and migrating data to the new LUNs. Caching is almost transparent and requires little if any downtime. EnhanceIO is based upon Flashcache.

^{Source:Stec-Inc}

bcache is the most worthless of all because it requires specially prepared (formatted) data partition. This makes it difficult (if possible) to attach cache to existing partition with data as one would need 200% capacity and to perform long data-moving in order to activate/deactivate caching.

The brilliance of EnhanceIO is that it doesn't need intermediate device at all and can be attached to any block device on-the-fly even when device is already mounted. Another super-cool thing is that you can attach EnhanceIO cache not just to partition but to partitioned block device to cache all its partitions at once. Just like flashcache enchanceio modules are built with DKMS and can be used with older kernels.

^{Source: Debian}

DM Cache Advantages

DM caches use a simplified architecture, which makes them adaptable and easy to customize. Users can adjust the block size and the cache capacity based on the amount of data it will have to handle or on the value of the data. If a particular application needs to store a great deal of data in sequence, users can configure the cache for that purpose. If a user wants to record information in a database simultaneously with the cache, that won't interfere with the cache's operations.

DM Cache Dis-Advantages

One drawback to using a DM cache is that the Linux operating system has limited space for storing metadata. If the cache is large, and includes lots of small blocks, that adds up to a lot of metadata for the stored information. To solve this problem, the user must increase the block size. Another possible problem is that, after a server crash, the cache metadata may no longer match the cache contents, though it is possible to restore the correct configuration eventually.

^{Source: Complements of Fraser Sherman}

So from the above information it’s clear that EnhanceIO is the way to go, but in my opinion since it’s based on Flashcache, I would go with flash cache. But I will definitely try both of them before making a final decision.

Feedback: I installed flashcache standard deb packages on Ubuntu about a year ago in write-back mode to cache /home (2TB) to a 50GB slice on a SSD. It has performed wonderfully with no problems. I have had a few X related "hangs" (ie not attributable to flashcache) and have used "Ctrl-Alt-PrtScr REISUB" a few times to recover with no problems and no corruption problems. I used flashcache because a) there were debs available and b) it did not require a backup and restore of /home.