Basic set up of dm-cache

I was starting on this info on get most out of your nvme, but also man lvmcache is a good resource.

I have (sorry) not more disks around:

/dev/sda2 (931G slow)
/dev//dev/nvme0n1 (372.6G fast)

Basic setup:

$ sudo pvcreate /dev/sda2
  Physical volume "/dev/sda2" successfully created.
$ sudo pvcreate /dev/nvme0n1
  Physical volume "/dev/nvme0n1" successfully created.
$ sudo vgcreate cache /dev/sda2 /dev/nvme0n1
  Volume group "cache" successfully created
$ sudo lvcreate -L 200G -n origin_device cache /dev/sda2
  Logical volume "origin_device" created
$ sudo lvcreate -L 60G -n cache_block cache /dev/nvme0n1
  Logical volume "cache_block" created.
$ sudo lvcreate -L 2G -n cache_meta cache /dev/nvme0n1
  Logical volume "cache_meta" created.
$ sudo lvconvert --type cache-pool /dev/cache/cache_block --poolmetadata /dev/cache/cache_meta
  WARNING: Converting logical volume cache/cache_block and cache/cache_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert cache/cache_block and cache/cache_meta? [y/n]: y
  Converted cache/cache_block to cache pool.
$ sudo lvconvert --type cache /dev/cache/origin_device --cachepool /dev/cache/cache_block
  Logical volume cache/origin_device is now cached.

After that you can use that device "as usual". I also created a non cached device as reference for a basic test:

$ sudo lvcreate -L 200G -n origin_device_reference cache /dev/sda2
  Logical volume "origin_device_reference" created.
$ sudo mkfs -t xfs /dev/cache/origin_device
$ sudo mkfs -t xfs /dev/cache/origin_device_reference

And mounted it

$ sudo mkdir /mnt/lv-xfs
$ sudo mkdir /mnt/lv-xfs-cached
$ sudo mount /dev/cache/origin_device_reference /mnt/lv-xfs
$ sudo mount /dev/cache/origin_device /mnt/lv-xfs-cached

After this my setup looked like this:

$ lsblk (filtered of other disks)
NAME                              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                 8:0    0 931.5G  0 disk 
|-sda2                              8:2    0   931G  0 part 
| |-cache-origin_device_reference 252:4    0   200G  0 lvm  /mnt/lv-xfs
| `-cache-origin_device_corig     252:3    0   200G  0 lvm  
|   `-cache-origin_device         252:0    0   200G  0 lvm  /mnt/lv-xfs-cached
nvme0n1                           259:0    0 372.6G  0 disk 
|-cache-cache_block_cdata         252:1    0    60G  0 lvm  
| `-cache-origin_device           252:0    0   200G  0 lvm  /mnt/lv-xfs-cached
`-cache-cache_block_cmeta         252:2    0     2G  0 lvm  
  `-cache-origin_device           252:0    0   200G  0 lvm  /mnt/lv-xfs-cached

$ sudo dmsetup table
cache-cache_block_cdata: 0 125829120 linear 259:0 2048
cache-origin_device_reference: 0 419430400 linear 8:2 423626752
cache-cache_block_cmeta: 0 4194304 linear 259:0 125831168
cache-origin_device: 0 419430400 cache 252:2 252:1 252:3 128 1 writethrough smq 0
cache-origin_device_corig: 0 419430400 linear 8:2 2048

Please be aware that dm-cache has evolved a lot. There are still many guides that recommend to tune the cache with "dmsetup message ...", but that all is part of the old "mq" policy. See Kernel Doc. These days the Stochastic multiqueue (smq) is the default, considered superior and comes without any tuning knobs. That went as far as to drop "mq" since Kernel 4.6 and make is an alias to the smq policy.

So very basic benchmarking with two slow sync io sequential disk crawlers and two aio random hot spots (and not all of the crawlers fitting onto the cache, but the hotspots easily do). There are more details on the results if you want to look. Results are much better than without cache, but the testcase was in no way sophisticated enough to check that in any detail.

Uncached
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.10     0.20  259.95  126.45  1840.00   599.32    12.63    65.96  170.43  126.56  260.62   2.59 100.00
dm-4              0.00     0.00  260.05  126.65  1840.00   599.32    12.62    65.99  170.37  126.53  260.39   2.59 100.00
   READ: io=1109.4MB, aggrb=1891KB/s
  WRITE: io=370212KB, aggrb=616KB/s


Cached
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.85    1.75  395.75   112.00  1679.40     9.01    33.18   83.44   82.97   83.44   2.52 100.00
nvme0n1         755.05     0.00 159339.95    0.25 873790.40    16.00    10.97    25.14    0.16    0.16    0.00   0.01 100.12
dm-0              0.00     0.00 156881.90  395.95 873903.00  1679.40    11.13    58.35    0.37    0.16   84.19   0.01 100.12
dm-1              0.00     0.00 160095.25    0.25 873791.00    16.00    10.92    25.41    0.16    0.16    0.00   0.01 100.10
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    1.75  396.60   112.00  1679.40     8.99    34.50   86.51   82.97   86.52   2.51 100.00
   READ: io=415116MB, aggrb=708356KB/s
  WRITE: io=1004.8MB, aggrb=1714KB/s

This shall not become a discussion on bcache, dm-cache, ... as I stated at the beginning I usually prefer bcache as well but that is not the point. But if OTOH you have any recommendations for dm-cache to add please feel free to use the comment section.