I recently learned that Ubuntu uses disk cache / filesystem cache (size is displayed in system monitor). I couldn't see any documentation that says there is a default limit on how much RAM can be used for disk cache. So, I assumed that disk cache size can grow till it completely consumes RAM.
However, I noticed something that says otherwise. I'm training a deep neural network with a large dataset. On one machine with 32GB RAM, disk cache grows till 27GB (rest other applications are using 3-4 GB and hence, around 1GB of RAM is still free). Because of this disk caching, the subsequent training epochs are much faster. But on another machine with 64GB RAM, disk cache grows only till 18GB (same dataset, same program, more than 30GB RAM is still free). Because of this, it is not able to cache the entire dataset and hence my subsequent training epochs are not benefitting from this disk caching.
I don't understand why the second machine is not using the additional RAM to cache the entire dataset. The only reason I can think of is that there is a limit set on how much RAM can be used for disk cache on the second machine, but not on the first machine. Is this true? If so, how can I increase this limit?
Idk if this matters, the first machine has Ubuntu 18.04 and the second one has Ubuntu 20.04.
PS: I searched on this site, unix.stackexchange, stackoverflow and google, but couldn't find anything (may be I'm not using the right keywords?)