Questions[free](server)

Management decided to downscale prod server.

Current size:   Standard B16ms (16 vcpus, 64 GiB memory)
Desired size:   Standard B8ms  (8 vcpus, 32 GiB memory)

CPU is mostly idle (output is from sar -u)

10:20:01        CPU     %user     %nice   %system   %iowait    %steal     %idle
10:30:01        all     15.46      0.00      4.24      0.19      0.00     80.11
10:40:01        all     17.11      0.04      4.21      0.18      0.00     78.47
10:50:01        all     19.29      0.00      4.76      0.52      0.00     75.43

What I'm concerned by, it's the memory:

root@precious-server:~# free -h
              total        used        free      shared  buff/cache   available
Mem:            62G        7.9G         26G        510M         28G         53G
Swap:           0B          0B          0B

It seems I have 53GB available, however free value is only 26GB.

Sar reports 53%-57% is used(%memused field), out of which only ~15%-18% is being committed (%commit field):

10:20:01    kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
10:30:01        28.9G     55.2G     33.9G     53.9%      2.6G     19.8G      9.7G     15.4%     24.1G      3.7G     11.5M
10:40:01        27.0G     53.3G     35.8G     57.0%      2.6G     19.8G     11.6G     18.5%     26.0G      3.7G      1.7M
10:50:01        26.8G     53.1G     36.0G     57.3%      2.6G     19.8G     11.8G     18.7%     26.2G      3.7G      4.3M

Grafana, reports only 15% of memory is being used:

So, you see my confusion? free command says I have 53GB available, sar says memory used is 52%, while Grafana says 15% used. I'm not sure I can do the downscale :)

What are your considerations gentlemen?

I'm aware of this question https://stackoverflow.com/questions/41224738/how-to-calculate-system-memory-usage-from-proc-meminfo-like-htop Answer seems outdated regarding memory usage and this question seems more complex.

htop definitely now doesn't calculate total memory via MemTotal - MemFree anymore. There https://access.redhat.com/solutions/406773 I found a soluation that could fit. According to that it would be calculated via MemTotal - MemFree - Buffers - Cached - Slab but that leads to a too low memory usage compared to htop.

Memory usage in htop seems to be quite related as free -m shows in cell Mem: used

But if you create a ramdisk, things change a lot:

mkdir mnt
sudo mount -t tmpfs none mnt
cd mnt 
sudo dd if=/dev/urandom of=test bs=32M status=progress

Above will create a RAM disk 50 % size of installed memory and fill this ramdisk.

htop now shows a bit more than half of the physical RAM used, in test case 3.82 GB of 7.44 GB used. However free -m shows still only 104 MB used compared to 102 MB before creating and filling the RAM disk. So 104 MB used is definitely wrong, because half of the physical system memory can't be available anymore since it is used by RAM disk.

In /proc/meminfo size of RAM disk seems to be included in Cached, so for calculating used memory cached shouldn't be substracted.

So why does free -m report such a wrong free memory value? By default there are various tmpfs on distributions like Debian or Ubuntu which then means free -m gives wrong impression of free memory.

Memory calculation of htop seems quite reasonable. So how can the htop value be calculated from /proc/meminfo?

From what I could understand from this change in Linux reported memory usage, /proc/meminfo reports MemFree as currently unused RAM, and MemAvailable as RAM currently available to new processes without swapping.

From this, shouldn't MemFree be contained in MemAvailable, thus being always less or equal than it? I guess unused RAM is also available to new processes without swapping. Or am I misunderstanding the "Free memory cannot be taken below the low watermark" (4th chunk from patch) part?

These values match, as expected, free and top outputs.

I saw this in an Exadata virtual machine with these specs:

cat /etc/oracle-release 
Oracle Linux Server release 7.8

uname -r
4.14.35-1902.306.2.1.el7uek.x86_64

rpm -q sysstat
sysstat-10.1.5-19.el7.x86_64

Full /proc/meminfo at an arbitrary moment, if it helps:

MemTotal:       19995464 kB
MemFree:         2963684 kB
MemAvailable:    1063344 kB
Buffers:          406124 kB
Cached:          2609256 kB
SwapCached:         9648 kB
Active:          4632640 kB
Inactive:        2348256 kB
Active(anon):    3882372 kB
Inactive(anon):   700524 kB
Active(file):     750268 kB
Inactive(file):  1647732 kB
Unevictable:      634040 kB
Mlocked:          634044 kB
SwapTotal:      16777212 kB
SwapFree:       16579068 kB
Dirty:              1684 kB
Writeback:             0 kB
AnonPages:       4592704 kB
Mapped:           384108 kB
Shmem:            484212 kB
Slab:             511064 kB
SReclaimable:     249660 kB
SUnreclaim:       261404 kB
KernelStack:       23184 kB
PageTables:        96648 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    24602016 kB
Committed_AS:    5737344 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:    2122
HugePages_Free:      590
HugePages_Rsvd:        1
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      833536 kB
DirectMap2M:    15943680 kB
DirectMap1G:     5242880 kB

Thanks,
Emerson

I have a java application that runs on a Linux server with physical memory(RAM) allocated as 12GB where I would see the normal utilization over a period of time as below.

sys> free -h
              total        used        free      shared  buff/cache   available
Mem:            11G        7.8G        1.6G        9.0M        2.2G        3.5G
Swap:            0B          0B          0B

Recently on increasing the load of the application, I could see the RAM utilization is almost full, and available space is very less where I could face some slowness but still application continues to work fine.

sys> free -h
              total        used        free      shared  buff/cache   available
Mem:            11G         11G        134M         17M        411M        240M
Swap:            0B          0B          0B
sys> free -h
              total        used        free      shared  buff/cache   available
Mem:            11G         11G        145M         25M        373M        204M
Swap:            0B          0B          0B

I referred to https://www.linuxatemyram.com/ where it suggested the below point.

Warning signs of a genuine low memory situation that you may want to look into:

• available memory (or "free + buffers/cache") is close to zero
• swap used increases or fluctuates.
• dmesg | grep oom-killer shows the OutOfMemory-killer at work

From the above points, I don't see any OOM issue at the application level and the swap was also disabled. so neglecting the two points. One point which troubles me was available memory is less than zero where I need a clarification

Questions:

1. In case available is close to 0, will it end up in a System crash?
2. Does it mean I need to upgrade the RAM when the available memory goes less?
3. On what basis the RAM memory should be allocated/increased?
4. Do we have any official recommendations/guidelines that need to follow for RAM memory allocation?

I'm serving on Images ( JPG|JPEG|PNG ) from an apache server. after sometime, the system cache becomes sooo populated that no other programs are able to run -- like recently I noticed that one of the php Cronjob got terminated because of no memory available to process a large number of files.

Can I configure apache2 in such a way that the images served aren't cached in RAM at all, and they should go directly form the storage subsystem. I know it will increase IO, but System will be able to perform other functions.

Secondly, as far as I know, these caches are circular in their operation. whenever there is need for some RAM space, older ( LRU? ) objects are kicked out of the RAM and needy programs gets inserts. I'm not seeing it happening here.

Its a Debian-8 Linux System, running Dell-R320 with 24 Cores + 15GB RAM. all filled up sooner or later.