How to Increase Ubuntu's Virtual Memory and/or Swap for Matlab?

You cannot dedicate swap for a software. What you can do is create a shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits.

Here's an example script which creates a swap of 10Mb in the /tmp directory, mounts it, starts R (I don't have matlab), wait R exits, umount the swap file and delete it.

Please note that: - you will have a warning because the swap file is not owned by root. That's because the system will use if for any software, maybe not ran by you, and you can read on this file... I let you fix it. - if you [ctrl]-[c] the script, or logoff, or etc., the swap will remain mounted. I let you fix it as well.

#!/usr/bin/env bash

SWAP_FILE=/tmp/my_swap_file
SIZE_MB=10
TO_RUN="R"

dd if=/dev/zero of=${SWAP_FILE} bs=1M count=${SIZE_MB}
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}

OK, quite a list you have there. Let me respond inline

1. How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?.

I don't like the concept of this script at all. That you have an external harddrive that you're trying to use as swap is just a bad idea. If you really intend to do this on a regular basis then resize your partitions to put a proper swap partition in, add a swap file, or just buy a bigger internal disk.

2. How to put warnings if the matrix size exceeds the swap size?

Just do do the math. If you know the size of the matrix before the program begins then compute the size in MiB and compare it to the available swap.

3. How to have a progress bar in computing your huge matrix in Matlab?

matlab has an API right? I don't think this is the right forum for that question. Even if you had an API, you'll be blocking on IO via swap so it'll just be a jerky progress bar that doesn't actually reflect reality.

4. How to kill busy progress and/or swapon -s/swapoff in iteration (2)?

You don't. just because you're done with the computation doesn't mean the operating system is done with the resources you allocated. When it's done writing out to swap, it'll free up. You've consumed so much memory that lots of applications can't get the memory they need so they're using swap too. Just leave it on and let the operating system do it's thing. Before you perform your next run clear the caches.

echo 3 > /proc/sys/vm/drop_caches 

There's probably more to it than that, I'm not a Linux VM expert. It would be worthwhile to investigate how the SLAB/SLUB allocator works and how to tune it for your large memory requirements. You may be able to MLOCK matlab into memory. That forces the OS to reserve memory for you, or it just doesn't start, you also have to unlock it when you're done. I can do this with the C API just fine but I not sure how you would do that outside of a process I can't recompile, that would require some research.

Finally, this is the sort of stuff EC2 was made for. It looks like 16G is what you need, a m4.4xlarge has 64G ram @ $0.958 per Hour. That's less than a cup coffee. Script your install of matlab using a juju charm or similar and turn the whole thing into a computation as a service.

Is 16G 16 GB?

• Yes, normally when we leave off the suffix we mean base2 numbers in units of bytes. If you want to concise you would write 16GiB.

"I need matrices that are > 100 GB. I do not know if you can do it with EC2."

• EC2 has machines with up to 2TB main memory. See for yourself. https://aws.amazon.com/ec2/instance-types/

Should you clear your caches also by echo 3 > /proc/sys/vm/drop_caches?

• Yes, it doesn't hurt to always do that. See Documentation/sysctl/vm.txt in the linux kernel.

How can you MLOCK Matlab into memory?

• man mlock. Though I goofed when I quoted that. This call assures that you can allocate all the memory you want and keep in from being swapped out, it'll never use virtual memory. That's not what you want.

I think you can bind C API to Matlab. - - Do you have any idea of turning off swap if there is any failure in the processes?

• I'm going to be frank here, the concept of micromanaging swap files in the manner you propose is ridiculous. The operating system's job is to manage resources and hand them out in a fair & consistent manner. Once you give it more resources, it's going to use them as it sees fit. You don't get to tell it when you're done and yank resources out from underneath it, the OS tells you when it's done.

When I ask the OS for an memory address space, sometimes it doesn't always succeed, that doesn't mean I can't try again. That matlab can't figure out to call malloc twice is matlab's problem.

So, to affect the change you want, if that 100G of space is really at a premium then you need to figure out how to tell the operating system to trim it's memory footprint (by clearing caches for starters) so the memory manager doesn't feel the need to use the additional swap space that it was provided. Then and only then can you ask the memory manager to release the swap file.

It's easy to grow things like memory and disks, it's a lot harder to shrink them. Shrinking forces a re-balancing of every user who has resources allocated in that space. If I instead said "I have a 100TB storage array but now I only need 60TB, why is it when I remove 40TB of disk that the array stops working?" Well, the answer would be obvious right?

So here are your options as I see it.

1. investigate the matlab C API to see if you can get better control over how memory is allocated for these massive working sets.

2. refactor your computation to compute what you have now using sub-matrices or some other sparse data representation.

3. write your own program in C/C++ using the plethora of linear algebra libraries out there to perform the computation and use malloc or mmap anonymous to allocate the address space you need.

Use zswap if you have swap portions on HDD/SSD. Tho module zram is without swap portions on HDD/SSD, so Hakala's answer is not applicable. See the thread zram vs zswap vs zcache Ultimate guide: when to use which one for explanations. Setup zswap as described in the thread How to Activate Zswap Successfully for Matlab Computation in Ubuntu 16.04?

• Replace the corresponding line with the following line in /etc/default/grub

  # https://wiki.archlinux.org/index.php/Zswap
  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lzo"
  

• Run sudo update-grub.

I would at least test how well compressed RAM (zram kernel module, available since kernel version 3.14) performs.

Following archlinux wiki instructions

modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm
echo 4G > /sys/block/zram0/disksize
mkswap --label zram0 /dev/zram0
swapon --priority 100 /dev/zram0

My guess is that compressed RAM should be faster than disk I/O.

To keep the change also after the restart, place boot time commands in /etc/rc.local and run sudo systemctl enable rc-local.service.

This is how you can expand your SWAP memory using an external hard drive:

• First, write down your actual SWAP memory by running:

  free -m
  

• Second, have at hand the folder of your HDD. It should be something like /media/myhdd.

• Decide the size of extra SWAP you want to add. Let's say, X GB.
• Calculate the amount of bytes that quantity is. Using GB, this is: Y=X*1024^3, where Y is the result of your calculation.
• Choose the block size of the file (it is in bytes). Let's use the default here: 4096 (See more about this here).
• Calculate the number of blocks the file will have: Z=Y/4096

• Create a file of size X GB using the following command in a terminal:

  dd if=/dev/zero of=/media/myhdd/swapfile bs=4096 count=Z
  

• Create the SWAP in the file using:

  sudo mkswap /media/myhdd/swapfile -f
  

• Finally, enable the SWAP:

  sudo swapon -p 1000 /media/myhdd/swapfile
  

Now your SWAP has increased. Check again with free -m

We can set up this in a non-interactive script (sudo powers needed):

#!/bin/bash

### Inputs ###

swap_GB=$(expr 1) # Enter here size of the swap memory to create, in GB.
swap_bs=$(expr 4096) # Enter here block size, in bytes (must be a multiple of 8).
HDD_folder="/media/myhdd/" # Enter absolute path of HDD inside the brackets.

### Swap creation ###

swap_size=$(expr $swap_GB \* 1024 \* 1024 \* 1024 / $swap_bs)
dd if=/dev/zero of=${HDD_folder}"/swapfile" bs=$swap_bs count=$swap_size
mkswap ${HDD_folder}"/swapfile" -f
swapon -p 1000 ${HDD_folder}"/swapfile"

### EOF ###

PS: please optimize/correct if possible. As said, it's my first script ever :)