0

Situation: to study the effect of zram's virtual memory compression (wiki) on memory's size and its speed; when swap is not on HDD/SSD; use zswap if you want to have swap on HDD/SSD
Motivation: I experience no effect on the memory's size but increased amount of errors from systemd/... with runit i.e. possible stability issues with the compression, although paging should be avoided more
Code

# http://askubuntu.com/a/800645/25388
modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm
echo 4G > /sys/block/zram0/disksize
mkswap --label zram0 /dev/zram0
swapon --priority 100 /dev/zram0

Characteristics

  • Command cat /sys/block/zram0/orig_data_size gives 4096.

  • RAM full and swap full. So computation ends when total memory consumption reaches the high about 20 GB, failing the computation.

    enter image description here

  • Command swapon baed on cat /proc/swaps

    NAME       TYPE      SIZE USED PRIO
/dev/sda3  partition   8G   0B   -1
/dev/zram0 partition   4G   0B  100

  • Command free -m 

                 total        used        free      shared  buff/cache   available
Mem:         7925         1390        5465      172     1069         6074
Swap:        12226        0           12226

  • Command df -k | grep sda2 about current memory use in SSD, possibility to extend it for Swap. 

    Filesystem     1K-blocks     Used      Available Use% Mounted on
/dev/sda2      108299224     76880916  25893932  75%  /
% df -h
/dev/sda2      104G          74G       25G       75%  /

Test code: any memory-heavy Matlab computation.

Output: the size of matrix does not increase with the implementation. Expected output: increase in memory size.

System: Linux Ubuntu 16.04 64 bit
Hardware: Macbook Air 2013-mid

Léo Léopold Hertz 준영
  • 6,928
  • 2
    some useful things to know about swap and about zram: 1. anything swapped out stays swapped out in swap until it's needed again (or discarded, e.g. the program exits). 2. zram uses memory, so when used as swap it effectively gives you transparently compressed & decompressed RAM that's only a little slower to access than normal RAM (but a LOT faster than SSD or HDD swap), but see point 1 (swapped stuff stays swapped). BTW, disk storage has nothing to do with memory or RAM so the df output is irrelevant...showing free -m would be more useful. – cas Jul 24 '16 at 09:30
  • 1
    also btw, unless you like to divide by 1048576 every time, swapon without the -s option gives more useful output. – cas Jul 24 '16 at 09:31
  • 2
    btw, a quick google tells me that the Macbook Air has soldered-on RAM so can't be upgraded (it could come with a maximum of 8GB which had to be chosen at purchase time). If you really need to run large matlab jobs, I suggest running them on a remote machine that has 16 or 32GB RAM or more. Perhaps you have access to a cluster or other large machine via ssh. if not, build yourself a linux box with as much RAM as you can stuff into it (32 or 64GB or more). 16GB DDR4 sticks are around $100 each these days and most DDR4-capable motherboards can take up to four of them. – cas Jul 24 '16 at 09:44
  • 1
    for older DDR3 motherboards, 8GB sticks are more readily available than 16GB but still cost around $100 per 16GB....but most m/bs are still limited to 4 DIMMs, so a max of 32GB rather than 64GB. prices are approximate, in AUD, and current at time of writing. – cas Jul 24 '16 at 09:45
  • 1
    you set the swap priority when you run swapon (or in /etc/fstab). AFAIK, to change the priority of a swap device you have to run swapoff on it first, and then swapon again with the new priority. – cas Jul 24 '16 at 10:25
  • Yes. Do you know about zram and its effect on the size of total RAM size? I need to compute with big matrices. – Léo Léopold Hertz 준영 Jul 24 '16 at 10:28
  • as for increasing RAM with zram....that's a tricky question. It does effectively increase available RAM (like any swap space does) but it does so by using some of your RAM. Depending on the type of data you're swapping, you might get 10% or 50% or even better compression (e.g. over 90% on plain text or similar). or you may get no compression for some data. and, as previously mentioned, stuff that gets swapped out stays in swap until the kernel has a reason to swap it back in again. Overall, if you're using even moderately compressible data you're better off with zram than without it. – cas Jul 24 '16 at 10:29
  • 1
    zram is not magic. it can help, but it still won't let you run a job that needs, e.g., 32GB of RAM on a system with only 8GB RAM + another 8GB of SSD or HDD swap. – cas Jul 24 '16 at 10:30
  • @cas I have 30 GB free space still in my SSD. I am trying to turn some of it to swap here http://unix.stackexchange.com/q/297767/16920 So probably, we can get 8 GB RAM + 20 GB of SSD swap. – Léo Léopold Hertz 준영 Jul 24 '16 at 10:32
  • 1
    you probably want to use zswap. This works like zram but can also use an SSD or HDD. see http://askubuntu.com/questions/471912/zram-vs-zswap-vs-zcache-ultimate-guide-when-to-use-which-one – cas Jul 24 '16 at 10:44
  • Finally, swapping is something to be avoided if at all possible. if you can upgrade your system's RAM, then do so. If you can offload the RAM-hungry task to another machine, then do so. Swap is always a last resort, because it's slow. If you need to run large matlab etc jobs frequently then no amount of swap space will relieve the frustration and tedium of waiting for it to finish (or make your machine interactively usable while it's thrashing data in and out of swap)...which is why i suggested building a linux box you can connect to via ssh to run large memory-hungry jobs on. – cas Jul 24 '16 at 10:51

1 Answers1

1

I stop optimising swap memory as instructed by cas, focused on other things. Please, note cas' comment

Finally, swapping is something to be avoided if at all possible. if you can upgrade your system's RAM, then do so. If you can offload the RAM-hungry task to another machine, then do so. Swap is always a last resort, because it's slow. If you need to run large matlab etc jobs frequently then no amount of swap space will relieve the frustration and tedium of waiting for it to finish (or make your machine interactively usable while it's thrashing data in and out of swap)...which is why i suggested building a linux box you can connect to via ssh to run large memory-hungry jobs on.

Léo Léopold Hertz 준영
  • 6,928