Thomas Browne's questions

I have 24GB of available RAM in my machine, but I usually use far less than 8GB of it. However, sometimes I'm involved with large data files and I suddenly need 40GB, and on infrequent occasions, I'd like to be able to access even more (up to 60GB). This is mainly using numpy and Apache Arrow.

Right now I have a 20GB ssd swap file, and that works fine, but is obviously slow. I'd like to replace it with zRAM, and assuming say I get double compression on 16GB, that's 32GB, plus the leftover 8GB = 40GB total in "ram". Ideally though, I'd like to go even bigger than 40GB on occasion, and wonder if it's possible for the stuff that doesn't fit into zRAM to spillover further into a "traditional" SSD swap file or partition.

Is this doable? Obviously I'd want the zRAM to be prioritised over the disk swapfile.

Thinking ZFS had my back, yesterday I started messing with Gnome, Compiz, and dconf editor. After several hours, Gnome sure looked good with all sorts of addins, but after a reboot, my (complex, 3 monitor) setup went berserk, with Wayland very unhappy. So I figured I'd just go back to my Grub snapshot history to early that evening before the tinkering started (cos apt had made a few snapshots then).

Select older root snapshot (system and user), Reboot, seems okay. But then.... sudo apt update && sudo apt upgrade .... whole system gets confused. Starts messing with kernels and graphic driver updates. Apt was very unhappy. Very strange. And smashes the whole display driver setup, to the extent I couldn't recover the system and spent a half day today reinstalling 20.04. This time with ext 4 (though I was quite happy with ZFS up to now).

So I'm wondering. Are ZFS snapshots really recommended for "rolling back" the OS on a root partition? Or is it really good only for a data drive or a home directory. Does apt get confused in some way ie can this kind of rollback create unrecoverably inconsistent states?

Straight from cold Fossa boot, until usable user interface:

Blender 2.91.2: 4 seconds

Gimp: 11 seconds

Boxy SVG: 22 seconds

Spotify: 43 seconds

The latter two seem like electron apps and take forever. Since these electron apps seem to be increasingly common, is there a way to speed up their snap load times? Spotify in particular is egregious.

Using Ubuntu 20.04 Focal, with a dual monitor configuration, is it possible for X (or Wayland) to be configured only to use one monitor, and the other one to be pure TTY (as in, not a terminal emulator, just default boot-style TTY, unknown/unused by any GUI).

The reason I'd like this is because I have a very high resolution 5k x 2k monitor as primary, and my secondary one is much lower DPI. I don't like the idea that X-scaling "uses more resources" because this suggests some on-the-fly interpolation scheme which, apart from introduced latency, will almost certainly not play nice with sub-pixel font rendering, and will probably screw up integer scaling on graphical programs. Wayland by contrast just doesn't work with this configuration - multiple problems and basically crashes. So essentially I'd like to code in a terminal on the low-res TTY monitor, and have the main high DPI monitor run X (or wayland) with 100% scaling, no screwing around, though I'd probably raise the DPI setting.

Is this possible on one video card (Radeon VII). If not, would it be possible if running a secondary video card at the same time?

If it is possible, how would I switch focus of the keyboard between TTY monitor and GUI(Gnome) monitor? Run some kind of bash script to toggle focus maybe?

I am running a Jetson Nano with Jetpack 4.4 which (Ubuntu 18.04), by default, has 4x495 Meg ZRAM partitions for swap space. The problem is that for a data science project I need MORE RAM than the 4gig installed on the Jetson (and whatever I get from compressed ZRAM - not much). So I'd like to make an 8gb swap file instead of the ZRAM swap.

How do I disable the ZRAM swap altogether so that I can create a swap file?

I've tried

sudo apt remove --purge zram-config
sudo dpkg --purge zrawmswap-enabler
sudo dpkg --purge zram-config
sudo service zramswap stop

But none of it works. ZRAM swap is still enabled on reboot:

Alternatively can I just create a disk swap file and expect it to replace ZRAM?

On multiple versions of Ubuntu, Vim and Neovim display a horrible gap on the right hand side of the terminal when it is maximised.

This is Gnome terminal:

And this is Konsole:

This does not happen when the terminal is not maximised because it only then only expands to the right by what apears to be full character pixel widths in the case of Gnome terminal at least. This issue also only occurs with Vim colour schemes which paint the background a different colour to the terminal background, such as in the gnome picture above, dw_green, or for Konsole, aqua.

Is there any way to alter Vim/Neovim behaviour, to force them to paint right to the edge even when there isn't a full character's space there?

Can ZFS be used like a virtual machine snapshot? Using, say, VirtualBox or VMM, I can snapshot a non-running VM before running and installing new software on it, and always revert back to a working configuration if I screw things up. Can I do this using ZFS?

I've seen plenty of examples of snapshotting my home directory which I can understand, but given the "fluid" nature of some of the files underneath root "/" is it unsafe to sudo zfs snapshot -r /@myrootsnapshot1 while the boot drive is mounted and Ubuntu is running?

And even more importantly, would a restore of such snapshot be safe? The objective is to snapshot before installing new software or trying out new configs, and roll back if I encounter problems.

If it's not safe, what is the recommended workflow for this objective?

I have 64GB installed but htop shows 20GB in use:

Running ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n gives me the largest processes using only a few 100 megabytes and adding up the whole output only gets me to 2.8GB (ps aux | awk '{print $6/1024}' | paste -s -d+ - | bc). This is more or less what I was getting with Ubuntu 19.04 from which I upgraded yesterday - 3 to 4GB used when no applications are running. So why is 20GB in use on htop?

Now it's true that I have installed ZFS (total 1.5 GB of SSD drives, in 3 pools, one of which is compressed), and I've been moving some pretty big files around so I could understand if there was some cache allocation. The htop Mem bar is mostly green which means "memory in use" as opposed to buffer (blue) or cache (orange) so it's quite concerning.

Is this ZFS using up a lot of RAM, and if so, will it release some if other applications need it?

EDIT

Here is the output of smem:

tbrowne@RyVe:~$ smem -tw
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory      20762532     435044   20327488 
userspace memory            2290448     519736    1770712 
free memory                42823220   42823220          0 
----------------------------------------------------------
                           65876200   43778000   22098200 

So it's "kernel dynamic memory" that's the culprit. Why so much?

EDIT 2 --> seems to be linked to huge file creation

I rebooted and RAM usage was circa 5GB. Even running a bunch of tags in firefox, running a few VMs, taking RAM up to 20GB, then closing all the applications it dropped back to 5GB. Then I created a big file in Python (1.8G of random number CSV), then concatenated that to itself 40x to produce a 72GB file:

tbrowne@RyVe:~$ python3
Python 3.8.2 (default, Mar 13 2020, 10:14:16) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pandas as pd
>>> pd.DataFrame(np.random.rand(10000, 10000)).to_csv("bigrand.csv")
>>> quit()
tbrowne@RyVe:~$ for i in {1..40}; do cat bigrand.csv >> biggest.csv; done

Now after it's all done, and nothing running on the machine, I have 34G in use by the kernel!

FINAL EDIT (to test the answer)

This python 3 script (you need to pip3 install numpy) will allocate about 1GB at a time until it fails. And as per the answer below, as soon as you run it, kernel memory gets freed so I was able to allocate 64 GB before it got killed (I have very little swap). In other words it confirms that ZFS will release memory when it's needed.

import numpy as np
xx = np.random.rand(10000, 12500)
import sys
sys.getsizeof(xx)
# 1000000112
# that's about 1 GB
ll = []
index = 1
while True:
    print(index)
    ll.append(np.random.rand(10000, 12500))
    index = index + 1

Seriously I didn't upgrade to 19.10 simply because I love Dingo's wallpaper so much that I didn't want to say goodbye.

With 20.04 coming up, and with Disco's batteries running out for the last time as its repos start failing, looks like my favourite Ubuntu wallpaper ever is reaching twilight. Goodbye Dear Friend.

Whoever did this design, IMO deserves a medal. The melding of animal and technical. The box back of the head reminiscent of a power supply, the subtly gaping mouth in ecstasy of the music..... the Cylon forehead.... the head tilted towards the stars...it was pure art.. with fuschia vectors.

Of course I could port Dingo to 20.04 but that would feel like an organ transplant. Better to let him vector through to the other side...

But I would like to know who designed him. Because you added a little something to my life, and you cemented my loyalty to Ubuntu. There was once a time when Apple owned this "we get Art" ethos. Ubuntu now arguably surpasses them.

I have a myuserid@remotemachine on my network to which I have access via public/private key ssh (ie, doesn't prompt for password, logs me straight in). Is there a way, from Python (or bash, or any other standard linux command) on my local computer, to read one of myuserid's environment variables on remotemachine without needing to run a pre-prepared remote script?

The up-arrow doesn't seem to work for command history in Konsole which I have installed in Ubuntu 19.04. Is this a KDE thing or should I change a setting somewhere?

Here are the general settings of my profile:

As a scientific software developer exploring GPUs, and particularly, the fast-moving ROCm ecosystem for AMD, as well as Vulkan, I'm frequently loading and testing new cutting-edge libraries, many of which are complicated and sometimes unstable. I don't want to clutter up my "base" system with all these installs, some of which may do stuff that hurts other bits of the system. I also frequently need to uninstall and reinstall. This problem is well known in Python development where the "virtual environment" (aka virtualenv) system has been developed, allowing custom "pip" environments with different package versions and even python versions inside them. So my question is:

Is a python-style virtualenv possible on Linux, and in particular, Ubuntu, which allows for apt (or snap or flatpak or indeed, source-based) installations that are sealed off from the base system?

Like Python virtualenvs, it would have to meet the following criteria:

a) All the base system packages should be visible from inside the "virtualenv".

b) Anything installed within the "virtualenv" should not be able to affect anything outside.

c) When I delete the virtualenv, it should remove all traces of itself and anything I've ever done inside it regarding software installations.

d) All the hardware must be visible, including GPU(s).

e) not necessary: different Linux kernel versions analogous to different python versions which virtualenvs allow.

I realise that there may be file-level problems with point c) because if you create files and directories in a python-level virtualenv, they'll remain afterwards. So source-based package installations may be problematic in this regard but I'm willing to be flexible on this front.

One technology which I don't want to use is virtualization. This does not meet criterion a) above and frequently doesn't meet d). My "base" system is already quite complex with frequently used things and I don't want to have to reinstall them each time. Grub-based multiple OS partitions also doesn't meet criterion a).

Can docker do this? I've never been able to figure out how to create a kind of "bash command line" within a container that I can then start installing stuff into which is then sealed off nicely. Do I have to go through a trial-and-error "docker compose" style workflow to meet this objective?

What about KVM, Snap, flatpak, or even lower level, cgroups? Some kind of file-system-level technology perhaps like ZFS or btrfs?

What are my options?

I know I can redirect all terminal output to a file, but does that work the other way around?

Can I have a terminal window open, in Tmux or elsewhere, that is receiving redirected output from a file in real time? IE if another process is writing to the file, this is being directed to a terminal window?

Use case: REPL swamped by output

The reason I would like this is I would like to see the output of some code in a REPL (Erlang it turns out) which has many processes creating terminal output stuff in background, but this output is very rapid and I "lose" my REPL command line constantly as it is drowned by the output. So I'd like those processes to output to a file instead, but, in another terminal, I still want to see what that output is in real time.

I am trying to run

setxkbmap -option caps:escape 

in my rc.local file under /etc/init.d/ in Kubuntu 15.04, and no matter where I put it, beginning or end, or with exec in front of it, it is not running. Basically, I want to be able to run gvim with caps as escape, and I don't really want to put this into .bashrc as that will run every time I open a terminal (unnecessary) and also I would like to be able to run gvim without first opening bash, and still having caps as escape.

This command works fine in the command line.

The whole rc.local looks like this:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          rc.local
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Run /etc/rc.local if it exist
### END INIT INFO


/usr/bin/setxkbmap -option caps:escape
PATH=/sbin:/usr/sbin:/bin:/usr/bin

. /lib/init/vars.sh
. /lib/lsb/init-functions

do_start() {
       if [ -x /etc/rc.local ]; then
               [ "$VERBOSE" != no ] && log_begin_msg 
"Running local boot scripts (/etc/rc.local)"
               /etc/rc.local
               ES=$?
               [ "$VERBOSE" != no ] && log_end_msg 
$ES
               return $ES
       fi
}

case "$1" in
   start)
       do_start
       ;;
   restart|reload|force-reload)
       echo "Error: argument '$1' not supported" >&2
       exit 3
       ;;
   stop)
       ;;
   *)
       echo "Usage: $0 start|stop" >&2
       exit 3
       ;;
esac

Am I doing something wrong? Anything else to do to get the same result? I am not running Unity so compizconfig won't work.

I am currently running Ubuntu 13 Desktop edition on a second PC on my desk, which has one small monitor. For work reasons my primary machine must be Windows(7), and it has multiple monitors, of varying resolutions.

I wish to have my Ubuntu Desktop fill up all those Windows monitors at their native resolutions so that I can basically make Ubuntu my primary desktop, without physically plugging in the keyboard and monitors into the Ubuntu box. Both machines are connected within a fire walled internal 1Gb network, and both have fixed IP addresses.

Nomachine NX is working pretty well up to now, but seems limited to aping the exact resolution of the (small) monitor I have on the Ubuntu box. VNC the same.

I do not wish to use a VM solution such as VMWare (though this does work), as I need more than one computer for reasons of data analysis throughput.

Is this possible? Is it viable? And how exactly do I go about it?

I have already downloaded and run MobaXterm, and running terminals over SSH seems to work fine. I have also run Firefox and other Ubuntu-machine applications in windows on my Windows machine, by calling them from the SSH terminal command line. But no luck so far getting the full Unity monty across my Windows monitors.