Persistent live drive with a partition for persistence

Yes, there are some web sites describing new tools and new features in [new versions of] old tools that can create a persistent live drive from an iso file with Ubuntu 19.10 and the corresponding Ubuntu family flavours.

• A new feature or fixed bug ('feature request' or 'bug fix' according to this link) makes it possible to use a partition for persistence, when booting directly from a cloned or extracted copy from the iso file.

This makes it much easier to create a live drive with a partition for persistence with Ubuntu 19.10.

• In previous versions of Ubuntu it was possible to use a file in a FAT32 file system for persistence. Its size is limited to 4 GiB. It was possible to boot via grub in a separate partition (from the content of the iso file) and use a partition for persistence. But this is more complicated so not suitable for manual methods.

• In 19.10 and future versions the size of a partition for persistence is only limited by the size of the drive (USB pendrive, SSD, HDD, memory card).

• Debian 10 live iso files have also this feature, and can be used to create persistent live drives with a partition for persistence.

• Newest new: In the developing Focal Fossa, to be released as 20.04 LTS, the default label of the partition for persistence is changed from casper-rw to writable. This is implemented in mkusb-plug version 2.5.5, (which is bundled with mkusb version 12.4.3 in the PPA).

Tools

Ubuntu and Debian

• mkusb - works with all current versions of Ubuntu

If you run standard Ubuntu live, you need an extra instruction to get the repository Universe. (Kubuntu, Lubuntu ... Xubuntu have the repository Universe activated automatically.)

    sudo add-apt-repository universe  # only for standard Ubuntu

    sudo add-apt-repository ppa:mkusb/ppa  # and press Enter
    sudo apt-get update
    sudo apt-get install mkusb mkusb-nox usb-pack-efi

• mkusb-minp - small stand-alone shellscript that can use this new feature in Ubuntu 19.10. It works with Debian 10 too.

Please read the warnings when you try older versions or re-spins (distros based on Ubuntu)

Beginning of dialogue:

End of dialogue:

• mkusb-plug - small set of shellscripts that can use this new feature in Ubuntu 19.10. It works with Debian 10 too. I think you will find things easier with this new and very safe tool with a graphical user interface.

Windows

• Update: Rufus - a well-known tool that can use this new feature in Ubuntu 19.10. Be sure to use the current version (Rufus 3.9 or newer),

• works well to create live-only drives with all current Ubuntu versions,

• can create persistent live drives with Ubuntu 19.10 and future versions. Please upgrade to version 3.9 (or newer), which creates a good ext3 file system and can make good persistent live drives.

• Please read the changelog at the web site, and the warnings (pop-up windows), they can help you avoid serious mistakes.

• Edit: Rufus 3.9 has arrived :-)

  

• Simple method where the iso file is edited before cloning

This new feature in Ubuntu 19.10 can be used also from Windows,

• Use HxD (a binary editor) to edit the iso file in order to replace two cosmetic boot options 'quiet splash' with 'persistent ' (replace 12 characters with 12 characters)

• Use Win32 Disk Imager to clone the edited iso file

• The first time the cloned drive is booted, the Ubuntu system will create a casper-rw partition with an ext4 file system automatically. Simple and robust!

• There are detailed descriptions at these links:

  • wiki.ubuntu.com/Win32DiskImager/iso2usb/persistent

  

  • wiki.ubuntu.com/Win32DiskImager/iso2usb

  

MacOS and Windows and Linux

• Unetbootin - you have to create a casper-rw partition for persistence manually and remove or rename the casper-rw file

See this link describing how to add a persistent partition to a UNetbootin live/persistent USB (Ubuntu 19.10+)

Please notice 'Space used to preserve files across reboots ...'

[![enter image description here][10]][10]

• Compressed image file

You can extract and clone a compressed image file directly according to the following link,

Xubuntu Core 20.04 LTS with mkusb 12.4.5 (mkusb-dus and mkusb-plug)

This works in all operating systems where you have

• a tool that can extract a file compressed with xz
• a cloning tool

.

Manual method

You need only a few manual steps to create a persistent live drive with a partition for persistence, when you are running Ubuntu.

• Edit the iso file to replace quiet splash with persistent . Yes, you can edit the binary iso file and replace 12 characters with 12 other characters and flash the output to the target device (usually a USB pendrive). sed can do it.

• Create a partition 'behind' the flashed copy of the edited iso file. fdisk can do it.

• Create an ext2 file system in this partition and put the label casper-rw on this partition. mkfs.ext2 can do it.

• Flush the buffers. sync can do it.

This is what mkusb-minp is doing plus a lot of checking, that wrap a safety belt around the process. In other words, you get help to write to the correct target device and get warnings when there might be problems.

You can do it yourself and feel every step, or you can read the code of the shellscript mkusb-minp and understand the details.

Please notice that I use dus and select 'Cloning' in the following example. You can use another cloning tool, but if you use a simple tool like dd, please double-check that you are cloning to the currect device, and that all partitions on that device are unmounted.

strings lubuntu-19.10-desktop-amd64.iso |grep 'quiet splash'  # check that 'quiet splash' is there to be replaced by 'persistent  ' (12 characters)
sed 's/quiet splash/persistent  /' lubuntu-19.10-desktop-amd64.iso > persistent-lubuntu-19.10-desktop-amd64.iso  # yes, sed works with binary files
ls -l *19.10*  # check that the size is the same
strings persistent-lubuntu-19.10-desktop-amd64.iso |grep 'persistent  '  # check that 'persistent' is there now
dus persistent-lubuntu-19.10-desktop-amd64.iso  # I use mkusb-dus, you can use the Ubuntu Startup Disk Creator or another cloning tool
sudo lsblk -fm  # It is important to check the device letter of the target drive (the USB drive, that you want persistent live
sudo fdisk /dev/sdx  # x is the device letter of the target drive, please double-check that you have the correct letter
n           # new partition
p           # primary
<Return>    # default: 3
<Return>    # default: next free sector
<Return>    # default: last addressable sector
w           # write and quit
sudo lsblk -fm  # check that things look good and verify that partition #3 is the correct partition to be used to store the persistent data
sudo mkfs.ext2 -L casper-rw /dev/sdx3  # put label and file system into the partition of persistence
sudo lsblk -fm  # check that things look good
sync  # flush the buffers and wait for prompt

Links

• Cloned USB drive with Lubuntu Eoan can be made persistent live
• Method developed by F. Hauri which works for Debian 10

There are general manual methods that can be easily modified from using a file for persistence to a partition for persistence. See the following link,

• help.ubuntu.com/community/Installation/iso2usb/diy

A cloned Ubuntu 19.10+ live drive is not really live-only

Isn't live the default for a cloned drive?

Well, it used to be, and it is still live but not live-only in 19.10.

If you want to make a purely live-only drive, where nothing will be preserved after shutdown and reboot, you can

• use mkusb-minp with the option -n or mkusb-plug and select 'No-persistent live drive'.

• or manually modify the built-in boot options to replace 'quiet splash' with 'nopersistent'

Link

• How can I get a live-only drive with Ubuntu 19.10 and newer versions?

I wrote a bash script (one liner) to edit live USB disk to change "quiet splash" to "persistent " based on the answer by @sudodus.

Disclaimer: it worked on my system, use at your own risk. It is for bash, may not work on other shells w/out edit.

After boot from USB extra partition writable was created w/out my intervention for overlay for persistence, if not fdisk can be used.

First variant of code is not efficient as it starts search with grep all over after each edit; but it can be stopped via ctrl-c when some changes are made.

device_to_edit=/dev/sdb; string_to_edit="quiet splas\x68"; new_string="persistent  "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else while [ -z $to_end ]; do offset_to_edit=$(sudo grep --only-matching --byte-offset --max-count=1 --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = "0" ; print $1 ; exit }'); if [ -z "$offset_to_edit" ]; then echo -e "\n end of file is reached" ; to_end="yes" ; else block_to_edit=$(($offset_to_edit/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)) ; fi ; done ; echo -e "\n$ed_count editions made"; fi

In second variant of code grep is done only one time, but grep goes through complete file (e.g. block device, USB) which can take significant time before any editing (on my system it took ~ 15 minutes for 8Gb drive, I added date for the sake of that curiosity):

date; device_to_edit=/dev/sdb; string_to_edit="quiet splash"; new_string="persistent  "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else sudo grep --only-matching --byte-offset --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = ":" ; print $1 }' | { IFS=$'\n' read -d '' -a offsets_to_edit ; for offset in ${offsets_to_edit[@]}; do block_to_edit=$(($offset/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)); done; echo -e "\n$ed_count editions made"; } ; fi; date

Notes:

Meaning of the scripts parameters and syntax could be understood by reading man bash, man awk, man sed, info '(coreutils) dd invocation', help read. Why (list) / { list; } needed in pipe is explained in https://stackoverflow.com/questions/2746553/read-values-into-a-shell-variable-from-a-pipe ("A pipeline may spawn a subshell, where the environment is inherited by value, not by reference"), working with two blocks is needed in case string starts in one block and ends in next one.

The scrips include three safety checks: lengths of original and replacement strings are same, block size is sufficiently large, device size is not larger then 20Gb (against accidental edit of main drive, need to be edited for larger flash storage, code is $((file_size/1024/1024/1024)) -gt 20).

There are some redundancies in the scripts, i.e. FS = ":" ; in awk, for some reason field separator did not work as I expected and I had to add --field-separator ":", also in grep --perl-regexp is needed to use hex e.g. \xaa values, but this edit does not have them.

Rufus is the easiest tool for making a persistent live drive with Ubuntu 19.10

(But not the necessarily the best).

For making a Persistent drive not limited to 4GB in Ubuntu, mkusb is easiest, (and best), to use. (What else is there)?

For making a Persistent drive not limited to 4GB in Windows 10, Rufus seems to currently be the easiest tool to use.

That does not make it the best tool though.

I did some simple experiments comparing the latest version of Rufus and mkusb:

First I wiped a 4GB USB and then ran Rufus in Windows 10, rebooted, and had a look ar the /casper-rw/upper/ folder with Disk Usage Analyzer.

Then I deleted the casper-rw partition and made a new one.

I rebooted the drive a couple times, without making any changes or saving anything, and again had a look at the repopulated Persistent partition with DUA.

I again wiped the drive and created another Persistent USB with mkusb, rebooted and had another look with DUA.

Again I deleted the casper-rw partition and made a new one.

I rebooted the drive a couple times, without making any changes, and again had a look at the repopulated Persistent partition with DUA.

As can be seen from the attached charts, mkusb seems to be superior in the use of disk space, and grub 2 is more straight forward than syslinux.

Persistent USB made with Rufus

Persistent USB made with Rufus with Repopulated casper-rw partition

Persistent USB made with mkusb

Persistent USB made with mkusb with Repopulated casper-rw partition

Edit: following the trail of persistent disk space usage.

I took a look at disk space usage in the casper-rw partition of Rufus and mkusb freshly made persistent USB's

Rufus var folder

Contents of Rufus snaps folder

mkusb var folder

Contents of mkusb snaps folder

Data not yet confirmed

Edit: Rufus 3.9.1624

Ran Disk Usage Analyzer with the latest version of Rufus, Results were close to Rufus 3.8. It apears that mkusb still makes a more space efficient Persistent drive.

Persistent USB made with Rufus 3.9 Rufus 3.9 var folder

Again, Data and methodology has not been confirmed.