Can ZFS be used like a virtual machine snapshot? Using, say, VirtualBox or VMM, I can snapshot a non-running VM before running and installing new software on it, and always revert back to a working configuration if I screw things up. Can I do this using ZFS?
I've seen plenty of examples of snapshotting my home directory which I can understand, but given the "fluid" nature of some of the files underneath root "/" is it unsafe to sudo zfs snapshot -r /@myrootsnapshot1 while the boot drive is mounted and Ubuntu is running?
And even more importantly, would a restore of such snapshot be safe? The objective is to snapshot before installing new software or trying out new configs, and roll back if I encounter problems.
If it's not safe, what is the recommended workflow for this objective?
Short answer: Yes, you can.
Long answer: There is one caveat - the consistency of the snapshots will be "crash level" consistency. This is usually not a problem because modern operating systems and file systems go to great lengths to ensure that an unexpected power failure won't corrupt the file system, and the worst that might happen is that the most recent few seconds worth of transactions might go missions.
I have sanoid taking snapshots of my various file systems, including my VM backing zvols, and have never had any kind of a corruption issue arise from a rollback of a VM.
There may be a handful of edge cases (e.g. with databases when you have your database data directory split up over several ZFS file systems), but even that just means that you have to take a consistent snapshot of everything (i.e. zfs snapshot -r, as you propose above), to make sure that the data and the logs don't get out of sync with each other.