liori's questions

It has been established that journald logs are huge. On this specific system, the logs grow about 3GB per week. For audit purposes I would like to preserve system logs for a longer time than what can I comfortably store on that system; this would only be "just-in-case" storage that doesn't have to be quick to retrieve from. What would be the best way to archive these logs in a space-efficient way?

I am thinking of periodically doing some kind of journalctl | xz >>/var/log/old-logs.txt.xz, but that would be quite wasteful in terms of CPU usage, re-archiving the same parts of data again and again and might miss logs if they get deleted by journalctl, so maybe there's a better way?

I want to use LVM on raw devices to be able to dynamically create RAID0 and RAID1 volumes depending on my current needs. I've recently learned that the write intent bitmap feature of mdadm reduces available IOPS a lot. In my case I'm getting only a third of IOPS that I should be getting.

Is it possible to create an LVM RAID1 logical volume without a write intent bitmap? I can deal with drawbacks, as the volumes I plan to create will never be large, hence I do not expect that rebuild time will be too long.

This is Debian Buster.

A large, USB-connected HDD with btrfs contains a 1TB file (a disk image). The first btrfs scrub after writing that file found 3 logical bad blocks developed somewhere in the middle of that file:

[ 7702.964265] BTRFS warning (device sdd1): checksum error at logical
5473719291904 on dev /dev/sdd1, sector 222940168, root 5, inode 1245769,
offset 97110921216, length 4096, links 1 (path: dysk/dysk.bin)
[ 7702.964274] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 17, gen 0
[ 7702.964278] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 5473719291904 on dev /dev/sdd1
[…]
[ 9588.625906] BTRFS warning (device sdd1): checksum error at logical
5241172611072 on dev /dev/sdd1, sector 691494312, root 5, inode 1245769,
offset 310632271872, length 4096, links 1 (path: dysk/dysk.bin)
[ 9588.625916] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 18, gen 0
[ 9588.625920] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 5241172611072 on dev /dev/sdd1
[…]
[20545.302898] BTRFS warning (device sdd1): checksum error at logical
3991747321856 on dev /dev/sdd1, sector 3185809200, root 5, inode 1245769,
offset 907925676032, length 4096, links 1 (path: dysk/dysk.bin)
[20545.302908] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 19, gen 0
[20545.302912] BTRFS error (device sdd1): unable to fixup (regular) error
at logical 3991747321856 on dev /dev/sdd1

No backups for this file, so no perfect recovery possible. The full area of the btrfs partition can be read without I/O error, so I think there are two possible cases: either the data block was written incorrectly (unrecoverable) or an incorrect checksum is stored (fix should be simple: recompute the checksum). Given that the first case is already lost, how can I ask btrfs to recompute the checksum of a data block?

I am a user of a Linux-based computing cluster that runs a task queuing system PBSPro. PBSPro likes to know how much RAM should be available for a task to make sure it will execute correctly; for example by qsub -l select=1:mem=4GB someapp I declare that I want to submit someapp for execution on a node that has at least 4GB of free memory. The tighter bound I can provide, the faster my application will be scheduled for execution.

How can I estimate how much memory will someapp need?

I can make a test run and watch htop, carefully watching my process' RSS, but is there any tool or method that would make it more automatic? Let say, return the maximum amount of memory paged in to the process over the whole time of its execution?

I am just a normal user on the cluster, with no root access. I am only running a single process with potentially several threads; even if someapp will fork(), I don't care whether the child process memory is counted or not.

I'm running a small Debian compute cluster on 8-core PCs with 16GB of RAM. I'm running batches of about 1k tasks (each batch has a total expected time of a month to be finished). A single task is single-threaded (so I can run multiple of them in parallel on each PC), does not consume much IO (loads several megabytes of data on start and dumps several megabytes of data on exit; does not communicate with outside world otherwise), its run time is unknown (from few minutes to ~week), its memory consumption is unknown (ranging from several megabytes to ~8GB; usage may grow slowly or quickly). I'd like to as many such tasks as possible in parallel on a single PC, but I want to avoid excessive swapping.

So I got an idea: I could monitor the memory usage of these tasks and suspend (kill -SIGSTOP) or hibernate (using a tool like CryoPID) tasks which consume too much memory to restart them later. By memory usage I mean number of “active virtual pages”, or number of allocated, not shared memory pages that have actually been touched (these tasks may allocate memory without using them).

I started looking for some tools to do that. I know that I can ulimit or run a task inside a memory-limited cgroup, but—if I understand them correctly—these solutions will kill the process instead of suspending it. I want to avoid killing them, because I would need to start them from scratch later, and that means wasted time. Also, they cannot actually measure the number of active virtual pages.

I could use real virtual machines, but they seem to have significant overhead in this case—having separate kernel, memory allocations, etc. would decrease the available memory; I'd have to run 8 of them. Also, as far as I know, they'd add computational overhead too.

I imagine that a tool that would implement such behavior would hook up some function to a page fault notification that would decide on each page fault whether it is time to suspend the process or not. But I don't know any tool that would work this way either.

Are there other choices?

I started 4 processes doing heavy computations. The machine has 4 full cores, and each process is single-threaded, so they don't compete for CPU. However I realized that they require more memory than physically exist in this system (16GB). Now their allocated address space combined is about 20GB, so each of them has part of their address space swapped out.

So I decided to suspend one of those processes and I wanted to put its memory pages to swap, so that physical memory would serve only the three other processes. I assume that this will eventually happen while the three alive processes will try to access swapped-out pages. But I'd like to force it so that the computations wont have to wait for IO.

My questions:

1. Is it possible to force a suspended process' memory go to swap? (so that there wont be IO when alive processes will do new allocations)

2. Is it possible to force an alive process' memory to come back to RAM?

This is Debian SID.

I've got a rather strange problem with sudo. Basically, it authenticates, but sometimes just doesn't launch the command provided. For example:

liori@marvin:~$ sudo whoami
root
liori@marvin:~$ sudo whoami
root
liori@marvin:~$ sudo whoami
liori@marvin:~$ sudo whoami
liori@marvin:~$ sudo whoami
liori@marvin:~$ 

I wrote a test case which demonstrates this problem:

liori@marvin:~$ sudo whoami; for i in `seq 100`; do echo -n ':' ; sudo whoami ; done ; echo
root
::::::::::::root
:::::root
:::::root
:::::::::::root
::::::::::root
::::::::::::::::::::::::::::::::root
:::root
::::root
::::root
:::root
:::root
::root
:root
:::::
liori@marvin:~$ 

Of course, the expected output is a series of lines, each one starting with exactly one colon character. I don't have any clue where to begin debugging this problem.

For each attempt (whether the command was actually run or not), I get an entry in syslog:

Apr 11 19:47:40 marvin systemd-logind[806]: New session c1079 of user root.
Apr 11 19:47:40 marvin systemd-logind[806]: Removed session c1079.

This is Debian SID. I started observing this behavior few days ago, after a somewhat bigger update (I update this system maybe once a month), and after moving the system from one hard disk to another (using rsync -av --del).

Any help will be appreciated.

I've got a backup process which reads quite a lot of data from disk, and therefore uses up all memory for disk cache, swapping off apps and other cached data in the process. There is no benefit to keep backuped data in disk cache after the backup is finished, so I'd like to limit the amount of data which could be cached by kernel for that process. Is it possible?

Thanks,

We've got a standard PHP-based shared hosting for some non-profit organizations. Users can upload over FTP any PHP apps, and we're worrying that some of them aren't properly maintained, f.e. updated.

Are there any scripts or programs that could automatically find what apps are installed, where are they and what versions?

I'd imagine them to work like anti-virus scanners, that is: have a database of "signatures" (for example a set of strings to be matched against .php files). It has been suggested that simple md5/sha1 sums won't work, as many of the configuration files are modified by user.