Questions[bareos](server)

I had an ubuntu server 18.04 running with bareos and mysql database. Monthly I saved data on tapedrive and everything was working fine.

Unfortunately I had a servercrash without a backup of the dump of the database (I know.)

Now I set up an Ubuntu 20.04 with postgres database, since bareos is not supporting mysql anymore.

My idea was to Bscan my tapes to recover Jobs and to access my data on the tapes.

During the bscan I get: bscan: stored/bscan.cc:789-0 Could not find Job for SessId=57 SessTime=1611327800 record.

I was using following command: bscan -V "tape1"|"tape2" -B postgres -u postgres -p "password" -v -s -S -m /dev/st0

The command didn't show me an error, but I'm not able to restore the data after the BSCAN.

Does bareos have a problem with the data on the tapes is from an Mysql server and cannot recover into the postgres database?

On my old server on mysql I was able to recover the whole database from tape when I tested that.

Problem

After building a Proof-of-Concept setup of Bareos, I can add and update various components like Clients, Devices, etc. But when modifying a FileSet, those changes are not reflected in the running Bareos config.

Details and research

PoC is built on Bareos 20.0.1, installed on a CentOS Linux 8.3 box. Till now all components, concepts and tasks are working as excepted. I can run backup and restore jobs succesfully.

When running bconsole, show filesets the updated fileset is visible (so it is read from config).

When running bconsole, list filesets the previous original fileset is listed. None of my modifications (i.e. added exclusions) is reflected over here.

When running jobs for new clients or rerunning previous jobs, I see no diferrence in FileSets in logging and no effects of my modifications (fewer files due to added exclusions).

Details:

• Config file location: /etc/bareos/bareos-dir.d/fileset/XX.conf
• Config file ownership: bareos:bareos

What I've tried

I've tried to reload/restart Bareos in a few ways, with no difference:

• bconsole, reload
• systemctl restart bareos-dir
• reboot server

Tried to educate myself by Bareos and Bacula official documentation of various versions, but couldn't find a specific part which describes this behaviour.

Using bpipe plugin with Bareos Backup I get an error:

Fatal error: bpipe-fd: Pipe read error: ERR=Error 0

after

zfs send -R dpool/some_dataset

finished, executed from bash script.
But only iff dpool/some_dataset has child datasets (there is dpool/some_dataset/child1 for example). The -R option includes the child within the zfs send. This is my script zfs_create_send_snapshot.sh:

#!/bin/sh
#
# create recursive ZFS snapshot for given dataset and pipe
# replication stream to stdout, then delete snapshot

if [ -z "$1" ]; then
  (>&2 echo "ERROR: missing dataset name argument")
  exit 1
fi

DDD=`date +%y%m%d%H%M`
SNAPNAME=$1@bareos_${DDD}
(>&2 echo "creating ZFS snapshot ${SNAPNAME}")

zfs snapshot -r ${SNAPNAME}
(>&2 echo "sending ZFS snapshot ${SNAPNAME}")
zfs send -R ${SNAPNAME}
RC=$?
(>&2 echo "deleting ZFS snapshot ${SNAPNAME}")
(>&2 zfs destroy -r ${SNAPNAME})
exit ${RC}

It's executed by bareos fileset like this:

Plugin = "bpipe:file=/tmp/zfs_snap.bin:reader=/etc/bareos zfs_create_send_snapshot.sh dpool/some_dataset:writer=/etc/bareos/writer.sh /tmp/zfs_snap.bin"

The job fails with broken pipe error only if dpool/some_dataset has child zfs datasets. Otherwise everything is fine. And it seems to be only a side-effect: The backup job writes complete zfs snapshot stream to tape until -- just erroneously finishing by error. It happens on openindiana/illumos. Recent Bareos client 17.2 compiled from git sources.

This looks very strange. I use Bacula and now BareOS for more than 10 years, but now one system experiences strange behaviour and I can't find out why and how to fix.

When it runs daily backups, it works fine, until it reaches the BackupCatalog job, which is configured to run after everything else.

This job looks like it was successfully terminated (JobStatus=T in the list jobs table):

*list jobs
...
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+
| JobId | Name          | Client       | StartTime           | Type | Level | JobFiles | JobBytes        | JobStatus |
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+
...
| 5,475 | BackupCatalog | kantor-fd    | 2019-12-04 02:56:40 | B    | F     |       21 |      27,364,860 | T         |
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+

However, in messages and in the log file I don't see the usual summary for this last job. The log file finishes like this:

19-Nov 02:32 kantor-dir JobId 5398: shell command: run BeforeJob "/usr/lib/bareos/scripts/make_catalog_backup.pl Kantor"
19-Nov 02:33 kantor-dir JobId 5398: Start Backup JobId 5398, Job=BackupCatalog.2019-11-18_23.10.00_10
19-Nov 02:33 kantor-dir JobId 5398: Using Device "FileStorage" to write.
19-Nov 02:33 kantor-sd JobId 5398: Volume "Kantor-2018-01-08_08:48:50" previously written, moving to end of data.
19-Nov 02:33 kantor-sd JobId 5398: Ready to append to end of Volume "Kantor-2018-01-08_08:48:50" size=4716094462
19-Nov 02:33 kantor-sd JobId 5398: Elapsed time=00:00:05, Transfer rate=5.663 M Bytes/second

And that's all. Note, RunAfterJob script seem to had not been executed. But if I execute it by hand, it works (the exported catalog database file gets removed). This is not the only job with RunAfterJob script, however.

I expected it to show something like this in the end. All other jobs have it:

19-Nov 02:32 kantor-dir JobId 5397: Bareos kantor-dir 16.2.6 (02Jun17):
  Build OS:               x86_64-pc-linux-gnu debian Debian GNU/Linux buster/sid
  JobId:                  5397
  Job:                    FTP.2019-11-18_23.05.00_09
...
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

19-Nov 02:32 kantor-dir JobId 5397: Begin pruning Jobs older than 1 month 10 days .
...

Also, the director status looks strange:

*status dir
kantor-dir Version: 16.2.6 (02 June 2017) x86_64-pc-linux-gnu debian Debian GNU/Linux buster/sid
Daemon started 03-Dec-19 11:10. Jobs: run=4, running=1 mode=0 db=mysql
 Heap: heap=135,168 smbytes=222,459 max_bytes=236,758 bufs=543 max_bufs=594

Scheduled Jobs:
...
====

Running Jobs:
Console connected at 04-Dec-19 09:03
 JobId Level   Name                       Status
======================================================================
  5475 Full    BackupCatalog.2019-12-03_23.10.00_08 has terminated
====

Terminated Jobs:

 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
...
  5471  Incr      6,591    7.499 G  OK       03-Dec-19 23:15 termsrv
  5472  Incr        427    11.37 G  OK       03-Dec-19 23:44 1C
  5473  Incr          3    3.198 G  OK       04-Dec-19 02:56 Oracle
  5474  Incr      5,797    2.600 G  OK       04-Dec-19 02:56 FTP


Client Initiated Connections (waiting for jobs):
...
====

i.e. the said job listed in the "running jobs", but it says it is terminated. It doesn't listed in the "terminated jobs", as if director still has something to finish.

It hanged in this state for six hours. I also see some strangeness with times (StartTime for it in the table and in the log file differs by half of hour, however, system date and MySQL's select NOW(); are in sync).

After director restart, director status looks more appropriate:

Running Jobs:
Console connected at 04-Dec-19 09:06
No Jobs running.
====
No Terminated Jobs.

This all started two weeks ago. If I leave it hanging, all following scheduled jobs will wait indefinitely for this stuck job, meaning, no backups will be performed.

I feel this could be issue with this job's RunAfterJob script, but it is standard shipped script. If I run in by the hand, it works. The job definition itself is also standard shipped, the only modification is that I added compression=GZIP in the FileSet, but I do that every time and this never caused any problems.

What to look for? How to fix?

Update:

The problem disappeared. I don't understand, why. Backups work for at least two days. Nothing appears to be stuck.

I have a BareOS installation with very little modification to the default config files. There are Full, Incremental, and Differential backups being performed. Most clients appear to be being backed up as expected.

However, one of my clients appears to be repeatedly backing up over 10% of the overall filesystem in every incremental cycle.

How do I find the largest files and folders that are being backed up repeatedly?

BAT does not appear to be very helpful here, since it only lists the size of the file node itself, rather than the entire folder size. I'm effectively looking for a du command that works within the BareOS framework for a specific backup attempt.