Questions[amanda](server)

We have a few hundred systems backing up to a tape library using AMANDA. Every day we get an email with a long list of STRANGE backup results. Almost all of these are because a log file was being written or a file-based cache was being updated on a web server. For example (5 machines showing a variety of STRANGE messages):

STRANGE DUMP DETAILS:
  /-- av1 / lev 2 STRANGE
  sendbackup: start [avion:/ level 2]
  sendbackup: info BACKUP=/bin/tar
  sendbackup: info RECOVER_CMD=/bin/tar -xpGf - ...
  sendbackup: info end
  ? /bin/tar: ./boot: directory is on a different filesystem; not dumped
  ? /bin/tar: ./dev: directory is on a different filesystem; not dumped
  ? /bin/tar: ./fs: directory is on a different filesystem; not dumped
  ? /bin/tar: ./proc: directory is on a different filesystem; not dumped
  ? /bin/tar: ./run: directory is on a different filesystem; not dumped
  ? /bin/tar: ./sys: directory is on a different filesystem; not dumped
  ? /bin/tar: ./wwwusers: directory is on a different filesystem; not dumped
  | /bin/tar: ./tmp/.X11-unix/X0: socket ignored
  | /bin/tar: ./tmp/gpg-v9zbWU/S.gpg-agent: socket ignored
  | Total bytes written: 795084800 (759MiB, 740KiB/s)
  sendbackup: size 776450
  sendbackup: end
  \--------
  /-- gitlab / lev 3 STRANGE
  sendbackup: start [gitlab:/ level 3]
  sendbackup: info BACKUP=/bin/tar
  sendbackup: info RECOVER_CMD=/bin/tar -xpGf - ...
  sendbackup: info end
  | /bin/tar: ./tmp/gitaly-ruby215330643/socket.0: socket ignored
  | /bin/tar: ./tmp/gitaly-ruby215330643/socket.1: socket ignored
  | /bin/tar: ./var/lib/gssproxy/default.sock: socket ignored
  ? /bin/tar: ./var/log/gitlab/gitlab-workhorse/current: file changed as we read it
  ? /bin/tar: ./var/log/gitlab/nginx/gitlab_access.log: file changed as we read it
  | /bin/tar: ./var/opt/gitlab/gitaly/gitaly.socket: socket ignored
  | /bin/tar: ./var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: socket ignored
  | /bin/tar: ./var/opt/gitlab/gitlab-workhorse/socket: socket ignored
  | /bin/tar: ./var/opt/gitlab/postgresql/.s.PGSQL.5432: socket ignored
  | /bin/tar: ./var/opt/gitlab/redis/redis.socket: socket ignored
  | /bin/tar: ./var/spool/postfix/private/anvil: socket ignored
  | /bin/tar: ./var/spool/postfix/private/bounce: socket ignored
  | /bin/tar: ./var/spool/postfix/private/defer: socket ignored
  | /bin/tar: ./var/spool/postfix/private/discard: socket ignored
  | /bin/tar: ./var/spool/postfix/private/error: socket ignored
  | /bin/tar: ./var/spool/postfix/private/lmtp: socket ignored
  Look in the '/var/log/amanda/log.error/gitlab._.3.20180606231049.errout' file for full error messages
  \--------
  /-- haproxy00-127 / lev 3 STRANGE
  sendbackup: start [haproxy00-127:/ level 3]
  sendbackup: info BACKUP=/bin/tar
  sendbackup: info RECOVER_CMD=/bin/tar -xpGf - ...
  sendbackup: info end
  ? /bin/tar: ./var/log/messages: file changed as we read it
  Look in the '/var/log/amanda/log.error/haproxy00-127._.3.20180607003248.errout' file for full error messages
  \--------
  /-- imap00 /srv/dovecot lev 0 STRANGE
  sendbackup: start [imap00:/srv/dovecot level 0]
  sendbackup: info BACKUP=/bin/tar
  sendbackup: info RECOVER_CMD=/bin/tar -xpGf - ...
  sendbackup: info end
  ? /bin/tar: ./a1/Maildir/new/1528332018.M.imap00.example.com,S=6309,W=6430: File removed before we read it
  ? /bin/tar: ./a1/Maildir/new/1528335571.M.imap00.example.com,S=8299,W=8451: File removed before we read it
  ? /bin/tar: ./a2/Maildir/cur/1528306776.M.imap00.example.com,S=13517,W=13770\:2,Scd: File removed before we read it
  ? /bin/tar: ./a2/Maildir/cur/1528332257.M.imap00.example.com,S=22257,W=22562\:2,STcd: File removed before we read it
  ? /bin/tar: ./a3/Maildir/cur/1528287062.M.imap00.example.com,S=10343,W=10558\:2,: File removed before we read it
  ? /bin/tar: ./a3/Maildir/cur/1528297383.M.imap00.example.com,S=4179,W=4265\:2,: File removed before we read it
  ? /bin/tar: ./a3/Maildir/cur/1528312816.M.imap00.example.com,S=29364,W=29796\:2,: File removed before we read it
  ? /bin/tar: ./a3/Maildir/new/1528327304.M.imap00.example.com,S=52059,W=53029: File removed before we read it
  ? /bin/tar: ./a4/Maildir/.folder/dovecot.index.log.2: File removed before we read it
  ? /bin/tar: ./a4/Maildir/.Personal/dovecot.index.log.2: File removed before we read it
  ? /bin/tar: ./a4/Maildir/.Trash/cur/1527783480.M.imap00.example.com,S=31976,W=32433\:2,Sab: File removed before we read it
  ? /bin/tar: ./a4/Maildir/cur/1528336066.M.imap00.example.com,S=122100,W=124740\:2,ab: File removed before we read it
  ? /bin/tar: ./a4/Maildir/cur/1528337059.M.imap00.example.com,S=28553,W=28964\:2,ab: File removed before we read it
  Look in the '/var/log/amanda/log.error/imap00._srv_dovecot.0.20180606220538.errout' file for full error messages
  \--------
  /-- webserver1 / lev 2 STRANGE
  sendbackup: start [webserver1:/ level 2]
  sendbackup: info BACKUP=/bin/tar
  sendbackup: info RECOVER_CMD=/bin/tar -xpGf - ...
  sendbackup: info end
  ? /bin/tar: ./var/log/httpd/kemp_access_log: file changed as we read it
  ? /bin/tar: ./var/log/httpd/wiki.example.com_access_ssl.log: file changed as we read it
  ? /bin/tar: ./var/log/httpd/www.example.com_access.log: file changed as we read it
  ? /bin/tar: ./var/log/httpd/www.example.com_ssl_access_ssl.log: file changed as we read it
  Look in the '/var/log/amanda/log.error/webserver1._.2.20180607004437.errout' file for full error messages
  \--------

(brought to you by Amanda version 3.3.3)

Relevant parts of our current configuration:

define dumptype nocomp-root-tar {
    program "GNUTAR"
    comment "Root partitions with gtar"
    compress none
    estimate server
    priority low
    exclude list "/etc/amanda/amanda.exclude.list"
    auth "bsd"
}
define dumptype nocomp-user-tar {
    program "GNUTAR"
    comment "Non-root partitions with gtar"
    compress none
    estimate server
    priority medium
    exclude list "/etc/amanda/amanda.exclude.list"
    auth "bsd"
}

We don't want to exclude these files from backup just because they might change or disappear. We also can't disable the IMAP server while backups are running. It looks like I can change our systems to use amgtar as their backup method which will let us eliminate all these messages.

define application-tool app_amgtar {
    plugin "amgtar"

    property        "NORMAL" ": socket ignored$"
    property append "NORMAL" "file changed as we read it$"
    property append "NORMAL" ": directory is on a different filesystem; not dumped$"
    property append "NORMAL" ": File removed before we read it$"

}
define dumptype nocomp-root-amgtar {
    program "APPLICATION"
    application "app_amgtar"
    comment "Root partitions with amgtar"
    compress none
    estimate server
    priority low
    exclude list "/etc/amanda/amanda.exclude.list"
    auth "bsd"
}
...

Are there any downsides to amgtar?

Will it be more difficult to recover older backups if we make this change?

The documentation seems to say that only GNUTAR supports exclude list. Does amgtar also support it since it uses GNUTAR underneath?

I was in the middle of an amanda backup, and after about 60GB, it exited with this error:

  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device
  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device

But it doesn't tell me which device. And I can't find any device that is full.

My backup ended after 60GB

-rw------- 1 amanda backup 36616372224 2016-11-02 23:42 00001.localhost._home.0
-rw------- 1 amanda backup 22800531456 2016-11-03 00:03 00002.localhost._home.0

My amanda.conf says

tapetype "HARD-DISK"
define tapetype HARD-DISK {
    comment "Dump onto hard disk"
    length 150 gbytes
}

My backup drive has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1             2.7T  2.3T  277G  90% /storage

The disk being backed up has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/HermanGroup-Home
                       74G   57G   14G  81% /home

But now everything I try to do with amanda fails:

FAILURE DUMP SUMMARY:
  newlaptop.local.net /home lev 9  FAILED [too many taper retries]
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device

How do I find out what happened?

UPDATE:

(in response to suggestion)

$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/root             262944   25713  237231   10% /
/dev/mapper/HermanGroup-Usr
                      655360  282687  372673   44% /usr
/dev/mapper/HermanGroup-Tmp
                      262144    2595  259549    1% /tmp
tmpfs                 219740       1  219739    1% /dev/shm
/dev/sdc1            183148544   28045 183120499    1% /storage
/dev/mapper/HermanGroup-Herman
                     3276800  298420 2978380   10% /herman
/dev/mapper/HermanGroup-Mysql
                      131072     191  130881    1% /var/lib/mysql
/dev/mapper/HermanGroup-UsrX11
                      393216   68534  324682   18% /usr/X11
/dev/mapper/HermanGroup-Home
                     4915200 1148476 3766724   24% /home
/dev/mapper/HermanGroup-WinGCC
                       65536    3536   62000    6% /usr/local/wingcc
/dev/mapper/HermanGroup-AmandaHolding
                      720896      11  720885    1% /storage/amanda/holding
192.168.42.10:/var/spool/mail
                      655360      17  655343    1% /var/spool/mail

Im attempting to setup AMANDA to use a TL4000 system with 3 drives and 45 slots. I've run into a few issues that have me stymied:

1. amlabel seems to work about half the time. The rest I get messages like Error reading volume label: Tape device /dev/nst0 is not ready or is empty. Often rerunning the command will get a success.

2. When I run amcheck daily I get errors like this:

   ERROR: server.edu /var: data-path is AMANDA but device do not support it
   ERROR: server.edu /etc: data-path is AMANDA but device do not support it

3. (related to label issue) The amcheck also gives me errors like this:

   slot ?: Slot 34, label 'backup-008', mismatch barcode between changer '000001L6' and tapelist file '000008L6'
   ERROR: Slot 34, label 'backup-008', mismatch barcode between changer '000001L6' and tapelist file '000008L6'

Somewhere there is a file of settings that is trying to tie the specific tape to a changer location but I can't find it. I know this is doable as I'm attempting to retire a 10 year old tape system that has used AMANDA all along. 400 tapes into 40 slots. It's been too long since this was configured to find the clues.

Suggestions on what I'm missing here?

I have inherited a bunch of old backup tapes that contain some data that needs to be recovered. The problem is that the amanda implementation used to create these backups has since been obliterated.

I am very new to amanda, but was under the impression that the config files are required to recover backed up data.

Is there any easy way to get this data off? If not, what steps should I take?

I want to (or better have to) do Backups of my KVM Virtual Machine images. I have searched for days for a good Backup Soloution. I know amanda is a very good solution. It would be kinf if someone kenn tell me if the following is supported:

1. Trigger the Creation of LVM Snapshot (by invoking a Shell Script that I will write for that purpose)
2. Do a Differential/Delta Backup on my KVM LVM qcow2 sparse file. => I only want to copy the actually changed bits/bytes (=Delta Backup). And it has to support that the file to be backuped up is a sparse file. (Rsync seems to have some kind of problems in regard to this (if the file does not exist yet on the other side... Then it will create a full file, not a sparse file))
3. Release the LVM Snapshot (By invoking a Script that I will write for that purpose)

It's strange, I have nowhere found any documentation about this fact when searching the internet. Zmanda (Commercial Edition) has support vom XEN VM Backup (but not for KVM as far as I can tell)...