Home / server / Questions / 812928
In Process
hymie
Asked: 2016-11-04 02:22:52 +0800 CST

amanda: No space left on device -- which device?

  • 772

I was in the middle of an amanda backup, and after about 60GB, it exited with this error:

  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device
  localhost /home lev 0  FAILED [data write: Connection reset by peer]
  localhost /home lev 0  partial taper: No space left on device: No space left on device

But it doesn't tell me which device. And I can't find any device that is full.

My backup ended after 60GB

-rw------- 1 amanda backup 36616372224 2016-11-02 23:42 00001.localhost._home.0
-rw------- 1 amanda backup 22800531456 2016-11-03 00:03 00002.localhost._home.0

My amanda.conf says

tapetype "HARD-DISK"
define tapetype HARD-DISK {
    comment "Dump onto hard disk"
    length 150 gbytes
}

My backup drive has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1             2.7T  2.3T  277G  90% /storage

The disk being backed up has space

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/HermanGroup-Home
                       74G   57G   14G  81% /home

But now everything I try to do with amanda fails:

FAILURE DUMP SUMMARY:
  newlaptop.local.net /home lev 9  FAILED [too many taper retries]
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device
  newlaptop.local.net /home lev 9  partial taper: No space left on device: No space left on device

How do I find out what happened?

UPDATE:

(in response to suggestion)

$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/root             262944   25713  237231   10% /
/dev/mapper/HermanGroup-Usr
                      655360  282687  372673   44% /usr
/dev/mapper/HermanGroup-Tmp
                      262144    2595  259549    1% /tmp
tmpfs                 219740       1  219739    1% /dev/shm
/dev/sdc1            183148544   28045 183120499    1% /storage
/dev/mapper/HermanGroup-Herman
                     3276800  298420 2978380   10% /herman
/dev/mapper/HermanGroup-Mysql
                      131072     191  130881    1% /var/lib/mysql
/dev/mapper/HermanGroup-UsrX11
                      393216   68534  324682   18% /usr/X11
/dev/mapper/HermanGroup-Home
                     4915200 1148476 3766724   24% /home
/dev/mapper/HermanGroup-WinGCC
                       65536    3536   62000    6% /usr/local/wingcc
/dev/mapper/HermanGroup-AmandaHolding
                      720896      11  720885    1% /storage/amanda/holding
192.168.42.10:/var/spool/mail
                      655360      17  655343    1% /var/spool/mail
amanda

4 Answers

  1. I'm guessing your filesystem is running out of inodes. Try out df -i.

    • 0

  2. It sounds like that your /home partition is being used while dumping? Then perhaps moved to your storage when finished. The message localhost /home lev 0 partial taper: No space left on device: No space left on device seem quite clear to me, that /home is being used somehow.

    • 0

  3. You seem to be using LVM for your volumes on your machines to be backed up. Are these home directories highly transactional? I ask because it is typical to create an LVM snapshot when backing up. This is the case in most backup applications that can utilize LVM on the target machine for backup consistency. Amanda backup is one such product. That snapshot may be running out of space, if it's being created at all. Could you provide the output of vgdisplay and lvdisplay for all affected volume groups and logical volumes?

    If it is indeed a snapshot causing this problem, your solution would be to add space to your volume group to compensate for the write transactions taking place during the backup process. This problem likely has nothing to do with your filesystem occupancy.

    • 0

  4. I don't know if this is "the answer" or just a coincidence, but...

    • the size of a virtual tape is defined as 150GB
    • there are two virtual tape drives
    • the free space left on the drive when I got the error was just under 300GB, which is 2 x 150GB
    • erasing some old backups seems to have resolved the issue (at least for now).

    So perhaps it's a bug (or feature) of amanda that it always expects to find enough free space to have one virtual tape loaded in each virtual tape drive.

    • 0