Home / server / Questions / 778268
Accepted
NathanTheGr8
Asked: 2016-05-22 17:36:25 +0800 CST

zpool showing more storage than it should

  • 772

I just expanded my zpool named storage with 3 HGST 6TB drives in raidz1. I thought this would add only about 11.x TB of storage (since they are really 6000GB drives) how ever now zpool list shows I have 27.1 TB total. This seems more than I should have. I am terrified I did some thing wrong and I have lost parity on my home server array. I added them to the pool with

sudo zpool add -f storage raidz1 drive1 drive2 drive3

Have I done something wrong and if so is it possible to revert my change?

sudo zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  27.1T  9.32T  17.8T         -      -    34%  1.00x  ONLINE  -

sudo zpool status

NAME                                          STATE     READ WRITE CKSUM
    storage                                       ONLINE       0     0     0
      mirror-0                                    ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX11DA40H4ZK  ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6410414  ONLINE       0     0     0
      mirror-1                                    ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65355TK  ONLINE       0     0     0
        ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4TP7L0  ONLINE       0     0     0
      raidz1-2                                    ONLINE       0     0     0
        wwn-0x5000cca255c02505                    ONLINE       0     0     0
        wwn-0x5000cca255c039a7                    ONLINE       0     0     0
        wwn-0x5000cca255c03b95                    ONLINE       0     0     0

errors: No known data errors

sudo fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 250.1 GB, 250059350016 bytes**This is the OS Drive**
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcb1189ee

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1   488397167   244198583+  ee  GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdf'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdf: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdg'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdg: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdh'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdh: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

sudo lshw -class disk

        *-disk:0
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.0.0
         bus info: scsi@0:0.0.0
         logical name: /dev/sdf
         version: T517
         serial: K1G0HW1B
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=62a489e5-51af-c94f-9535-235b7bbe2d15 sectorsize=4096
    *-disk:1
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.1.0
         bus info: scsi@0:0.1.0
         logical name: /dev/sdg
         version: T517
         serial: K1G09VRB
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=a7295fde-f0f8-7f41-8546-7ee02490d72b sectorsize=4096
    *-disk:2
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.2.0
         bus info: scsi@0:0.2.0
         logical name: /dev/sdh
         version: T517
         serial: K1G0HB3B
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=4677c5c9-364a-c943-a8d5-90d2159f3cf0 sectorsize=4096
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@1:0.0.0
         logical name: /dev/sda
         version: 82.0
         serial: WD-WX91D65355TK
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=e4026c6a-61c5-9a4a-adaf-ebcdf8bba68c sectorsize=4096
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68L
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@2:0.0.0
         logical name: /dev/sdb
         version: 82.0
         serial: WD-WXB1HB4TP7L0
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=8b576966-3d94-424b-aae9-9fb35c0e070a sectorsize=4096
    *-disk
         description: ATA Disk
         product: Samsung SSD 840
         physical id: 0.0.0
         bus info: scsi@4:0.0.0
         logical name: /dev/sdc
         version: EXT0
         serial: S1DBNEADA00610E
         size: 232GiB (250GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=8d48ca17-2e53-4ee4-a040-e80362e68eca sectorsize=512
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@5:0.0.0
         logical name: /dev/sdd
         version: 82.0
         serial: WD-WX11DA40H4ZK
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=5811c203-99cd-5342-bb8f-c838f896b64e sectorsize=4096
  *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@6:0.0.0
         logical name: /dev/sde
         version: 82.0
         serial: WD-WX51D6410414
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=c24e5950-cd95-f94b-826a-21a004221d32 sectorsize=4096
ubuntu raid zfs

1 Answers

  1. Best Answer

    Oh no! You did something kinda bad...

    Please read: http://nex7.blogspot.com/2013/03/readme1st.html

    Two issues to note:

    Under RAIDZ1|2|3, the zpool command reflects the size of the disks comprising the pool, including parity. zfs list will show usable space. That's where the discrepancy comes from.

    Also, what you ended up doing was... expanding a 4-disk RAID 1+0 (ZFS mirrors) by adding a RAIDZ1 group of 3 disks.

    So your mirrors mean that the RAID1 pairs are striped together. Fine. But by adding a RAIDZ1 group, you now have a stripe across two mirror sets and a RAIDZ1.

    Failure of either two disks in a mirror group or two disks in the RAIDZ1 group would result in total pool failure.

    You should have added an even number if disks by creating more mirror pairs. Right now, you have no option to revert the change because your data is scattered across the disk groups. This is probably a backup.rebuild.restore situation.

    • 3