[PLUG] ZFS mirror expansion fails to see larger disk (solved)

["Lee H. Marzke" <lee@marzke.net>]

I have a FreeNAS system with 6 x 2TB drives for my VMware lab cluster

I just solved a pool size expansion issue I've had for a while, so I though I'd share it
as the solution was not on-line.

This is FreeNAS 11.1U4 / on FreeBSD,  with GPT partition tables, and USB flash boot.
The pool is 3 mirrors of 2 x 2TB disks each,  with an Intel 320 SSD for L2Arc, and Intel 3700 SSD for SLOG

When my FreeNAS ZFS system had a failed 2TB disk,  I replaced it with a 4TB disk,  and
naturally didn't see a pool size increase.    However, when I replaced the 2nd disk
in the mirror,  I also didn't see a pool size increase as expected.

The ZFS pool had autoexpand enabled, so this should have worked.

root@freenas:~ # zpool get all 
NAME          PROPERTY                       VALUE                          SOURCE
bigpool       size                           5.61T                          -
bigpool       capacity                       67%                            -
bigpool       altroot                        /mnt                           local
bigpool       health                         ONLINE                         -
bigpool       autoexpand                     on                             local

Lots of people online complaining about expansion not working, with lots of hints.

I tried running the following to expand the disk but it didn't work.
zpool online -e bigpool /dev/diskID   

Turns out gpart showed the real problem here.  Note the [CORRUPT] label on da1
while the 2nd 4TB disk (da5) shows no problem and correct sizes.

root@freenas:~ # gpart show
=>        34  3907029101  da0  GPT  (1.8T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)
  3907029128           7       - free -  (3.5K)

=>        34  4294967228  da1  GPT  (3.6T) [CORRUPT]
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  4290772824    2  freebsd-zfs  (2.0T)
  4294967256           6       - free -  (3.0K)
=>        34  3907029101  da2  GPT  (1.8T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)
  3907029128           7       - free -  (3.5K)

=>        34  3907029101  da3  GPT  (1.8T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)
  3907029128           7       - free -  (3.5K)

=>        34  3907029101  da4  GPT  (1.8T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  3902834696    2  freebsd-zfs  (1.8T)
  3907029128           7       - free -  (3.5K)

=>        34  7814037101  da5  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)


The solution to this was easily fixed with gpart, but takes quite a few
steps - as outlined here:

http://plone.4aero.com/Members/lmarzke/howto/freenas-zfs-mirror-expansion-issue

Not sure what caused this issue on da1, when the other 4TB disk da5 didn't have the
same issue.  

- Lee

-- 
PS:  I'm looking for a new VMware data-center contract or full-time.
https://www.linkedin.com/in/leemarzke/

"Between subtle shading and the absence of light lies the nuance of iqlusion..."  - Kryptos

Lee Marzke,  lee@marzke.net
IT Consultant, VMware, VCenter, SAN storage, infrastructure, SW CM

BEGIN:VCARD
VERSION:3.0
FN:Marzke\, Lee
N:Marzke;Lee;;;
ADR;TYPE=home,postal,parcel:;;119 Pepper Dr.;Collegeville;PA;19426;
TEL;TYPE=car,voice:610-564-4932
TEL;TYPE=cell,voice:610-564-4932
TEL;TYPE=voice:252-627-9531
TEL;TYPE=work,voice:800-393-5217
EMAIL;TYPE=internet:lee@marzke.net
EMAIL;TYPE=internet:lmarzke@4aero.com
ORG:4aero
TITLE:Infrastructure Consultant
NOTE:252-Marzke1 (2526279531)
CATEGORIES:Family
REV:2018-02-20T14:25:15Z
UID:f5f810f6-3347-4915-9a40-567cf4591450:86081
END:VCARD

BEGIN:VCARD
VERSION:3.0
FN:Marzke\, Lee
N:Marzke;Lee;;;
ADR;TYPE=home,postal,parcel:;;119 Pepper Dr.;Collegeville;PA;19426;
TEL;TYPE=car,voice:610-564-4932
TEL;TYPE=cell,voice:610-564-4932
TEL;TYPE=voice:252-627-9531
TEL;TYPE=work,voice:800-393-5217
EMAIL;TYPE=internet:lee@marzke.net
EMAIL;TYPE=internet:lmarzke@4aero.com
ORG:4aero
TITLE:Infrastructure Consultant
NOTE:252-Marzke1 (2526279531)
CATEGORIES:Family
REV:2018-02-20T14:25:15Z
UID:f5f810f6-3347-4915-9a40-567cf4591450:86081
END:VCARD

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug