Re: [PLUG] XFS on CentOS-5.9

["Gavin W. Burris" <bug@sas.upenn.edu>]

Hi JP,

I have multiple 100TB file systems in production.  I would strongly
recommend going 64-bit, because with such a large filesystem you are
likely to allocate large amounts of memory for a filesystem repair at
some point.  I actually have two 100GB swap partitions preconfigured for
this likelihood.  I believe the general rule is to have 2GB of memory
for every 1TB of filesystem, varying with your inode count.

I would recommend putting XFS on top of LVM, so that you can stripe
across multiple devices.  LVM will also allow you to grow the volume
when needed.

Another thing to look out for are subdirectory NFS shares.  You will be
unable to share a subdirectory of a large XFS filesystem.  Make sure you
are OK sharing and mounting the root of the target NFS filesystem,
otherwise it fails.  I believe this is a shortcoming in the current NFS
code, not being able to address subdirectory inodes past the 32-bit
range.

Cheers.


On Wed, Jun 12, 2013 at 05:40:01PM -0400, JP Vossen wrote:
> Short version: what, if anything, is going to bit me for using a
> 26TB XFS filesystem on CentOS-5.9?
> 
> Details:
> 
> I recently had a problem trying to format a partition larger than
> 16TB on CentOS-5.9 on a Dell PE720 with 8x4TB drives in a PERC H710
> hardware RAID5.  Neither ext3 nor ext4 would work, even though in
> theory, and possibly with newer 'e4fsprogs', ext4 should work (after
> 'yum install e4fsprogs' since that's not in the stock install).  I
> had a multitude of problems, all of which boiled down to:
> 	1) CentOS-5.9 is *old*
> 	2) 32-bits is not enough
> 
> I am running x86_64, but with CentOS-5 at least, there are still far
> too many i386 tools and libs installed and I was hitting:
> 	mked4fs 1.41.12 (17-May-2010)
> 	mkfs.ext4: size of device /dev/sdb1 too big to be expressed in 32
> bits using a blocksize of 4096
> 
> Interestingly, I was hitting that at 17.6TB, *not* the "16TB limit"
> that everyone else talks about (yes, I know the meaning of "TB" is
> fuzzy . As far as I can tell, using 8k block sizes on i386 or x86_64
> is not allowed, attempts to remove the 32-bit libs failed for
> dependencies, and "-O 64bit" also failed as above.
> 	mkfs.ext4 -O 64bit,has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
> -m2 -L /data /dev/sdb1
> 
> On the other hand, XFS was backported into the CentOS stock repos,
> and that Just Worked, so far at least:
> 	yum install xfsprogs
> 	parted -s /dev/sdb mklabel gpt
> 	parted -s -- /dev/sdb mkpart primary xfs 0 -1
> 	  # -s = script mode, no prompts
> 	  # -- = end of options, rest are arguments
> 	  # Start at 0 and end at '-1' = whole disk, no matter what size
> 	mkfs.xfs -fL /data /dev/sdb1
> 	  # -f  = Force
> 	  # -L  = Set new volume label
> 
> That worked and was *far* faster, seconds to format 26TB as opposed
> to 20-30 minutes for ext4 to format 17.6TB on pretty fast hardware.
> 
> So...what, if anything, is going to bit me for using this?
> 
> Thanks,
> JP
> ----------------------------|:::======|-------------------------------
> JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
> My Account, My Opinions     |=========|      http://www.jpsdomain.org/
> ----------------------------|=========|-------------------------------
> "Microsoft Tax" = the additional hardware & yearly fees for the add-on
> software required to protect Windows from its own poorly designed and
> implemented self, while the overhead incidentally flattens Moore's Law.
> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
> 

-- 
Gavin W. Burris
Senior Systems Programmer
Information Security and Unix Systems
School of Arts and Sciences
University of Pennsylvania
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug