Gavin W. Burris on 9 May 2014 08:43:00 -0700


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] iSCSI storage appliance(s)


Hi, Carl.

How about a simple, big, rack-mount server with lots of disk bays?  Take
a look at something like this:
http://www.siliconmechanics.com/i50434/4u-storage-server.php

You can get it configured with CentOS and a MegaRAID controller, which
has a command line and GUI utility.  36 2TB hot-swap drives.  Make three
virtual disks in sets of 12.  You could do RAID 6, or even RAID 6 plus a
hot spare.  Keep some cold spares on site.  When you get an email
warning of a failed disk, swap it out.  Done.  Put XFS on it and you
have a 50TB NFS server.

Cheers.


On Fri 05/09/14 11:25AM -0400, Carl Johnson wrote:
> What kind of hardware do you plan to use?
> C.O.T.S x86 server grade stuff mostly. Need more specifics?
> 
> Roughly how much storage do you plan to manage? We're going to start with
> about 20TB. It's tough to plan how much we'll need though, as this is for
> CCTV DVR's with motion detection. Hence the scalablity requirement. I'm
> told that the pesky insurance company says their system needs to be capable
> of retaining three years of recordings.  So it's tough to gauge how much
> we'll really need to do this again, because of different camera frame
> rates/resolutions/how much motion etc. you get the idea.....
> 
> Is ISCSI the only thing you'd like to do? I had thought about using NFS,
> but I ended up using iSCSI because I thought it'd fit better. I was trying
> to avoid layers of abstraction/complexity. Am I wrong?
> 
> My main reason for not going the ZFS route is what you confirmed. Easy
> scalability and RAM both of which potentially change the hardware scope the
> most and, therefore, the cost.
> 
> If a web UI is a lower priority for you, it sounds like this system will
> be run by a reasonably technically proficient person.
> SystemS, probably 15ish in total all said and done. But yeah....Hi, I'm
> Carl, nice to meet you. ;-). There may be occasion where I'll need to talk
> someone else thru say, a disk replacement, via phone or something so hence
> the webUI need.
> 
> As it is right now, I've got two boxes in a test system. One, the storage
> box, is running CentOS. The other, the DVR itself, is an Ubuntu 12LTS box.
> I may try a wash/rinse/repeat on the storage box with OMV though and see if
> I like it or not. Though honestly, after reading thru your response, I'm
> probably going to go either the OMV/debian route or ditch the appliance
> overlay completely and use Centos/SoftRAID/Btrfs.
> 
> Can you elaborate on the webmin idea? Specifically, what *.wbm's do I need
> to do all that I'm asking? That may be something else I'll have to test
> drive too.
> 
> 
> 
> 
> On Fri, May 9, 2014 at 5:05 AM, PaulNM <plug@paulscrap.com> wrote:
> 
> > On 05/08/2014 01:32 PM, Carl Johnson wrote:
> > > Who's familiar with any of the NAS distros out there?
> > > FreeNAS/NAS4Free/NAPP-it/Openfiler/Openmediavault.....etc.?
> > >
> >
> > What kind of hardware do you plan to use? Roughly how much storage do
> > you plan to manage? Is ISCSI the only thing you'd like to do?
> >
> > I have more personal experience with FreeNAS/NAS4Free than the others
> > (except for the Webmin approach I'll mention later).  Actually, to be
> > precise, I've never used NAS4Free. It's a continuation of older versions
> > of FreeNAS that I have used, though.
> >
> > Openfiler appears to be a dead project.  Their last release is ~3 years
> > old and there doesn't appear to be any real work going on.
> >
> > Never heard of or used NAPP-it, so can't really comment on it. It
> > appears to be opensolaris/openindiana based? The site isn't very clear.
> >
> > Never heard of OpenMediaVault (OMV) either, though it looks *really*
> > interesting as it's based on Debian.  Not thrilled that they're still
> > using Squeeze as a base so close to when security support is ending.
> > Yes, I know Squeeze now has long term support, but that's a *very*
> > recent change.  Apparently there is a procedure to install OMV on
> > Wheezy, though.
> >
> > I've done a project where we used a Debian install with Webmin. This
> > approach is nice in that there's more flexibility to add other services
> > down the road. Also, I have a great deal of experience managing Debian
> > machines, so it's more comfortable for me. Webmin makes it easier for
> > the less technical people to check up on things and handle simpler
> > tasks. (I'll call this the WebMin approach.)
> >
> >
> > > What I'd like to have :
> > > 1. Flexibility of adding to the total unit capacity with drives of
> > > different capacities.
> >
> > FreeNAS can handle this fine, it prefers using ZFS pools. (If you're
> > familiar with LVM, ZFS is somewhat similar in concept but with more
> > features.) NAS4Free and NAPP-it should be the same for the same reasons.
> >
> > OMV and the Webmin approach I mentioned are both linux based.  You can
> > easily used LVM, RAID, or some combination of both.
> >
> > That said:
> >
> > You are aware that RAID/RAIDZ implementations are limited by the
> > smallest member of their array/volume, right? No matter what solution
> > you end up using, you'll hit this limitation.  There is unRAID, but
> > that's not so good redundancy-wise.  ISCSI would be problematic with
> > unRAID, and you have to pay if you use more than 3 drives.
> >
> >
> > > 2. Fault tolerance of at least one drive failure; two preferred.
> >
> > Here's where it gets tricky. ZFS does support setting up a mirror as
> > well as a few software raid implementations (RAIDZ1/RAIDZ2/RAIDZ3).
> >
> > What it doesn't support is adding drives to an existing RAIDZ set. Not a
> > problem if you're starting with all the drives you plan to use, but if
> > you ever want to add more drives to the RAIDZ:
> > You'll need to backup the data,
> > destroy the old RAIDZ,
> > create a new RAIDZ consisting of the drives from the old one and any new
> > drives,
> > restore the backup.
> >
> > The other option is to add drives in pairs/triplets and make them
> > separate RAIDZ volumes.
> >
> > MDADM (Linux RAID) can very easily add drives to existing arrays.
> > You'll have to expand any LVM volume and filesystem on it afterwards.
> >
> > > 3. Presenting the storage via an iSCSI target.
> >
> > Trivial in FreeNAS/NAS4Free. NAPP-it can apparently do this as well. OMV
> > has a plugin for this, as does Webmin.
> >
> > > 4. Adding and/or replacing disks without taking the ISCSI target offline.
> >
> > If the target is a RAIDZ or RAID volume, then yes.
> >
> > > 5. Admin/management via a web UI (not nearly as important as the other
> > > four, if I have to use the CLI, so be it.)
> >
> > All of the examples at the top are geared towards web UI, though many
> > also let you use a terminal or ssh in.
> >
> > >
> > > Pros/Cons/Suggestions/Thoughts/Tar/Feathers?
> > >
> >
> > The problem with ZFS is that it has many great features, but not all
> > apply at once. I was looking into it for a major project and got really
> > excited reading about all the great support it has for adding drives
> > expanding pools, snapshots, and RAIDZ.  It wasn't until I got into the
> > details via a test VM that I found out about RAIDZ volumes not being
> > expandable.
> >
> > You also need to make sure that whatever OS you use has a version of ZFS
> > that supports the feature(s) you want to use. I wouldn't mess with ZFS
> > on linux at all.
> >
> > Also, ZFS isn't really recommended for 32-bit systems.  You can do it,
> > but I really don't advise it if you'll be dealing with large amounts of
> > storage.  Especially if combined with low amounts of RAM.
> >
> > On the other hand, LVM and Linux RAID are very mature approaches with
> > easy to use tools.
> >
> > If a web UI is a lower priority for you, it sounds like this system will
> > be run by a reasonably technically proficient person. The older I get,
> > and the more projects I get under my belt, the less I like the
> > all-in-one or "appliance" approaches.
> >
> > If you just do a standard install of a distro, you'll get continuous
> > security updates and a great deal of flexibility.  The downside is it
> > takes a little more know-how to get things setup. The really nice thing
> > about Webmin vs some of the other admin interfaces like cpanel/plesk/etc
> > is that Webmin doesn't really mess with the installed system or make
> > specialized customizations to it.  It's really just a GUI that edits the
> > config files for you, while still giving you the option to edit them
> > yourself. I'm curious where OMV falls on this spectrum.
> >
> >
> > - PaulNM
> >
> >
> >
> >
> >
> >
> >
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group         --
> > http://www.phillylinux.org
> > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion  --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >

> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug


-- 
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug