| Doug Stewart on 8 Aug 2018 19:37:06 -0700 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
| Re: [PLUG] Virtualization clusters & shared storage |
JP: Merely conceptual overlap and not really practical for your purposes. I think I was trying to say that other people see the same issue as you do, just in a different venue. -- Doug Stewart > On Aug 8, 2018, at 9:23 PM, JP Vossen <jp@jpsdomain.org> wrote: > > Firth, thanks to Lee and Andy for the term I didn't know for what I mean: hyper-converged infrastructure. > https://en.wikipedia.org/wiki/Hyper-converged_infrastructure > > It looks like Proxmox does that: > https://pve.proxmox.com/wiki/Hyper-converged_Infrastructure > > Thanks Keith, I had seen LizardFS but was not aware of the implications. > > Doug, Kubernetes is not in play, though it might be in the future. Or I may be missing your point. > > Lee, thanks for the insights and VMware updates, as always. :-) > > I've used FreeNAS in the past and like it, but I'm not sure I'm explaining my thoughts as well as I'd like. But let me try this: > > (I wrote this next part before I learned that "hyper-converged" is what I mean, but I'm leaving it here in case it's still useful.) > > Assume for simplicity that I have 5 R710s with 24M RAM, 8 CPUs and 6x 2TB drives in RAID5 for 10TB local storage each. > > 3 node cluster with redundant FreeNAS: > 1. VM node1: CPU/RAM used for VMs, 10TB local space wasted > 2. VM node2: CPU/RAM used, 10TB local space wasted > 3. VM node3: CPU/RAM used, 10TB local space wasted > 4. FreeNAS 1: CPU/RAM somewhat wasted, 10TB space available > 5. FreeNAS 2: CPU/RAM somewhat wasted, 0TB space available since it's a mirror > > 3 node cluster with -on-node-shared-storage- hyper-converged storage: > 1. VM node1: CPU/RAM used, 20TB local/shared space available + > 2. VM node2: CPU/RAM used, 20TB local/shared space available + > 3. VM node3: CPU/RAM used, 20TB local/shared space available + > + For 20TB I'm assuming (3-1) * 10TB, for some kind of parity space loss. If it was REALLY smart, it would keep the store for the local VMs local while still replicating, but that would require a lot more hooks into the entire system, not just some kind of replicated system. > > But the point here is that with my idea I have 2x the disk with 3/5ths the servers. Or put another way, I can now do a 5 node cluster with even more CPU, RAM and space dedicated to actually running VMs, and not lose 2/5ths of the nodes to just storing the VMs. > > That said, I'd thought about the rebuild overhead, but not in depth, and that--and general "parity" or redundancy however implemented--are significant. So my 2/5ths comparisons are not 100% fair. But still, the idea apparently does have merit. > > >> On 08/08/2018 07:50 PM, Lee H. Marzke wrote: >> JP, if you want cheap storage for your lab , I think you can't beat FreeNAS or >> equiv rackmount solutions from https://www.ixsystems.com. I run my lab on >> FreeNAS and a Dell 2950 server with 6x2TB disks and 2 x SSD. If you put storage >> into the servers you will find all sorts of edge cases that you hadn't planned on. >> Just taking down a server for quick RAM swap, will cause it to need to rebuild >> using lots of network/CPU. If you have TB of fast SSD storage on multiple servers and >> don't have 10GB connectivity between hosts, or you have slow HD's you will have pain. >> Generally you try to migrate data off nodes prior to maintenance - which may take several days. >> The VMware solutions have changed a lot, and though it does not meet JP's >> needs for *free*, they may fit someone else's needs for a reliable / highly >> available solution. Of course ESXi and VSAN are free for 60 day trial after >> install for lab use. >> First there is a VMware Migrate *both* mode where you can migrate both the Hypervisor and >> then storage in one go, where the two storage units are not connected across Hypervisors. >> Needless to say this takes a long time to sequentially move memory , then disk to the remote >> server, and it doesn't help improve HA. >> Next VMware VSAN is caching on really fast and VMware is hiring like mad >> to fill new VSAN technical sales roles Nationwide. VSAN uses storage in each >> host ( minimum of one SSD and one HD ) and uses high-performance object >> storage on each compute node. All VM objects are stored on two hosts >> minimum, with vSAN taking care of all the distribution. The hosts must be >> linked on a 1GB ( pref 10GB ) private network for back-end communication. >> Writes are sent and committed to two nodes before being acknowledged. >> You get one big storage pool - and allocate storage to VM's as you like - >> with no sub LUNs or anything else to manage. If you have 4 or more hosts >> instead of mirroring data over 2 hosts, you can do erasure coding ( equiv >> of RAID 5/6 but with disks spread out across hosts. So now your not >> losing 50% of your storage , but have more intensive CPU and network >> operations. The vSAN software is pre-installed into ESX these days - just >> need to activate it and apply a license after the 60 free day trial. >> Not sure why you say FreeNAS is wasting CPU in more nodes, as those CPU cycles >> would be used locally in the Hyperconverged solutions as well ( perhaps taking 10% >> to 20% cycles away from a host for storage and replication ) so you may need more / larger >> hosts in a hyperconverged solution to make up for that. Remember mirroring >> takes little CPU, but waste's 50% of your storage, any erasure coding is >> much more CPU intensive, and more network intensive. >> The other solutions mentioned except a ZFS server are likely way too complex >> for a lab storage solution. Is a company really going to give >> a lab team 6 months of effort to put together storage that may or may >> not perform ? Can you do a business justification to spend dozens of MM of >> effort just to save the $20K on an entry level TrueNAS ZFS ? >> Lee >> ----- Original Message ----- >>> From: "Vossen JP" <jp@jpsdomain.org> >>> To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org> >>> Sent: Wednesday, 8 August, 2018 17:13:17 >>> Subject: [PLUG] Virtualization clusters & shared storage >>> I have a question about virtualization cluster solutions. One thing >>> that has always bugged me is that VM vMotion/LiveMigration features >>> require shared storage, which makes sense, but they always seem to >>> assume that shared storage is external, as in a NAS or SAN. What would >>> be REALLY cool is a system that uses the cluster members "local" storage >>> as JBOD that becomes the shared storage. Maybe that's how some of >>> solutions work (via Ceph, GlusterFS or ZFS?) and I've missed it, but >>> that seems to me to be a great solution for the lab & SOHO market. >>> >>> What I mean is, say I have at least 2 nodes in a cluster, though 3+ >>> would be better. Each node would have at least 2 partitions, one for >>> the OS/Hypervisor/whatever and the other for shared & replicated >>> storage. The "shared & replicated" partition would be, well, shared & >>> replicated across the cluster, providing shared storage without needing >>> an external NAS/SAN. >>> >>> This is important to me because we have a lot of hardware sitting around >>> that has a lot of local storage. It's basically all R710/720/730 with >>> PERC RAID and 6x or 8x drive bays full of 1TB to 4TB drives. While I >>> *can* allocate some nodes for FreeNAS or something, that increases my >>> required node count and wastes the CPU & RAM in the NAS nodes while also >>> wasting a ton of local storage on the host nodes. It would be more >>> resource efficient to just use the "local" storage that's already >>> spinning. The alternative we're using now (that sucks) is that the >>> hypervisors are all just stand-alone with local storage. I'd rather get >>> all the cluster advantages without the NAS/SAN issues >>> (connectivity/speed, resilience, yet more rack space & boxes). >>> >>> Are there solutions that work that way and I've just missed it? >>> >>> >>> Related, I'm aware of these virtualization environment tools, any more >>> good ones? >>> 1. OpenStack, but this is way too complicated and overkill >>> 2. Proxmox sounds very cool >>> 3. Cloudstack likewise, except it's Java! :-( >>> 4. Ganeti was interesting but it looks like it may have stalled out >>> around 2016 >>> 5. https://en.wikipedia.org/wiki/OVirt except it's Java and too limited >>> 6. https://en.wikipedia.org/wiki/OpenNebula with some Java and might do >>> on-node-shared-storage? >>> 7. Like AWS: https://en.wikipedia.org/wiki/Eucalyptus_(software) except >>> it's Java >>> >>> I'm asking partly for myself to replace my free but not F/OSS ESXi >>> server at home and partly for a work lab that my team needs to rebuild >>> in the next few months. We have a mishmash right now, much of it ESXi. >>> We have a lot of hardware laying around, but we have *no budget* for >>> licenses for anything. I know Lee will talk about the VMware starter >>> packs and deals like that but we not only have no budget, that kind of >>> thing is a nightmare politically and procedurally and is a no-go; it's >>> free or nothing. And yes I know that free costs money in terms of >>> people time, but that's already paid for and while we're already busy, >>> this is something that has to happen. >>> >>> Also we might like to branch out from ESXi anyway... We are doing a >>> some work in AWS, but that's not a solution here, though cross cloud >>> tools like Terraform (and Ansible) are in use and the more we can use >>> them here too the better. > Thanks, > JP > -- ------------------------------------------------------------------- > JP Vossen, CISSP | http://www.jpsdomain.org/ | http://bashcookbook.com/ > ___________________________________________________________________________ > Philadelphia Linux Users Group -- http://www.phillylinux.org > Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce > General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug