Rich Freeman via plug on 17 Jan 2023 13:00:35 -0800 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
Re: [PLUG] Assessing interest in potential PLUG topics |
On Tue, Jan 17, 2023 at 2:55 PM brent saner via plug <plug@lists.phillylinux.org> wrote: > > Oh hey, one of my wheelhouses! Yeah, I don't have too much to add to what you said. Maybe you should do the Ceph talk, though I'm happy to do it. :) > It's not hard and virtually bulletproof once up, but BOY, that learning curve is a 95 degree incline. That's part of why I think it makes for an interesting talk. It has a really nice design that is intended to eliminate bottlenecks to scalability, which is why it is so popular on large installs. I've mentioned it in passing in other talks like my LizardFS one. (I actually recently migrated to MooseFS which could be another talk, but honestly it is mostly the same info at a superficial level as for LizardFS. I could probably summarize the pros/cons I've found in an email. I'm sure Keith would appreciate that if nobody else does... > The RedHat docs are decent. The upstream docs are... uh. They're not good. Yeah, I discovered that when I wanted to grant a single client account access to both block/object resources and cephfs. The official docs don't tell you how to do that with a single account (doing it with two is easy with their directions though). > In short, I chose Ceph over Longhorn because (again, as of a bit over a year ago) it: > > - Didn't have the same amount of stability, recovery, and "self-healing" features Ceph does (did) > - Didn't have native Linux kernel support like Ceph > -- And no feasible/practical Windows block client support ($dayjob is mixed-platform) > - Just didn't have the same amount of third-party integrations Ceph did/does So, I played with Longhorn not long after I installed k8s as I was looking for block storage options as I wasn't sure if I wanted to accept the overhead required to store small files (databases/etc) on MooseFS. Longhorn was very easy to install and definitely felt more k8s-native than Rook/Ceph. However, I have no idea how it scales (I'm guessing not nearly as well), and it really only solved block storage (PVs/PVCs in k8s speak). I don't think it has object store or filesystem options. It did seem easy to maintain assuming it does what you need and doesn't eat your data. IMO Ceph is the solution you should be choosing for distributed storage unless you have a REALLY good reason not to at this point. EVERYBODY seems to be using it these days, assuming they're not using some proprietary solution. Rook orchestrates Ceph on k8s, but definitely doesn't feel as k8s-native as longhorn. I ended up playing around with ceph command line operations to do things, as not quite everything is abstracted in a k8s Custom Resource. That isn't entirely a bad thing though as ceph is quite capable. The thing about Ceph is that it has a lot of components that all need to find each other and authenticate and so on, so it is often deployed with some kind of orchestration solution. I was playing around with an Ansible playbook for it that had something like 200 plays in it - edit a config file and watch it go nuts with ssh. Rook is just one of those orchestration solutions, but of course k8s is all about orchestration so it is a pretty good one. Create your cluster Custom Resource and in the yaml you tell it where to find drives for OSDs and then watch it start spawning monitors, OSDs, jobs to go format bluestore devices, and so on. Create a filesystem custom resource and watch some metadata servers spawn. It has some automagic options that I'm not sure I'd trust, but in theory you could just plug a bare hard drive in any k8s node and it will notice it, format it with bluestore, and spawn an OSD. (I don't think I'd want it to actually do that - and that is an option.) You can also run the Ceph cluster purely inside of k8s in which case it isn't exposed to the network, or you can run it with host networking which makes it very easy to access from outside of k8s, which would be important to me. As you mentioned I don't think that is really an option for Longhorn, though I think you can tell Longhorn to maybe mount a volume on a node at the host level - maybe. Really getting data into and out of PVs seems like a bit of a pain in k8s if your storage solution doesn't provide some kind of back door like Ceph would. -- Rich ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug