Re: [PLUG] Assessing interest in potential PLUG topics

[Rich Freeman via plug <plug@lists.phillylinux.org>]

On Tue, Jan 17, 2023 at 2:55 PM brent saner via plug
<plug@lists.phillylinux.org> wrote:
>
> Oh hey, one of my wheelhouses!

Yeah, I don't have too much to add to what you said.  Maybe you should
do the Ceph talk, though I'm happy to do it.  :)

> It's not hard and virtually bulletproof once up, but BOY, that learning curve is a 95 degree incline.

That's part of why I think it makes for an interesting talk.  It has a
really nice design that is intended to eliminate bottlenecks to
scalability, which is why it is so popular on large installs.  I've
mentioned it in passing in other talks like my LizardFS one.  (I
actually recently migrated to MooseFS which could be another talk, but
honestly it is mostly the same info at a superficial level as for
LizardFS.  I could probably summarize the pros/cons I've found in an
email.  I'm sure Keith would appreciate that if nobody else does...

> The RedHat docs are decent. The upstream docs are... uh. They're not good.

Yeah, I discovered that when I wanted to grant a single client account
access to both block/object resources and cephfs.  The official docs
don't tell you how to do that with a single account (doing it with two
is easy with their directions though).

> In short, I chose Ceph over Longhorn because (again, as of a bit over a year ago) it:
>
> - Didn't have the same amount of stability, recovery, and "self-healing" features Ceph does (did)
> - Didn't have native Linux kernel support like Ceph
> -- And no feasible/practical Windows block client support ($dayjob is mixed-platform)
> - Just didn't have the same amount of third-party integrations Ceph did/does

So, I played with Longhorn not long after I installed k8s as I was
looking for block storage options as I wasn't sure if I wanted to
accept the overhead required to store small files (databases/etc) on
MooseFS.

Longhorn was very easy to install and definitely felt more k8s-native
than Rook/Ceph.  However, I have no idea how it scales (I'm guessing
not nearly as well), and it really only solved block storage (PVs/PVCs
in k8s speak).  I don't think it has object store or filesystem
options.  It did seem easy to maintain assuming it does what you need
and doesn't eat your data.

IMO Ceph is the solution you should be choosing for distributed
storage unless you have a REALLY good reason not to at this point.
EVERYBODY seems to be using it these days, assuming they're not using
some proprietary solution.

Rook orchestrates Ceph on k8s, but definitely doesn't feel as
k8s-native as longhorn.  I ended up playing around with ceph command
line operations to do things, as not quite everything is abstracted in
a k8s Custom Resource.  That isn't entirely a bad thing though as ceph
is quite capable.

The thing about Ceph is that it has a lot of components that all need
to find each other and authenticate and so on, so it is often deployed
with some kind of orchestration solution.  I was playing around with
an Ansible playbook for it that had something like 200 plays in it -
edit a config file and watch it go nuts with ssh.  Rook is just one of
those orchestration solutions, but of course k8s is all about
orchestration so it is a pretty good one.  Create your cluster Custom
Resource and in the yaml you tell it where to find drives for OSDs and
then watch it start spawning monitors, OSDs, jobs to go format
bluestore devices, and so on.  Create a filesystem custom resource and
watch some metadata servers spawn.  It has some automagic options that
I'm not sure I'd trust, but in theory you could just plug a bare hard
drive in any k8s node and it will notice it, format it with bluestore,
and spawn an OSD.  (I don't think I'd want it to actually do that -
and that is an option.)  You can also run the Ceph cluster purely
inside of k8s in which case it isn't exposed to the network, or you
can run it with host networking which makes it very easy to access
from outside of k8s, which would be important to me.  As you mentioned
I don't think that is really an option for Longhorn, though I think
you can tell Longhorn to maybe mount a volume on a node at the host
level - maybe.  Really getting data into and out of PVs seems like a
bit of a pain in k8s if your storage solution doesn't provide some
kind of back door like Ceph would.

-- 
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug