Re: [PLUG] Collecting k8s events

[Rich Freeman via plug <plug@lists.phillylinux.org>]

On Wed, Jun 14, 2023 at 1:20 AM Will <staticphantom@gmail.com> wrote:
>
> I would look into Loki with Graphana, Prometheus, OpenSearch, and for tracing I would consider Jaeger.
>
> I need to take time and talk to you on IRC for how fluentd works and related to set up a backend. For the backend, if you want for a time series database that isn't Loki, look at TimeScale.
>

You know I'm always up for a chat...

I ended up going with Grafana as my first solution.  This was largely
due to looking into Elasticsearch and what it takes to run it.  Due to
the full-text indexing and use of Java it apparently needs gigabytes
of RAM.  That would make sense if I had thousands of applications, but
it just is too heavy for what I'm doing. Really just dumping
everything into a text file somewhere and using grep would probably
have been sufficient.

Grafana claims to be easy to deploy and have minimal resource
requirements.  The big caveat is that it has minimal indexing.
Typically the workflow for logs is that you use the indexed fields to
narrow the data down to the time/application/node/container/etc that
you're interested in, then you have it select just that data and do a
full index of it, and then you can query whatever fields it extracts
from the log data to get the lines you're interested in.  So it is a
slower search workflow, but the assumption with logs is that 99% of
the time you don't look at them anyway, so you're optimizing for
storage and not for retrieval.

I was looking up the instructions for deploying the server side, and
their site was prominently advertising the free tier of their cloud
service, which actually seems like it would be sufficient.  So I just
deployed the agents and pointed them at the cloud and it was pretty
easy to get most of it working (I'm not sure that all the components
are actually working, but logs are, and the metrics are really just a
bonus).  My main concern is whether I really want to be sending all
those logs to a random company as they contain personal info.

I think I might focus on some other things first, but I wouldn't be
surprised if I deployed Grafana+Loki locally and redirect the agents
to point to them.  That shouldn't be too hard since it is all in a
configmap.  I think it might also require prometheus but the docs are
a little fuzzy on that (at least from the little I've read so far).

The main issue I see with fluentd is that it doesn't seem to have an
interface to grab k8s events.  It just looks at container logs in
/var/log/containers and those don't contain the events.  It would need
to talk to the k8s API to pull those.  Grafana agent apparently does
that.

-- 
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug