Re: [PLUG] Disk is Full. How to Move /var?

[Rich Freeman <r-plug@thefreemanclan.net>]

On Mon, Oct 15, 2018 at 10:59 AM Soren Harward <stharward@gmail.com> wrote:
>
> On Mon, Oct 15, 2018 at 10:40 AM Fred Stluka <fred@bristle.com> wrote:
>>
>> Historically, I thought "var" was named for "variable" because it
>> was the one directory where it as most likely for truly huge files
>> to be put.
>
>
> That may have been its original purpose, and is still true on servers
> where the mail/news/print spools, daemon "home" directories like
> /var/www and /var/lib/mysql, and logs reside. Most package managers
> — Gentoo included — also track their state somewhere in /var,
> usually /var/db.  But for a desktop system, where the bulky data is
> (or should be) under /home, /var shouldn't be more than a few GB, most
> of which will be logs.

FHS goes into the gory details, but /var is basically where packages
can store persistent state of one kind or another.  You could view it
as /home for daemons.

The other examples Fred gave were  /bin, /lib, /opt, /sbin, /usr.  All
of these are read-only directories except when installing packages.
Nothing except the package manager should be writing to them, and
other than when installing packages these should be able to be mounted
read-only.  Places like /usr/lib or /usr/share are used to store
various types of application-specific STATIC data, but not anything
that changes at runtime.

/var can get quite large on some servers, such as mail servers (that
don't use .maildir in /home), database servers, web servers, and so
on, since these daemons typically serve data stored in /var.  On a
typical desktop install it tends to be small.

>
> The other things that gets stored under /var are small, variable files
> describing system state, like the stuff in /var/run and /var/lock .
> These are the files that I'm concerned about getting mangled early in
> the boot process.  Most distros are gradually moving to using /run
> (which is mounted as a tmpfs) instead of /var/lock or /var/run; Gentoo
> w/ systemd /var/lock and /var/run are both symlinked to /run.  But for
> the symlink to behave properly, /var itself has to be mounted.

At least in the past Gentoo has not tended to like having either /var
or /usr be symlinks.  I think portage used to throw errors in this
situation.  Bind mounts generally behave better.

Officially I believe Gentoo only supports /usr or /var on a separate
filesystem from / if it is mounted early during boot, and ensuring
this is the user's responsibility.  That isn't to say that all systems
break if these directories aren't on root, but if they do any
resulting bugs can be marked INVALID at the maintainer's discretion.

I don't have my entire /var off of root, but I do have a number of
directories there stored elsewhere, since my root is an SSD.  I mount
everything from an initramfs, so by the time the system is booting
from the main root everything is available.  If you're making /var a
different filesystem then you almost certainly want to have it mounted
early during boot.

The typical solution to this is an initramfs.  Dracut makes it really
easy to mount directories before pivoting to the new root.  Often it
detects these situations in your fstab automatically and does the
right thing.  If not modules like fstab-sys make it pretty easy to
force things to mount during early boot (basically you stick an extra
fstab.sys in /etc which contains anything you want mounted in early
boot and dracut will make it so).  With Gentoo+openrc I believe there
is a script you can use to do the same thing that runs very early in
boot before anything needs /var/run and so on to be around.

> Anyway, I think that moving /var itself off the root filesystem is
> not worth the potential hazard of screwed up boot state because even
> if it is taking a ton of space, my experience is that (1) there are
> ways easy ways to recover that space like deleting old logs and/or (2)
> there's only one or two big subdirs of /var (like /var/lib/mysql or
> /var/spool/mail) that are eating most of the space, and can be moved
> to HDD while leaving the small stuff like /var/run, /var/lock, and
> /var/log on the root filesystem so that they're available immediately
> on boot.

This is usually the simplest approach.

If a daemon requires one of these directories then ensure the
directory is mounted before the daemon runs (with systemd this is
easily done with an AFTER dependency on a mount unit in a drop-in,
with openrc you could make a dep on the service that does the mounting
if that isn't already the default order).

And of course make sure your logs are rotating.  Almost any linux box
will run out of space eventually without log rotation, whether it be
via logrotate, or journald, or whatever.

As others have already pointed out, playing with /usr or /var is best
done when booted from alternate media, or from a shell in an initramfs
or something.

--
Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug