Re: [PLUG] Fixing /boot/ too small

[JP Vossen via plug <plug@lists.phillylinux.org>]

To close the loop on this one, I *finally* got out to NJ to fix this, 
and the following process *worked*!  The process at the very bottom 
worked for the simple case but NOT with mdadm RAID-1.

Note, special cases:
1. Old computer using BIOS, no UEFI
2. But it is using mdadm RAID-1, which is why it had a too-small /boot/
3. Not tested with LVM, LUKS or anything else

Have a good backup and a TESTED rescue USB stick before you start! 
Being near another computer with an internet connection is also a Good 
Thing.

Process to swap /boot/ from partition onto /:
sudo -s
cd /
cp -a /boot /boot.new
mkdir /boot.old
vi /etc/fstab
	# And change it to boot.old, see below
umount /boot
rmdir boot
mv boot.new boot
dpkg-reconfigure mdadm        # Originally missed, pretty sure CRITICAL!
#update-initramfs -u -k all    # Probably not needed
grub-install /dev/sda         # Originally missed and this is CRITICAL!
grub-install /dev/sdb         # Originally missed and this is CRITICAL!
update-grub
Reboot with crossed fingers

The /etc/fstab/ edit is something like commenting out line 11 and adding 
line 12, but it depends on your system.  You don't need to call it 
"boot.old" actually, rename as appropriate to whatever you'll use it 
for, if anything.  "Scratch" maybe?  Whatever.

```
$ cat -n fstab
...
     7  # <file system> <mount point>   <type>  <options>       <dump> 
<pass>
     8  # / was on /dev/md1 during installation
     9  UUID=1c6962e0-7887-4900-bf3e-2dff7cfb47b6 /               ext4 
  errors=remount-ro 0       1
    10  # /boot was on /dev/md0 during installation
    11  #UUID=2ec8d4f8-1510-4a5e-8f65-6e6d67a7a3d7 /boot           ext4 
   defaults        0       2
    12  UUID=2ec8d4f8-1510-4a5e-8f65-6e6d67a7a3d7 /boot.old        ext4 
   defaults        0       2
```

I have slightly mixed testing records for the `dpkg-reconfigure mdadm` 
and `grub-install /dev/sd{a,b}` parts.  I know the `grub-install` is 
needed, which I found surprising given the `update-grub` but it just did 
not work without the re-installs.  I *think* the `dpkg-reconfigure 
mdadm` is also needed, but I did not test without it.  Bottom line, 
neither hurts and at least one is needed, so just do it.


IF you muck it up and break it (like I did, a bunch of times), it's not 
hard to fix it!  This is the fix for the mdadm RAID-1, if you have LVM 
or LUKS, you have to install and fiddle with them instead of `mdadm`.

I booted from my Mint-19 USB stick and got a terminal, then:
sudo -s
apt install mdadm 		# [1]
mdadm --assemble --scan
mount /dev/md1 /mnt		# Your numbers may vary
for dir in dev proc run sys; do mount -o bind /$dir /mnt/$dir; done
chroot /mnt
# now either fix stuff or do the process as above...whatever
reboot

It gets tedious doing that again and again until you get it fixed right, 
so you can write a script to put someplace you can `scp` it from the 
Live-USB when you reboot into it.

[1] Do NOT `apt update` before `apt install mdadm`!  When I did that I 
got a too-new-for-USB-stick-system `mdadm` that hung on install and did 
not work!


On 4/7/20 11:24 PM, JP Vossen wrote:
> Summary: I need to fix some Mint-19 workstations where /boot/ is too small.
>
> Solution: below but needs sanity check.
>
> Details:
>
> I have some workstations I installed a long time ago and /boot/ is now 
> too small.  200M was fine when I built them, but not anymore.  I did a 
> hacked RAID-1 mdadm install because whatever installer (Ubuntu or Mint) 
> didn't do RAID, but I didn't do LVM.  I've never had to re-install 
> because...RAID-1!  I've replaced all the disks over the years by 
> swapping over to bigger ones, re-mirroring, and growing /.
>
> The too small /boot/ is now causing a lot of problems because every time 
> I run OS updates that want `update-initramfs` it fails due to lack of 
> space, which fails the update, which fails the Ansible task.  And then I 
> am sad.
>
> Fix: All the disk space is allocated, and they are not using LVM anyway, 
> so I can't just make /boot/ bigger.  (Well, if I was swapping to bigger 
> disks I probably could, but that's not soon enough.)  But I *CAN* just 
> move /boot/ from its own md device and partition into '/' where it 
> probably should have been in the first place.  I'd then be wasting the 
> old /boot/ space, but I can live with that or find something else to put 
> in there.  Maybe make it /tmp/.
>
> I've done the following process on a sacrificial Mint-19 VM and rebooted 
> and it Just Worked.  BUT that VM was not running mdadm because I didn't 
> feel like hacking the install again.  I've also done the *process* on 2 
> of the workstations, but not *rebooted* yet.  One node is 1.5 hours away 
> in NJ...
>
> On one hand, this is a very simple and straightforward process, even 
> with mdadm in the mix.  OTOH, GRUB and the boot process are subtle and 
> quick to anger.
>
> Solution:

This should work for a "simple" system, like without mdadm raid...

> sudo -s
> cp -a /boot /boot.new
> mkdir /boot.old
> touch /boot.new/z-boot.new /boot.old/z-boot.old /boot/z-boot.orig
>      # So I know what used to be where
> vi /etc/fstab
>      # To change the mount for /boot/ to /boot.old/
> umount /boot
> rmdir /boot
> mv /boot.new /boot
> update-initramfs -u -k all
> update-grub
> ### reboot?!?

Later,
JP
--  -------------------------------------------------------------------
JP Vossen, CISSP | http://www.jpsdomain.org/ | http://bashcookbook.com/
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug