System Recovery

Recovery Strategy

When a machine will not boot or behaves badly, work from the outside in: get a shell, see the disk, then repair the smallest broken thing. Most recoveries follow the same arc: boot rescue media, mount or `chroot` into the install, fix the bootloader, filesystem, or config, then reboot.

Symptom	Reach for
No bootloader	$ grub-install
Kernel can't find root device	$ update-initramfs
Boot drops to emergency mode	$ findmnt
Lost root password	$ passwd
Dirty filesystem	$ fsck
Broken packages	$ dpkg
Failing disk	$ smartctl
Deleted files	$ testdisk

Before you change anything on a failing disk, image it first. See the Backup & Imaging basics page for `ddrescue` and `dd`.

Boot from Rescue Media

Almost every repair starts from a second environment, so a broken install is not running while you fix it. Boot a live USB (any distribution's installer works) or your distribution's dedicated rescue image, then drop to a terminal.Write a live image to a USB stick from another machine.

$ dd if=linux.iso of=/dev/sdc bs=4M status=progress conv=fsync

Many distributions also ship a rescue entry in GRUB, and systemd offers built-in recovery shells you can request from the boot menu (see *Rescue & Emergency Mode* below). Use a live USB when the bootloader itself is gone.Match the live system's architecture and, ideally, its toolset to the broken install. An Arch live USB has `mkinitcpio`; a Debian one has `update-initramfs`.

See the Disks

Before mounting anything, identify the partitions. `lsblk` shows the block-device tree, and `blkid` prints filesystem types and UUIDs so you mount the right thing.

$ lsblk -f

$ blkid

$ fdisk -l

`findmnt` shows what is already mounted, useful when a live system auto-mounts disks.

$ findmnt

Command	Shows
lsblk -f	Devices, filesystems, labels, mountpoints
blkid	UUID and type per partition
fdisk -l	Partition tables and sizes
findmnt	Current mount tree

Activate LVM & Encrypted Volumes

If the root filesystem lives on LVM or LUKS, the live system will not see it until you bring it up. Unlock an encrypted partition first, then scan for volume groups.

$ cryptsetup open /dev/sda2 cryptroot

$ vgscan

$ vgchange -ay

$ lvscan

After this, logical volumes appear under `/dev/mapper/` and the unlocked LUKS device under `/dev/mapper/cryptroot`, ready to mount.

Mount the Broken System

Mount the root partition somewhere, then layer `/boot` (and the EFI partition) on top so paths match the real system.

$ mount /dev/sda2 /mnt

$ mount /dev/sda1 /mnt/boot/efi

If you only need to copy data off, stop here and grab the files. If the root filesystem refuses to mount cleanly, run a check on it first (see *Filesystem Repair*).Mount read-only with `mount -o ro` when you only want to rescue data from a questionable disk, so you cannot make things worse.

chroot into the System

To run the broken system's own tools (its `grub`, its package manager, its `passwd`), `chroot` into it. First bind-mount the kernel's pseudo-filesystems so those tools work, including the EFI variables for UEFI bootloader fixes.

$ mount --rbind /dev /mnt/dev

$ mount --rbind /proc /mnt/proc

$ mount --rbind /sys /mnt/sys

$ mount --rbind /run /mnt/run

$ chroot /mnt /bin/bash

On Arch and its derivatives, `arch-chroot` does all of this in one step.

$ arch-chroot /mnt

When you finish, `exit` the chroot and unmount everything in reverse, recursively.

$ umount -R /mnt

The bind mounts are not optional. Without `/dev`, `/proc`, and `/sys` mounted, tools like `grub-install` and `update-initramfs` fail in confusing ways. After each `--rbind`, run `mount --make-rslave /mnt/dev` (and the same for the others) so the later `umount -R /mnt` cannot propagate back and unmount the live system's own `/dev` and `/sys`.

Repair the Bootloader

A missing or broken bootloader leaves you at a `grub>` prompt or with no menu at all. From inside the chroot, reinstall GRUB to the disk, then regenerate its config.For a legacy BIOS system, install to the whole disk (not a partition).

$ grub-install /dev/sda

For a UEFI system, install to the EFI partition with the EFI target.

$ grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

Then rebuild the menu so it lists the installed kernels. Debian and Ubuntu wrap this as `update-grub`.

$ grub-mkconfig -o /boot/grub/grub.cfg

$ update-grub

Inspect or clean up UEFI boot entries with `efibootmgr` when the firmware boots the wrong thing.

$ efibootmgr -v

Rebuild the initramfs

If a kernel update was interrupted or the early-boot image is broken (a "cannot find root device" or kernel panic on boot), regenerate the initramfs from inside the chroot. The command depends on the distribution.

Distribution	Rebuild command
Debian / Ubuntu	update-initramfs -u -k all
Arch	mkinitcpio -P
Fedora / RHEL	dracut -f --regenerate-all

$ update-initramfs -u -k all

$ mkinitcpio -P

$ dracut -f --regenerate-all

Rescue & Emergency Mode

You do not always need a live USB. systemd ships two recovery targets you can request at the GRUB menu: rescue (single-user, most services stopped) and emergency (a bare shell, root mounted read-only, almost nothing else).At the menu, highlight the entry, press `e` to edit, append to the `linux` line, then press `Ctrl-X` (or `F10`) to boot it:

$ systemd.unit=rescue.target

$ systemd.unit=emergency.target

To bypass init entirely and get a root shell with no password, append this instead, then remount the root filesystem writable.

$ init=/bin/bash

$ mount -o remount,rw /

From an already-running system, you can switch into these modes directly, and resume a normal boot once the problem is fixed.

$ systemctl rescue

$ systemctl emergency

$ systemctl default

Careful: after `init=/bin/bash`, the normal shutdown path is gone. Run `sync` and reboot with the magic SysRq keys or `mount -o remount,ro /` before a hard reset, so you do not corrupt the filesystem.

Reset a Forgotten Root Password

The simplest path is from a chroot: mount the system, `chroot` in, and just set the password.

$ chroot /mnt /bin/bash

$ passwd root

Without rescue media, use the `init=/bin/bash` trick above, remount root read-write, change the password, then reboot.

$ mount -o remount,rw /

$ passwd root

$ exec /sbin/init

On SELinux systems (Fedora, RHEL): run `touch /.autorelabel` after changing the password this way. Otherwise the rewritten `/etc/shadow` keeps a wrong security label and can lock you out again on the next boot.

Filesystem Repair

A dirty or corrupt filesystem can block boot. Always check an unmounted filesystem, never the live root. Boot rescue media, leave the partition unmounted, then run the right checker.For ext2/3/4, `fsck` (or `e2fsck`) replays the journal and fixes structure. `-f` forces a full check, `-y` answers yes to every prompt.

$ fsck -f /dev/sda2

$ e2fsck -fy /dev/sda2

XFS uses its own repair tool, which needs the filesystem unmounted.

$ xfs_repair /dev/sda2

Btrfs checks read-only by default; `--repair` is a genuine last resort that can make damage worse.

$ btrfs check /dev/sda2

$ btrfs check --repair /dev/sda2

Filesystem	Check / repair
ext2/3/4	fsck -f, e2fsck -fy
XFS	xfs_repair
Btrfs	btrfs check, --repair (last resort)
FAT / exFAT	fsck.vfat -a, fsck.exfat

Careful: `fsck` on a mounted, writable filesystem will corrupt it. If you must check the root device, do it from rescue media or before it is remounted read-write.

Fix a Broken /etc/fstab

A typo or a missing device in `/etc/fstab` drops the boot into emergency mode. From the emergency shell, remount root writable and fix or comment out the bad line.

$ mount -o remount,rw /

Edit `/etc/fstab`, comment out the offending mount, or add the `nofail` option so a missing device no longer blocks boot.

$ UUID=... /data ext4 defaults,nofail 0 2

Verify the file parses before rebooting; `mount -a` tries every entry and reports failures.

$ mount -a

$ findmnt --verify

Match devices by UUID (from `blkid`), not by `/dev/sdX`, which can change between boots and is a common cause of fstab boot failures.

Repair Broken Package State

An interrupted upgrade can leave packages half-configured and the system unbootable or unusable. Run these from a chroot if the system will not boot, or directly if it limps to a shell.On Debian and Ubuntu, finish pending configuration and resolve broken dependencies.

$ dpkg --configure -a

$ apt --fix-broken install

On Fedora and RHEL, check and re-synchronize the package set.

$ dnf check

$ dnf distro-sync

Rebuild a corrupt RPM database if `rpm` itself errors out.

$ rpm --rebuilddb

Diagnose with Logs

Before guessing, read what the system recorded. `journalctl` shows the boot log; `-x` adds explanations, `-b` selects a boot, and `-b -1` shows the previous (failed) boot.

$ journalctl -xb

$ journalctl -b -1 -p err

From a live USB, point `journalctl` at the broken system's logs instead of the live one's.

$ journalctl -D /mnt/var/log/journal -xb

The kernel ring buffer catches hardware and driver errors that crash early.

$ dmesg --level=err,warn

Check Disk Health

If the same errors keep coming back, the drive may be dying. `smartctl` reads the disk's own SMART data, the fastest way to confirm hardware failure.

$ smartctl -H /dev/sda

$ smartctl -a /dev/sda

$ smartctl -t short /dev/sda

`badblocks` scans the surface for unreadable sectors. Use the read-only mode on a disk with data on it.

$ badblocks -sv /dev/sda

A failing Reallocated_Sector_Ct or Current_Pending_Sector in `smartctl -a` means: stop, image the disk with `ddrescue`, and replace it. Repairs on dying hardware only buy minutes.

Recover Deleted Files & Partitions

When the problem is lost data rather than a broken boot, work on a copy of the disk, never the original. `testdisk` rebuilds damaged partition tables and recovers whole partitions.

$ testdisk /dev/sda

`photorec` ignores the filesystem and carves files back by their signatures, which works even on a reformatted disk.

$ photorec /dev/sda

For ext3/ext4, `extundelete` can recover recently deleted files using the journal.

$ extundelete /dev/sda2 --restore-all

The moment you realize files were deleted, unmount the filesystem or power off. Continued writes overwrite the very blocks you are trying to recover.

Last Resort: Rescue, then Rebuild

If the install is too broken to repair, salvage what you can and start fresh. Image the failing disk before it gets worse, then mount the image to extract files at your own pace.

$ ddrescue /dev/sda rescue.img rescue.map

A whole-disk image holds a partition table, not a single filesystem, so attach it read-only with `losetup -P` to expose its partitions, then mount the one you need.

$ losetup -rfP --show rescue.img

$ mount -o ro /dev/loop0p2 /mnt

Copy the data you need off the image, reinstall the OS, then restore from backups. A clean reinstall is often faster and safer than chasing a deeply corrupted system.Every recovery is easier with a current backup and a tested restore. The work you do here is the price of not having one: see the Backup & Imaging page.