Recovery

Recovery

System recovery and rollback procedures for NullOS.

Overview

NixOS provides powerful recovery capabilities through generations, allowing you to roll back changes or recover from a broken system.

---

Boot Menu Recovery

Selecting Previous Generation

1. Boot system 2. Access GRUB menu (press Escape or Shift during boot if hidden) 3. Select "NixOS - All configurations" 4. Choose a previous generation 5. Boot into that generation

Your system will boot with the older configuration.

---

Making Rollback Permanent

After booting into an old generation:

# Make it the default
sudo nixos-rebuild switch --rollback

# Or switch to specific generation
sudo nix-env --switch-generation 42 --profile /nix/var/nix/profiles/system
sudo /nix/var/nix/profiles/system/bin/switch-to-configuration switch

---

Viewing Generations

List System Generations

sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

List Home Manager Generations

home-manager generations

---

Recovery from Live USB

When System Won't Boot

1. Boot from NixOS live USB

2. Mount system partitions:

sudo mount /dev/nvme0n1p2 /mnt
sudo mount /dev/nvme0n1p1 /mnt/boot

3. Enter system:

sudo nixos-enter --root /mnt

4. Rollback or fix:

nixos-rebuild switch --rollback
# Or edit configuration and rebuild

5. Reboot:

exit
reboot

---

Fixing Broken Configuration

Configuration Won't Build

1. Identify the error:

sudo nixos-rebuild switch --flake .#hostname --show-trace

2. Check recent changes:

git log --oneline -10
git diff HEAD~1

3. Revert changes:

git revert HEAD
# Or
git reset --hard HEAD~1

4. Rebuild:

sudo nixos-rebuild switch --flake .#hostname

System Builds But Won't Boot

1. Boot into previous generation from GRUB

2. Check what changed:

nix-diff \
  /nix/var/nix/profiles/system-42-link \
  /nix/var/nix/profiles/system-43-link

3. Fix and rebuild:

# Edit configuration
sudo nixos-rebuild switch --flake .#hostname

---

Recovering Lost Files

From Restic Backup

# List available backups
sudo restic -r sftp:server:/backups snapshots

# Find file
sudo restic -r sftp:server:/backups ls latest | grep filename

# Restore file
sudo restic -r sftp:server:/backups restore latest \
  --target /tmp/restore \
  --include /path/to/file

From Git History

# Find deleted file
git log --all --full-history -- path/to/file

# Restore from commit
git checkout commit-hash -- path/to/file

---

Password Recovery

Reset User Password

From live USB:

# Mount and enter system
sudo mount /dev/sdXn /mnt
sudo mount /dev/sdXm /mnt/boot
sudo nixos-enter --root /mnt

# Reset password
passwd username

# Exit and reboot
exit
reboot

Root Password

Same procedure as user password:

passwd root

---

Disk Issues

Root Partition Full

1. Boot into recovery

2. Clean up:

# Remove old generations
sudo nix-collect-garbage -d

# Optimize store
sudo nix-store --optimise

# Remove unused packages
sudo nix-store --gc

3. Check space:

df -h /

Corrupted Nix Store

# Verify store
sudo nix-store --verify --check-contents

# Repair
sudo nix-store --verify --check-contents --repair

---

Boot Loader Issues

GRUB Not Showing

# From live USB, mount and enter
sudo nixos-enter --root /mnt

# Reinstall GRUB
sudo nixos-rebuild switch

# Or manually
sudo grub-install /dev/sdX
sudo nixos-rebuild switch

systemd-boot Issues

# Reinstall bootloader
bootctl install

---

Network Recovery

No Network After Boot

# Check interface
ip link

# Bring up interface (replace wlan0/eth0 with your interface)
sudo ip link set wlan0 up

# Or use NetworkManager (recommended)
sudo systemctl restart NetworkManager
cmcli c

---

Database Recovery

Generation Database

NixOS keeps metadata about generations:

# Rebuild database
sudo nix-store --verify

# Check for corruption
sudo nix-store --verify --check-contents

---

Emergency Shell

Accessing Emergency Shell

1. Add to kernel parameters: systemd.unit=emergency.target

2. Or at GRUB: Press 'e', add to linux line

3. Boot - drops to root shell

4. Make fixes

5. Continue boot:

systemctl default

---

Reinstallation (Last Resort)

Preserving Data

1. Backup home directory
2. Save NixOS configuration
3. Note installed packages

Clean Reinstall

# From live USB
# Format partitions
sudo mkfs.ext4 /dev/sdXn
sudo mkfs.fat -F 32 /dev/sdXm

# Mount
sudo mount /dev/sdXn /mnt
sudo mkdir -p /mnt/boot
sudo mount /dev/sdXm /mnt/boot

# Restore config
# ... copy NixOS config to /mnt/etc/nixos

# Install
sudo nixos-install

# Restore home
# ... restore from backup

---

Prevention

Regular Backups

• Enable automated Restic backups
• Test restores monthly
• Keep configuration in Git

Configuration Management

# Before big changes
git add .
git commit -m "Working config before X change"

# Test changes
sudo nixos-rebuild test --flake .#hostname

# If successful, make permanent
sudo nixos-rebuild switch --flake .#hostname

Keep Old Generations

# Don't delete all generations
sudo nix-collect-garbage --delete-older-than 30d

---

Recovery Checklist

When system breaks:

• Can you boot into previous generation?
• Do you have a live USB ready?
• Is your NixOS config in Git?
• Do you have recent backups?
• Can you access emergency shell?
• Do you know your disk partition layout?

---

Emergency Preparation

Create Recovery USB

# Download NixOS ISO
wget https://channels.nixos.org/nixos-unstable/latest-nixos-minimal-x86_64-linux.iso

# Write to USB
sudo dd if=nixos.iso of=/dev/sdX bs=4M status=progress

Document Your System

Keep notes on:

• Partition layout
• Encryption setup
• Network configuration
• Hardware details

Save Configuration

# Backup config regularly
rsync -av /etc/nixos/ /mnt/backup/nixos-config/

# Or use Git
cd ~/NullOS
git add .
git commit -m "Current working config"
git push

---

Next Steps

• Backup - Automated backup setup
• Troubleshooting - Common issues
• Installation - Fresh install guide