--- title: "LNbits Borg Backup Solution - Setup Guide" author: "LNbits Documentation" date: "2025-10-25" mainfont: "DejaVu Sans" monofont: "DejaVu Sans Mono" fontsize: 10pt geometry: margin=0.75in toc: true toc-depth: 2 numbersections: true colorlinks: true linkcolor: blue --- # LNbits Borg Backup Solution - Setup Guide **Infrastructure:** - **LNbits Server:** NixOS with SQLite database - **Backup Device:** Raspberry Pi or RockPro64 (Raspberry Pi OS / Armbian / Debian) - **Data Location:** `/var/lib/lnbits/data` --- ## Table of Contents 1. [Quick Overview](#quick-overview) 2. [Part 1: Setup Backup Device](#part-1-setup-backup-device-pirockpro64) 3. [Part 2: Configure NixOS Server](#part-2-configure-nixos-server) 4. [Part 3: Testing & Verification](#part-3-testing--verification) 5. [Part 4: Recovery Procedures](#part-4-recovery-procedures) 6. [Part 5: Maintenance](#part-5-maintenance) 7. [Troubleshooting](#troubleshooting) --- ## Quick Overview ### What You'll Get - **Automated hourly backups** from LNbits server to Pi/RockPro64 - **90%+ storage savings** through deduplication - **Encryption** with AES-256 - **Point-in-time recovery** from any backup - **Email alerts** on failures - **Simple commands** for recovery ### Architecture ``` LNbits Server (NixOS) Backup Device (Pi/RockPro64) ┌────────────────────┐ ┌──────────────────────────┐ │ SQLite Database │ │ USB Drive (2-4TB) │ │ /var/lib/lnbits/ │ SSH │ /mnt/borg-backups/ │ │ │────────>│ │ │ Borg Client │ Hourly │ Borg Repository │ │ Automated Backups │ │ Deduplicated & Encrypted │ └────────────────────┘ └──────────────────────────┘ ``` ### Time Required - **Part 1** (Backup Device): 20-30 minutes - **Part 2** (NixOS Server): 15-20 minutes - **Part 3** (Testing): 10 minutes - **Total**: ~1 hour --- ## Part 1: Setup Backup Device (Pi/RockPro64) ### Hardware Requirements **Raspberry Pi 4 or RockPro64:** - 2GB+ RAM - Gigabit Ethernet - USB 3.0 or SATA external drive (2-4TB recommended) - Reliable power supply **OS:** Raspberry Pi OS (64-bit), Armbian, or Debian-based Linux ### Step 1.1: Install Operating System **For Raspberry Pi:** ```bash # Use Raspberry Pi Imager # Install: Raspberry Pi OS Lite (64-bit) # Enable SSH during setup ``` **For RockPro64:** ```bash # Download Armbian # Flash to eMMC or SD card # Enable SSH ``` ### Step 1.2: Initial Setup **Connect and update:** ```bash # SSH into your device ssh pi@192.168.1.100 # or your device IP # Update system sudo apt update sudo apt upgrade -y # Install required packages sudo apt install -y borgbackup openssh-server vim # Verify Borg installation borg --version # Should show: borg 1.2.x or newer ``` ### Step 1.3: Setup External Drive **Connect your USB/SATA drive and format:** ```bash # Identify the drive lsblk # Look for your drive, e.g., sda # CAUTION: This will ERASE the drive! # Replace /dev/sda with your actual drive sudo fdisk /dev/sda # Press: n (new partition), p (primary), Enter, Enter, Enter # Press: w (write) # Format as ext4 sudo mkfs.ext4 -L borg-backups /dev/sda1 # Create mount point sudo mkdir -p /mnt/borg-backups # Mount the drive sudo mount /dev/disk/by-label/borg-backups /mnt/borg-backups # Verify mount df -h | grep borg-backups ``` **Setup automatic mounting:** ```bash # Get UUID of drive sudo blkid /dev/sda1 # Add to /etc/fstab for auto-mount on boot sudo nano /etc/fstab # Add this line (replace UUID with yours): UUID=your-uuid-here /mnt/borg-backups ext4 defaults,nofail 0 2 # Save (Ctrl+O, Enter, Ctrl+X) # Test fstab sudo umount /mnt/borg-backups sudo mount -a df -h | grep borg-backups # Should show mounted ``` ### Step 1.3b: Using an Existing Drive (Alternative) **If you have an existing SSD/drive with other files** (skip if you formatted a new drive above): ```bash # Identify where your drive is mounted df -h # Example: /dev/sda1 mounted at /mnt/my_ssd # Create borg backup directory on existing drive sudo mkdir -p /mnt/my_ssd/borg-backups # IMPORTANT: Fix parent directory permissions # The borg user needs to traverse /mnt/my_ssd to reach /mnt/my_ssd/borg-backups # Check current permissions ls -ld /mnt/my_ssd # Option A: Add traverse permission for all users (safest, recommended) # This only allows listing/traversing, NOT reading other files sudo chmod o+x /mnt/my_ssd/ # Option B: Add borg user to the group that owns the drive # First check who owns it: ls -ld /mnt/my_ssd # If owned by group 'users' or 'disk', add borg to that group: sudo usermod -aG users borg # Replace 'users' with actual group # Create symlink for consistency with documentation paths (optional) sudo ln -s /mnt/my_ssd/borg-backups /mnt/borg-backups # Verify borg user can access it sudo -u borg ls /mnt/my_ssd/borg-backups # Should work without "Permission denied" ``` **Important Notes:** - Using `chmod o+x` on `/mnt/my_ssd` is safe - it only allows directory traversal - Your other files remain protected (other users can't read them) - The borg user will only have access to the `borg-backups` subdirectory ### Step 1.4: Create Borg User ```bash # Create dedicated user for backups sudo useradd -m -s /bin/bash borg # Give ownership of backup directory # If using new drive: sudo chown -R borg:borg /mnt/borg-backups # If using existing drive with subdirectory: sudo chown -R borg:borg /mnt/my_ssd/borg-backups # Setup SSH for borg user sudo mkdir -p /home/borg/.ssh sudo chmod 700 /home/borg/.ssh sudo touch /home/borg/.ssh/authorized_keys sudo chmod 600 /home/borg/.ssh/authorized_keys sudo chown -R borg:borg /home/borg/.ssh ``` ### Step 1.5: Configure SSH (Optional Security Hardening) ```bash # Edit SSH config sudo nano /etc/ssh/sshd_config # Recommended settings: # PermitRootLogin no # PasswordAuthentication no # PubkeyAuthentication yes # Restart SSH sudo systemctl restart sshd ``` ### Step 1.6: Test and Finalize ```bash # Verify borg user can access backup directory sudo su - borg ls -la /mnt/borg-backups exit # Check system resources free -h df -h /mnt/borg-backups ``` **✓ Part 1 Complete!** Your backup device is ready. --- ## Part 2: Configure NixOS Server ### Step 2.1: Generate SSH Key for Backups **On LNbits server:** ```bash # Generate dedicated SSH key for Borg backups sudo ssh-keygen -t ed25519 -f /root/.ssh/borg_backup_key -N "" -C "borg-backup@lnbits" # Display public key sudo cat /root/.ssh/borg_backup_key.pub # Copy this output ``` ### Step 2.2: Add Public Key to Backup Device **On backup device (Pi/RockPro64):** ```bash # Add public key to borg user's authorized_keys sudo su - borg nano ~/.ssh/authorized_keys # Paste the public key from previous step # Save and exit (Ctrl+O, Enter, Ctrl+X) exit ``` ### Step 2.3: Test SSH Connection **On LNbits server:** ```bash # Test connection (replace with your backup device IP) sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 # Should connect without password # Type 'exit' to disconnect ``` ### Step 2.4: Initialize Borg Repository **On backup device:** ```bash # Switch to borg user sudo su - borg # Initialize repository with encryption # If using NEW dedicated drive: borg init --encryption=repokey-blake2 /mnt/borg-backups/lnbits # If using EXISTING drive with subdirectory (from Step 1.3b): borg init --encryption=repokey-blake2 /mnt/my_ssd/borg-backups/lnbits # You'll be prompted for a passphrase # IMPORTANT: Use a strong passphrase and save it securely! # Example: "correct-horse-battery-staple-lightning-2025" # Export repository key for disaster recovery # Adjust path to match your repository location above: borg key export /mnt/my_ssd/borg-backups/lnbits ~/lnbits-borg-key.txt # Display the key cat ~/lnbits-borg-key.txt # IMPORTANT: Copy this key to a secure location! # You'll need it for disaster recovery # Delete the key file from backup device after saving elsewhere rm ~/lnbits-borg-key.txt ``` **Save these securely (password manager + paper backup):** 1. Borg repository passphrase 2. Repository key (from `borg-lnbits-key.txt`) 3. SSH private key (from `/root/.ssh/borg_backup_key`) ### Step 2.5: Create Passphrase File on NixOS Server **On LNbits server:** ```bash # Create secrets directory sudo mkdir -p /root/secrets # Create passphrase file sudo nano /root/secrets/borg-passphrase # Type your Borg passphrase (the one you just created) # Save and exit # Secure the file sudo chmod 400 /root/secrets/borg-passphrase sudo chown root:root /root/secrets/borg-passphrase # Verify sudo cat /root/secrets/borg-passphrase ``` ### Step 2.6: Enable Backup in Machine Configuration **Edit your main NixOS configuration:** ```bash sudo nano configuration.nix ``` ```nix imports = [ # ... # read docs/lnbits-borg-backup-guide.md ./borg-lnbits.nix # ... ]; ``` **Modify the Machine `borg-lnbits.nix` file:** **Save and exit** ### Step 2.8: Apply Configuration ```bash # Test configuration (dry run) nix-build ./krops.nix -A prod-atio && ./result ``` **On your lnbits server** ```bash # Verify services started systemctl status borgbackup-job-lnbits.timer systemctl status borgbackup-job-lnbits.service ``` **✓ Part 2 Complete!** Your NixOS server is configured for automated backups. --- ## Part 3: Testing & Verification ### Test 1: Manual Backup ```bash # Trigger first backup manually sudo systemctl start borgbackup-job-lnbits.service # Watch logs in real-time journalctl -u borgbackup-job-lnbits.service -f # Press Ctrl+C when done ``` **Expected output:** ``` === LNbits Backup Starting: Fri Oct 25 15:30:00 2025 === Creating SQLite snapshot... Snapshot created: 5242880 bytes ------------------------------------------------------------------------------ Archive name: lnbits-2025-10-25_15:30:00 Time (start): Fri, 2025-10-25 15:30:01 Time (end): Fri, 2025-10-25 15:30:12 Duration: 11.23 seconds Number of files: 145 Original size Compressed size Deduplicated size This archive: 78.45 MB 23.12 MB 23.12 MB All archives: 78.45 MB 23.12 MB 23.12 MB === Backup Complete: Fri Oct 25 15:30:12 2025 === ``` ### Test 2: List Backups ```bash # List all backups lnbits-borg-list # Expected output: # lnbits-2025-10-25_15:30:00 Fri, 2025-10-25 15:30:00 [...] ``` ### Test 3: Repository Info ```bash # Show repository statistics lnbits-borg-info # Check storage savings lnbits-borg-info | grep "All archives" ``` ### Test 4: Verify Backup Contents ```bash # List files in the backup LATEST=$(lnbits-borg-list --last 1 --short) lnbits-borg-list ::$LATEST | head -20 ``` ### Test 5: Test Recovery ```bash # Restore to temporary location LATEST=$(lnbits-borg-list --last 1 --short) lnbits-borg-restore $LATEST /tmp/test-restore # Verify database file exists ls -lh /tmp/test-restore/tmp/lnbits-snapshot.sqlite3 # Check database integrity sqlite3 /tmp/test-restore/tmp/lnbits-snapshot.sqlite3 "PRAGMA integrity_check;" # Should output: ok # Cleanup rm -rf /tmp/test-restore ``` ### Test 6: Verify Automatic Schedule ```bash # Check next scheduled backup systemctl list-timers | grep borg # Should show something like: # Fri 2025-10-25 16:00:00 CEST 5min left n/a n/a borgbackup-job-lnbits.timer ``` **✓ Part 3 Complete!** All tests passed, backups are working! --- ## Part 4: Recovery Procedures ### Scenario 1: Restore Database After Corruption ```bash # 1. Stop LNbits sudo systemctl stop lnbits # 2. Backup corrupted database sudo mv /var/lib/lnbits/data/database.sqlite3 \ /var/lib/lnbits/data/database.sqlite3.bad # 3. List available backups lnbits-borg-list # 4. Choose a backup (e.g., from 2 hours ago) BACKUP="lnbits-2025-10-25_13:00:00" # 5. Restore sudo lnbits-borg-restore $BACKUP /tmp/recovery # 6. Copy database back sudo cp /tmp/recovery/tmp/lnbits-snapshot.sqlite3 \ /var/lib/lnbits/data/database.sqlite3 # 7. Fix permissions sudo chown lnbits:lnbits /var/lib/lnbits/data/database.sqlite3 # 8. Verify integrity sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;" # 9. Start LNbits sudo systemctl start lnbits # 10. Cleanup sudo rm -rf /tmp/recovery ``` ### Scenario 2: Browse Backups ```bash # Mount repository sudo lnbits-borg-mount /mnt/borg-browse # Browse all backups ls /mnt/borg-browse/ # Enter specific backup cd /mnt/borg-browse/lnbits-2025-10-25_10:00:00/var/lib/lnbits/data/ # Copy specific files sudo cp -a extensions/myextension /var/lib/lnbits/data/extensions/ # Unmount when done sudo borg umount /mnt/borg-browse ``` ### Scenario 3: Point-in-Time Recovery ```bash # List backups from specific date lnbits-borg-list | grep "2025-10-24" # Choose backup before incident lnbits-borg-restore lnbits-2025-10-24_14:00:00 /tmp/recovery # Follow "Restore Database" steps above ``` --- ## Part 5: Maintenance ### Daily Checks ```bash # Check last backup status systemctl status borgbackup-job-lnbits.service # View recent logs journalctl -u borgbackup-job-lnbits.service --since today # Verify latest backup lnbits-borg-list --last 1 ``` ### Weekly Tasks ```bash # Repository statistics lnbits-borg-info # Storage usage lnbits-borg-info | grep "All archives" # Backup device disk space ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups" ``` ### Monthly Tasks ```bash # Compact repository (reclaim space) export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase) export BORG_RSH="ssh -i /root/.ssh/borg_backup_key" sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits # Full integrity check sudo borg check --verify-data borg@192.168.1.100:/mnt/borg-backups/lnbits ``` ### Monitoring Script Create a simple status script: ```bash sudo nano /usr/local/bin/borg-status ``` ```bash #!/usr/bin/env bash export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase) export BORG_RSH="ssh -i /root/.ssh/borg_backup_key" REPO="borg@192.168.1.100:/mnt/borg-backups/lnbits" echo "=== LNbits Borg Backup Status ===" echo "" echo "Last Backup:" borg list --last 1 $REPO echo "" echo "Repository Size:" borg info $REPO | grep "All archives" echo "" echo "Backup Device Storage:" ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups | tail -1" echo "" echo "Recent Backups:" borg list $REPO --last 5 --short ``` ```bash sudo chmod +x /usr/local/bin/borg-status ``` Run anytime: `sudo borg-status` --- ## Troubleshooting ### Issue: "Permission denied (publickey)" **Solution:** ```bash # Test SSH connection sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 # If fails, verify public key on backup device ssh pi@192.168.1.100 # Login with your regular user sudo cat /home/borg/.ssh/authorized_keys # Should contain the public key from /root/.ssh/borg_backup_key.pub # Fix permissions if needed sudo chmod 700 /home/borg/.ssh sudo chmod 600 /home/borg/.ssh/authorized_keys sudo chown -R borg:borg /home/borg/.ssh ``` ### Issue: "Repository does not exist" **Solution:** ```bash # Verify repository exists on backup device ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 \ "ls -la /mnt/borg-backups/lnbits" # Should show Borg repository files # If missing, reinitialize (ONLY if truly lost!) ``` ### Issue: Backups are slow **For Raspberry Pi:** ```nix # In /etc/nixos/configuration.nix services.lnbits-backup.compression = "lz4"; # Use fast compression ``` **Check network speed:** ```bash # On LNbits server ping 192.168.1.100 # Install iperf3 on both devices # On backup device: iperf3 -s # On LNbits server: iperf3 -c 192.168.1.100 ``` ### Issue: Out of disk space ```bash # Check disk usage ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups" # Manually prune old backups export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase) export BORG_RSH="ssh -i /root/.ssh/borg_backup_key" sudo borg prune \ --keep-hourly=12 \ --keep-daily=4 \ --keep-weekly=2 \ borg@192.168.1.100:/mnt/borg-backups/lnbits # Compact repository sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits ``` ### Issue: Snapshot too small error ```bash # Check database health sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;" # Check disk space df -h /var/lib/lnbits/data df -h /tmp # Verify LNbits is running sudo systemctl status lnbits ``` --- ## Quick Reference Commands ```bash # List backups lnbits-borg-list # Show repository info lnbits-borg-info # Restore backup lnbits-borg-restore [destination] # Mount for browsing lnbits-borg-mount [mount-point] # Manual backup sudo systemctl start borgbackup-job-lnbits.service # View logs journalctl -u borgbackup-job-lnbits.service -f # Check timer systemctl status borgbackup-job-lnbits.timer ``` --- ## Storage Estimates **Typical LNbits installation:** - Initial backup: 500 MB - 2 GB - Daily growth (with dedup): 10-50 MB - 6 months of backups: 10-25 GB - **Recommended**: 2TB drive minimum **For larger deployments:** | Database Size | 6-Month Backup | Recommended Drive | |---------------|----------------|-------------------| | < 2 GB | 10-20 GB | 2 TB | | 2-10 GB | 20-50 GB | 2-4 TB | | > 10 GB | 50-200 GB | 4 TB | --- ## Security Checklist - [x] Strong Borg passphrase (20+ characters) - [x] Passphrase backed up (password manager + paper) - [x] Repository key exported and backed up - [x] SSH key protected (600 permissions) - [x] Backup device SSH hardened (no root, no passwords) - [x] Repository encrypted (repokey-blake2) - [x] Regular integrity checks (weekly automatic) --- ## Next Steps 1. **Offsite Backup** (Recommended within 30 days) - Setup cloud storage (S3, Backblaze B2) - Or sync to remote VPS - Use rclone or rsync from backup device 2. **Monitoring** - Setup email alerts - Consider Prometheus/Grafana - Create health check dashboard 3. **Testing** - Quarterly recovery drills - Verify disaster recovery procedures - Update documentation --- **Setup Complete!** Your LNbits server now has enterprise-grade backup protection with Borg.