krops-multi-deploy/docs/lnbits-borg-backup-guide.md

title	author	date	mainfont	monofont	fontsize	geometry	toc	toc-depth	numbersections	colorlinks	linkcolor
LNbits Borg Backup Solution - Setup Guide	LNbits Documentation	2025-10-25	DejaVu Sans	DejaVu Sans Mono	10pt	margin=0.75in	true	2	true	true	blue

LNbits Borg Backup Solution - Setup Guide

Infrastructure:

• LNbits Server: NixOS with SQLite database
• Backup Device: Raspberry Pi or RockPro64 (Raspberry Pi OS / Armbian / Debian)
• Data Location: /var/lib/lnbits/data

---

Table of Contents

1. Quick Overview
2. Part 1: Setup Backup Device
3. Part 2: Configure NixOS Server
4. Part 3: Testing & Verification
5. Part 4: Recovery Procedures
6. Part 5: Maintenance
7. Troubleshooting

---

Quick Overview

What You'll Get

• Automated hourly backups from LNbits server to Pi/RockPro64
• 90%+ storage savings through deduplication
• Encryption with AES-256
• Point-in-time recovery from any backup
• Email alerts on failures
• Simple commands for recovery

Architecture

LNbits Server (NixOS)          Backup Device (Pi/RockPro64)
┌────────────────────┐         ┌──────────────────────────┐
│ SQLite Database    │         │ USB Drive (2-4TB)        │
│ /var/lib/lnbits/   │  SSH    │ /mnt/borg-backups/       │
│                    │────────>│                          │
│ Borg Client        │ Hourly  │ Borg Repository          │
│ Automated Backups  │         │ Deduplicated & Encrypted │
└────────────────────┘         └──────────────────────────┘

Time Required

• Part 1 (Backup Device): 20-30 minutes
• Part 2 (NixOS Server): 15-20 minutes
• Part 3 (Testing): 10 minutes
• Total: ~1 hour

---

Part 1: Setup Backup Device (Pi/RockPro64)

Hardware Requirements

Raspberry Pi 4 or RockPro64:

• 2GB+ RAM
• Gigabit Ethernet
• USB 3.0 or SATA external drive (2-4TB recommended)
• Reliable power supply

OS: Raspberry Pi OS (64-bit), Armbian, or Debian-based Linux

Step 1.1: Install Operating System

For Raspberry Pi:

# Use Raspberry Pi Imager
# Install: Raspberry Pi OS Lite (64-bit)
# Enable SSH during setup

For RockPro64:

# Download Armbian
# Flash to eMMC or SD card
# Enable SSH

Step 1.2: Initial Setup

Connect and update:

# SSH into your device
ssh pi@192.168.1.100  # or your device IP

# Update system
sudo apt update
sudo apt upgrade -y

# Install required packages
sudo apt install -y borgbackup openssh-server vim

# Verify Borg installation
borg --version
# Should show: borg 1.2.x or newer

Step 1.3: Setup External Drive

Connect your USB/SATA drive and format:

# Identify the drive
lsblk
# Look for your drive, e.g., sda

# CAUTION: This will ERASE the drive!
# Replace /dev/sda with your actual drive
sudo fdisk /dev/sda
# Press: n (new partition), p (primary), Enter, Enter, Enter
# Press: w (write)

# Format as ext4
sudo mkfs.ext4 -L borg-backups /dev/sda1

# Create mount point
sudo mkdir -p /mnt/borg-backups

# Mount the drive
sudo mount /dev/disk/by-label/borg-backups /mnt/borg-backups

# Verify mount
df -h | grep borg-backups

Setup automatic mounting:

# Get UUID of drive
sudo blkid /dev/sda1

# Add to /etc/fstab for auto-mount on boot
sudo nano /etc/fstab

# Add this line (replace UUID with yours):
UUID=your-uuid-here /mnt/borg-backups ext4 defaults,nofail 0 2

# Save (Ctrl+O, Enter, Ctrl+X)

# Test fstab
sudo umount /mnt/borg-backups
sudo mount -a
df -h | grep borg-backups  # Should show mounted

Step 1.3b: Using an Existing Drive (Alternative)

If you have an existing SSD/drive with other files (skip if you formatted a new drive above):

# Identify where your drive is mounted
df -h
# Example: /dev/sda1 mounted at /mnt/my_ssd

# Create borg backup directory on existing drive
sudo mkdir -p /mnt/my_ssd/borg-backups

# IMPORTANT: Fix parent directory permissions
# The borg user needs to traverse /mnt/my_ssd to reach /mnt/my_ssd/borg-backups

# Check current permissions
ls -ld /mnt/my_ssd

# Option A: Add traverse permission for all users (safest, recommended)
# This only allows listing/traversing, NOT reading other files
sudo chmod o+x /mnt/my_ssd/

# Option B: Add borg user to the group that owns the drive
# First check who owns it:
ls -ld /mnt/my_ssd
# If owned by group 'users' or 'disk', add borg to that group:
sudo usermod -aG users borg  # Replace 'users' with actual group

# Create symlink for consistency with documentation paths (optional)
sudo ln -s /mnt/my_ssd/borg-backups /mnt/borg-backups

# Verify borg user can access it
sudo -u borg ls /mnt/my_ssd/borg-backups
# Should work without "Permission denied"

Important Notes:

• Using chmod o+x on /mnt/my_ssd is safe - it only allows directory traversal
• Your other files remain protected (other users can't read them)
• The borg user will only have access to the borg-backups subdirectory

Step 1.4: Create Borg User

# Create dedicated user for backups
sudo useradd -m -s /bin/bash borg

# Give ownership of backup directory
# If using new drive:
sudo chown -R borg:borg /mnt/borg-backups
# If using existing drive with subdirectory:
sudo chown -R borg:borg /mnt/my_ssd/borg-backups

# Setup SSH for borg user
sudo mkdir -p /home/borg/.ssh
sudo chmod 700 /home/borg/.ssh
sudo touch /home/borg/.ssh/authorized_keys
sudo chmod 600 /home/borg/.ssh/authorized_keys
sudo chown -R borg:borg /home/borg/.ssh

Step 1.5: Configure SSH (Optional Security Hardening)

# Edit SSH config
sudo nano /etc/ssh/sshd_config

# Recommended settings:
# PermitRootLogin no
# PasswordAuthentication no
# PubkeyAuthentication yes

# Restart SSH
sudo systemctl restart sshd

Step 1.6: Test and Finalize

# Verify borg user can access backup directory
sudo su - borg
ls -la /mnt/borg-backups
exit

# Check system resources
free -h
df -h /mnt/borg-backups

✓ Part 1 Complete! Your backup device is ready.

---

Part 2: Configure NixOS Server

Step 2.1: Generate SSH Key for Backups

On LNbits server:

# Generate dedicated SSH key for Borg backups
sudo ssh-keygen -t ed25519 -f /root/.ssh/borg_backup_key -N "" -C "borg-backup@lnbits"

# Display public key
sudo cat /root/.ssh/borg_backup_key.pub
# Copy this output

Step 2.2: Add Public Key to Backup Device

On backup device (Pi/RockPro64):

# Add public key to borg user's authorized_keys
sudo su - borg
nano ~/.ssh/authorized_keys

# Paste the public key from previous step
# Save and exit (Ctrl+O, Enter, Ctrl+X)

exit

Step 2.3: Test SSH Connection

On LNbits server:

# Test connection (replace with your backup device IP)
sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100

# Should connect without password
# Type 'exit' to disconnect

Step 2.4: Initialize Borg Repository

On backup device:

# Switch to borg user
sudo su - borg

# Initialize repository with encryption
# If using NEW dedicated drive:
borg init --encryption=repokey-blake2 /mnt/borg-backups/lnbits

# If using EXISTING drive with subdirectory (from Step 1.3b):
borg init --encryption=repokey-blake2 /mnt/my_ssd/borg-backups/lnbits

# You'll be prompted for a passphrase
# IMPORTANT: Use a strong passphrase and save it securely!
# Example: "correct-horse-battery-staple-lightning-2025"

# Export repository key for disaster recovery
# Adjust path to match your repository location above:
borg key export /mnt/my_ssd/borg-backups/lnbits ~/lnbits-borg-key.txt

# Display the key
cat ~/lnbits-borg-key.txt

# IMPORTANT: Copy this key to a secure location!
# You'll need it for disaster recovery

# Delete the key file from backup device after saving elsewhere
rm ~/lnbits-borg-key.txt

Save these securely (password manager + paper backup):

1. Borg repository passphrase
2. Repository key (from borg-lnbits-key.txt)
3. SSH private key (from /root/.ssh/borg_backup_key)

Step 2.5: Create Passphrase File on NixOS Server

On LNbits server:

# Create secrets directory
sudo mkdir -p /root/secrets

# Create passphrase file
sudo nano /root/secrets/borg-passphrase
# Type your Borg passphrase (the one you just created)
# Save and exit

# Secure the file
sudo chmod 400 /root/secrets/borg-passphrase
sudo chown root:root /root/secrets/borg-passphrase

# Verify
sudo cat /root/secrets/borg-passphrase

Step 2.6: Enable Backup in Machine Configuration

Edit your main NixOS configuration:

sudo nano configuration.nix

  imports = [
    # ...

    # read docs/lnbits-borg-backup-guide.md
    ./borg-lnbits.nix

    # ...
  ];

Modify the Machine borg-lnbits.nix file:

Save and exit

Step 2.8: Apply Configuration

# Test configuration (dry run)
nix-build ./krops.nix -A prod-atio && ./result

On your lnbits server

# Verify services started
systemctl status borgbackup-job-lnbits.timer
systemctl status borgbackup-job-lnbits.service

✓ Part 2 Complete! Your NixOS server is configured for automated backups.

---

Part 3: Testing & Verification

Test 1: Manual Backup

# Trigger first backup manually
sudo systemctl start borgbackup-job-lnbits.service

# Watch logs in real-time
journalctl -u borgbackup-job-lnbits.service -f

# Press Ctrl+C when done

Expected output:

=== LNbits Backup Starting: Fri Oct 25 15:30:00 2025 ===
Creating SQLite snapshot...
Snapshot created: 5242880 bytes
------------------------------------------------------------------------------
Archive name: lnbits-2025-10-25_15:30:00
Time (start): Fri, 2025-10-25 15:30:01
Time (end):   Fri, 2025-10-25 15:30:12
Duration: 11.23 seconds
Number of files: 145
                       Original size      Compressed size    Deduplicated size
This archive:               78.45 MB             23.12 MB              23.12 MB
All archives:               78.45 MB             23.12 MB              23.12 MB
=== Backup Complete: Fri Oct 25 15:30:12 2025 ===

Test 2: List Backups

# List all backups
lnbits-borg-list

# Expected output:
# lnbits-2025-10-25_15:30:00    Fri, 2025-10-25 15:30:00 [...]

Test 3: Repository Info

# Show repository statistics
lnbits-borg-info

# Check storage savings
lnbits-borg-info | grep "All archives"

Test 4: Verify Backup Contents

# List files in the backup
LATEST=$(lnbits-borg-list --last 1 --short)
lnbits-borg-list ::$LATEST | head -20

Test 5: Test Recovery

# Restore to temporary location
LATEST=$(lnbits-borg-list --last 1 --short)
lnbits-borg-restore $LATEST /tmp/test-restore

# Verify database file exists
ls -lh /tmp/test-restore/tmp/lnbits-snapshot.sqlite3

# Check database integrity
sqlite3 /tmp/test-restore/tmp/lnbits-snapshot.sqlite3 "PRAGMA integrity_check;"
# Should output: ok

# Cleanup
rm -rf /tmp/test-restore

Test 6: Verify Automatic Schedule

# Check next scheduled backup
systemctl list-timers | grep borg

# Should show something like:
# Fri 2025-10-25 16:00:00 CEST  5min left   n/a          n/a    borgbackup-job-lnbits.timer

✓ Part 3 Complete! All tests passed, backups are working!

---

Part 4: Recovery Procedures

Scenario 1: Restore Database After Corruption

# 1. Stop LNbits
sudo systemctl stop lnbits

# 2. Backup corrupted database
sudo mv /var/lib/lnbits/data/database.sqlite3 \
        /var/lib/lnbits/data/database.sqlite3.bad

# 3. List available backups
lnbits-borg-list

# 4. Choose a backup (e.g., from 2 hours ago)
BACKUP="lnbits-2025-10-25_13:00:00"

# 5. Restore
sudo lnbits-borg-restore $BACKUP /tmp/recovery

# 6. Copy database back
sudo cp /tmp/recovery/tmp/lnbits-snapshot.sqlite3 \
        /var/lib/lnbits/data/database.sqlite3

# 7. Fix permissions
sudo chown lnbits:lnbits /var/lib/lnbits/data/database.sqlite3

# 8. Verify integrity
sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;"

# 9. Start LNbits
sudo systemctl start lnbits

# 10. Cleanup
sudo rm -rf /tmp/recovery

Scenario 2: Browse Backups

# Mount repository
sudo lnbits-borg-mount /mnt/borg-browse

# Browse all backups
ls /mnt/borg-browse/

# Enter specific backup
cd /mnt/borg-browse/lnbits-2025-10-25_10:00:00/var/lib/lnbits/data/

# Copy specific files
sudo cp -a extensions/myextension /var/lib/lnbits/data/extensions/

# Unmount when done
sudo borg umount /mnt/borg-browse

Scenario 3: Point-in-Time Recovery

# List backups from specific date
lnbits-borg-list | grep "2025-10-24"

# Choose backup before incident
lnbits-borg-restore lnbits-2025-10-24_14:00:00 /tmp/recovery

# Follow "Restore Database" steps above

---

Part 5: Maintenance

Daily Checks

# Check last backup status
systemctl status borgbackup-job-lnbits.service

# View recent logs
journalctl -u borgbackup-job-lnbits.service --since today

# Verify latest backup
lnbits-borg-list --last 1

Weekly Tasks

# Repository statistics
lnbits-borg-info

# Storage usage
lnbits-borg-info | grep "All archives"

# Backup device disk space
ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups"

Monthly Tasks

# Compact repository (reclaim space)
export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"
sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits

# Full integrity check
sudo borg check --verify-data borg@192.168.1.100:/mnt/borg-backups/lnbits

Monitoring Script

Create a simple status script:

sudo nano /usr/local/bin/borg-status

#!/usr/bin/env bash

export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"
REPO="borg@192.168.1.100:/mnt/borg-backups/lnbits"

echo "=== LNbits Borg Backup Status ==="
echo ""
echo "Last Backup:"
borg list --last 1 $REPO
echo ""
echo "Repository Size:"
borg info $REPO | grep "All archives"
echo ""
echo "Backup Device Storage:"
ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups | tail -1"
echo ""
echo "Recent Backups:"
borg list $REPO --last 5 --short

sudo chmod +x /usr/local/bin/borg-status

Run anytime: sudo borg-status

---

Troubleshooting

Issue: "Permission denied (publickey)"

Solution:

# Test SSH connection
sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100

# If fails, verify public key on backup device
ssh pi@192.168.1.100  # Login with your regular user
sudo cat /home/borg/.ssh/authorized_keys
# Should contain the public key from /root/.ssh/borg_backup_key.pub

# Fix permissions if needed
sudo chmod 700 /home/borg/.ssh
sudo chmod 600 /home/borg/.ssh/authorized_keys
sudo chown -R borg:borg /home/borg/.ssh

Issue: "Repository does not exist"

Solution:

# Verify repository exists on backup device
ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 \
  "ls -la /mnt/borg-backups/lnbits"

# Should show Borg repository files
# If missing, reinitialize (ONLY if truly lost!)

Issue: Backups are slow

For Raspberry Pi:

# In /etc/nixos/configuration.nix
services.lnbits-backup.compression = "lz4";  # Use fast compression

Check network speed:

# On LNbits server
ping 192.168.1.100

# Install iperf3 on both devices
# On backup device:
iperf3 -s

# On LNbits server:
iperf3 -c 192.168.1.100

Issue: Out of disk space

# Check disk usage
ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups"

# Manually prune old backups
export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"

sudo borg prune \
  --keep-hourly=12 \
  --keep-daily=4 \
  --keep-weekly=2 \
  borg@192.168.1.100:/mnt/borg-backups/lnbits

# Compact repository
sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits

Issue: Snapshot too small error

# Check database health
sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;"

# Check disk space
df -h /var/lib/lnbits/data
df -h /tmp

# Verify LNbits is running
sudo systemctl status lnbits

---

Quick Reference Commands

# List backups
lnbits-borg-list

# Show repository info
lnbits-borg-info

# Restore backup
lnbits-borg-restore <archive-name> [destination]

# Mount for browsing
lnbits-borg-mount [mount-point]

# Manual backup
sudo systemctl start borgbackup-job-lnbits.service

# View logs
journalctl -u borgbackup-job-lnbits.service -f

# Check timer
systemctl status borgbackup-job-lnbits.timer

---

Storage Estimates

Typical LNbits installation:

• Initial backup: 500 MB - 2 GB
• Daily growth (with dedup): 10-50 MB
• 6 months of backups: 10-25 GB
• Recommended: 2TB drive minimum

For larger deployments:

Database Size	6-Month Backup	Recommended Drive
< 2 GB	10-20 GB	2 TB
2-10 GB	20-50 GB	2-4 TB
> 10 GB	50-200 GB	4 TB

---

Security Checklist

• Strong Borg passphrase (20+ characters)
• Passphrase backed up (password manager + paper)
• Repository key exported and backed up
• SSH key protected (600 permissions)
• Backup device SSH hardened (no root, no passwords)
• Repository encrypted (repokey-blake2)
• Regular integrity checks (weekly automatic)

---

Next Steps

1. Offsite Backup (Recommended within 30 days)

   • Setup cloud storage (S3, Backblaze B2)
   • Or sync to remote VPS
   • Use rclone or rsync from backup device

2. Monitoring

   • Setup email alerts
   • Consider Prometheus/Grafana
   • Create health check dashboard

3. Testing

   • Quarterly recovery drills
   • Verify disaster recovery procedures
   • Update documentation

---

Setup Complete! Your LNbits server now has enterprise-grade backup protection with Borg.