fix gitignore, add example machine-specific

Sets the custom frontend URL to dynamically point to a subdomain based on the domain configuration. This ensures correct routing and accessibility of the application's frontend.

.gitignore

@@ -2,7 +2,6 @@ build/
 node_modules/
 dist/
 result
-machine-specific
 web-app
 lnbits
 lnbits-extensions
@@ -15,11 +14,6 @@ krops.nix
 # Copy example-build-local.nix to build-local.nix and customize
 build-local.nix
 
-# Machine-specific configurations (user creates these)
-# Keep example-machine as a template
-config/machines/*
-!config/machines/example-machine/
-
 # Secrets - only ignore unencrypted secrets
 # Encrypted .age files are SAFE to commit
 secrets/**/!(*.age)

CLAUDE.md

@@ -0,0 +1,224 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a NixOS multi-machine deployment system using **krops** with **Nix 25.05**. It manages multiple machines with different configurations, each potentially running services like LNBits (Lightning Network), pict-rs (image hosting), and custom web applications with machine-specific builds.
+
+## Key Architecture Concepts
+
+### Two-Stage Deployment Model
+
+This project uses a **two-stage deployment process**:
+
+1. **Local Build Stage** (`build-local.nix`): Builds web applications locally with machine-specific assets (`.env` files, images)
+2. **Remote Deploy Stage** (`krops.nix`): Deploys NixOS configurations and pre-built artifacts to target machines
+
+### Configuration Inheritance Pattern
+
+- **`config/shared.nix`**: Base configuration inherited by all machines (takes `domain` parameter)
+- **`config/machines/{machine-name}/configuration.nix`**: Machine-specific entry point that:
+  - Defines the `domain` variable
+  - Imports `shared.nix` with the domain
+  - Imports machine-specific modules (bootloader, hardware, services)
+
+### LNBits Flake Integration
+
+The project uses a sophisticated LNBits deployment via Nix flakes:
+
+- **Source deployed to**: `/var/src/lnbits-src/` via krops symlink
+- **Flake reference**: `builtins.getFlake "path:/var/src/lnbits-src"` (mutable local source)
+- **How it works**: Uses `uv2nix` to convert `uv.lock` into a reproducible Nix venv in `/nix/store`
+- **Runtime**: Systemd service runs `/nix/store/xxx-lnbits-env/bin/lnbits` with `LNBITS_PATH` pointing to source
+- **Lock file**: `flake.lock` is deployed with the source and automatically used by Nix
+
+See `docs/lnbits-flake-explanation.md` for detailed explanation.
+
+#### Ensuring flake.lock is Used
+
+Nix automatically uses `flake.lock` when present. To verify:
+
+```bash
+# Check flake.lock exists locally
+ls -lh lnbits/flake.lock
+
+# After deployment, verify on target machine
+ssh root@machine "ls -lh /var/src/lnbits-src/flake.lock"
+
+# View locked input versions
+nix flake metadata path:/var/src/lnbits-src
+```
+
+To update flake inputs (do this locally before deploying):
+```bash
+cd lnbits/
+nix flake update
+# Or update specific input:
+nix flake lock --update-input nixpkgs
+```
+
+**Important**: The krops `.file` source type copies all files including `flake.lock`. Since `lnbits/` is a symlink, krops follows it and deploys the entire directory tree.
+
+### Krops Source Deployment
+
+Files are deployed to target machines under `/var/src/`:
+
+- `/var/src/config-shared` → `config/shared.nix`
+- `/var/src/config-machine` → `config/machines/{machine-name}/`
+- `/var/src/web-app-dist` → `build/{machine-name}/dist/`
+- `/var/src/lnbits-src` → `lnbits/` (full source with flake)
+- `/var/src/lnbits-extensions` → `lnbits-extensions/`
+
+## Common Development Commands
+
+### Building Web Applications Locally
+
+```bash
+# Build for a specific machine
+nix-build ./build-local.nix -A machine1 && ./result/bin/build-machine1
+
+# Build for all machines
+nix-build ./build-local.nix -A all && ./result/bin/build-all
+```
+
+This copies web-app source to `./build/{machine}/`, adds machine-specific `.env` and images, then runs `npm run build`.
+
+### Deploying to Machines
+
+```bash
+# Deploy to a specific machine
+nix-build ./krops.nix -A machine1 && ./result
+
+# Deploy to all machines
+nix-build ./krops.nix -A all && ./result
+```
+
+### Complete Workflow (Build + Deploy)
+
+```bash
+# 1. Build web-apps locally
+nix-build ./build-local.nix -A all && ./result/bin/build-all
+
+# 2. Deploy to all machines
+nix-build ./krops.nix -A all && ./result
+```
+
+## Project Structure
+
+```
+.
+├── krops.nix                     # Main deployment config (gitignored)
+├── example-krops.nix             # Template for krops.nix
+├── build-local.nix               # Local web-app build scripts (gitignored)
+├── example-build-local.nix       # Template for build-local.nix
+├── config/
+│   ├── shared.nix                # Shared config (takes domain parameter)
+│   ├── nginx.nix                 # Nginx + ACME + fail2ban
+│   ├── lnbits.nix                # LNBits flake integration
+│   ├── pict-rs.nix               # Image hosting service
+│   └── machines/                 # Per-machine configs (gitignored)
+│       ├── example-machine/      # Template (committed)
+│       │   ├── configuration.nix # Sets domain, imports shared + modules
+│       │   ├── boot.nix          # Bootloader config
+│       │   └── example-service.nix # WireGuard and service examples
+│       ├── machine1/             # Your machines (gitignored)
+│       └── machine2/
+├── build/                        # Generated web-app builds (gitignored)
+├── machine-specific/             # Machine-specific web-app assets (symlink)
+│   └── {machine-name}/
+│       ├── env/.env              # Environment variables
+│       └── images/               # Logos and images
+├── web-app/                      # Shared web-app source (symlink)
+├── lnbits/                       # LNBits source with flake (symlink)
+└── lnbits-extensions/            # Custom LNBits extensions (symlink)
+```
+
+## Adding a New Machine
+
+1. **Create machine configuration**:
+   ```bash
+   cp -r config/machines/example-machine config/machines/new-machine
+   ```
+
+2. **Edit `config/machines/new-machine/configuration.nix`**:
+   - Set `domain = "yourdomain.com"`
+   - Add `hardware-configuration.nix` from `nixos-generate-config`
+   - Customize `boot.nix` for your bootloader
+
+3. **Create machine-specific web-app assets** (if needed):
+   ```bash
+   mkdir -p machine-specific/new-machine/{env,images}
+   # Add .env file and images
+   ```
+
+4. **Update `build-local.nix`**:
+   - Add `new-machine = buildForMachine "new-machine";` to outputs
+   - Add to `all` build script
+
+5. **Update `krops.nix`**:
+   - Add machine deployment block
+   - Add to `inherit` list and `all` script
+
+6. **Build and deploy**:
+   ```bash
+   nix-build ./build-local.nix -A new-machine && ./result/bin/build-new-machine
+   nix-build ./krops.nix -A new-machine && ./result
+   ```
+
+## Machine-Specific Services
+
+To add services that only run on certain machines (e.g., WireGuard on one machine):
+
+1. Create `config/machines/{machine}/custom-service.nix`
+2. Import it in `config/machines/{machine}/configuration.nix`
+3. Deploy only affects that machine
+
+See `config/machines/example-machine/example-service.nix` for WireGuard and other examples.
+
+## Service Configuration Details
+
+### Virtual Hosts Pattern
+
+All services use domain-based virtual hosts defined in `shared.nix` or service modules:
+
+- Web app: `app.${domain}`
+- LNBits: `lnbits.${domain}`
+- Pict-rs: `img.${domain}`
+
+All automatically get SSL via Let's Encrypt ACME.
+
+### LNBits Extensions
+
+Two deployment options in `config/lnbits.nix`:
+
+- **Option 1 (Symlink)**: Replace `/var/lib/lnbits/extensions` entirely (deletes UI-installed extensions)
+- **Option 2 (Merge)**: Copy deployed extensions alongside UI-installed ones using rsync
+
+Currently using Option 1 (symlink) - see commented code in `config/lnbits.nix:96-122`.
+
+### Nginx Configuration
+
+- `recommendedProxySettings = false` (disabled for WebSocket compatibility)
+- WebSocket support configured in LNBits vhost with upgrade headers
+- ACME email: `admin@aiolabs.dev` (set in `config/nginx.nix:16`)
+
+## Important Files for Understanding
+
+- **`DEPLOYMENT-GUIDE.md`**: Comprehensive deployment instructions
+- **`docs/lnbits-flake-explanation.md`**: How LNBits flake deployment works (uv2nix, path: references, etc.)
+- **`example-krops.nix`** and **`example-build-local.nix`**: Templates for configuration
+
+## Configuration Management
+
+- **Gitignored**: `krops.nix`, `build-local.nix`, `config/machines/*` (except example-machine), `build/`, `machine-specific/*`
+- **Committed**: Example configs, shared modules, service definitions
+- **Secrets**: Managed separately (see `secrets/` directory with age encryption)
+
+## Notes for AI Assistants
+
+- When adding machines, update BOTH `krops.nix` and `build-local.nix`
+- Always use the `domain` parameter pattern - it's passed to all modules
+- LNBits requires deploying the FULL source tree (including flake.nix, uv.lock) to `/var/src/lnbits-src`
+- Web-app builds happen locally to support machine-specific configurations
+- The example-machine configuration is the canonical template - keep it updated

config/lnbits.nix

@@ -25,6 +25,8 @@ in
       LNBITS_ADMIN_UI = "true";
       AUTH_ALLOWED_METHODS = "user-id-only, username-password";
       LNBITS_BACKEND_WALLET_CLASS = "FakeWallet";
+      LNBITS_BASEURL="https://lnbits.${domain}/";
+      FORWARDED_ALLOW_IPS = "*";
       LNBITS_SITE_TITLE = "AIO";
       LNBITS_SITE_TAGLINE = "Open Source Lightning Payments Platform";
       LNBITS_SITE_DESCRIPTION = "A lightning wallet for the community";
@@ -32,14 +34,19 @@ in
       LNBITS_DEFAULT_WALLET_NAME = "AIO Wallet";
       LNBITS_EXTENSIONS_MANIFESTS =
         "https://raw.githubusercontent.com/lnbits/lnbits-extensions/main/extensions.json";
-      LNBITS_EXTENSIONS_DEFAULT_INSTALL =
-        "nostrclient,nostrmarket,nostrrelay,lnurlp,events";
-      LNBITS_ADMIN_EXTENSIONS = "ngrok,nostrclient,nostrrelay";
-      LNBITS_USER_DEFAULT_EXTENSIONS = "lnurlp,nostrmarket,events";
-      FORWARDED_ALLOW_IPS = "*";
+      # LNBITS_EXTENSIONS_DEFAULT_INSTALL =
+      #   "nostrclient,nostrmarket,nostrrelay,lnurlp,events";
+      LNBITS_ADMIN_EXTENSIONS = "ngrok,nostrclient,nostrrelay,satmachineadmin";
+      LNBITS_USER_DEFAULT_EXTENSIONS = "lnurlp,nostrmarket,events,lndhub";
+      LNBITS_CUSTOM_FRONTEND_URL = "https://app.${domain}";
     };
   };
 
+  # Make openssh and sshpass available to lnbits service
+  systemd.services.lnbits = {
+    path = with pkgs; [ openssh sshpass ];
+  };
+
   services.nginx = {
     # Add the connection upgrade map
     appendHttpConfig = ''
@@ -54,7 +61,7 @@ in
       enableACME = true;
       locations = {
         # WebSocket endpoints with additional headers that LNbits might expect
-        "~ ^/(api/v1/ws/|.*relay.*/)" = {
+        "~ ^/(api/v1/ws/|.*relay.*/|.*/ws$)" = {
           proxyPass = "http://127.0.0.1:5000";
           extraConfig = ''
             # WebSocket configuration
@@ -98,17 +105,17 @@ in
   # This will DELETE any extensions installed via the LNBits UI.
   #
   # Option 1: Replace extensions directory entirely (use with caution)
-  systemd.tmpfiles.rules = [
-    # Set permissions on source directory so lnbits user can read it
-    "d /var/src/lnbits-extensions 0755 lnbits lnbits - -"
-    # Create symlink with proper ownership
-    "L+ /var/lib/lnbits/extensions - lnbits lnbits - /var/src/lnbits-extensions"
-  ];
+  # systemd.tmpfiles.rules = [
+  #   # Set permissions on source directory so lnbits user can read it
+  #   "d /var/src/lnbits-extensions 0755 lnbits lnbits - -"
+  #   # Create symlink with proper ownership
+  #   "L+ /var/lib/lnbits/extensions - lnbits lnbits - /var/src/lnbits-extensions"
+  # ];
   #
   # Option 2: Manually merge deployed extensions with existing ones
   # Copy deployed extensions into the extensions directory without replacing it:
   # systemd.tmpfiles.rules = [
-  #   "d /var/src/lnbits-extensions 0755 lnbits lnbits - -"
+  #   "d /var/src/lnbits-extensions 0755 root root - -"
   # ];
   # systemd.services.lnbits-copy-extensions = {
   #   description = "Copy deployed LNBits extensions";
@@ -116,7 +123,7 @@ in
   #   wantedBy = [ "lnbits.service" ];
   #   serviceConfig = {
   #     Type = "oneshot";
-  #     ExecStart = "${pkgs.rsync}/bin/rsync -av /var/src/lnbits-extensions/ /var/lib/lnbits/extensions/";
+  #     ExecStart = "${pkgs.bash}/bin/bash -c '${pkgs.rsync}/bin/rsync -av /var/src/lnbits-extensions/ /var/lib/lnbits/extensions/ && ${pkgs.coreutils}/bin/chown -R lnbits:lnbits /var/lib/lnbits/extensions/'";
   #   };
   # };
 }

config/machines/example-machine/borg-lnbits.nix

@@ -0,0 +1,35 @@
+{ config, pkgs, ... }:
+
+{
+  imports = [
+    ./modules/lnbits-backup.nix  # Add this line
+  ];
+
+  # Enable LNbits Borg backup
+  services.lnbits-backup = {
+    enable = true;
+
+    # Repository path - update with your actual path
+    # Option 1: Traditional format
+    # repository = "borg@192.168.1.100:/mnt/my_ssd/borg-backups/lnbits";
+
+    # Option 2: Full SSH URL format (note double slash for absolute path)
+    repository = "ssh://borg@<your-backup-ip>//mnt/borg-backups/lnbits";
+
+    # Backup schedule
+    schedule = "*:0/15";  # Backup schedule (hourly, daily, or systemd timer format)
+
+    # Compression with auto-detection
+    # "auto" skips compression for already-compressed files (images, videos, etc.)
+    compression = "auto,lz4";      # For Raspberry Pi (faster)
+    # compression = "auto,zstd";   # For RockPro64 (better compression)
+
+    # Retention policy, leave all of these
+    retention = {
+      hourly = 24;   # Last 24 hours
+      daily = 7;     # Last 7 days
+      weekly = 4;    # Last 4 weeks
+      monthly = 6;   # Last 6 months
+    };
+  };
+}

config/machines/example-machine/configuration.nix

@@ -16,6 +16,10 @@ in
     # Import boot configuration (bootloader settings)
     ./boot.nix
 
+    # BORG backup
+    # read docs/lnbits-borg-backup-guide.md
+    # ./borg-lnbits.nix
+
     # Import any machine-specific services
     # Comment out or remove if not needed
     # ./example-service.nix

config/machines/example-machine/modules/lnbits-backup.nix

@@ -0,0 +1,177 @@
+{ config, lib, pkgs, ... }:
+
+with lib;
+
+let
+  cfg = config.services.lnbits-backup;
+
+in {
+  options.services.lnbits-backup = {
+    enable = mkEnableOption "LNbits Borg Backup Service";
+
+    dataPath = mkOption {
+      type = types.path;
+      default = "/var/lib/lnbits/data";
+      description = "Path to LNbits data directory";
+    };
+
+    repository = mkOption {
+      type = types.str;
+      example = "ssh://borg@192.168.1.100//mnt/my_ssd/borg-backups/lnbits";
+      description = "Borg repository location (SSH URL or user@host:/path format)";
+    };
+
+    schedule = mkOption {
+      type = types.str;
+      default = "hourly";
+      description = "Backup schedule (hourly, daily, or systemd timer format)";
+    };
+
+    compression = mkOption {
+      type = types.str;
+      default = "auto,lz4";
+      description = "Compression: auto,lz4 (Pi), auto,zstd (RockPro64). 'auto' skips already-compressed files.";
+    };
+
+    passphraseFile = mkOption {
+      type = types.path;
+      default = "/root/secrets/borg-passphrase";
+      description = "Path to Borg passphrase file";
+    };
+
+    sshKeyFile = mkOption {
+      type = types.path;
+      default = "/root/.ssh/borg_backup_key";
+      description = "SSH private key for repository access";
+    };
+
+    retention = {
+      hourly = mkOption { type = types.int; default = 24; };
+      daily = mkOption { type = types.int; default = 7; };
+      weekly = mkOption { type = types.int; default = 4; };
+      monthly = mkOption { type = types.int; default = 6; };
+    };
+
+    alertEmail = mkOption {
+      type = types.nullOr types.str;
+      default = null;
+      description = "Email for alerts (optional)";
+    };
+  };
+
+  config = mkIf cfg.enable {
+    services.borgbackup.jobs.lnbits = {
+      paths = [
+        cfg.dataPath
+        "/tmp/lnbits-snapshot.sqlite3"
+      ];
+
+      exclude = [ "*.log" "*.pyc" "__pycache__" "*.tmp" ];
+      repo = cfg.repository;
+
+      encryption = {
+        mode = "repokey-blake2";
+        passCommand = "cat ${cfg.passphraseFile}";
+      };
+
+      compression = cfg.compression;
+      startAt = cfg.schedule;
+      persistentTimer = true;  # Run missed backups after reboots
+
+      prune.keep = {
+        inherit (cfg.retention) hourly daily weekly monthly;
+      };
+
+      preHook = ''
+        echo "=== LNbits Backup Starting: $(date) ==="
+
+        if [ -f "${cfg.dataPath}/database.sqlite3" ]; then
+          echo "Creating SQLite snapshot..."
+          ${pkgs.sqlite}/bin/sqlite3 "${cfg.dataPath}/database.sqlite3" \
+            ".backup '/tmp/lnbits-snapshot.sqlite3'"
+
+          SIZE=$(stat -c%s "/tmp/lnbits-snapshot.sqlite3")
+          echo "Snapshot created: $SIZE bytes"
+
+          if [ "$SIZE" -lt 1000 ]; then
+            echo "ERROR: Snapshot too small!"
+            rm -f /tmp/lnbits-snapshot.sqlite3
+            exit 1
+          fi
+        else
+          echo "ERROR: Database not found!"
+          exit 1
+        fi
+      '';
+
+      postHook = ''
+        rm -f /tmp/lnbits-snapshot.sqlite3
+
+        # Weekly integrity check (Sundays)
+        if [ $(date +%u) -eq 7 ]; then
+          echo "Running weekly integrity check..."
+          ${pkgs.borgbackup}/bin/borg check --repository-only ${cfg.repository}
+        fi
+
+        echo "=== Backup Complete: $(date) ==="
+      '';
+
+      environment = {
+        BORG_RSH = "ssh -i ${cfg.sshKeyFile} -o StrictHostKeyChecking=accept-new";
+      };
+    };
+
+    # Install required packages and helper commands
+    environment.systemPackages = with pkgs; [
+      borgbackup
+      sqlite
+
+      (writeShellScriptBin "lnbits-borg-list" ''
+        export BORG_PASSPHRASE=$(cat ${cfg.passphraseFile})
+        export BORG_RSH="ssh -i ${cfg.sshKeyFile}"
+        ${borgbackup}/bin/borg list ${cfg.repository} "$@"
+      '')
+
+      (writeShellScriptBin "lnbits-borg-info" ''
+        export BORG_PASSPHRASE=$(cat ${cfg.passphraseFile})
+        export BORG_RSH="ssh -i ${cfg.sshKeyFile}"
+        ${borgbackup}/bin/borg info ${cfg.repository} "$@"
+      '')
+
+      (writeShellScriptBin "lnbits-borg-restore" ''
+        if [ $# -lt 1 ]; then
+          echo "Usage: lnbits-borg-restore <archive-name> [destination]"
+          echo ""
+          echo "Available archives:"
+          lnbits-borg-list
+          exit 1
+        fi
+
+        ARCHIVE="$1"
+        DEST="''${2:-/tmp/lnbits-restore}"
+
+        export BORG_PASSPHRASE=$(cat ${cfg.passphraseFile})
+        export BORG_RSH="ssh -i ${cfg.sshKeyFile}"
+
+        mkdir -p "$DEST"
+        cd "$DEST"
+        echo "Restoring to: $DEST"
+        ${borgbackup}/bin/borg extract ${cfg.repository}::$ARCHIVE
+        echo "Done! Restored files in: $DEST"
+      '')
+
+      (writeShellScriptBin "lnbits-borg-mount" ''
+        MOUNT="''${1:-/mnt/borg-browse}"
+
+        export BORG_PASSPHRASE=$(cat ${cfg.passphraseFile})
+        export BORG_RSH="ssh -i ${cfg.sshKeyFile}"
+
+        mkdir -p "$MOUNT"
+        ${borgbackup}/bin/borg mount ${cfg.repository} "$MOUNT"
+        echo "Mounted at: $MOUNT"
+        echo "Unmount: borg umount $MOUNT"
+      '')
+    ];
+  };
+}
+

docs/lnbits-borg-backup-guide.md

@@ -0,0 +1,815 @@
+---
+title: "LNbits Borg Backup Solution - Setup Guide"
+author: "LNbits Documentation"
+date: "2025-10-25"
+mainfont: "DejaVu Sans"
+monofont: "DejaVu Sans Mono"
+fontsize: 10pt
+geometry: margin=0.75in
+toc: true
+toc-depth: 2
+numbersections: true
+colorlinks: true
+linkcolor: blue
+---
+
+# LNbits Borg Backup Solution - Setup Guide
+
+**Infrastructure:**
+- **LNbits Server:** NixOS with SQLite database
+- **Backup Device:** Raspberry Pi or RockPro64 (Raspberry Pi OS / Armbian / Debian)
+- **Data Location:** `/var/lib/lnbits/data`
+
+---
+
+## Table of Contents
+
+1. [Quick Overview](#quick-overview)
+2. [Part 1: Setup Backup Device](#part-1-setup-backup-device-pirockpro64)
+3. [Part 2: Configure NixOS Server](#part-2-configure-nixos-server)
+4. [Part 3: Testing & Verification](#part-3-testing--verification)
+5. [Part 4: Recovery Procedures](#part-4-recovery-procedures)
+6. [Part 5: Maintenance](#part-5-maintenance)
+7. [Troubleshooting](#troubleshooting)
+
+---
+
+## Quick Overview
+
+### What You'll Get
+
+- **Automated hourly backups** from LNbits server to Pi/RockPro64
+- **90%+ storage savings** through deduplication
+- **Encryption** with AES-256
+- **Point-in-time recovery** from any backup
+- **Email alerts** on failures
+- **Simple commands** for recovery
+
+### Architecture
+
+```
+LNbits Server (NixOS)          Backup Device (Pi/RockPro64)
+┌────────────────────┐         ┌──────────────────────────┐
+│ SQLite Database    │         │ USB Drive (2-4TB)        │
+│ /var/lib/lnbits/   │  SSH    │ /mnt/borg-backups/       │
+│                    │────────>│                          │
+│ Borg Client        │ Hourly  │ Borg Repository          │
+│ Automated Backups  │         │ Deduplicated & Encrypted │
+└────────────────────┘         └──────────────────────────┘
+```
+
+### Time Required
+
+- **Part 1** (Backup Device): 20-30 minutes
+- **Part 2** (NixOS Server): 15-20 minutes
+- **Part 3** (Testing): 10 minutes
+- **Total**: ~1 hour
+
+---
+
+## Part 1: Setup Backup Device (Pi/RockPro64)
+
+### Hardware Requirements
+
+**Raspberry Pi 4 or RockPro64:**
+- 2GB+ RAM
+- Gigabit Ethernet
+- USB 3.0 or SATA external drive (2-4TB recommended)
+- Reliable power supply
+
+**OS:** Raspberry Pi OS (64-bit), Armbian, or Debian-based Linux
+
+### Step 1.1: Install Operating System
+
+**For Raspberry Pi:**
+```bash
+# Use Raspberry Pi Imager
+# Install: Raspberry Pi OS Lite (64-bit)
+# Enable SSH during setup
+```
+
+**For RockPro64:**
+```bash
+# Download Armbian
+# Flash to eMMC or SD card
+# Enable SSH
+```
+
+### Step 1.2: Initial Setup
+
+**Connect and update:**
+
+```bash
+# SSH into your device
+ssh pi@192.168.1.100  # or your device IP
+
+# Update system
+sudo apt update
+sudo apt upgrade -y
+
+# Install required packages
+sudo apt install -y borgbackup openssh-server vim
+
+# Verify Borg installation
+borg --version
+# Should show: borg 1.2.x or newer
+```
+
+### Step 1.3: Setup External Drive
+
+**Connect your USB/SATA drive and format:**
+
+```bash
+# Identify the drive
+lsblk
+# Look for your drive, e.g., sda
+
+# CAUTION: This will ERASE the drive!
+# Replace /dev/sda with your actual drive
+sudo fdisk /dev/sda
+# Press: n (new partition), p (primary), Enter, Enter, Enter
+# Press: w (write)
+
+# Format as ext4
+sudo mkfs.ext4 -L borg-backups /dev/sda1
+
+# Create mount point
+sudo mkdir -p /mnt/borg-backups
+
+# Mount the drive
+sudo mount /dev/disk/by-label/borg-backups /mnt/borg-backups
+
+# Verify mount
+df -h | grep borg-backups
+```
+
+**Setup automatic mounting:**
+
+```bash
+# Get UUID of drive
+sudo blkid /dev/sda1
+
+# Add to /etc/fstab for auto-mount on boot
+sudo nano /etc/fstab
+
+# Add this line (replace UUID with yours):
+UUID=your-uuid-here /mnt/borg-backups ext4 defaults,nofail 0 2
+
+# Save (Ctrl+O, Enter, Ctrl+X)
+
+# Test fstab
+sudo umount /mnt/borg-backups
+sudo mount -a
+df -h | grep borg-backups  # Should show mounted
+```
+
+### Step 1.3b: Using an Existing Drive (Alternative)
+
+**If you have an existing SSD/drive with other files** (skip if you formatted a new drive above):
+
+```bash
+# Identify where your drive is mounted
+df -h
+# Example: /dev/sda1 mounted at /mnt/my_ssd
+
+# Create borg backup directory on existing drive
+sudo mkdir -p /mnt/my_ssd/borg-backups
+
+# IMPORTANT: Fix parent directory permissions
+# The borg user needs to traverse /mnt/my_ssd to reach /mnt/my_ssd/borg-backups
+
+# Check current permissions
+ls -ld /mnt/my_ssd
+
+# Option A: Add traverse permission for all users (safest, recommended)
+# This only allows listing/traversing, NOT reading other files
+sudo chmod o+x /mnt/my_ssd/
+
+# Option B: Add borg user to the group that owns the drive
+# First check who owns it:
+ls -ld /mnt/my_ssd
+# If owned by group 'users' or 'disk', add borg to that group:
+sudo usermod -aG users borg  # Replace 'users' with actual group
+
+# Create symlink for consistency with documentation paths (optional)
+sudo ln -s /mnt/my_ssd/borg-backups /mnt/borg-backups
+
+# Verify borg user can access it
+sudo -u borg ls /mnt/my_ssd/borg-backups
+# Should work without "Permission denied"
+```
+
+**Important Notes:**
+- Using `chmod o+x` on `/mnt/my_ssd` is safe - it only allows directory traversal
+- Your other files remain protected (other users can't read them)
+- The borg user will only have access to the `borg-backups` subdirectory
+
+### Step 1.4: Create Borg User
+
+```bash
+# Create dedicated user for backups
+sudo useradd -m -s /bin/bash borg
+
+# Give ownership of backup directory
+# If using new drive:
+sudo chown -R borg:borg /mnt/borg-backups
+# If using existing drive with subdirectory:
+sudo chown -R borg:borg /mnt/my_ssd/borg-backups
+
+# Setup SSH for borg user
+sudo mkdir -p /home/borg/.ssh
+sudo chmod 700 /home/borg/.ssh
+sudo touch /home/borg/.ssh/authorized_keys
+sudo chmod 600 /home/borg/.ssh/authorized_keys
+sudo chown -R borg:borg /home/borg/.ssh
+```
+
+### Step 1.5: Configure SSH (Optional Security Hardening)
+
+```bash
+# Edit SSH config
+sudo nano /etc/ssh/sshd_config
+
+# Recommended settings:
+# PermitRootLogin no
+# PasswordAuthentication no
+# PubkeyAuthentication yes
+
+# Restart SSH
+sudo systemctl restart sshd
+```
+
+### Step 1.6: Test and Finalize
+
+```bash
+# Verify borg user can access backup directory
+sudo su - borg
+ls -la /mnt/borg-backups
+exit
+
+# Check system resources
+free -h
+df -h /mnt/borg-backups
+```
+
+**✓ Part 1 Complete!** Your backup device is ready.
+
+---
+
+## Part 2: Configure NixOS Server
+
+### Step 2.1: Generate SSH Key for Backups
+
+**On LNbits server:**
+
+```bash
+# Generate dedicated SSH key for Borg backups
+sudo ssh-keygen -t ed25519 -f /root/.ssh/borg_backup_key -N "" -C "borg-backup@lnbits"
+
+# Display public key
+sudo cat /root/.ssh/borg_backup_key.pub
+# Copy this output
+```
+
+### Step 2.2: Add Public Key to Backup Device
+
+**On backup device (Pi/RockPro64):**
+
+```bash
+# Add public key to borg user's authorized_keys
+sudo su - borg
+nano ~/.ssh/authorized_keys
+
+# Paste the public key from previous step
+# Save and exit (Ctrl+O, Enter, Ctrl+X)
+
+exit
+```
+
+### Step 2.3: Test SSH Connection
+
+**On LNbits server:**
+
+```bash
+# Test connection (replace with your backup device IP)
+sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100
+
+# Should connect without password
+# Type 'exit' to disconnect
+```
+
+### Step 2.4: Initialize Borg Repository
+
+**On backup device:**
+
+```bash
+# Switch to borg user
+sudo su - borg
+
+# Initialize repository with encryption
+# If using NEW dedicated drive:
+borg init --encryption=repokey-blake2 /mnt/borg-backups/lnbits
+
+# If using EXISTING drive with subdirectory (from Step 1.3b):
+borg init --encryption=repokey-blake2 /mnt/my_ssd/borg-backups/lnbits
+
+# You'll be prompted for a passphrase
+# IMPORTANT: Use a strong passphrase and save it securely!
+# Example: "correct-horse-battery-staple-lightning-2025"
+
+# Export repository key for disaster recovery
+# Adjust path to match your repository location above:
+borg key export /mnt/my_ssd/borg-backups/lnbits ~/lnbits-borg-key.txt
+
+# Display the key
+cat ~/lnbits-borg-key.txt
+
+# IMPORTANT: Copy this key to a secure location!
+# You'll need it for disaster recovery
+
+# Delete the key file from backup device after saving elsewhere
+rm ~/lnbits-borg-key.txt
+```
+
+**Save these securely (password manager + paper backup):**
+1. Borg repository passphrase
+2. Repository key (from `borg-lnbits-key.txt`)
+3. SSH private key (from `/root/.ssh/borg_backup_key`)
+
+### Step 2.5: Create Passphrase File on NixOS Server
+
+**On LNbits server:**
+
+```bash
+# Create secrets directory
+sudo mkdir -p /root/secrets
+
+# Create passphrase file
+sudo nano /root/secrets/borg-passphrase
+# Type your Borg passphrase (the one you just created)
+# Save and exit
+
+# Secure the file
+sudo chmod 400 /root/secrets/borg-passphrase
+sudo chown root:root /root/secrets/borg-passphrase
+
+# Verify
+sudo cat /root/secrets/borg-passphrase
+```
+
+
+### Step 2.6: Enable Backup in Machine Configuration
+
+**Edit your main NixOS configuration:**
+
+```bash
+sudo nano configuration.nix
+```
+
+```nix
+  imports = [
+    # ...
+
+    # read docs/lnbits-borg-backup-guide.md
+    ./borg-lnbits.nix
+
+    # ...
+  ];
+```
+
+
+**Modify the Machine `borg-lnbits.nix` file:**
+
+**Save and exit**
+
+### Step 2.8: Apply Configuration
+
+```bash
+# Test configuration (dry run)
+nix-build ./krops.nix -A prod-atio && ./result
+```
+
+**On your lnbits server**
+
+```bash
+# Verify services started
+systemctl status borgbackup-job-lnbits.timer
+systemctl status borgbackup-job-lnbits.service
+```
+
+**✓ Part 2 Complete!** Your NixOS server is configured for automated backups.
+
+---
+
+## Part 3: Testing & Verification
+
+### Test 1: Manual Backup
+
+```bash
+# Trigger first backup manually
+sudo systemctl start borgbackup-job-lnbits.service
+
+# Watch logs in real-time
+journalctl -u borgbackup-job-lnbits.service -f
+
+# Press Ctrl+C when done
+```
+
+**Expected output:**
+```
+=== LNbits Backup Starting: Fri Oct 25 15:30:00 2025 ===
+Creating SQLite snapshot...
+Snapshot created: 5242880 bytes
+------------------------------------------------------------------------------
+Archive name: lnbits-2025-10-25_15:30:00
+Time (start): Fri, 2025-10-25 15:30:01
+Time (end):   Fri, 2025-10-25 15:30:12
+Duration: 11.23 seconds
+Number of files: 145
+                       Original size      Compressed size    Deduplicated size
+This archive:               78.45 MB             23.12 MB              23.12 MB
+All archives:               78.45 MB             23.12 MB              23.12 MB
+=== Backup Complete: Fri Oct 25 15:30:12 2025 ===
+```
+
+### Test 2: List Backups
+
+```bash
+# List all backups
+lnbits-borg-list
+
+# Expected output:
+# lnbits-2025-10-25_15:30:00    Fri, 2025-10-25 15:30:00 [...]
+```
+
+### Test 3: Repository Info
+
+```bash
+# Show repository statistics
+lnbits-borg-info
+
+# Check storage savings
+lnbits-borg-info | grep "All archives"
+```
+
+### Test 4: Verify Backup Contents
+
+```bash
+# List files in the backup
+LATEST=$(lnbits-borg-list --last 1 --short)
+lnbits-borg-list ::$LATEST | head -20
+```
+
+### Test 5: Test Recovery
+
+```bash
+# Restore to temporary location
+LATEST=$(lnbits-borg-list --last 1 --short)
+lnbits-borg-restore $LATEST /tmp/test-restore
+
+# Verify database file exists
+ls -lh /tmp/test-restore/tmp/lnbits-snapshot.sqlite3
+
+# Check database integrity
+sqlite3 /tmp/test-restore/tmp/lnbits-snapshot.sqlite3 "PRAGMA integrity_check;"
+# Should output: ok
+
+# Cleanup
+rm -rf /tmp/test-restore
+```
+
+### Test 6: Verify Automatic Schedule
+
+```bash
+# Check next scheduled backup
+systemctl list-timers | grep borg
+
+# Should show something like:
+# Fri 2025-10-25 16:00:00 CEST  5min left   n/a          n/a    borgbackup-job-lnbits.timer
+```
+
+**✓ Part 3 Complete!** All tests passed, backups are working!
+
+---
+
+## Part 4: Recovery Procedures
+
+### Scenario 1: Restore Database After Corruption
+
+```bash
+# 1. Stop LNbits
+sudo systemctl stop lnbits
+
+# 2. Backup corrupted database
+sudo mv /var/lib/lnbits/data/database.sqlite3 \
+        /var/lib/lnbits/data/database.sqlite3.bad
+
+# 3. List available backups
+lnbits-borg-list
+
+# 4. Choose a backup (e.g., from 2 hours ago)
+BACKUP="lnbits-2025-10-25_13:00:00"
+
+# 5. Restore
+sudo lnbits-borg-restore $BACKUP /tmp/recovery
+
+# 6. Copy database back
+sudo cp /tmp/recovery/tmp/lnbits-snapshot.sqlite3 \
+        /var/lib/lnbits/data/database.sqlite3
+
+# 7. Fix permissions
+sudo chown lnbits:lnbits /var/lib/lnbits/data/database.sqlite3
+
+# 8. Verify integrity
+sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;"
+
+# 9. Start LNbits
+sudo systemctl start lnbits
+
+# 10. Cleanup
+sudo rm -rf /tmp/recovery
+```
+
+### Scenario 2: Browse Backups
+
+```bash
+# Mount repository
+sudo lnbits-borg-mount /mnt/borg-browse
+
+# Browse all backups
+ls /mnt/borg-browse/
+
+# Enter specific backup
+cd /mnt/borg-browse/lnbits-2025-10-25_10:00:00/var/lib/lnbits/data/
+
+# Copy specific files
+sudo cp -a extensions/myextension /var/lib/lnbits/data/extensions/
+
+# Unmount when done
+sudo borg umount /mnt/borg-browse
+```
+
+### Scenario 3: Point-in-Time Recovery
+
+```bash
+# List backups from specific date
+lnbits-borg-list | grep "2025-10-24"
+
+# Choose backup before incident
+lnbits-borg-restore lnbits-2025-10-24_14:00:00 /tmp/recovery
+
+# Follow "Restore Database" steps above
+```
+
+---
+
+## Part 5: Maintenance
+
+### Daily Checks
+
+```bash
+# Check last backup status
+systemctl status borgbackup-job-lnbits.service
+
+# View recent logs
+journalctl -u borgbackup-job-lnbits.service --since today
+
+# Verify latest backup
+lnbits-borg-list --last 1
+```
+
+### Weekly Tasks
+
+```bash
+# Repository statistics
+lnbits-borg-info
+
+# Storage usage
+lnbits-borg-info | grep "All archives"
+
+# Backup device disk space
+ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups"
+```
+
+### Monthly Tasks
+
+```bash
+# Compact repository (reclaim space)
+export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
+export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"
+sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits
+
+# Full integrity check
+sudo borg check --verify-data borg@192.168.1.100:/mnt/borg-backups/lnbits
+```
+
+### Monitoring Script
+
+Create a simple status script:
+
+```bash
+sudo nano /usr/local/bin/borg-status
+```
+
+```bash
+#!/usr/bin/env bash
+
+export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
+export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"
+REPO="borg@192.168.1.100:/mnt/borg-backups/lnbits"
+
+echo "=== LNbits Borg Backup Status ==="
+echo ""
+echo "Last Backup:"
+borg list --last 1 $REPO
+echo ""
+echo "Repository Size:"
+borg info $REPO | grep "All archives"
+echo ""
+echo "Backup Device Storage:"
+ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups | tail -1"
+echo ""
+echo "Recent Backups:"
+borg list $REPO --last 5 --short
+```
+
+```bash
+sudo chmod +x /usr/local/bin/borg-status
+```
+
+Run anytime: `sudo borg-status`
+
+---
+
+## Troubleshooting
+
+### Issue: "Permission denied (publickey)"
+
+**Solution:**
+
+```bash
+# Test SSH connection
+sudo ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100
+
+# If fails, verify public key on backup device
+ssh pi@192.168.1.100  # Login with your regular user
+sudo cat /home/borg/.ssh/authorized_keys
+# Should contain the public key from /root/.ssh/borg_backup_key.pub
+
+# Fix permissions if needed
+sudo chmod 700 /home/borg/.ssh
+sudo chmod 600 /home/borg/.ssh/authorized_keys
+sudo chown -R borg:borg /home/borg/.ssh
+```
+
+### Issue: "Repository does not exist"
+
+**Solution:**
+
+```bash
+# Verify repository exists on backup device
+ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 \
+  "ls -la /mnt/borg-backups/lnbits"
+
+# Should show Borg repository files
+# If missing, reinitialize (ONLY if truly lost!)
+```
+
+### Issue: Backups are slow
+
+**For Raspberry Pi:**
+
+```nix
+# In /etc/nixos/configuration.nix
+services.lnbits-backup.compression = "lz4";  # Use fast compression
+```
+
+**Check network speed:**
+
+```bash
+# On LNbits server
+ping 192.168.1.100
+
+# Install iperf3 on both devices
+# On backup device:
+iperf3 -s
+
+# On LNbits server:
+iperf3 -c 192.168.1.100
+```
+
+### Issue: Out of disk space
+
+```bash
+# Check disk usage
+ssh -i /root/.ssh/borg_backup_key borg@192.168.1.100 "df -h /mnt/borg-backups"
+
+# Manually prune old backups
+export BORG_PASSPHRASE=$(cat /root/secrets/borg-passphrase)
+export BORG_RSH="ssh -i /root/.ssh/borg_backup_key"
+
+sudo borg prune \
+  --keep-hourly=12 \
+  --keep-daily=4 \
+  --keep-weekly=2 \
+  borg@192.168.1.100:/mnt/borg-backups/lnbits
+
+# Compact repository
+sudo borg compact borg@192.168.1.100:/mnt/borg-backups/lnbits
+```
+
+### Issue: Snapshot too small error
+
+```bash
+# Check database health
+sudo sqlite3 /var/lib/lnbits/data/database.sqlite3 "PRAGMA integrity_check;"
+
+# Check disk space
+df -h /var/lib/lnbits/data
+df -h /tmp
+
+# Verify LNbits is running
+sudo systemctl status lnbits
+```
+
+---
+
+## Quick Reference Commands
+
+```bash
+# List backups
+lnbits-borg-list
+
+# Show repository info
+lnbits-borg-info
+
+# Restore backup
+lnbits-borg-restore <archive-name> [destination]
+
+# Mount for browsing
+lnbits-borg-mount [mount-point]
+
+# Manual backup
+sudo systemctl start borgbackup-job-lnbits.service
+
+# View logs
+journalctl -u borgbackup-job-lnbits.service -f
+
+# Check timer
+systemctl status borgbackup-job-lnbits.timer
+```
+
+---
+
+## Storage Estimates
+
+**Typical LNbits installation:**
+
+- Initial backup: 500 MB - 2 GB
+- Daily growth (with dedup): 10-50 MB
+- 6 months of backups: 10-25 GB
+- **Recommended**: 2TB drive minimum
+
+**For larger deployments:**
+
+| Database Size | 6-Month Backup | Recommended Drive |
+|---------------|----------------|-------------------|
+| < 2 GB | 10-20 GB | 2 TB |
+| 2-10 GB | 20-50 GB | 2-4 TB |
+| > 10 GB | 50-200 GB | 4 TB |
+
+---
+
+## Security Checklist
+
+- [x] Strong Borg passphrase (20+ characters)
+- [x] Passphrase backed up (password manager + paper)
+- [x] Repository key exported and backed up
+- [x] SSH key protected (600 permissions)
+- [x] Backup device SSH hardened (no root, no passwords)
+- [x] Repository encrypted (repokey-blake2)
+- [x] Regular integrity checks (weekly automatic)
+
+---
+
+## Next Steps
+
+1. **Offsite Backup** (Recommended within 30 days)
+   - Setup cloud storage (S3, Backblaze B2)
+   - Or sync to remote VPS
+   - Use rclone or rsync from backup device
+
+2. **Monitoring**
+   - Setup email alerts
+   - Consider Prometheus/Grafana
+   - Create health check dashboard
+
+3. **Testing**
+   - Quarterly recovery drills
+   - Verify disaster recovery procedures
+   - Update documentation
+
+---
+
+**Setup Complete!** Your LNbits server now has enterprise-grade backup protection with Borg.
+

machine-specific/example-machine/env/.env

@@ -0,0 +1,27 @@
+# App Branding
+VITE_APP_NAME=YourApp
+
+VITE_PICTRS_BASE_URL=https://img.yourdomain.com
+
+# Lightning Address Domain (optional)
+# Override the domain used for Lightning Addresses
+# If not set, domain will be extracted from VITE_LNBITS_BASE_URL
+# Example: mydomain.com will show addresses as username@mydomain.com
+VITE_LIGHTNING_DOMAIN=
+
+VITE_NOSTR_RELAYS=["wss://lnbits.yourdomain.com/nostrrelay/test1"]
+
+VITE_LNBITS_BASE_URL=https://lnbits.yourdomain.com
+# Invoice/Read Key below
+VITE_API_KEY=
+
+VITE_ADMIN_PUBKEYS=["<your-pubkey"]
+
+# Push Notifications
+VITE_VAPID_PUBLIC_KEY=
+VITE_PUSH_NOTIFICATIONS_ENABLED=true
+
+# Market Configuration
+VITE_MARKET_NADDR=
+VITE_LIGHTNING_ENABLED=true
+VITE_MARKET_DEFAULT_CURRENCY=sat