diff --git a/misc-docs/Lamassu-Database-Analysis.md b/Lamassu-Database-Analysis.md similarity index 100% rename from misc-docs/Lamassu-Database-Analysis.md rename to Lamassu-Database-Analysis.md diff --git a/misc-docs/EMERGENCY_PROTOCOLS.md b/misc-docs/EMERGENCY_PROTOCOLS.md deleted file mode 100644 index e408774..0000000 --- a/misc-docs/EMERGENCY_PROTOCOLS.md +++ /dev/null @@ -1,1362 +0,0 @@ -# Satoshi Machine Admin - Emergency Protocols -## DCA System Failure Recovery Guide - -**Document Version**: 1.0 -**Last Updated**: 2025-10-19 -**Extension Version**: v0.0.1 -**Status**: Production - ---- - -## Table of Contents - -1. [Critical Failure Scenarios](#critical-failure-scenarios) -2. [Emergency Protocol Checklist](#emergency-protocol-checklist) -3. [Recovery Procedures](#recovery-procedures) -4. [Prevention Measures](#prevention-measures) -5. [Monitoring & Alerts](#monitoring--alerts) -6. [Contact Information](#contact-information) - ---- - -## Critical Failure Scenarios - -### 1. Duplicate Transaction Processing ⚠️ CRITICAL - -**Risk Level**: 🔴 **CRITICAL** -**Impact**: Same Lamassu transaction processed twice → double distribution to clients → financial loss - -#### Detection Methods - -1. **Dashboard Monitoring**: - - Sudden large balance deductions from client accounts - - Multiple distribution entries for same timestamp - - Commission wallet receiving duplicate amounts - -2. **Database Query**: -```sql --- Find duplicate transactions -SELECT transaction_id, COUNT(*) as count, - STRING_AGG(id::text, ', ') as record_ids -FROM satoshimachine.lamassu_transactions -GROUP BY transaction_id -HAVING COUNT(*) > 1; -``` - -3. **Automated Alert Triggers**: - - Same `txn_id` appears in multiple processing cycles - - Client balance drops more than expected based on deposit ratios - -#### Immediate Response - -1. ✅ **STOP POLLING IMMEDIATELY** - Disable automatic background task -2. ✅ Document all duplicate entries with screenshots -3. ✅ Identify affected clients and amounts -4. ✅ Calculate total over-distribution amount - -#### Recovery Steps - -```sql --- Step 1: Identify duplicate distributions -SELECT lt.transaction_id, lt.id, lt.created_at, lt.base_amount, - COUNT(dp.id) as distribution_count -FROM satoshimachine.lamassu_transactions lt -LEFT JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -GROUP BY lt.id -HAVING COUNT(dp.id) > (SELECT COUNT(*) FROM satoshimachine.dca_clients WHERE remaining_balance > 0); - --- Step 2: Calculate over-distributed amounts per client -SELECT client_id, - SUM(amount_sats) as total_received, - -- Manual calculation of expected amount needed here -FROM satoshimachine.dca_payments -WHERE lamassu_transaction_id IN (SELECT id FROM duplicates_table) -GROUP BY client_id; -``` - -**Manual Correction**: -1. Calculate correct distribution amounts -2. Create compensating negative adjustments (if supported) OR -3. Deduct from future distributions until balanced -4. Document all corrections in audit log -5. Notify affected clients if material amount - -#### Prevention Measures - -**Required Code Changes**: -```python -# Add to transaction_processor.py BEFORE processing -existing = await get_lamassu_transaction_by_txid(txn_id) -if existing: - logger.warning(f"⚠️ Transaction {txn_id} already processed, skipping") - return None -``` - -**Required Database Change**: -```sql -ALTER TABLE satoshimachine.lamassu_transactions -ADD CONSTRAINT unique_transaction_id UNIQUE (transaction_id); -``` - ---- - -### 2. SSH/Database Connection Loss - -**Risk Level**: 🟡 **MEDIUM** -**Impact**: Polling stops → transactions not processed → clients not receiving Bitcoin on time - -#### Detection Methods - -1. **Dashboard Indicators**: - - No new records in transaction history for > 24 hours - - "Test Connection" button fails in admin configuration - - Background task logs show SSH connection errors - -2. **Database Query**: -```sql --- Check last successful poll -SELECT MAX(created_at) as last_transaction, - EXTRACT(EPOCH FROM (NOW() - MAX(created_at)))/3600 as hours_since_last -FROM satoshimachine.lamassu_transactions; - --- If hours_since_last > 24, investigate immediately -``` - -3. **Log File Check**: -```bash -# Check LNBits logs for SSH errors -tail -n 100 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log | grep -i "ssh\|connection" -``` - -#### Immediate Response - -1. ✅ Verify network connectivity to Lamassu server -2. ✅ Test SSH credentials manually: -```bash -ssh postgres@ -p -i -``` -3. ✅ Check firewall rules and network changes -4. ✅ Verify Lamassu server is running and accessible - -#### Recovery Steps - -**Option A: Credential Issues** -1. Regenerate SSH keys if compromised -2. Update authorized_keys on Lamassu server -3. Update configuration in admin dashboard -4. Test connection before re-enabling polling - -**Option B: Network Issues** -1. Coordinate with network admin to restore connectivity -2. Verify IP whitelisting if applicable -3. Test connection stability before resuming - -**Option C: Lamassu Server Issues** -1. Contact Lamassu administrator -2. Verify PostgreSQL service is running -3. Check database is accessible - -**Post-Recovery**: -```python -# System automatically catches up using last_polled_at timestamp -# Run manual poll to process missed transactions -POST /api/v1/dca/manual-poll -``` - -#### Prevention Measures - -1. **SSH Key Authentication** (more reliable than password): -```bash -# Generate dedicated key for this service -ssh-keygen -t ed25519 -f ~/.ssh/satmachine_lamassu -C "satmachine-polling" -``` - -2. **Connection Monitoring**: Implement daily health check -3. **Retry Logic**: Add exponential backoff in polling code -4. **Alert System**: Email/SMS when polling fails for > 2 hours - ---- - -### 3. Payment Distribution Failures - -**Risk Level**: 🔴 **CRITICAL** -**Impact**: Commission deducted and client balances reduced, but transfers fail → money stuck in limbo - -#### Detection Methods - -1. **Dashboard Monitoring**: - - Client balances decrease but payment status shows "failed" - - Commission wallet balance doesn't increase as expected - - Error notifications in payment processing - -2. **Database Query**: -```sql --- Find stuck/failed payments (older than 1 hour, not completed) -SELECT dp.id, dp.client_id, dp.amount_sats, dp.status, dp.created_at, - c.username, dp.payment_hash -FROM satoshimachine.dca_payments dp -JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -WHERE dp.status != 'completed' - AND dp.created_at < NOW() - INTERVAL '1 hour' -ORDER BY dp.created_at DESC; -``` - -3. **Wallet Balance Check**: -```sql --- Compare expected vs actual commission wallet balance -SELECT - SUM(commission_amount) as total_commission_expected, - -- Manually check actual wallet balance in LNBits -FROM satoshimachine.lamassu_transactions; -``` - -#### Immediate Response - -1. ✅ **STOP POLLING** - Prevent more transactions from failing -2. ✅ Identify root cause: - - Insufficient balance in source wallet? - - Invalid wallet adminkey? - - LNBits API issues? - - Network connectivity to LNBits? - -3. ✅ Document all failed payments with IDs and amounts - -#### Recovery Steps - -**Step 1: Verify Wallet Configuration** -```bash -# Test that commission wallet is accessible -curl -X GET https:///api/v1/wallet \ - -H "X-Api-Key: " -``` - -**Step 2: Check Wallet Balance** -- Ensure commission wallet has sufficient balance -- Verify no wallet locks or restrictions - -**Step 3: Manual Retry Process** - -**Option A: Retry via API** (if retry endpoint exists) -```python -# Retry failed payment -POST /api/v1/dca/payments/{payment_id}/retry -``` - -**Option B: Manual Recreation** (if no retry available) -1. Query failed payment details -2. Mark original payment as "cancelled" -3. Create new payment entry with same parameters -4. Process through normal payment flow -5. Update client records - -**Step 4: Verify Reconciliation** -```sql --- After recovery, verify all clients balance correctly -SELECT - c.id, - c.username, - c.remaining_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_payments -FROM satoshimachine.dca_clients c -LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' -LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id -GROUP BY c.id; -``` - -#### Prevention Measures - -**Required Code Changes**: -```python -# Add pre-flight check in transaction_processor.py -async def verify_commission_wallet_accessible(): - """Verify commission wallet exists and is accessible before processing""" - try: - response = await wallet_api_call(commission_wallet_id) - if not response.ok: - raise Exception("Commission wallet not accessible") - return True - except Exception as e: - logger.error(f"Pre-flight check failed: {e}") - return False - -# Add transaction rollback on distribution failure -async def process_transaction_with_rollback(): - transaction_record = None - try: - # Deduct balances - # Create distributions - # Transfer funds - # Mark complete - except Exception as e: - # ROLLBACK: Restore client balances - # Mark transaction as failed - # Alert admin -``` - -**Monitoring**: -1. Alert if any payment remains in non-completed status > 15 minutes -2. Daily reconciliation of commission wallet balance -3. Automated testing of wallet accessibility - ---- - -### 4. Balance Discrepancies - -**Risk Level**: 🟡 **MEDIUM** -**Impact**: Client balances don't match deposit/payment records → accounting errors, audit failures - -#### Detection Methods - -**Full Reconciliation Query**: -```sql --- Complete balance reconciliation report -SELECT - c.id, - c.username, - c.remaining_balance as current_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_distributed, - COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as calculated_balance, - c.remaining_balance - (COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0)) as discrepancy -FROM satoshimachine.dca_clients c -LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' -LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id -GROUP BY c.id, c.username, c.remaining_balance -HAVING ABS(c.remaining_balance - (COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0))) > 1; -``` - -#### Immediate Response - -1. ✅ Run full reconciliation query above -2. ✅ Export complete audit trail to CSV -3. ✅ Document discrepancy amounts per client -4. ✅ Identify pattern (all clients affected? specific time period?) - -#### Recovery Steps - -**For Each Discrepancy**: - -1. **Trace Transaction History**: -```sql --- Get complete transaction history for client -SELECT 'DEPOSIT' as type, id, amount, status, created_at, confirmed_at -FROM satoshimachine.dca_deposits -WHERE client_id = -UNION ALL -SELECT 'PAYMENT' as type, id, amount_sats, status, created_at, NULL -FROM satoshimachine.dca_payments -WHERE client_id = -ORDER BY created_at; -``` - -2. **Manual Recalculation**: - - Sum all confirmed deposits - - Subtract all completed payments - - Compare to current balance - - Identify missing/extra transactions - -3. **Correction Methods**: - -**Option A: Adjustment Entry** (Recommended) -```sql --- Create compensating deposit for positive discrepancy -INSERT INTO satoshimachine.dca_deposits (client_id, amount, status, note) -VALUES (, , 'confirmed', 'Balance correction - reconciliation 2025-10-19'); - --- OR Create compensating payment for negative discrepancy -INSERT INTO satoshimachine.dca_payments (client_id, amount_sats, status, note) -VALUES (, , 'completed', 'Balance correction - reconciliation 2025-10-19'); -``` - -**Option B: Direct Balance Update** (Use with extreme caution) -```sql --- ONLY if audit trail is complete and discrepancy is unexplained -UPDATE satoshimachine.dca_clients -SET remaining_balance = , - updated_at = NOW() -WHERE id = ; - --- MUST document in separate audit log -``` - -#### Prevention Measures - -**Daily Automated Reconciliation**: -```python -# Add to tasks.py -async def daily_reconciliation_check(): - """Run daily at 00:00 UTC""" - discrepancies = await find_balance_discrepancies() - if discrepancies: - await send_alert_to_admin(discrepancies) - await log_reconciliation_report(discrepancies) -``` - -**Database Constraints**: -```sql --- Prevent negative balances -ALTER TABLE satoshimachine.dca_clients -ADD CONSTRAINT positive_balance CHECK (remaining_balance >= 0); - --- Prevent confirmed deposits with zero amount -ALTER TABLE satoshimachine.dca_deposits -ADD CONSTRAINT positive_deposit CHECK (amount > 0); -``` - -**Audit Enhancements**: -- Store before/after balance with each transaction -- Implement change log table for all balance modifications -- Automated snapshots of all balances daily - ---- - -### 5. Commission Calculation Errors - -**Risk Level**: 🟠 **HIGH** -**Impact**: Wrong commission rate applied → over/under collection → revenue loss or client overcharge - -#### Detection Methods - -**Verification Query**: -```sql --- Verify all commission calculations are mathematically correct -SELECT - id, - transaction_id, - crypto_atoms, - commission_percentage, - discount, - base_amount, - commission_amount, - -- Recalculate expected values - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as expected_base, - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) as expected_commission, - -- Calculate differences - base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as base_difference, - commission_amount - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) as commission_difference -FROM satoshimachine.lamassu_transactions -WHERE ABS(base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) > 1 - OR ABS(commission_amount - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))))) > 1; -``` - -**Manual Spot Check**: -``` -Example: 2000 GTQ → 266,800 sats (3% commission, 0% discount) - -Expected calculation: -- Effective commission = 0.03 * (100 - 0) / 100 = 0.03 -- Base amount = 266800 / (1 + 0.03) = 258,835 sats -- Commission = 266800 - 258835 = 7,965 sats - -Verify these match database values. -``` - -#### Immediate Response - -1. ✅ Run verification query to identify all affected transactions -2. ✅ Calculate total under/over collection amount -3. ✅ Determine if pattern (all transactions? specific time period? specific commission rate?) -4. ✅ **STOP POLLING** if active miscalculation detected - -#### Recovery Steps - -**Step 1: Quantify Impact** -```sql --- Total revenue impact -SELECT - COUNT(*) as affected_transactions, - SUM(commission_difference) as total_revenue_impact, - MIN(created_at) as first_occurrence, - MAX(created_at) as last_occurrence -FROM ( - -- Use verification query from above -) as calc_check -WHERE ABS(commission_difference) > 1; -``` - -**Step 2: Client Impact Assessment** -```sql --- Which clients were affected and by how much -SELECT - c.id, - c.username, - COUNT(lt.id) as affected_transactions, - SUM(lt.base_amount) as total_distributed, - SUM(expected_base) as should_have_distributed, - SUM(expected_base - lt.base_amount) as client_impact -FROM satoshimachine.lamassu_transactions lt -JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -WHERE -- filter for affected transactions -GROUP BY c.id; -``` - -**Step 3: Correction** - -**If Under-Collected Commission**: -- Revenue lost to business -- Client received correct amount -- No client-facing correction needed -- Document for accounting - -**If Over-Collected Commission**: -- Client under-distributed -- Create compensating payments to affected clients: -```sql --- Add to client balances -UPDATE satoshimachine.dca_clients c -SET remaining_balance = remaining_balance + adjustment.amount -FROM ( - -- Calculate adjustment per client - SELECT client_id, SUM(should_have_distributed - actual_distributed) as amount - FROM affected_transactions_analysis - GROUP BY client_id -) adjustment -WHERE c.id = adjustment.client_id; -``` - -**Step 4: Fix Code Bug** -- Identify root cause in `transaction_processor.py` -- Add unit test for failed scenario -- Deploy fix -- Verify with test transaction - -#### Prevention Measures - -**Unit Tests** (Add to test suite): -```python -def test_commission_calculation_scenarios(): - """Test edge cases for commission calculation""" - test_cases = [ - # (crypto_atoms, commission%, discount%, expected_base, expected_commission) - (266800, 0.03, 0.0, 258835, 7965), - (100000, 0.05, 10.0, 95694, 4306), # With discount - (1, 0.03, 0.0, 0, 1), # Minimum amount - # Add more edge cases - ] - for case in test_cases: - result = calculate_commission(*case[:3]) - assert result.base_amount == case[3] - assert result.commission_amount == case[4] -``` - -**Calculation Verification** (Add to processing): -```python -# After calculation, verify math is correct -calculated_total = base_amount + commission_amount -assert abs(calculated_total - crypto_atoms) <= 1, "Commission calculation error detected" -``` - -**Audit Trail**: -- Store all calculation parameters with each transaction -- Log formula used and intermediate values -- Enable post-processing verification - ---- - -### 6. Wallet Key Rotation/Invalidation - -**Risk Level**: 🟠 **HIGH** -**Impact**: Commission wallet adminkey changes → can't receive commission payments → processing halts - -#### Detection Methods - -1. **API Error Responses**: - - Payment API returns 401/403 authentication errors - - "Invalid API key" messages in logs - -2. **Wallet Balance Check**: -```sql --- Commission not accumulating despite transactions -SELECT - SUM(commission_amount) as expected_total_commission, - -- Compare to actual wallet balance in LNBits dashboard -FROM satoshimachine.lamassu_transactions -WHERE created_at > ''; -``` - -3. **Manual Test**: -```bash -# Test commission wallet adminkey -curl -X GET https:///api/v1/wallet \ - -H "X-Api-Key: " - -# Should return wallet details, not auth error -``` - -#### Immediate Response - -1. ✅ **STOP POLLING** - No point processing if can't distribute -2. ✅ Identify if key was intentionally rotated or compromised -3. ✅ Obtain new valid adminkey for commission wallet -4. ✅ Verify source wallet adminkey is also still valid - -#### Recovery Steps - -**Step 1: Update Configuration** -1. Access admin dashboard -2. Navigate to Configuration section -3. Update commission wallet adminkey -4. Update source wallet adminkey if also affected -5. **Test Connection** before saving - -**Step 2: Verify Configuration Persisted** -```sql --- Check configuration was saved correctly -SELECT commission_wallet_id, - LEFT(commission_wallet_adminkey, 10) || '...' as key_preview, - updated_at -FROM satoshimachine.lamassu_config -ORDER BY updated_at DESC -LIMIT 1; -``` - -**Step 3: Reprocess Failed Transactions** -- Identify transactions that failed due to auth errors -- Mark for retry or manually reprocess -- Verify commission payments complete successfully - -**Step 4: Resume Operations** -1. Test with manual poll first -2. Verify single transaction processes completely -3. Re-enable automatic polling -4. Monitor for 24 hours - -#### Prevention Measures - -**Key Management Documentation**: -- Document which LNBits wallets are used for commission/source -- Store backup admin keys in secure location (password manager) -- Define key rotation procedure with testing steps -- Require testing in staging before production changes - -**Configuration Validation**: -```python -# Add to config save endpoint in views_api.py -async def validate_wallet_keys(commission_key, source_key): - """Test wallet keys before saving configuration""" - # Test commission wallet - commission_valid = await test_wallet_access(commission_key) - if not commission_valid: - raise ValueError("Commission wallet key is invalid") - - # Test source wallet (if applicable) - if source_key: - source_valid = await test_wallet_access(source_key) - if not source_valid: - raise ValueError("Source wallet key is invalid") - - return True -``` - -**Automated Monitoring**: -- Daily health check of wallet accessibility -- Alert if wallet API calls start failing -- Backup key verification in secure environment - ---- - -## Emergency Protocol Checklist - -Use this checklist when ANY error is detected in the DCA system. - -### Phase 1: Immediate Actions (First 5 Minutes) - -- [ ] **Stop Automatic Processing** - - Disable background polling task - - Verify no transactions are currently processing - - Document time polling was stopped - -- [ ] **Assess Severity** - - Is money at risk? (duplicate processing, failed payments) - - Are clients affected? (missing distributions, balance errors) - - Is this blocking operations? (connection loss, wallet issues) - -- [ ] **Initial Documentation** - - Take screenshots of error messages - - Note exact timestamp of error detection - - Record current system state (balances, last transaction, etc.) - -- [ ] **Notify Stakeholders** - - Alert system administrator - - Notify clients if distributions will be delayed > 24 hours - - Escalate if financial impact > threshold - -### Phase 2: Investigation (15-30 Minutes) - -- [ ] **Collect Diagnostic Information** - - Run relevant SQL queries from scenarios above - - Check LNBits logs: `/lnbits/data/logs/` - - Review recent configuration changes - - Test external connections (SSH, wallets) - -- [ ] **Identify Root Cause** - - Match symptoms to failure scenarios above - - Determine if human error, system failure, or external issue - - Estimate scope of impact (time range, # clients, # transactions) - -- [ ] **Document Findings** - - Record root cause analysis - - List all affected records (transaction IDs, client IDs) - - Calculate financial impact (over/under distributed amounts) - - Take database snapshots for audit trail - -### Phase 3: Recovery (30 Minutes - 2 Hours) - -- [ ] **Fix Root Cause** - - Apply code fix if bug - - Update configuration if settings issue - - Restore connection if network issue - - Refer to specific recovery steps in scenarios above - -- [ ] **Data Correction** - - Run reconciliation queries - - Execute correction SQL statements - - Verify all client balances are accurate - - Ensure audit trail is complete - -- [ ] **Verification** - - Test fix with single transaction - - Verify wallets are accessible - - Confirm connections are stable - - Run full reconciliation report - -### Phase 4: Resumption (After Verification) - -- [ ] **Gradual Restart** - - Process one manual poll successfully - - Monitor for errors during processing - - Verify distributions complete end-to-end - - Check commission payments arrive correctly - -- [ ] **Re-enable Automation** - - Turn on background polling task - - Set monitoring alerts - - Document in system log - -- [ ] **Enhanced Monitoring** - - Watch closely for 24 hours - - Run reconciliation after next 3-5 transactions - - Verify no recurrence of issue - -### Phase 5: Post-Incident (24-48 Hours After) - -- [ ] **Complete Post-Mortem** - - Document full timeline of incident - - Record exact root cause and fix applied - - Calculate total impact (financial, time, clients affected) - - Identify what went well and what could improve - -- [ ] **Implement Safeguards** - - Add prevention measures from scenario sections - - Implement new monitoring/alerts - - Add automated tests for this failure mode - - Update runbooks and documentation - -- [ ] **Stakeholder Communication** - - Send incident report to management - - Notify affected clients if applicable - - Document lessons learned - - Update emergency contact procedures if needed - ---- - -## Recovery Procedures - -### Full System Reconciliation - -Run this complete reconciliation procedure weekly or after any incident: - -```sql --- ============================================ --- FULL SYSTEM RECONCILIATION REPORT --- ============================================ - --- 1. Client Balance Reconciliation -WITH client_financials AS ( - SELECT - c.id, - c.username, - c.remaining_balance as current_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_payments, - COUNT(DISTINCT d.id) as deposit_count, - COUNT(DISTINCT p.id) as payment_count - FROM satoshimachine.dca_clients c - LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' - LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id - GROUP BY c.id -) -SELECT - id, - username, - current_balance, - total_deposits, - total_payments, - (total_deposits - total_payments) as calculated_balance, - (current_balance - (total_deposits - total_payments)) as discrepancy, - deposit_count, - payment_count, - CASE - WHEN ABS(current_balance - (total_deposits - total_payments)) <= 1 THEN '✅ OK' - ELSE '⚠️ MISMATCH' - END as status -FROM client_financials -ORDER BY ABS(current_balance - (total_deposits - total_payments)) DESC; - --- 2. Transaction Processing Verification -SELECT - COUNT(*) as total_transactions, - SUM(crypto_atoms) as total_sats_processed, - SUM(base_amount) as total_distributed, - SUM(commission_amount) as total_commission, - MIN(created_at) as first_transaction, - MAX(created_at) as last_transaction -FROM satoshimachine.lamassu_transactions; - --- 3. Failed/Pending Payments Check -SELECT - status, - COUNT(*) as count, - SUM(amount_sats) as total_amount, - MIN(created_at) as oldest, - MAX(created_at) as newest -FROM satoshimachine.dca_payments -GROUP BY status -ORDER BY - CASE status - WHEN 'failed' THEN 1 - WHEN 'pending' THEN 2 - WHEN 'completed' THEN 3 - END; - --- 4. Unconfirmed Deposits Check (sitting too long) -SELECT - id, - client_id, - amount, - status, - created_at, - EXTRACT(EPOCH FROM (NOW() - created_at))/3600 as hours_pending -FROM satoshimachine.dca_deposits -WHERE status = 'pending' - AND created_at < NOW() - INTERVAL '48 hours' -ORDER BY created_at; - --- 5. Commission Calculation Verification (sample check) -SELECT - id, - transaction_id, - crypto_atoms, - commission_percentage, - discount, - base_amount, - commission_amount, - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as expected_base, - base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as difference -FROM satoshimachine.lamassu_transactions -ORDER BY created_at DESC -LIMIT 20; -``` - -### Emergency Access Procedures - -**If Admin Dashboard Inaccessible**: - -1. **Direct Database Access**: -```bash -# Connect to LNBits database -sqlite3 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/database.sqlite - -# Or PostgreSQL if used -psql -h localhost -U lnbits -d lnbits -``` - -2. **Direct Configuration Update**: -```sql --- Update Lamassu config directly -UPDATE satoshimachine.lamassu_config -SET polling_enabled = false -WHERE id = (SELECT MAX(id) FROM satoshimachine.lamassu_config); -``` - -3. **Manual Client Balance Update**: -```sql --- ONLY in emergency when dashboard unavailable -UPDATE satoshimachine.dca_clients -SET remaining_balance = -WHERE id = ; --- MUST document this action in incident log -``` - -**If Background Task Won't Stop**: - -```bash -# Find LNBits process -ps aux | grep lnbits - -# Restart LNBits service (will stop all background tasks) -systemctl restart lnbits -# OR if running manually: -pkill -f lnbits -uv run lnbits -``` - -### Data Export for Audit - -**Complete Audit Trail Export**: - -```bash -# Export all DCA-related tables to CSV -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.lamassu_transactions;" \ - > lamassu_transactions_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_payments;" \ - > dca_payments_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_deposits;" \ - > dca_deposits_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_clients;" \ - > dca_clients_export_$(date +%Y%m%d_%H%M%S).csv -``` - -**Combined Audit Report**: - -```sql --- Complete transaction-to-distribution audit trail -SELECT - lt.id as transaction_id, - lt.transaction_id as lamassu_txn_id, - lt.created_at as transaction_time, - lt.crypto_atoms as total_sats, - lt.fiat_code, - lt.fiat_amount, - lt.commission_percentage, - lt.discount, - lt.base_amount as distributable_sats, - lt.commission_amount, - c.id as client_id, - c.username as client_name, - dp.amount_sats as client_received, - dp.status as payment_status, - dp.payment_hash -FROM satoshimachine.lamassu_transactions lt -LEFT JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -LEFT JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -ORDER BY lt.created_at DESC, c.username; -``` - ---- - -## Prevention Measures - -### Required Immediate Implementations - -#### 1. Idempotency Protection (CRITICAL) - -**File**: `transaction_processor.py` - -```python -async def process_lamassu_transaction(txn_data: dict) -> Optional[LamassuTransaction]: - """Process transaction with idempotency check""" - - # CRITICAL: Check if already processed - existing = await get_lamassu_transaction_by_txid(txn_data['id']) - if existing: - logger.warning(f"⚠️ Transaction {txn_data['id']} already processed at {existing.created_at}, skipping") - return None - - # Continue with processing... -``` - -#### 2. Database Constraints - -**File**: `migrations.py` - -```sql --- Add unique constraint on transaction_id -ALTER TABLE satoshimachine.lamassu_transactions -ADD CONSTRAINT unique_transaction_id UNIQUE (transaction_id); - --- Prevent negative balances -ALTER TABLE satoshimachine.dca_clients -ADD CONSTRAINT positive_balance CHECK (remaining_balance >= 0); - --- Ensure positive amounts -ALTER TABLE satoshimachine.dca_deposits -ADD CONSTRAINT positive_deposit CHECK (amount > 0); - -ALTER TABLE satoshimachine.dca_payments -ADD CONSTRAINT positive_payment CHECK (amount_sats > 0); -``` - -#### 3. Transaction Status Tracking - -**File**: `models.py` - -```python -class LamassuTransaction(BaseModel): - # ... existing fields ... - status: str = "pending" # pending, processing, completed, failed - error_message: Optional[str] = None - processed_at: Optional[datetime] = None -``` - -#### 4. Pre-flight Wallet Validation - -**File**: `transaction_processor.py` - -```python -async def validate_system_ready() -> Tuple[bool, str]: - """Validate system is ready to process transactions""" - - # Check commission wallet accessible - try: - commission_wallet = await get_wallet(config.commission_wallet_id) - if not commission_wallet: - return False, "Commission wallet not accessible" - except Exception as e: - return False, f"Commission wallet error: {str(e)}" - - # Check for stuck payments - stuck_payments = await get_stuck_payments(hours=2) - if stuck_payments: - return False, f"{len(stuck_payments)} payments stuck for >2 hours" - - # Check database connectivity - try: - await db_health_check() - except Exception as e: - return False, f"Database health check failed: {str(e)}" - - return True, "System ready" - -# Call before processing -ready, message = await validate_system_ready() -if not ready: - logger.error(f"System not ready: {message}") - await send_alert_to_admin(message) - return -``` - -### Automated Monitoring Implementation - -#### Daily Reconciliation Task - -**File**: `tasks.py` - -```python -@scheduler.scheduled_job("cron", hour=0, minute=0) # Daily at midnight UTC -async def daily_reconciliation(): - """Run daily balance reconciliation and report discrepancies""" - - logger.info("Starting daily reconciliation...") - - discrepancies = await find_balance_discrepancies() - - if discrepancies: - report = generate_reconciliation_report(discrepancies) - await send_alert_to_admin("⚠️ Balance Discrepancies Detected", report) - await log_reconciliation_issue(discrepancies) - else: - logger.info("✅ Daily reconciliation passed - all balances match") - - # Check for stuck payments - stuck_payments = await get_stuck_payments(hours=24) - if stuck_payments: - await send_alert_to_admin( - f"⚠️ {len(stuck_payments)} Stuck Payments Detected", - format_stuck_payments_report(stuck_payments) - ) -``` - -#### Connection Health Monitor - -```python -@scheduler.scheduled_job("interval", hours=2) # Every 2 hours -async def check_system_health(): - """Monitor system health and alert on issues""" - - issues = [] - - # Check last successful poll - last_poll = await get_last_successful_poll_time() - if last_poll and (datetime.utcnow() - last_poll).total_seconds() > 86400: # 24 hours - issues.append(f"No successful poll in {(datetime.utcnow() - last_poll).total_seconds() / 3600:.1f} hours") - - # Check wallet accessibility - try: - await test_wallet_access(config.commission_wallet_adminkey) - except Exception as e: - issues.append(f"Commission wallet inaccessible: {str(e)}") - - # Check database connectivity - try: - await test_lamassu_connection() - except Exception as e: - issues.append(f"Lamassu database connection failed: {str(e)}") - - if issues: - await send_alert_to_admin("⚠️ System Health Check Failed", "\n".join(issues)) -``` - -### Alert Configuration - -**File**: `config.py` or environment variables - -```python -# Alert settings -ALERT_EMAIL = "admin@yourdomain.com" -ALERT_WEBHOOK = "https://hooks.slack.com/..." # Slack/Discord webhook -ALERT_PHONE = "+1234567890" # For critical alerts (optional) - -# Alert thresholds -MAX_STUCK_PAYMENT_HOURS = 2 -MAX_POLL_DELAY_HOURS = 24 -MAX_BALANCE_DISCREPANCY_SATS = 100 -``` - ---- - -## Monitoring & Alerts - -### Dashboard Indicators to Watch - -#### Critical Indicators (Check Daily) -- ✅ **Last Successful Poll Time** - Should be within last 2-4 hours (based on polling interval) -- ✅ **Failed Payment Count** - Should be 0; investigate immediately if > 0 -- ✅ **Commission Wallet Balance** - Should increase proportionally with transactions -- ✅ **Active Clients with Balance** - Cross-reference with expected DCA participants - -#### Warning Indicators (Check Weekly) -- ⚠️ **Pending Deposits > 48 hours** - May indicate confirmation workflow issue -- ⚠️ **Client Balance Reconciliation** - Run full reconciliation report -- ⚠️ **Average Commission %** - Verify matches configured rates -- ⚠️ **Transaction Processing Time** - Should complete within minutes, not hours - -### Automated Alert Triggers - -Implement these alerts in production: - -| Alert | Severity | Trigger Condition | Response Time | -|-------|----------|-------------------|---------------| -| Duplicate Transaction Detected | 🔴 CRITICAL | Same `transaction_id` inserted twice | Immediate | -| Payment Stuck > 15 minutes | 🔴 CRITICAL | Payment status not "completed" after 15min | < 30 minutes | -| Polling Failed > 24 hours | 🟠 HIGH | No new transactions in 24 hours | < 2 hours | -| Balance Discrepancy > 100 sats | 🟠 HIGH | Reconciliation finds error > threshold | < 4 hours | -| Wallet Inaccessible | 🟠 HIGH | Commission wallet returns auth error | < 1 hour | -| Database Connection Failed | 🟡 MEDIUM | Cannot connect to Lamassu DB | < 4 hours | -| Commission Calculation Anomaly | 🟡 MEDIUM | Calculated amount differs from formula | < 24 hours | - -### Log Files to Monitor - -**Location**: `/home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/` - -**Key Log Patterns**: -```bash -# Critical errors -grep -i "error\|exception\|failed" lnbits.log | tail -100 - -# Transaction processing -grep "LamassuTransactionProcessor" lnbits.log | tail -50 - -# Payment distribution -grep "dca_payment\|distribution" lnbits.log | tail -50 - -# SSH connection issues -grep -i "ssh\|connection\|timeout" lnbits.log | tail -50 - -# Wallet API calls -grep "wallet.*api\|payment_hash" lnbits.log | tail -50 -``` - -### Manual Checks (Weekly) - -**Sunday 00:00 UTC - Weekly Audit**: - -1. [ ] Run full reconciliation SQL report -2. [ ] Export all tables to CSV for backup -3. [ ] Verify commission wallet balance matches sum of commission_amount -4. [ ] Check for any pending deposits > 7 days old -5. [ ] Review last 20 transactions for calculation correctness -6. [ ] Test database connection from admin dashboard -7. [ ] Test manual poll to verify end-to-end flow -8. [ ] Review error logs for any concerning patterns - ---- - -## Contact Information - -### System Access - -**LNBits Admin Dashboard**: -- URL: `https:///satoshimachine` -- Requires superuser authentication - -**Database Access**: -```bash -# LNBits database -sqlite3 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/database.sqlite - -# Direct table access -sqlite3 /path/to/db "SELECT * FROM satoshimachine.;" -``` - -**Log Files**: -```bash -# Main logs -tail -f /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log - -# Error logs only -tail -f /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log | grep -i error -``` - -### Emergency Escalation - -**Level 1 - System Administrator** (First Contact): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -**Level 2 - Technical Lead** (If L1 unavailable): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -**Level 3 - Business Owner** (Financial impact > $X): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -### External Contacts - -**Lamassu Administrator**: -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- SSH access issues, database access, ATM questions - -**LNBits Infrastructure**: -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Wallet issues, API problems, system downtime - -**Accountant/Auditor**: -- Name: _______________________ -- Email: _______________________ -- For balance discrepancies requiring financial reconciliation - ---- - -## Appendix: Quick Reference Commands - -### Emergency Stop - -```bash -# Stop LNBits service (stops all background tasks) -systemctl stop lnbits - -# Or kill process -pkill -f lnbits -``` - -### Emergency Database Disable Polling - -```sql --- Disable automatic polling -UPDATE satoshimachine.lamassu_config -SET polling_enabled = false; -``` - -### Quick Balance Check - -```sql --- All client balances summary -SELECT id, username, remaining_balance, created_at -FROM satoshimachine.dca_clients -ORDER BY remaining_balance DESC; -``` - -### Last 10 Transactions - -```sql -SELECT id, transaction_id, created_at, crypto_atoms, base_amount, commission_amount -FROM satoshimachine.lamassu_transactions -ORDER BY created_at DESC -LIMIT 10; -``` - -### Failed Payments - -```sql -SELECT * FROM satoshimachine.dca_payments -WHERE status != 'completed' -ORDER BY created_at DESC; -``` - -### Export Everything (Backup) - -```bash -#!/bin/bash -# Emergency full backup -DATE=$(date +%Y%m%d_%H%M%S) -BACKUP_DIR="emergency_backup_${DATE}" -mkdir -p $BACKUP_DIR - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.lamassu_transactions;" \ - > ${BACKUP_DIR}/lamassu_transactions.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_payments;" \ - > ${BACKUP_DIR}/dca_payments.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_deposits;" \ - > ${BACKUP_DIR}/dca_deposits.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_clients;" \ - > ${BACKUP_DIR}/dca_clients.csv - -echo "Backup complete in ${BACKUP_DIR}/" -``` - ---- - -## Document Change Log - -| Version | Date | Author | Changes | -|---------|------|--------|---------| -| 1.0 | 2025-10-19 | Claude Code | Initial emergency protocols document | -| | | | | -| | | | | - ---- - -## Sign-Off - -This document has been reviewed and approved for use in production emergency response: - -**System Administrator**: _____________________ Date: _______ - -**Technical Lead**: _____________________ Date: _______ - -**Business Owner**: _____________________ Date: _______ - ---- - -**END OF DOCUMENT** - -*Keep this document accessible at all times. Print and store in emergency response binder.* -*Review and update quarterly or after any major incident.* diff --git a/misc-docs/EMERGENCY_PROTOCOLS_PRINT.md b/misc-docs/EMERGENCY_PROTOCOLS_PRINT.md deleted file mode 100644 index eb5b6b9..0000000 --- a/misc-docs/EMERGENCY_PROTOCOLS_PRINT.md +++ /dev/null @@ -1,1362 +0,0 @@ -# Satoshi Machine Admin - Emergency Protocols -## DCA System Failure Recovery Guide - -**Document Version**: 1.0 -**Last Updated**: 2025-10-19 -**Extension Version**: v0.0.1 -**Status**: Production - ---- - -## Table of Contents - -1. [Critical Failure Scenarios](#critical-failure-scenarios) -2. [Emergency Protocol Checklist](#emergency-protocol-checklist) -3. [Recovery Procedures](#recovery-procedures) -4. [Prevention Measures](#prevention-measures) -5. [Monitoring & Alerts](#monitoring--alerts) -6. [Contact Information](#contact-information) - ---- - -## Critical Failure Scenarios - -### 1. Duplicate Transaction Processing WARNING CRITICAL - -**Risk Level**: CRITICAL **CRITICAL** -**Impact**: Same Lamassu transaction processed twice → double distribution to clients → financial loss - -#### Detection Methods - -1. **Dashboard Monitoring**: - - Sudden large balance deductions from client accounts - - Multiple distribution entries for same timestamp - - Commission wallet receiving duplicate amounts - -2. **Database Query**: -```sql --- Find duplicate transactions -SELECT transaction_id, COUNT(*) as count, - STRING_AGG(id::text, ', ') as record_ids -FROM satoshimachine.lamassu_transactions -GROUP BY transaction_id -HAVING COUNT(*) > 1; -``` - -3. **Automated Alert Triggers**: - - Same `txn_id` appears in multiple processing cycles - - Client balance drops more than expected based on deposit ratios - -#### Immediate Response - -1. [OK] **STOP POLLING IMMEDIATELY** - Disable automatic background task -2. [OK] Document all duplicate entries with screenshots -3. [OK] Identify affected clients and amounts -4. [OK] Calculate total over-distribution amount - -#### Recovery Steps - -```sql --- Step 1: Identify duplicate distributions -SELECT lt.transaction_id, lt.id, lt.created_at, lt.base_amount, - COUNT(dp.id) as distribution_count -FROM satoshimachine.lamassu_transactions lt -LEFT JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -GROUP BY lt.id -HAVING COUNT(dp.id) > (SELECT COUNT(*) FROM satoshimachine.dca_clients WHERE remaining_balance > 0); - --- Step 2: Calculate over-distributed amounts per client -SELECT client_id, - SUM(amount_sats) as total_received, - -- Manual calculation of expected amount needed here -FROM satoshimachine.dca_payments -WHERE lamassu_transaction_id IN (SELECT id FROM duplicates_table) -GROUP BY client_id; -``` - -**Manual Correction**: -1. Calculate correct distribution amounts -2. Create compensating negative adjustments (if supported) OR -3. Deduct from future distributions until balanced -4. Document all corrections in audit log -5. Notify affected clients if material amount - -#### Prevention Measures - -**Required Code Changes**: -```python -# Add to transaction_processor.py BEFORE processing -existing = await get_lamassu_transaction_by_txid(txn_id) -if existing: - logger.warning(f"WARNING Transaction {txn_id} already processed, skipping") - return None -``` - -**Required Database Change**: -```sql -ALTER TABLE satoshimachine.lamassu_transactions -ADD CONSTRAINT unique_transaction_id UNIQUE (transaction_id); -``` - ---- - -### 2. SSH/Database Connection Loss - -**Risk Level**: MEDIUM **MEDIUM** -**Impact**: Polling stops → transactions not processed → clients not receiving Bitcoin on time - -#### Detection Methods - -1. **Dashboard Indicators**: - - No new records in transaction history for > 24 hours - - "Test Connection" button fails in admin configuration - - Background task logs show SSH connection errors - -2. **Database Query**: -```sql --- Check last successful poll -SELECT MAX(created_at) as last_transaction, - EXTRACT(EPOCH FROM (NOW() - MAX(created_at)))/3600 as hours_since_last -FROM satoshimachine.lamassu_transactions; - --- If hours_since_last > 24, investigate immediately -``` - -3. **Log File Check**: -```bash -# Check LNBits logs for SSH errors -tail -n 100 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log | grep -i "ssh\|connection" -``` - -#### Immediate Response - -1. [OK] Verify network connectivity to Lamassu server -2. [OK] Test SSH credentials manually: -```bash -ssh postgres@ -p -i -``` -3. [OK] Check firewall rules and network changes -4. [OK] Verify Lamassu server is running and accessible - -#### Recovery Steps - -**Option A: Credential Issues** -1. Regenerate SSH keys if compromised -2. Update authorized_keys on Lamassu server -3. Update configuration in admin dashboard -4. Test connection before re-enabling polling - -**Option B: Network Issues** -1. Coordinate with network admin to restore connectivity -2. Verify IP whitelisting if applicable -3. Test connection stability before resuming - -**Option C: Lamassu Server Issues** -1. Contact Lamassu administrator -2. Verify PostgreSQL service is running -3. Check database is accessible - -**Post-Recovery**: -```python -# System automatically catches up using last_polled_at timestamp -# Run manual poll to process missed transactions -POST /api/v1/dca/manual-poll -``` - -#### Prevention Measures - -1. **SSH Key Authentication** (more reliable than password): -```bash -# Generate dedicated key for this service -ssh-keygen -t ed25519 -f ~/.ssh/satmachine_lamassu -C "satmachine-polling" -``` - -2. **Connection Monitoring**: Implement daily health check -3. **Retry Logic**: Add exponential backoff in polling code -4. **Alert System**: Email/SMS when polling fails for > 2 hours - ---- - -### 3. Payment Distribution Failures - -**Risk Level**: CRITICAL **CRITICAL** -**Impact**: Commission deducted and client balances reduced, but transfers fail → money stuck in limbo - -#### Detection Methods - -1. **Dashboard Monitoring**: - - Client balances decrease but payment status shows "failed" - - Commission wallet balance doesn't increase as expected - - Error notifications in payment processing - -2. **Database Query**: -```sql --- Find stuck/failed payments (older than 1 hour, not completed) -SELECT dp.id, dp.client_id, dp.amount_sats, dp.status, dp.created_at, - c.username, dp.payment_hash -FROM satoshimachine.dca_payments dp -JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -WHERE dp.status != 'completed' - AND dp.created_at < NOW() - INTERVAL '1 hour' -ORDER BY dp.created_at DESC; -``` - -3. **Wallet Balance Check**: -```sql --- Compare expected vs actual commission wallet balance -SELECT - SUM(commission_amount) as total_commission_expected, - -- Manually check actual wallet balance in LNBits -FROM satoshimachine.lamassu_transactions; -``` - -#### Immediate Response - -1. [OK] **STOP POLLING** - Prevent more transactions from failing -2. [OK] Identify root cause: - - Insufficient balance in source wallet? - - Invalid wallet adminkey? - - LNBits API issues? - - Network connectivity to LNBits? - -3. [OK] Document all failed payments with IDs and amounts - -#### Recovery Steps - -**Step 1: Verify Wallet Configuration** -```bash -# Test that commission wallet is accessible -curl -X GET https:///api/v1/wallet \ - -H "X-Api-Key: " -``` - -**Step 2: Check Wallet Balance** -- Ensure commission wallet has sufficient balance -- Verify no wallet locks or restrictions - -**Step 3: Manual Retry Process** - -**Option A: Retry via API** (if retry endpoint exists) -```python -# Retry failed payment -POST /api/v1/dca/payments/{payment_id}/retry -``` - -**Option B: Manual Recreation** (if no retry available) -1. Query failed payment details -2. Mark original payment as "cancelled" -3. Create new payment entry with same parameters -4. Process through normal payment flow -5. Update client records - -**Step 4: Verify Reconciliation** -```sql --- After recovery, verify all clients balance correctly -SELECT - c.id, - c.username, - c.remaining_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_payments -FROM satoshimachine.dca_clients c -LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' -LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id -GROUP BY c.id; -``` - -#### Prevention Measures - -**Required Code Changes**: -```python -# Add pre-flight check in transaction_processor.py -async def verify_commission_wallet_accessible(): - """Verify commission wallet exists and is accessible before processing""" - try: - response = await wallet_api_call(commission_wallet_id) - if not response.ok: - raise Exception("Commission wallet not accessible") - return True - except Exception as e: - logger.error(f"Pre-flight check failed: {e}") - return False - -# Add transaction rollback on distribution failure -async def process_transaction_with_rollback(): - transaction_record = None - try: - # Deduct balances - # Create distributions - # Transfer funds - # Mark complete - except Exception as e: - # ROLLBACK: Restore client balances - # Mark transaction as failed - # Alert admin -``` - -**Monitoring**: -1. Alert if any payment remains in non-completed status > 15 minutes -2. Daily reconciliation of commission wallet balance -3. Automated testing of wallet accessibility - ---- - -### 4. Balance Discrepancies - -**Risk Level**: MEDIUM **MEDIUM** -**Impact**: Client balances don't match deposit/payment records → accounting errors, audit failures - -#### Detection Methods - -**Full Reconciliation Query**: -```sql --- Complete balance reconciliation report -SELECT - c.id, - c.username, - c.remaining_balance as current_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_distributed, - COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as calculated_balance, - c.remaining_balance - (COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0)) as discrepancy -FROM satoshimachine.dca_clients c -LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' -LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id -GROUP BY c.id, c.username, c.remaining_balance -HAVING ABS(c.remaining_balance - (COALESCE(SUM(d.amount), 0) - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0))) > 1; -``` - -#### Immediate Response - -1. [OK] Run full reconciliation query above -2. [OK] Export complete audit trail to CSV -3. [OK] Document discrepancy amounts per client -4. [OK] Identify pattern (all clients affected? specific time period?) - -#### Recovery Steps - -**For Each Discrepancy**: - -1. **Trace Transaction History**: -```sql --- Get complete transaction history for client -SELECT 'DEPOSIT' as type, id, amount, status, created_at, confirmed_at -FROM satoshimachine.dca_deposits -WHERE client_id = -UNION ALL -SELECT 'PAYMENT' as type, id, amount_sats, status, created_at, NULL -FROM satoshimachine.dca_payments -WHERE client_id = -ORDER BY created_at; -``` - -2. **Manual Recalculation**: - - Sum all confirmed deposits - - Subtract all completed payments - - Compare to current balance - - Identify missing/extra transactions - -3. **Correction Methods**: - -**Option A: Adjustment Entry** (Recommended) -```sql --- Create compensating deposit for positive discrepancy -INSERT INTO satoshimachine.dca_deposits (client_id, amount, status, note) -VALUES (, , 'confirmed', 'Balance correction - reconciliation 2025-10-19'); - --- OR Create compensating payment for negative discrepancy -INSERT INTO satoshimachine.dca_payments (client_id, amount_sats, status, note) -VALUES (, , 'completed', 'Balance correction - reconciliation 2025-10-19'); -``` - -**Option B: Direct Balance Update** (Use with extreme caution) -```sql --- ONLY if audit trail is complete and discrepancy is unexplained -UPDATE satoshimachine.dca_clients -SET remaining_balance = , - updated_at = NOW() -WHERE id = ; - --- MUST document in separate audit log -``` - -#### Prevention Measures - -**Daily Automated Reconciliation**: -```python -# Add to tasks.py -async def daily_reconciliation_check(): - """Run daily at 00:00 UTC""" - discrepancies = await find_balance_discrepancies() - if discrepancies: - await send_alert_to_admin(discrepancies) - await log_reconciliation_report(discrepancies) -``` - -**Database Constraints**: -```sql --- Prevent negative balances -ALTER TABLE satoshimachine.dca_clients -ADD CONSTRAINT positive_balance CHECK (remaining_balance >= 0); - --- Prevent confirmed deposits with zero amount -ALTER TABLE satoshimachine.dca_deposits -ADD CONSTRAINT positive_deposit CHECK (amount > 0); -``` - -**Audit Enhancements**: -- Store before/after balance with each transaction -- Implement change log table for all balance modifications -- Automated snapshots of all balances daily - ---- - -### 5. Commission Calculation Errors - -**Risk Level**: HIGH **HIGH** -**Impact**: Wrong commission rate applied → over/under collection → revenue loss or client overcharge - -#### Detection Methods - -**Verification Query**: -```sql --- Verify all commission calculations are mathematically correct -SELECT - id, - transaction_id, - crypto_atoms, - commission_percentage, - discount, - base_amount, - commission_amount, - -- Recalculate expected values - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as expected_base, - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) as expected_commission, - -- Calculate differences - base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as base_difference, - commission_amount - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) as commission_difference -FROM satoshimachine.lamassu_transactions -WHERE ABS(base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100)))) > 1 - OR ABS(commission_amount - ROUND(crypto_atoms - (crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))))) > 1; -``` - -**Manual Spot Check**: -``` -Example: 2000 GTQ → 266,800 sats (3% commission, 0% discount) - -Expected calculation: -- Effective commission = 0.03 * (100 - 0) / 100 = 0.03 -- Base amount = 266800 / (1 + 0.03) = 258,835 sats -- Commission = 266800 - 258835 = 7,965 sats - -Verify these match database values. -``` - -#### Immediate Response - -1. [OK] Run verification query to identify all affected transactions -2. [OK] Calculate total under/over collection amount -3. [OK] Determine if pattern (all transactions? specific time period? specific commission rate?) -4. [OK] **STOP POLLING** if active miscalculation detected - -#### Recovery Steps - -**Step 1: Quantify Impact** -```sql --- Total revenue impact -SELECT - COUNT(*) as affected_transactions, - SUM(commission_difference) as total_revenue_impact, - MIN(created_at) as first_occurrence, - MAX(created_at) as last_occurrence -FROM ( - -- Use verification query from above -) as calc_check -WHERE ABS(commission_difference) > 1; -``` - -**Step 2: Client Impact Assessment** -```sql --- Which clients were affected and by how much -SELECT - c.id, - c.username, - COUNT(lt.id) as affected_transactions, - SUM(lt.base_amount) as total_distributed, - SUM(expected_base) as should_have_distributed, - SUM(expected_base - lt.base_amount) as client_impact -FROM satoshimachine.lamassu_transactions lt -JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -WHERE -- filter for affected transactions -GROUP BY c.id; -``` - -**Step 3: Correction** - -**If Under-Collected Commission**: -- Revenue lost to business -- Client received correct amount -- No client-facing correction needed -- Document for accounting - -**If Over-Collected Commission**: -- Client under-distributed -- Create compensating payments to affected clients: -```sql --- Add to client balances -UPDATE satoshimachine.dca_clients c -SET remaining_balance = remaining_balance + adjustment.amount -FROM ( - -- Calculate adjustment per client - SELECT client_id, SUM(should_have_distributed - actual_distributed) as amount - FROM affected_transactions_analysis - GROUP BY client_id -) adjustment -WHERE c.id = adjustment.client_id; -``` - -**Step 4: Fix Code Bug** -- Identify root cause in `transaction_processor.py` -- Add unit test for failed scenario -- Deploy fix -- Verify with test transaction - -#### Prevention Measures - -**Unit Tests** (Add to test suite): -```python -def test_commission_calculation_scenarios(): - """Test edge cases for commission calculation""" - test_cases = [ - # (crypto_atoms, commission%, discount%, expected_base, expected_commission) - (266800, 0.03, 0.0, 258835, 7965), - (100000, 0.05, 10.0, 95694, 4306), # With discount - (1, 0.03, 0.0, 0, 1), # Minimum amount - # Add more edge cases - ] - for case in test_cases: - result = calculate_commission(*case[:3]) - assert result.base_amount == case[3] - assert result.commission_amount == case[4] -``` - -**Calculation Verification** (Add to processing): -```python -# After calculation, verify math is correct -calculated_total = base_amount + commission_amount -assert abs(calculated_total - crypto_atoms) <= 1, "Commission calculation error detected" -``` - -**Audit Trail**: -- Store all calculation parameters with each transaction -- Log formula used and intermediate values -- Enable post-processing verification - ---- - -### 6. Wallet Key Rotation/Invalidation - -**Risk Level**: HIGH **HIGH** -**Impact**: Commission wallet adminkey changes → can't receive commission payments → processing halts - -#### Detection Methods - -1. **API Error Responses**: - - Payment API returns 401/403 authentication errors - - "Invalid API key" messages in logs - -2. **Wallet Balance Check**: -```sql --- Commission not accumulating despite transactions -SELECT - SUM(commission_amount) as expected_total_commission, - -- Compare to actual wallet balance in LNBits dashboard -FROM satoshimachine.lamassu_transactions -WHERE created_at > ''; -``` - -3. **Manual Test**: -```bash -# Test commission wallet adminkey -curl -X GET https:///api/v1/wallet \ - -H "X-Api-Key: " - -# Should return wallet details, not auth error -``` - -#### Immediate Response - -1. [OK] **STOP POLLING** - No point processing if can't distribute -2. [OK] Identify if key was intentionally rotated or compromised -3. [OK] Obtain new valid adminkey for commission wallet -4. [OK] Verify source wallet adminkey is also still valid - -#### Recovery Steps - -**Step 1: Update Configuration** -1. Access admin dashboard -2. Navigate to Configuration section -3. Update commission wallet adminkey -4. Update source wallet adminkey if also affected -5. **Test Connection** before saving - -**Step 2: Verify Configuration Persisted** -```sql --- Check configuration was saved correctly -SELECT commission_wallet_id, - LEFT(commission_wallet_adminkey, 10) || '...' as key_preview, - updated_at -FROM satoshimachine.lamassu_config -ORDER BY updated_at DESC -LIMIT 1; -``` - -**Step 3: Reprocess Failed Transactions** -- Identify transactions that failed due to auth errors -- Mark for retry or manually reprocess -- Verify commission payments complete successfully - -**Step 4: Resume Operations** -1. Test with manual poll first -2. Verify single transaction processes completely -3. Re-enable automatic polling -4. Monitor for 24 hours - -#### Prevention Measures - -**Key Management Documentation**: -- Document which LNBits wallets are used for commission/source -- Store backup admin keys in secure location (password manager) -- Define key rotation procedure with testing steps -- Require testing in staging before production changes - -**Configuration Validation**: -```python -# Add to config save endpoint in views_api.py -async def validate_wallet_keys(commission_key, source_key): - """Test wallet keys before saving configuration""" - # Test commission wallet - commission_valid = await test_wallet_access(commission_key) - if not commission_valid: - raise ValueError("Commission wallet key is invalid") - - # Test source wallet (if applicable) - if source_key: - source_valid = await test_wallet_access(source_key) - if not source_valid: - raise ValueError("Source wallet key is invalid") - - return True -``` - -**Automated Monitoring**: -- Daily health check of wallet accessibility -- Alert if wallet API calls start failing -- Backup key verification in secure environment - ---- - -## Emergency Protocol Checklist - -Use this checklist when ANY error is detected in the DCA system. - -### Phase 1: Immediate Actions (First 5 Minutes) - -- [ ] **Stop Automatic Processing** - - Disable background polling task - - Verify no transactions are currently processing - - Document time polling was stopped - -- [ ] **Assess Severity** - - Is money at risk? (duplicate processing, failed payments) - - Are clients affected? (missing distributions, balance errors) - - Is this blocking operations? (connection loss, wallet issues) - -- [ ] **Initial Documentation** - - Take screenshots of error messages - - Note exact timestamp of error detection - - Record current system state (balances, last transaction, etc.) - -- [ ] **Notify Stakeholders** - - Alert system administrator - - Notify clients if distributions will be delayed > 24 hours - - Escalate if financial impact > threshold - -### Phase 2: Investigation (15-30 Minutes) - -- [ ] **Collect Diagnostic Information** - - Run relevant SQL queries from scenarios above - - Check LNBits logs: `/lnbits/data/logs/` - - Review recent configuration changes - - Test external connections (SSH, wallets) - -- [ ] **Identify Root Cause** - - Match symptoms to failure scenarios above - - Determine if human error, system failure, or external issue - - Estimate scope of impact (time range, # clients, # transactions) - -- [ ] **Document Findings** - - Record root cause analysis - - List all affected records (transaction IDs, client IDs) - - Calculate financial impact (over/under distributed amounts) - - Take database snapshots for audit trail - -### Phase 3: Recovery (30 Minutes - 2 Hours) - -- [ ] **Fix Root Cause** - - Apply code fix if bug - - Update configuration if settings issue - - Restore connection if network issue - - Refer to specific recovery steps in scenarios above - -- [ ] **Data Correction** - - Run reconciliation queries - - Execute correction SQL statements - - Verify all client balances are accurate - - Ensure audit trail is complete - -- [ ] **Verification** - - Test fix with single transaction - - Verify wallets are accessible - - Confirm connections are stable - - Run full reconciliation report - -### Phase 4: Resumption (After Verification) - -- [ ] **Gradual Restart** - - Process one manual poll successfully - - Monitor for errors during processing - - Verify distributions complete end-to-end - - Check commission payments arrive correctly - -- [ ] **Re-enable Automation** - - Turn on background polling task - - Set monitoring alerts - - Document in system log - -- [ ] **Enhanced Monitoring** - - Watch closely for 24 hours - - Run reconciliation after next 3-5 transactions - - Verify no recurrence of issue - -### Phase 5: Post-Incident (24-48 Hours After) - -- [ ] **Complete Post-Mortem** - - Document full timeline of incident - - Record exact root cause and fix applied - - Calculate total impact (financial, time, clients affected) - - Identify what went well and what could improve - -- [ ] **Implement Safeguards** - - Add prevention measures from scenario sections - - Implement new monitoring/alerts - - Add automated tests for this failure mode - - Update runbooks and documentation - -- [ ] **Stakeholder Communication** - - Send incident report to management - - Notify affected clients if applicable - - Document lessons learned - - Update emergency contact procedures if needed - ---- - -## Recovery Procedures - -### Full System Reconciliation - -Run this complete reconciliation procedure weekly or after any incident: - -```sql --- ============================================ --- FULL SYSTEM RECONCILIATION REPORT --- ============================================ - --- 1. Client Balance Reconciliation -WITH client_financials AS ( - SELECT - c.id, - c.username, - c.remaining_balance as current_balance, - COALESCE(SUM(d.amount), 0) as total_deposits, - COALESCE(SUM(CASE WHEN p.status = 'completed' THEN p.amount_sats ELSE 0 END), 0) as total_payments, - COUNT(DISTINCT d.id) as deposit_count, - COUNT(DISTINCT p.id) as payment_count - FROM satoshimachine.dca_clients c - LEFT JOIN satoshimachine.dca_deposits d ON c.id = d.client_id AND d.status = 'confirmed' - LEFT JOIN satoshimachine.dca_payments p ON c.id = p.client_id - GROUP BY c.id -) -SELECT - id, - username, - current_balance, - total_deposits, - total_payments, - (total_deposits - total_payments) as calculated_balance, - (current_balance - (total_deposits - total_payments)) as discrepancy, - deposit_count, - payment_count, - CASE - WHEN ABS(current_balance - (total_deposits - total_payments)) <= 1 THEN '[OK] OK' - ELSE 'WARNING MISMATCH' - END as status -FROM client_financials -ORDER BY ABS(current_balance - (total_deposits - total_payments)) DESC; - --- 2. Transaction Processing Verification -SELECT - COUNT(*) as total_transactions, - SUM(crypto_atoms) as total_sats_processed, - SUM(base_amount) as total_distributed, - SUM(commission_amount) as total_commission, - MIN(created_at) as first_transaction, - MAX(created_at) as last_transaction -FROM satoshimachine.lamassu_transactions; - --- 3. Failed/Pending Payments Check -SELECT - status, - COUNT(*) as count, - SUM(amount_sats) as total_amount, - MIN(created_at) as oldest, - MAX(created_at) as newest -FROM satoshimachine.dca_payments -GROUP BY status -ORDER BY - CASE status - WHEN 'failed' THEN 1 - WHEN 'pending' THEN 2 - WHEN 'completed' THEN 3 - END; - --- 4. Unconfirmed Deposits Check (sitting too long) -SELECT - id, - client_id, - amount, - status, - created_at, - EXTRACT(EPOCH FROM (NOW() - created_at))/3600 as hours_pending -FROM satoshimachine.dca_deposits -WHERE status = 'pending' - AND created_at < NOW() - INTERVAL '48 hours' -ORDER BY created_at; - --- 5. Commission Calculation Verification (sample check) -SELECT - id, - transaction_id, - crypto_atoms, - commission_percentage, - discount, - base_amount, - commission_amount, - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as expected_base, - base_amount - ROUND(crypto_atoms / (1 + (commission_percentage * (100 - discount) / 100))) as difference -FROM satoshimachine.lamassu_transactions -ORDER BY created_at DESC -LIMIT 20; -``` - -### Emergency Access Procedures - -**If Admin Dashboard Inaccessible**: - -1. **Direct Database Access**: -```bash -# Connect to LNBits database -sqlite3 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/database.sqlite - -# Or PostgreSQL if used -psql -h localhost -U lnbits -d lnbits -``` - -2. **Direct Configuration Update**: -```sql --- Update Lamassu config directly -UPDATE satoshimachine.lamassu_config -SET polling_enabled = false -WHERE id = (SELECT MAX(id) FROM satoshimachine.lamassu_config); -``` - -3. **Manual Client Balance Update**: -```sql --- ONLY in emergency when dashboard unavailable -UPDATE satoshimachine.dca_clients -SET remaining_balance = -WHERE id = ; --- MUST document this action in incident log -``` - -**If Background Task Won't Stop**: - -```bash -# Find LNBits process -ps aux | grep lnbits - -# Restart LNBits service (will stop all background tasks) -systemctl restart lnbits -# OR if running manually: -pkill -f lnbits -uv run lnbits -``` - -### Data Export for Audit - -**Complete Audit Trail Export**: - -```bash -# Export all DCA-related tables to CSV -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.lamassu_transactions;" \ - > lamassu_transactions_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_payments;" \ - > dca_payments_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_deposits;" \ - > dca_deposits_export_$(date +%Y%m%d_%H%M%S).csv - -sqlite3 -header -csv /path/to/lnbits/database.sqlite \ - "SELECT * FROM satoshimachine.dca_clients;" \ - > dca_clients_export_$(date +%Y%m%d_%H%M%S).csv -``` - -**Combined Audit Report**: - -```sql --- Complete transaction-to-distribution audit trail -SELECT - lt.id as transaction_id, - lt.transaction_id as lamassu_txn_id, - lt.created_at as transaction_time, - lt.crypto_atoms as total_sats, - lt.fiat_code, - lt.fiat_amount, - lt.commission_percentage, - lt.discount, - lt.base_amount as distributable_sats, - lt.commission_amount, - c.id as client_id, - c.username as client_name, - dp.amount_sats as client_received, - dp.status as payment_status, - dp.payment_hash -FROM satoshimachine.lamassu_transactions lt -LEFT JOIN satoshimachine.dca_payments dp ON dp.lamassu_transaction_id = lt.id -LEFT JOIN satoshimachine.dca_clients c ON dp.client_id = c.id -ORDER BY lt.created_at DESC, c.username; -``` - ---- - -## Prevention Measures - -### Required Immediate Implementations - -#### 1. Idempotency Protection (CRITICAL) - -**File**: `transaction_processor.py` - -```python -async def process_lamassu_transaction(txn_data: dict) -> Optional[LamassuTransaction]: - """Process transaction with idempotency check""" - - # CRITICAL: Check if already processed - existing = await get_lamassu_transaction_by_txid(txn_data['id']) - if existing: - logger.warning(f"WARNING Transaction {txn_data['id']} already processed at {existing.created_at}, skipping") - return None - - # Continue with processing... -``` - -#### 2. Database Constraints - -**File**: `migrations.py` - -```sql --- Add unique constraint on transaction_id -ALTER TABLE satoshimachine.lamassu_transactions -ADD CONSTRAINT unique_transaction_id UNIQUE (transaction_id); - --- Prevent negative balances -ALTER TABLE satoshimachine.dca_clients -ADD CONSTRAINT positive_balance CHECK (remaining_balance >= 0); - --- Ensure positive amounts -ALTER TABLE satoshimachine.dca_deposits -ADD CONSTRAINT positive_deposit CHECK (amount > 0); - -ALTER TABLE satoshimachine.dca_payments -ADD CONSTRAINT positive_payment CHECK (amount_sats > 0); -``` - -#### 3. Transaction Status Tracking - -**File**: `models.py` - -```python -class LamassuTransaction(BaseModel): - # ... existing fields ... - status: str = "pending" # pending, processing, completed, failed - error_message: Optional[str] = None - processed_at: Optional[datetime] = None -``` - -#### 4. Pre-flight Wallet Validation - -**File**: `transaction_processor.py` - -```python -async def validate_system_ready() -> Tuple[bool, str]: - """Validate system is ready to process transactions""" - - # Check commission wallet accessible - try: - commission_wallet = await get_wallet(config.commission_wallet_id) - if not commission_wallet: - return False, "Commission wallet not accessible" - except Exception as e: - return False, f"Commission wallet error: {str(e)}" - - # Check for stuck payments - stuck_payments = await get_stuck_payments(hours=2) - if stuck_payments: - return False, f"{len(stuck_payments)} payments stuck for >2 hours" - - # Check database connectivity - try: - await db_health_check() - except Exception as e: - return False, f"Database health check failed: {str(e)}" - - return True, "System ready" - -# Call before processing -ready, message = await validate_system_ready() -if not ready: - logger.error(f"System not ready: {message}") - await send_alert_to_admin(message) - return -``` - -### Automated Monitoring Implementation - -#### Daily Reconciliation Task - -**File**: `tasks.py` - -```python -@scheduler.scheduled_job("cron", hour=0, minute=0) # Daily at midnight UTC -async def daily_reconciliation(): - """Run daily balance reconciliation and report discrepancies""" - - logger.info("Starting daily reconciliation...") - - discrepancies = await find_balance_discrepancies() - - if discrepancies: - report = generate_reconciliation_report(discrepancies) - await send_alert_to_admin("WARNING Balance Discrepancies Detected", report) - await log_reconciliation_issue(discrepancies) - else: - logger.info("[OK] Daily reconciliation passed - all balances match") - - # Check for stuck payments - stuck_payments = await get_stuck_payments(hours=24) - if stuck_payments: - await send_alert_to_admin( - f"WARNING {len(stuck_payments)} Stuck Payments Detected", - format_stuck_payments_report(stuck_payments) - ) -``` - -#### Connection Health Monitor - -```python -@scheduler.scheduled_job("interval", hours=2) # Every 2 hours -async def check_system_health(): - """Monitor system health and alert on issues""" - - issues = [] - - # Check last successful poll - last_poll = await get_last_successful_poll_time() - if last_poll and (datetime.utcnow() - last_poll).total_seconds() > 86400: # 24 hours - issues.append(f"No successful poll in {(datetime.utcnow() - last_poll).total_seconds() / 3600:.1f} hours") - - # Check wallet accessibility - try: - await test_wallet_access(config.commission_wallet_adminkey) - except Exception as e: - issues.append(f"Commission wallet inaccessible: {str(e)}") - - # Check database connectivity - try: - await test_lamassu_connection() - except Exception as e: - issues.append(f"Lamassu database connection failed: {str(e)}") - - if issues: - await send_alert_to_admin("WARNING System Health Check Failed", "\n".join(issues)) -``` - -### Alert Configuration - -**File**: `config.py` or environment variables - -```python -# Alert settings -ALERT_EMAIL = "admin@yourdomain.com" -ALERT_WEBHOOK = "https://hooks.slack.com/..." # Slack/Discord webhook -ALERT_PHONE = "+1234567890" # For critical alerts (optional) - -# Alert thresholds -MAX_STUCK_PAYMENT_HOURS = 2 -MAX_POLL_DELAY_HOURS = 24 -MAX_BALANCE_DISCREPANCY_SATS = 100 -``` - ---- - -## Monitoring & Alerts - -### Dashboard Indicators to Watch - -#### Critical Indicators (Check Daily) -- [OK] **Last Successful Poll Time** - Should be within last 2-4 hours (based on polling interval) -- [OK] **Failed Payment Count** - Should be 0; investigate immediately if > 0 -- [OK] **Commission Wallet Balance** - Should increase proportionally with transactions -- [OK] **Active Clients with Balance** - Cross-reference with expected DCA participants - -#### Warning Indicators (Check Weekly) -- WARNING **Pending Deposits > 48 hours** - May indicate confirmation workflow issue -- WARNING **Client Balance Reconciliation** - Run full reconciliation report -- WARNING **Average Commission %** - Verify matches configured rates -- WARNING **Transaction Processing Time** - Should complete within minutes, not hours - -### Automated Alert Triggers - -Implement these alerts in production: - -| Alert | Severity | Trigger Condition | Response Time | -|-------|----------|-------------------|---------------| -| Duplicate Transaction Detected | CRITICAL CRITICAL | Same `transaction_id` inserted twice | Immediate | -| Payment Stuck > 15 minutes | CRITICAL CRITICAL | Payment status not "completed" after 15min | < 30 minutes | -| Polling Failed > 24 hours | HIGH HIGH | No new transactions in 24 hours | < 2 hours | -| Balance Discrepancy > 100 sats | HIGH HIGH | Reconciliation finds error > threshold | < 4 hours | -| Wallet Inaccessible | HIGH HIGH | Commission wallet returns auth error | < 1 hour | -| Database Connection Failed | MEDIUM MEDIUM | Cannot connect to Lamassu DB | < 4 hours | -| Commission Calculation Anomaly | MEDIUM MEDIUM | Calculated amount differs from formula | < 24 hours | - -### Log Files to Monitor - -**Location**: `/home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/` - -**Key Log Patterns**: -```bash -# Critical errors -grep -i "error\|exception\|failed" lnbits.log | tail -100 - -# Transaction processing -grep "LamassuTransactionProcessor" lnbits.log | tail -50 - -# Payment distribution -grep "dca_payment\|distribution" lnbits.log | tail -50 - -# SSH connection issues -grep -i "ssh\|connection\|timeout" lnbits.log | tail -50 - -# Wallet API calls -grep "wallet.*api\|payment_hash" lnbits.log | tail -50 -``` - -### Manual Checks (Weekly) - -**Sunday 00:00 UTC - Weekly Audit**: - -1. [ ] Run full reconciliation SQL report -2. [ ] Export all tables to CSV for backup -3. [ ] Verify commission wallet balance matches sum of commission_amount -4. [ ] Check for any pending deposits > 7 days old -5. [ ] Review last 20 transactions for calculation correctness -6. [ ] Test database connection from admin dashboard -7. [ ] Test manual poll to verify end-to-end flow -8. [ ] Review error logs for any concerning patterns - ---- - -## Contact Information - -### System Access - -**LNBits Admin Dashboard**: -- URL: `https:///satoshimachine` -- Requires superuser authentication - -**Database Access**: -```bash -# LNBits database -sqlite3 /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/database.sqlite - -# Direct table access -sqlite3 /path/to/db "SELECT * FROM satoshimachine.;" -``` - -**Log Files**: -```bash -# Main logs -tail -f /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log - -# Error logs only -tail -f /home/padreug/AioLabs/Git/lnbits-extensions/lnbits/data/logs/lnbits.log | grep -i error -``` - -### Emergency Escalation - -**Level 1 - System Administrator** (First Contact): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -**Level 2 - Technical Lead** (If L1 unavailable): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -**Level 3 - Business Owner** (Financial impact > $X): -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Availability: _______________________ - -### External Contacts - -**Lamassu Administrator**: -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- SSH access issues, database access, ATM questions - -**LNBits Infrastructure**: -- Name: _______________________ -- Email: _______________________ -- Phone: _______________________ -- Wallet issues, API problems, system downtime - -**Accountant/Auditor**: -- Name: _______________________ -- Email: _______________________ -- For balance discrepancies requiring financial reconciliation - ---- - -## Appendix: Quick Reference Commands - -### Emergency Stop - -```bash -# Stop LNBits service (stops all background tasks) -systemctl stop lnbits - -# Or kill process -pkill -f lnbits -``` - -### Emergency Database Disable Polling - -```sql --- Disable automatic polling -UPDATE satoshimachine.lamassu_config -SET polling_enabled = false; -``` - -### Quick Balance Check - -```sql --- All client balances summary -SELECT id, username, remaining_balance, created_at -FROM satoshimachine.dca_clients -ORDER BY remaining_balance DESC; -``` - -### Last 10 Transactions - -```sql -SELECT id, transaction_id, created_at, crypto_atoms, base_amount, commission_amount -FROM satoshimachine.lamassu_transactions -ORDER BY created_at DESC -LIMIT 10; -``` - -### Failed Payments - -```sql -SELECT * FROM satoshimachine.dca_payments -WHERE status != 'completed' -ORDER BY created_at DESC; -``` - -### Export Everything (Backup) - -```bash -#!/bin/bash -# Emergency full backup -DATE=$(date +%Y%m%d_%H%M%S) -BACKUP_DIR="emergency_backup_${DATE}" -mkdir -p $BACKUP_DIR - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.lamassu_transactions;" \ - > ${BACKUP_DIR}/lamassu_transactions.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_payments;" \ - > ${BACKUP_DIR}/dca_payments.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_deposits;" \ - > ${BACKUP_DIR}/dca_deposits.csv - -sqlite3 -header -csv /path/to/database.sqlite \ - "SELECT * FROM satoshimachine.dca_clients;" \ - > ${BACKUP_DIR}/dca_clients.csv - -echo "Backup complete in ${BACKUP_DIR}/" -``` - ---- - -## Document Change Log - -| Version | Date | Author | Changes | -|---------|------|--------|---------| -| 1.0 | 2025-10-19 | Claude Code | Initial emergency protocols document | -| | | | | -| | | | | - ---- - -## Sign-Off - -This document has been reviewed and approved for use in production emergency response: - -**System Administrator**: _____________________ Date: _______ - -**Technical Lead**: _____________________ Date: _______ - -**Business Owner**: _____________________ Date: _______ - ---- - -**END OF DOCUMENT** - -*Keep this document accessible at all times. Print and store in emergency response binder.* -*Review and update quarterly or after any major incident.*