# BQL Price Notation Solution for SATS Tracking **Date**: 2025-01-12 **Status**: Testing **Context**: Explore price notation as alternative to metadata for SATS tracking --- ## Problem Recap Current approach stores SATS in metadata: ```beancount 2025-11-10 * "Groceries" Expenses:Food -360.00 EUR sats-equivalent: 337096 Liabilities:Payable:User-abc 360.00 EUR sats-equivalent: 337096 ``` **Issue**: BQL cannot access metadata, so balance queries require manual aggregation. --- ## Solution: Use Price Notation ### Proposed Format Post in actual transaction currency (EUR) with SATS as price: ```beancount 2025-11-10 * "Groceries" Expenses:Food -360.00 EUR @@ 337096 SATS Liabilities:Payable:User-abc 360.00 EUR @@ 337096 SATS ``` **What this means**: - Primary amount: `-360.00 EUR` (the actual transaction currency) - Total price: `337096 SATS` (the bitcoin equivalent value) - Transaction integrity preserved (posted in EUR as it occurred) - SATS tracked as price (queryable by BQL) --- ## Price Notation Options ### Option 1: Per-Unit Price (`@`) ```beancount Expenses:Food -360.00 EUR @ 936.38 SATS ``` **What it means**: Each EUR is worth 936.38 SATS **Total calculation**: 360 × 936.38 = 337,096.8 SATS **Precision**: May introduce rounding (336,696.8 vs 337,096) ### Option 2: Total Price (`@@`) ✅ RECOMMENDED ```beancount Expenses:Food -360.00 EUR @@ 337096 SATS ``` **What it means**: Total transaction value is 337,096 SATS **Total calculation**: Exact 337,096 SATS (no rounding) **Precision**: Preserves exact SATS amount from original calculation **Why `@@` is better for Castle:** - ✅ Preserves exact SATS amount (no rounding errors) - ✅ Matches current metadata storage exactly - ✅ Clearer intent: "this transaction equals X SATS total" --- ## How BQL Handles Prices ### Available Price Columns From BQL schema: - `price_number` - The numeric price amount (Decimal) - `price_currency` - The currency of the price (str) - `position` - Full posting (includes price) - `WEIGHT(position)` - Function that returns balance weight ### BQL Query Capabilities **Test Query 1: Access price directly** ```sql SELECT account, number, currency, price_number, price_currency WHERE account ~ 'User-375ec158' AND price_currency = 'SATS'; ``` **Expected Result** (if price notation works): ```json { "rows": [ ["Liabilities:Payable:User-abc", "360.00", "EUR", "337096", "SATS"] ] } ``` **Test Query 2: Aggregate SATS from prices** ```sql SELECT account, SUM(price_number) as total_sats WHERE account ~ 'User-' AND price_currency = 'SATS' AND flag != '!' GROUP BY account; ``` **Expected Result**: ```json { "rows": [ ["Liabilities:Payable:User-abc", "337096"] ] } ``` --- ## Testing Plan ### Step 1: Run Metadata Test ```bash cd /home/padreug/projects/castle-beancounter ./test_metadata_simple.sh ``` **What to look for**: - Does `meta` column exist in response? - Is `sats-equivalent` accessible in the data? **If YES**: Metadata IS accessible, simpler solution available **If NO**: Proceed with price notation approach ### Step 2: Test Current Data Structure ```bash ./test_bql_metadata.sh ``` This runs 6 tests: 1. Check metadata column 2. Check price columns 3. Basic position query 4. Test WEIGHT function 5. Aggregate positions 6. Aggregate weights **What to look for**: - Which columns are available? - What does `position` return for entries with prices? - Can we access `price_number` and `price_currency`? ### Step 3: Create Test Ledger Entry Add one test entry to your ledger: ```beancount 2025-01-12 * "TEST: Price notation test" Expenses:Test:PriceNotation -100.00 EUR @@ 93600 SATS Liabilities:Payable:User-TEST 100.00 EUR @@ 93600 SATS ``` Then query: ```bash curl -s "http://localhost:3333/castle-ledger/api/query" \ -G \ --data-urlencode "query_string=SELECT account, position, price_number, price_currency WHERE account ~ 'TEST'" \ | jq '.' ``` **Expected if working**: ```json { "data": { "rows": [ ["Expenses:Test:PriceNotation", "-100.00 EUR @@ 93600 SATS", "93600", "SATS"], ["Liabilities:Payable:User-TEST", "100.00 EUR @@ 93600 SATS", "93600", "SATS"] ], "types": [ {"name": "account", "type": "str"}, {"name": "position", "type": "Position"}, {"name": "price_number", "type": "Decimal"}, {"name": "price_currency", "type": "str"} ] } } ``` --- ## Migration Strategy (If Price Notation Works) ### Phase 1: Test on Sample Data 1. Create test ledger with mix of formats 2. Verify BQL can query price_number 3. Verify aggregation accuracy 4. Compare with manual method results ### Phase 2: Write Migration Script ```python #!/usr/bin/env python3 """ Migrate metadata sats-equivalent to price notation. Converts: Expenses:Food -360.00 EUR sats-equivalent: 337096 To: Expenses:Food -360.00 EUR @@ 337096 SATS """ import re from pathlib import Path def migrate_entry(entry_lines): """Migrate a single transaction entry.""" result = [] current_posting = None sats_value = None for line in entry_lines: # Check if this is a posting line if re.match(r'^\s{2,}\w+:', line): # If we have pending sats from previous posting, add it if current_posting and sats_value: # Add @@ notation to posting posting = current_posting.rstrip() posting += f" @@ {sats_value} SATS\n" result.append(posting) current_posting = None sats_value = None else: if current_posting: result.append(current_posting) current_posting = line # Check if this is sats-equivalent metadata elif 'sats-equivalent:' in line: match = re.search(r'sats-equivalent:\s*(-?\d+)', line) if match: sats_value = match.group(1) # Don't include metadata line in result else: # Other lines (date, narration, other metadata) if current_posting and sats_value: posting = current_posting.rstrip() posting += f" @@ {sats_value} SATS\n" result.append(posting) current_posting = None sats_value = None elif current_posting: result.append(current_posting) current_posting = None result.append(line) # Handle last posting if current_posting and sats_value: posting = current_posting.rstrip() posting += f" @@ {sats_value} SATS\n" result.append(posting) elif current_posting: result.append(current_posting) return result def migrate_ledger(input_file, output_file): """Migrate entire ledger file.""" with open(input_file, 'r') as f: lines = f.readlines() result = [] current_entry = [] in_transaction = False for line in lines: # Transaction start if re.match(r'^\d{4}-\d{2}-\d{2}\s+[*!]', line): in_transaction = True current_entry = [line] # Empty line ends transaction elif in_transaction and line.strip() == '': current_entry.append(line) migrated = migrate_entry(current_entry) result.extend(migrated) current_entry = [] in_transaction = False # Inside transaction elif in_transaction: current_entry.append(line) # Outside transaction else: result.append(line) # Handle last entry if file doesn't end with blank line if current_entry: migrated = migrate_entry(current_entry) result.extend(migrated) with open(output_file, 'w') as f: f.writelines(result) if __name__ == '__main__': import sys if len(sys.argv) != 3: print("Usage: migrate_ledger.py ") sys.exit(1) migrate_ledger(sys.argv[1], sys.argv[2]) print(f"Migrated {sys.argv[1]} -> {sys.argv[2]}") ``` ### Phase 3: Update Balance Query Methods Replace `get_user_balance_bql()` with price-based version: ```python async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]: """ Get user balance using price notation (SATS stored as @@ price). Returns: { "balance": int (sats from price_number), "fiat_balances": {"EUR": Decimal("100.50")}, "accounts": [{"account": "...", "sats": 150000}] } """ user_id_prefix = user_id[:8] # Query: Get EUR positions with SATS prices query = f""" SELECT account, number as eur_amount, price_number as sats_amount WHERE account ~ ':User-{user_id_prefix}' AND (account ~ 'Payable' OR account ~ 'Receivable') AND flag != '!' AND price_currency = 'SATS' """ result = await self.query_bql(query) total_sats = 0 fiat_balances = {} accounts_map = {} for row in result["rows"]: account_name, eur_amount, sats_amount = row # Parse amounts sats = int(Decimal(sats_amount)) if sats_amount else 0 eur = Decimal(eur_amount) if eur_amount else Decimal(0) total_sats += sats # Aggregate fiat if eur != 0: if "EUR" not in fiat_balances: fiat_balances["EUR"] = Decimal(0) fiat_balances["EUR"] += eur # Track per account if account_name not in accounts_map: accounts_map[account_name] = {"account": account_name, "sats": 0} accounts_map[account_name]["sats"] += sats return { "balance": total_sats, "fiat_balances": fiat_balances, "accounts": list(accounts_map.values()) } ``` ### Phase 4: Validation 1. Run both methods in parallel 2. Compare results for all users 3. Log any discrepancies 4. Investigate and fix differences 5. Once validated, switch to BQL method --- ## Advantages of Price Notation Approach ### 1. BQL Compatibility ✅ - `price_number` is a standard BQL column - Can aggregate: `SUM(price_number)` - Can filter: `WHERE price_currency = 'SATS'` ### 2. Transaction Integrity ✅ - Post in actual transaction currency (EUR) - SATS as secondary value (price) - Proper accounting: source currency preserved ### 3. Beancount Features ✅ - Price database automatically updated - Can query historical EUR/SATS rates - Reports can show both EUR and SATS values ### 4. Performance ✅ - BQL filters at source (no fetching all entries) - Direct column access (no metadata parsing) - Efficient aggregation (database-level) ### 5. Reporting Flexibility ✅ - Show EUR amounts in reports - Show SATS equivalents alongside - Filter by either currency - Calculate gains/losses if SATS price changes --- ## Potential Issues and Solutions ### Issue 1: Price vs Cost Confusion **Problem**: Beancount distinguishes between `@` price and `{}` cost **Solution**: Always use price (`@` or `@@`), never cost (`{}`) **Why**: - Cost is for tracking cost basis (investments, capital gains) - Price is for conversion rates (what we need) ### Issue 2: Precision Loss with `@` **Problem**: Per-unit price may have rounding ```beancount 360.00 EUR @ 936.38 SATS = 336,696.8 SATS (not 337,096) ``` **Solution**: Always use `@@` total price ```beancount 360.00 EUR @@ 337096 SATS = 337,096 SATS (exact) ``` ### Issue 3: Negative Numbers **Problem**: How to handle negative EUR with positive SATS? ```beancount -360.00 EUR @@ ??? SATS ``` **Solution**: Price is always positive (it's a rate, not an amount) ```beancount -360.00 EUR @@ 337096 SATS ✅ Correct ``` The sign applies to the position, price is the conversion factor. ### Issue 4: Historical Data **Problem**: Existing entries have metadata, not prices **Solution**: Migration script (see Phase 2) - One-time conversion - Validate with checksums - Keep backup of original --- ## Testing Checklist - [ ] Run `test_metadata_simple.sh` - Check if metadata is accessible - [ ] Run `test_bql_metadata.sh` - Full BQL capabilities test - [ ] Add test entry with `@@` notation to ledger - [ ] Query test entry with BQL to verify price_number access - [ ] Compare aggregation: metadata vs price notation - [ ] Test negative amounts with prices - [ ] Test zero amounts - [ ] Test multi-currency scenarios (EUR, USD with SATS prices) - [ ] Verify price database is populated correctly - [ ] Check that WEIGHT() function returns SATS value - [ ] Validate balances match current manual method --- ## Decision Matrix | Criteria | Metadata | Price Notation | Winner | |----------|----------|----------------|--------| | BQL Queryable | ❌ No | ✅ Yes | Price | | Transaction Integrity | ✅ EUR first | ✅ EUR first | Tie | | SATS Precision | ✅ Exact int | ✅ Exact (with @@) | Tie | | Migration Effort | ✅ None | ⚠️ Script needed | Metadata | | Performance | ❌ Manual loop | ✅ BQL optimized | Price | | Beancount Standard | ⚠️ Non-standard | ✅ Standard feature | Price | | Reporting Flexibility | ⚠️ Limited | ✅ Both currencies | Price | | Future Proof | ⚠️ Custom | ✅ Standard | Price | **Recommendation**: **Price Notation** if tests confirm BQL can access `price_number` --- ## Next Steps 1. **Run tests** (test_metadata_simple.sh and test_bql_metadata.sh) 2. **Review results** - Can BQL access price_number? 3. **Add test entry** with @@ notation 4. **Query test entry** - Verify aggregation works 5. **If successful**: - Write full migration script - Test on copy of production ledger - Validate balances match - Schedule migration (maintenance window) - Update balance query methods - Deploy and monitor 6. **If unsuccessful**: - Document why price notation doesn't work - Consider Beancount plugin approach - Or accept manual aggregation with caching --- **Document Status**: Awaiting test results **Next Action**: Run test scripts and report findings