castle/docs/BQL-PRICE-NOTATION-SOLUTION.md

BQL Price Notation Solution for SATS Tracking

Date: 2025-01-12 Status: Testing Context: Explore price notation as alternative to metadata for SATS tracking

---

Problem Recap

Current approach stores SATS in metadata:

2025-11-10 * "Groceries"
  Expenses:Food                 -360.00 EUR
    sats-equivalent: 337096
  Liabilities:Payable:User-abc   360.00 EUR
    sats-equivalent: 337096

Issue: BQL cannot access metadata, so balance queries require manual aggregation.

---

Solution: Use Price Notation

Proposed Format

Post in actual transaction currency (EUR) with SATS as price:

2025-11-10 * "Groceries"
  Expenses:Food                 -360.00 EUR @@ 337096 SATS
  Liabilities:Payable:User-abc   360.00 EUR @@ 337096 SATS

What this means:

• Primary amount: -360.00 EUR (the actual transaction currency)
• Total price: 337096 SATS (the bitcoin equivalent value)
• Transaction integrity preserved (posted in EUR as it occurred)
• SATS tracked as price (queryable by BQL)

---

Price Notation Options

Option 1: Per-Unit Price (@)

Expenses:Food  -360.00 EUR @ 936.38 SATS

What it means: Each EUR is worth 936.38 SATS Total calculation: 360 × 936.38 = 337,096.8 SATS Precision: May introduce rounding (336,696.8 vs 337,096)

Option 2: Total Price (@@) ✅ RECOMMENDED

Expenses:Food  -360.00 EUR @@ 337096 SATS

What it means: Total transaction value is 337,096 SATS Total calculation: Exact 337,096 SATS (no rounding) Precision: Preserves exact SATS amount from original calculation

Why @@ is better for Castle:

• ✅ Preserves exact SATS amount (no rounding errors)
• ✅ Matches current metadata storage exactly
• ✅ Clearer intent: "this transaction equals X SATS total"

---

How BQL Handles Prices

Available Price Columns

From BQL schema:

• price_number - The numeric price amount (Decimal)
• price_currency - The currency of the price (str)
• position - Full posting (includes price)
• WEIGHT(position) - Function that returns balance weight

BQL Query Capabilities

Test Query 1: Access price directly

SELECT account, number, currency, price_number, price_currency
WHERE account ~ 'User-375ec158'
  AND price_currency = 'SATS';

Expected Result (if price notation works):

{
  "rows": [
    ["Liabilities:Payable:User-abc", "360.00", "EUR", "337096", "SATS"]
  ]
}

Test Query 2: Aggregate SATS from prices

SELECT account,
       SUM(price_number) as total_sats
WHERE account ~ 'User-'
  AND price_currency = 'SATS'
  AND flag != '!'
GROUP BY account;

Expected Result:

{
  "rows": [
    ["Liabilities:Payable:User-abc", "337096"]
  ]
}

---

Testing Plan

Step 1: Run Metadata Test

cd /home/padreug/projects/castle-beancounter
./test_metadata_simple.sh

What to look for:

• Does meta column exist in response?
• Is sats-equivalent accessible in the data?

If YES: Metadata IS accessible, simpler solution available If NO: Proceed with price notation approach

Step 2: Test Current Data Structure

./test_bql_metadata.sh

This runs 6 tests:

1. Check metadata column
2. Check price columns
3. Basic position query
4. Test WEIGHT function
5. Aggregate positions
6. Aggregate weights

What to look for:

• Which columns are available?
• What does position return for entries with prices?
• Can we access price_number and price_currency?

Step 3: Create Test Ledger Entry

Add one test entry to your ledger:

2025-01-12 * "TEST: Price notation test"
  Expenses:Test:PriceNotation   -100.00 EUR @@ 93600 SATS
  Liabilities:Payable:User-TEST  100.00 EUR @@ 93600 SATS

Then query:

curl -s "http://localhost:3333/castle-ledger/api/query" \
  -G \
  --data-urlencode "query_string=SELECT account, position, price_number, price_currency WHERE account ~ 'TEST'" \
  | jq '.'

Expected if working:

{
  "data": {
    "rows": [
      ["Expenses:Test:PriceNotation", "-100.00 EUR @@ 93600 SATS", "93600", "SATS"],
      ["Liabilities:Payable:User-TEST", "100.00 EUR @@ 93600 SATS", "93600", "SATS"]
    ],
    "types": [
      {"name": "account", "type": "str"},
      {"name": "position", "type": "Position"},
      {"name": "price_number", "type": "Decimal"},
      {"name": "price_currency", "type": "str"}
    ]
  }
}

---

Migration Strategy (If Price Notation Works)

Phase 1: Test on Sample Data

1. Create test ledger with mix of formats
2. Verify BQL can query price_number
3. Verify aggregation accuracy
4. Compare with manual method results

Phase 2: Write Migration Script

#!/usr/bin/env python3
"""
Migrate metadata sats-equivalent to price notation.

Converts:
  Expenses:Food  -360.00 EUR
    sats-equivalent: 337096

To:
  Expenses:Food  -360.00 EUR @@ 337096 SATS
"""

import re
from pathlib import Path

def migrate_entry(entry_lines):
    """Migrate a single transaction entry."""
    result = []
    current_posting = None
    sats_value = None

    for line in entry_lines:
        # Check if this is a posting line
        if re.match(r'^\s{2,}\w+:', line):
            # If we have pending sats from previous posting, add it
            if current_posting and sats_value:
                # Add @@ notation to posting
                posting = current_posting.rstrip()
                posting += f" @@ {sats_value} SATS\n"
                result.append(posting)
                current_posting = None
                sats_value = None
            else:
                if current_posting:
                    result.append(current_posting)
                current_posting = line

        # Check if this is sats-equivalent metadata
        elif 'sats-equivalent:' in line:
            match = re.search(r'sats-equivalent:\s*(-?\d+)', line)
            if match:
                sats_value = match.group(1)
            # Don't include metadata line in result

        else:
            # Other lines (date, narration, other metadata)
            if current_posting and sats_value:
                posting = current_posting.rstrip()
                posting += f" @@ {sats_value} SATS\n"
                result.append(posting)
                current_posting = None
                sats_value = None
            elif current_posting:
                result.append(current_posting)
                current_posting = None

            result.append(line)

    # Handle last posting
    if current_posting and sats_value:
        posting = current_posting.rstrip()
        posting += f" @@ {sats_value} SATS\n"
        result.append(posting)
    elif current_posting:
        result.append(current_posting)

    return result

def migrate_ledger(input_file, output_file):
    """Migrate entire ledger file."""
    with open(input_file, 'r') as f:
        lines = f.readlines()

    result = []
    current_entry = []
    in_transaction = False

    for line in lines:
        # Transaction start
        if re.match(r'^\d{4}-\d{2}-\d{2}\s+[*!]', line):
            in_transaction = True
            current_entry = [line]

        # Empty line ends transaction
        elif in_transaction and line.strip() == '':
            current_entry.append(line)
            migrated = migrate_entry(current_entry)
            result.extend(migrated)
            current_entry = []
            in_transaction = False

        # Inside transaction
        elif in_transaction:
            current_entry.append(line)

        # Outside transaction
        else:
            result.append(line)

    # Handle last entry if file doesn't end with blank line
    if current_entry:
        migrated = migrate_entry(current_entry)
        result.extend(migrated)

    with open(output_file, 'w') as f:
        f.writelines(result)

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 3:
        print("Usage: migrate_ledger.py <input.beancount> <output.beancount>")
        sys.exit(1)

    migrate_ledger(sys.argv[1], sys.argv[2])
    print(f"Migrated {sys.argv[1]} -> {sys.argv[2]}")

Phase 3: Update Balance Query Methods

Replace get_user_balance_bql() with price-based version:

async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]:
    """
    Get user balance using price notation (SATS stored as @@ price).

    Returns:
        {
            "balance": int (sats from price_number),
            "fiat_balances": {"EUR": Decimal("100.50")},
            "accounts": [{"account": "...", "sats": 150000}]
        }
    """
    user_id_prefix = user_id[:8]

    # Query: Get EUR positions with SATS prices
    query = f"""
        SELECT
            account,
            number as eur_amount,
            price_number as sats_amount
        WHERE account ~ ':User-{user_id_prefix}'
        AND (account ~ 'Payable' OR account ~ 'Receivable')
        AND flag != '!'
        AND price_currency = 'SATS'
    """

    result = await self.query_bql(query)

    total_sats = 0
    fiat_balances = {}
    accounts_map = {}

    for row in result["rows"]:
        account_name, eur_amount, sats_amount = row

        # Parse amounts
        sats = int(Decimal(sats_amount)) if sats_amount else 0
        eur = Decimal(eur_amount) if eur_amount else Decimal(0)

        total_sats += sats

        # Aggregate fiat
        if eur != 0:
            if "EUR" not in fiat_balances:
                fiat_balances["EUR"] = Decimal(0)
            fiat_balances["EUR"] += eur

        # Track per account
        if account_name not in accounts_map:
            accounts_map[account_name] = {"account": account_name, "sats": 0}
        accounts_map[account_name]["sats"] += sats

    return {
        "balance": total_sats,
        "fiat_balances": fiat_balances,
        "accounts": list(accounts_map.values())
    }

Phase 4: Validation

1. Run both methods in parallel
2. Compare results for all users
3. Log any discrepancies
4. Investigate and fix differences
5. Once validated, switch to BQL method

---

Advantages of Price Notation Approach

1. BQL Compatibility ✅

• price_number is a standard BQL column
• Can aggregate: SUM(price_number)
• Can filter: WHERE price_currency = 'SATS'

2. Transaction Integrity ✅

• Post in actual transaction currency (EUR)
• SATS as secondary value (price)
• Proper accounting: source currency preserved

3. Beancount Features ✅

• Price database automatically updated
• Can query historical EUR/SATS rates
• Reports can show both EUR and SATS values

4. Performance ✅

• BQL filters at source (no fetching all entries)
• Direct column access (no metadata parsing)
• Efficient aggregation (database-level)

5. Reporting Flexibility ✅

• Show EUR amounts in reports
• Show SATS equivalents alongside
• Filter by either currency
• Calculate gains/losses if SATS price changes

---

Potential Issues and Solutions

Issue 1: Price vs Cost Confusion

Problem: Beancount distinguishes between @ price and {} cost Solution: Always use price (@ or @@), never cost ({})

Why:

• Cost is for tracking cost basis (investments, capital gains)
• Price is for conversion rates (what we need)

Issue 2: Precision Loss with @

Problem: Per-unit price may have rounding

360.00 EUR @ 936.38 SATS  = 336,696.8 SATS (not 337,096)

Solution: Always use @@ total price

360.00 EUR @@ 337096 SATS  = 337,096 SATS (exact)

Issue 3: Negative Numbers

Problem: How to handle negative EUR with positive SATS?

-360.00 EUR @@ ??? SATS

Solution: Price is always positive (it's a rate, not an amount)

-360.00 EUR @@ 337096 SATS  ✅ Correct

The sign applies to the position, price is the conversion factor.

Issue 4: Historical Data

Problem: Existing entries have metadata, not prices

Solution: Migration script (see Phase 2)

• One-time conversion
• Validate with checksums
• Keep backup of original

---

Testing Checklist

• Run test_metadata_simple.sh - Check if metadata is accessible
• Run test_bql_metadata.sh - Full BQL capabilities test
• Add test entry with @@ notation to ledger
• Query test entry with BQL to verify price_number access
• Compare aggregation: metadata vs price notation
• Test negative amounts with prices
• Test zero amounts
• Test multi-currency scenarios (EUR, USD with SATS prices)
• Verify price database is populated correctly
• Check that WEIGHT() function returns SATS value
• Validate balances match current manual method

---

Decision Matrix

Criteria	Metadata	Price Notation	Winner
BQL Queryable	❌ No	✅ Yes	Price
Transaction Integrity	✅ EUR first	✅ EUR first	Tie
SATS Precision	✅ Exact int	✅ Exact (with @@)	Tie
Migration Effort	✅ None	⚠️ Script needed	Metadata
Performance	❌ Manual loop	✅ BQL optimized	Price
Beancount Standard	⚠️ Non-standard	✅ Standard feature	Price
Reporting Flexibility	⚠️ Limited	✅ Both currencies	Price
Future Proof	⚠️ Custom	✅ Standard	Price

Recommendation: Price Notation if tests confirm BQL can access price_number

---

Next Steps

1. Run tests (test_metadata_simple.sh and test_bql_metadata.sh)
2. Review results - Can BQL access price_number?
3. Add test entry with @@ notation
4. Query test entry - Verify aggregation works
5. If successful:
   • Write full migration script
   • Test on copy of production ledger
   • Validate balances match
   • Schedule migration (maintenance window)
   • Update balance query methods
   • Deploy and monitor
6. If unsuccessful:
   • Document why price notation doesn't work
   • Consider Beancount plugin approach
   • Or accept manual aggregation with caching

---

Document Status: Awaiting test results Next Action: Run test scripts and report findings