castle/docs/BQL-PRICE-NOTATION-SOLUTION.md

# BQL Price Notation Solution for SATS Tracking

**Date**: 2025-01-12
**Status**: Testing
**Context**: Explore price notation as alternative to metadata for SATS tracking

---

## Problem Recap

Current approach stores SATS in metadata:
```beancount
2025-11-10 * "Groceries"
  Expenses:Food                 -360.00 EUR
    sats-equivalent: 337096
  Liabilities:Payable:User-abc   360.00 EUR
    sats-equivalent: 337096
```

**Issue**: BQL cannot access metadata, so balance queries require manual aggregation.

---

## Solution: Use Price Notation

### Proposed Format

Post in actual transaction currency (EUR) with SATS as price:

```beancount
2025-11-10 * "Groceries"
  Expenses:Food                 -360.00 EUR @@ 337096 SATS
  Liabilities:Payable:User-abc   360.00 EUR @@ 337096 SATS
```

**What this means**:
- Primary amount: `-360.00 EUR` (the actual transaction currency)
- Total price: `337096 SATS` (the bitcoin equivalent value)
- Transaction integrity preserved (posted in EUR as it occurred)
- SATS tracked as price (queryable by BQL)

---

## Price Notation Options

### Option 1: Per-Unit Price (`@`)

```beancount
Expenses:Food  -360.00 EUR @ 936.38 SATS
```

**What it means**: Each EUR is worth 936.38 SATS
**Total calculation**: 360 × 936.38 = 337,096.8 SATS
**Precision**: May introduce rounding (336,696.8 vs 337,096)

### Option 2: Total Price (`@@`) ✅ RECOMMENDED

```beancount
Expenses:Food  -360.00 EUR @@ 337096 SATS
```

**What it means**: Total transaction value is 337,096 SATS
**Total calculation**: Exact 337,096 SATS (no rounding)
**Precision**: Preserves exact SATS amount from original calculation

**Why `@@` is better for Castle:**
- ✅ Preserves exact SATS amount (no rounding errors)
- ✅ Matches current metadata storage exactly
- ✅ Clearer intent: "this transaction equals X SATS total"

---

## How BQL Handles Prices

### Available Price Columns

From BQL schema:
- `price_number` - The numeric price amount (Decimal)
- `price_currency` - The currency of the price (str)
- `position` - Full posting (includes price)
- `WEIGHT(position)` - Function that returns balance weight

### BQL Query Capabilities

**Test Query 1: Access price directly**
```sql
SELECT account, number, currency, price_number, price_currency
WHERE account ~ 'User-375ec158'
  AND price_currency = 'SATS';
```

**Expected Result** (if price notation works):
```json
{
  "rows": [
    ["Liabilities:Payable:User-abc", "360.00", "EUR", "337096", "SATS"]
  ]
}
```

**Test Query 2: Aggregate SATS from prices**
```sql
SELECT account,
       SUM(price_number) as total_sats
WHERE account ~ 'User-'
  AND price_currency = 'SATS'
  AND flag != '!'
GROUP BY account;
```

**Expected Result**:
```json
{
  "rows": [
    ["Liabilities:Payable:User-abc", "337096"]
  ]
}
```

---

## Testing Plan

### Step 1: Run Metadata Test

```bash
cd /home/padreug/projects/castle-beancounter
./test_metadata_simple.sh
```

**What to look for**:
- Does `meta` column exist in response?
- Is `sats-equivalent` accessible in the data?

**If YES**: Metadata IS accessible, simpler solution available
**If NO**: Proceed with price notation approach

### Step 2: Test Current Data Structure

```bash
./test_bql_metadata.sh
```

This runs 6 tests:
1. Check metadata column
2. Check price columns
3. Basic position query
4. Test WEIGHT function
5. Aggregate positions
6. Aggregate weights

**What to look for**:
- Which columns are available?
- What does `position` return for entries with prices?
- Can we access `price_number` and `price_currency`?

### Step 3: Create Test Ledger Entry

Add one test entry to your ledger:

```beancount
2025-01-12 * "TEST: Price notation test"
  Expenses:Test:PriceNotation   -100.00 EUR @@ 93600 SATS
  Liabilities:Payable:User-TEST  100.00 EUR @@ 93600 SATS
```

Then query:
```bash
curl -s "http://localhost:3333/castle-ledger/api/query" \
  -G \
  --data-urlencode "query_string=SELECT account, position, price_number, price_currency WHERE account ~ 'TEST'" \
  | jq '.'
```

**Expected if working**:
```json
{
  "data": {
    "rows": [
      ["Expenses:Test:PriceNotation", "-100.00 EUR @@ 93600 SATS", "93600", "SATS"],
      ["Liabilities:Payable:User-TEST", "100.00 EUR @@ 93600 SATS", "93600", "SATS"]
    ],
    "types": [
      {"name": "account", "type": "str"},
      {"name": "position", "type": "Position"},
      {"name": "price_number", "type": "Decimal"},
      {"name": "price_currency", "type": "str"}
    ]
  }
}
```

---

## Migration Strategy (If Price Notation Works)

### Phase 1: Test on Sample Data

1. Create test ledger with mix of formats
2. Verify BQL can query price_number
3. Verify aggregation accuracy
4. Compare with manual method results

### Phase 2: Write Migration Script

```python
#!/usr/bin/env python3
"""
Migrate metadata sats-equivalent to price notation.

Converts:
  Expenses:Food  -360.00 EUR
    sats-equivalent: 337096

To:
  Expenses:Food  -360.00 EUR @@ 337096 SATS
"""

import re
from pathlib import Path

def migrate_entry(entry_lines):
    """Migrate a single transaction entry."""
    result = []
    current_posting = None
    sats_value = None

    for line in entry_lines:
        # Check if this is a posting line
        if re.match(r'^\s{2,}\w+:', line):
            # If we have pending sats from previous posting, add it
            if current_posting and sats_value:
                # Add @@ notation to posting
                posting = current_posting.rstrip()
                posting += f" @@ {sats_value} SATS\n"
                result.append(posting)
                current_posting = None
                sats_value = None
            else:
                if current_posting:
                    result.append(current_posting)
                current_posting = line

        # Check if this is sats-equivalent metadata
        elif 'sats-equivalent:' in line:
            match = re.search(r'sats-equivalent:\s*(-?\d+)', line)
            if match:
                sats_value = match.group(1)
            # Don't include metadata line in result

        else:
            # Other lines (date, narration, other metadata)
            if current_posting and sats_value:
                posting = current_posting.rstrip()
                posting += f" @@ {sats_value} SATS\n"
                result.append(posting)
                current_posting = None
                sats_value = None
            elif current_posting:
                result.append(current_posting)
                current_posting = None

            result.append(line)

    # Handle last posting
    if current_posting and sats_value:
        posting = current_posting.rstrip()
        posting += f" @@ {sats_value} SATS\n"
        result.append(posting)
    elif current_posting:
        result.append(current_posting)

    return result

def migrate_ledger(input_file, output_file):
    """Migrate entire ledger file."""
    with open(input_file, 'r') as f:
        lines = f.readlines()

    result = []
    current_entry = []
    in_transaction = False

    for line in lines:
        # Transaction start
        if re.match(r'^\d{4}-\d{2}-\d{2}\s+[*!]', line):
            in_transaction = True
            current_entry = [line]

        # Empty line ends transaction
        elif in_transaction and line.strip() == '':
            current_entry.append(line)
            migrated = migrate_entry(current_entry)
            result.extend(migrated)
            current_entry = []
            in_transaction = False

        # Inside transaction
        elif in_transaction:
            current_entry.append(line)

        # Outside transaction
        else:
            result.append(line)

    # Handle last entry if file doesn't end with blank line
    if current_entry:
        migrated = migrate_entry(current_entry)
        result.extend(migrated)

    with open(output_file, 'w') as f:
        f.writelines(result)

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 3:
        print("Usage: migrate_ledger.py <input.beancount> <output.beancount>")
        sys.exit(1)

    migrate_ledger(sys.argv[1], sys.argv[2])
    print(f"Migrated {sys.argv[1]} -> {sys.argv[2]}")
```

### Phase 3: Update Balance Query Methods

Replace `get_user_balance_bql()` with price-based version:

```python
async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]:
    """
    Get user balance using price notation (SATS stored as @@ price).

    Returns:
        {
            "balance": int (sats from price_number),
            "fiat_balances": {"EUR": Decimal("100.50")},
            "accounts": [{"account": "...", "sats": 150000}]
        }
    """
    user_id_prefix = user_id[:8]

    # Query: Get EUR positions with SATS prices
    query = f"""
        SELECT
            account,
            number as eur_amount,
            price_number as sats_amount
        WHERE account ~ ':User-{user_id_prefix}'
        AND (account ~ 'Payable' OR account ~ 'Receivable')
        AND flag != '!'
        AND price_currency = 'SATS'
    """

    result = await self.query_bql(query)

    total_sats = 0
    fiat_balances = {}
    accounts_map = {}

    for row in result["rows"]:
        account_name, eur_amount, sats_amount = row

        # Parse amounts
        sats = int(Decimal(sats_amount)) if sats_amount else 0
        eur = Decimal(eur_amount) if eur_amount else Decimal(0)

        total_sats += sats

        # Aggregate fiat
        if eur != 0:
            if "EUR" not in fiat_balances:
                fiat_balances["EUR"] = Decimal(0)
            fiat_balances["EUR"] += eur

        # Track per account
        if account_name not in accounts_map:
            accounts_map[account_name] = {"account": account_name, "sats": 0}
        accounts_map[account_name]["sats"] += sats

    return {
        "balance": total_sats,
        "fiat_balances": fiat_balances,
        "accounts": list(accounts_map.values())
    }
```

### Phase 4: Validation

1. Run both methods in parallel
2. Compare results for all users
3. Log any discrepancies
4. Investigate and fix differences
5. Once validated, switch to BQL method

---

## Advantages of Price Notation Approach

### 1. BQL Compatibility ✅
- `price_number` is a standard BQL column
- Can aggregate: `SUM(price_number)`
- Can filter: `WHERE price_currency = 'SATS'`

### 2. Transaction Integrity ✅
- Post in actual transaction currency (EUR)
- SATS as secondary value (price)
- Proper accounting: source currency preserved

### 3. Beancount Features ✅
- Price database automatically updated
- Can query historical EUR/SATS rates
- Reports can show both EUR and SATS values

### 4. Performance ✅
- BQL filters at source (no fetching all entries)
- Direct column access (no metadata parsing)
- Efficient aggregation (database-level)

### 5. Reporting Flexibility ✅
- Show EUR amounts in reports
- Show SATS equivalents alongside
- Filter by either currency
- Calculate gains/losses if SATS price changes

---

## Potential Issues and Solutions

### Issue 1: Price vs Cost Confusion

**Problem**: Beancount distinguishes between `@` price and `{}` cost
**Solution**: Always use price (`@` or `@@`), never cost (`{}`)

**Why**:
- Cost is for tracking cost basis (investments, capital gains)
- Price is for conversion rates (what we need)

### Issue 2: Precision Loss with `@`

**Problem**: Per-unit price may have rounding
```beancount
360.00 EUR @ 936.38 SATS  = 336,696.8 SATS (not 337,096)
```

**Solution**: Always use `@@` total price
```beancount
360.00 EUR @@ 337096 SATS  = 337,096 SATS (exact)
```

### Issue 3: Negative Numbers

**Problem**: How to handle negative EUR with positive SATS?
```beancount
-360.00 EUR @@ ??? SATS
```

**Solution**: Price is always positive (it's a rate, not an amount)
```beancount
-360.00 EUR @@ 337096 SATS  ✅ Correct
```

The sign applies to the position, price is the conversion factor.

### Issue 4: Historical Data

**Problem**: Existing entries have metadata, not prices

**Solution**: Migration script (see Phase 2)
- One-time conversion
- Validate with checksums
- Keep backup of original

---

## Testing Checklist

- [ ] Run `test_metadata_simple.sh` - Check if metadata is accessible
- [ ] Run `test_bql_metadata.sh` - Full BQL capabilities test
- [ ] Add test entry with `@@` notation to ledger
- [ ] Query test entry with BQL to verify price_number access
- [ ] Compare aggregation: metadata vs price notation
- [ ] Test negative amounts with prices
- [ ] Test zero amounts
- [ ] Test multi-currency scenarios (EUR, USD with SATS prices)
- [ ] Verify price database is populated correctly
- [ ] Check that WEIGHT() function returns SATS value
- [ ] Validate balances match current manual method

---

## Decision Matrix

| Criteria | Metadata | Price Notation | Winner |
|----------|----------|----------------|--------|
| BQL Queryable | ❌ No | ✅ Yes | Price |
| Transaction Integrity | ✅ EUR first | ✅ EUR first | Tie |
| SATS Precision | ✅ Exact int | ✅ Exact (with @@) | Tie |
| Migration Effort | ✅ None | ⚠️ Script needed | Metadata |
| Performance | ❌ Manual loop | ✅ BQL optimized | Price |
| Beancount Standard | ⚠️ Non-standard | ✅ Standard feature | Price |
| Reporting Flexibility | ⚠️ Limited | ✅ Both currencies | Price |
| Future Proof | ⚠️ Custom | ✅ Standard | Price |

**Recommendation**: **Price Notation** if tests confirm BQL can access `price_number`

---

## Next Steps

1. **Run tests** (test_metadata_simple.sh and test_bql_metadata.sh)
2. **Review results** - Can BQL access price_number?
3. **Add test entry** with @@ notation
4. **Query test entry** - Verify aggregation works
5. **If successful**:
   - Write full migration script
   - Test on copy of production ledger
   - Validate balances match
   - Schedule migration (maintenance window)
   - Update balance query methods
   - Deploy and monitor
6. **If unsuccessful**:
   - Document why price notation doesn't work
   - Consider Beancount plugin approach
   - Or accept manual aggregation with caching

---

**Document Status**: Awaiting test results
**Next Action**: Run test scripts and report findings