castle/docs/SATS-EQUIVALENT-METADATA.md

# SATS-Equivalent Metadata Field

**Date**: 2025-01-12
**Status**: Current Architecture
**Location**: Beancount posting metadata

---

## Overview

The `sats-equivalent` metadata field is Castle's solution for **dual-currency tracking** in a fiat-denominated ledger. It preserves Bitcoin (satoshi) amounts alongside fiat currency amounts without violating accounting principles or creating multi-currency complexity in position balances.

### Quick Summary

- **Purpose**: Track Bitcoin/Lightning amounts in a EUR-denominated ledger
- **Location**: Beancount posting metadata (not position amounts)
- **Format**: String containing absolute satoshi amount (e.g., `"337096"`)
- **Primary Use**: Calculate user balances in satoshis (Castle's primary currency)
- **Key Principle**: Satoshis are for reference; EUR is the actual transaction currency

---

## The Problem: Dual-Currency Tracking

Castle needs to track both:
1. **Fiat amounts** (EUR, USD) - The actual transaction currency
2. **Bitcoin amounts** (satoshis) - The Lightning Network settlement currency

### Why Not Just Use SATS as Position Amounts?

**Accounting Reality**: When a user pays €36.93 cash for groceries, the transaction is denominated in EUR, not Bitcoin. Recording it as Bitcoin would:
- ❌ Misrepresent the actual transaction
- ❌ Create exchange rate volatility issues
- ❌ Complicate traditional accounting reconciliation
- ❌ Make fiat-based reporting difficult

**Castle's Philosophy**: Record transactions in their **actual currency**, with Bitcoin as supplementary data.

---

## Architecture: EUR-Primary Format

### Current Ledger Format

```beancount
2025-11-10 * "Groceries (36.93 EUR)" #expense-entry
  Expenses:Food:Supplies              36.93 EUR
    sats-equivalent: "39669"
    reference: "cash-payment-abc123"
  Liabilities:Payable:User-5987ae95  -36.93 EUR
    sats-equivalent: "39669"
```

**Key Components:**
- **Position Amount**: `36.93 EUR` - The actual transaction amount
- **Metadata**: `sats-equivalent: "39669"` - The Bitcoin equivalent at time of transaction
- **Sign**: The sign (debit/credit) is on the EUR amount; sats-equivalent is always absolute value

### How It's Created

In `views_api.py:839`:

```python
# If fiat currency is provided, use EUR-based format
if fiat_currency and fiat_amount:
    # EUR-based posting (current architecture)
    posting_metadata["sats-equivalent"] = str(abs(line.amount))

    # Apply the sign from line.amount to fiat_amount
    signed_fiat_amount = fiat_amount if line.amount >= 0 else -fiat_amount

    posting = {
        "account": account.name,
        "amount": f"{signed_fiat_amount:.2f} {fiat_currency}",
        "meta": posting_metadata if posting_metadata else None
    }
```

**Critical Details:**
- `line.amount` is always in satoshis internally
- The sign (debit/credit) transfers to the fiat amount
- `sats-equivalent` stores the **absolute value** of the satoshi amount
- Sign interpretation depends on account type (Asset/Liability/etc.)

---

## Usage: Balance Calculation

### Primary Use Case: User Balances

Castle's core function is tracking **who owes whom** in satoshis. The `sats-equivalent` metadata enables this.

**Flow** (`fava_client.py:220-248`):

```python
# Parse posting amount (EUR/USD)
fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', amount_str)
if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'):
    fiat_amount = Decimal(fiat_match.group(1))
    fiat_currency = fiat_match.group(2)

    # Track fiat balance
    fiat_balances[fiat_currency] += fiat_amount

    # Extract SATS equivalent from metadata
    posting_meta = posting.get("meta", {})
    sats_equiv = posting_meta.get("sats-equivalent")
    if sats_equiv:
        # Apply the sign from fiat_amount to sats_equiv
        sats_amount = int(sats_equiv) if fiat_amount > 0 else -int(sats_equiv)
        total_sats += sats_amount
```

**Sign Interpretation:**
- EUR amount is `36.93` (positive/debit) → sats is `+39669`
- EUR amount is `-36.93` (negative/credit) → sats is `-39669`

### Secondary Use: Journal Entry Display

When displaying transactions to users (`views_api.py:747-751`):

```python
# Extract sats equivalent from metadata
posting_meta = first_posting.get("meta", {})
sats_equiv = posting_meta.get("sats-equivalent")
if sats_equiv:
    amount_sats = abs(int(sats_equiv))
```

This allows the UI to show both EUR and SATS amounts for each transaction.

---

## Why Metadata Instead of Positions?

### The BQL Limitation

Beancount Query Language (BQL) **cannot access metadata**. This means:

```sql
-- ✅ This works (queries position amounts):
SELECT account, sum(position) WHERE account ~ 'User-5987ae95'
-- Returns: EUR positions (not useful for satoshi balances)

-- ❌ This is NOT possible:
SELECT account, sum(meta["sats-equivalent"]) WHERE account ~ 'User-5987ae95'
-- Error: BQL cannot access metadata
```

### Why Castle Accepts This Trade-off

**Performance Analysis** (see `docs/BQL-BALANCE-QUERIES.md`):
1. **Caching solves the bottleneck**: 60-80% performance improvement from caching account/permission lookups
2. **Iteration is necessary anyway**: Even with BQL, we'd need to iterate postings to access metadata
3. **Manual aggregation is fast**: The actual summation is not the bottleneck
4. **Database queries are the bottleneck**: Solved by Phase 1 caching, not BQL

**Architectural Correctness > Query Performance**:
- ✅ Transactions recorded in their actual currency
- ✅ No artificial multi-currency positions
- ✅ Clean accounting reconciliation
- ✅ Exchange rate changes don't affect historical records

---

## Alternative Considered: Price Notation

### Price Notation Format (Not Implemented)

```beancount
2025-11-10 * "Groceries"
  Expenses:Food                 -360.00 EUR @@ 337096 SATS
  Liabilities:Payable:User-abc   360.00 EUR @@ 337096 SATS
```

**Pros:**
- ✅ BQL can query prices (enables BQL aggregation)
- ✅ Standard Beancount syntax
- ✅ SATS trackable via price database

**Cons:**
- ❌ Semantically incorrect: `@@` means "total price paid", not "equivalent value"
- ❌ Implies currency conversion happened (it didn't)
- ❌ Confuses future readers about transaction nature
- ❌ Complicates Beancount's price database

**Decision**: Metadata is more semantically correct for "reference value" than price notation.

See `docs/BQL-PRICE-NOTATION-SOLUTION.md` for full analysis.

---

## Data Flow Example

### User Adds Expense

**User Action**: "I paid €36.93 cash for groceries"

**Castle's Internal Representation**:
```python
# User provides or Castle calculates:
fiat_amount = Decimal("36.93")  # EUR
fiat_currency = "EUR"
amount_sats = 39669  # Calculated from exchange rate

# Create journal entry line:
line = CreateEntryLine(
    account_id=expense_account.id,
    amount=amount_sats,  # Internal: always satoshis
    metadata={
        "fiat_currency": "EUR",
        "fiat_amount": "36.93"
    }
)
```

**Beancount Entry Created** (`views_api.py:835-849`):
```beancount
2025-11-10 * "Groceries (36.93 EUR)" #expense-entry
  Expenses:Food:Supplies              36.93 EUR
    sats-equivalent: "39669"
  Liabilities:Payable:User-5987ae95  -36.93 EUR
    sats-equivalent: "39669"
```

**Balance Calculation** (`fava_client.py:get_user_balance`):
```python
# Iterate all postings for user accounts
# For each posting:
#   - Parse EUR amount: -36.93 EUR (credit to liability)
#   - Extract sats-equivalent: "39669"
#   - Apply sign: -36.93 is negative → sats = -39669
#   - Accumulate: user_balance_sats += -39669

# Result: negative balance = Castle owes user
```

**User Balance Response**:
```json
{
  "user_id": "5987ae95",
  "balance": -39669,  // Castle owes user 39,669 sats
  "fiat_balances": {
    "EUR": "-36.93"   // Castle owes user €36.93
  }
}
```

---

## Implementation Details

### Where It's Set

**Primary Location**: `views_api.py:835-849` (Creating journal entries)

All EUR-based postings get `sats-equivalent` metadata:
- Expense entries (user adds liability)
- Receivable entries (admin records what user owes)
- Revenue entries (direct income)
- Payment entries (settling balances)

### Where It's Read

**Primary Location**: `fava_client.py:239-247` (Balance calculation)

Used in:
1. `get_user_balance()` - Calculate individual user balance
2. `get_all_user_balances()` - Calculate all user balances
3. `get_journal_entries()` - Display transaction amounts

### Data Type and Format

- **Type**: String (Beancount metadata values must be strings or numbers)
- **Format**: Absolute value, no sign, no decimal point
- **Examples**:
  - ✅ `"39669"` (correct)
  - ✅ `"1000000"` (1M sats)
  - ❌ `"-39669"` (incorrect: sign goes on EUR amount)
  - ❌ `"396.69"` (incorrect: satoshis are integers)

---

## Key Principles

### 1. Record in Transaction Currency

```beancount
# ✅ CORRECT: User paid EUR, record in EUR
Expenses:Food  36.93 EUR
  sats-equivalent: "39669"

# ❌ WRONG: Recording Bitcoin when user paid cash
Expenses:Food  39669 SATS {36.93 EUR}
```

### 2. Preserve Historical Values

The `sats-equivalent` is the **exact satoshi amount at transaction time**. It does NOT change when exchange rates change.

**Example:**
- 2025-11-10: User pays €36.93 → 39,669 sats (rate: 1074.19 sats/EUR)
- 2025-11-15: Exchange rate changes to 1100 sats/EUR
- **Metadata stays**: `sats-equivalent: "39669"` ✅
- **If we used current rate**: Would become 40,623 sats ❌

### 3. Separate Fiat and Sats Balances

Castle tracks TWO independent balances:
- **Satoshi balance**: Sum of `sats-equivalent` metadata (primary)
- **Fiat balances**: Sum of EUR/USD position amounts (secondary)

These are calculated independently and don't cross-convert.

### 4. Absolute Values in Metadata

The sign (debit/credit) lives on the position amount, NOT the metadata.

```beancount
# Debit (expense increases):
Expenses:Food  36.93 EUR        # Positive
  sats-equivalent: "39669"      # Absolute value

# Credit (liability increases):
Liabilities:Payable  -36.93 EUR # Negative
  sats-equivalent: "39669"      # Same absolute value
```

---

## Migration Path

### Future: If We Change to SATS-Primary Format

**Hypothetical future format:**
```beancount
; SATS as position, EUR as cost:
2025-11-10 * "Groceries"
  Expenses:Food                 39669 SATS {36.93 EUR}
  Liabilities:Payable:User-abc -39669 SATS {36.93 EUR}
```

**Benefits:**
- ✅ BQL can query SATS directly
- ✅ No metadata parsing needed
- ✅ Standard Beancount cost accounting

**Migration Script** (conceptual):
```python
# Read all postings with sats-equivalent metadata
# For each posting:
#   - Extract sats from metadata
#   - Extract EUR from position
#   - Rewrite as: "<sats> SATS {<eur> EUR}"
```

**Decision**: Not implementing now because:
1. Current architecture is semantically correct
2. Performance is acceptable with caching
3. Migration would break existing tooling
4. EUR-primary aligns with accounting reality

---

## Related Documentation

- `docs/BQL-BALANCE-QUERIES.md` - Why BQL can't query metadata and performance analysis
- `docs/BQL-PRICE-NOTATION-SOLUTION.md` - Alternative using price notation (not implemented)
- `beancount_format.py` - Functions that create entries with sats-equivalent metadata
- `fava_client.py:get_user_balance()` - How metadata is parsed for balance calculation

---

## Technical Summary

**Field**: `sats-equivalent`
**Type**: Metadata (string)
**Location**: Beancount posting metadata
**Format**: Absolute satoshi amount as string (e.g., `"39669"`)
**Purpose**: Track Bitcoin equivalent of fiat transactions
**Primary Use**: Calculate user satoshi balances
**Sign Handling**: Inherits from position amount (EUR/USD)
**Queryable via BQL**: ❌ No (BQL cannot access metadata)
**Performance**: ✅ Acceptable with caching (60-80% improvement)
**Architectural Status**: ✅ Current production format
**Future Migration**: Possible to SATS-primary if needed