update CLAUDE.md

CLAUDE.md

@@ -12,9 +12,11 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable
 
 **Double-Entry Accounting**: Every transaction affects at least two accounts. Debits must equal credits. Five account types: Assets, Liabilities, Equity, Revenue (Income), Expenses.
 
+**Fava/Beancount Backend**: Castle now uses [Fava](https://github.com/beancount/fava) as the primary accounting engine. Fava is a web interface for Beancount that provides a REST API for ledger operations. All accounting calculations (balance sheets, trial balances, account reports) are delegated to Fava/Beancount. Castle formats transactions as Beancount entries and submits them via Fava's API.
+
+**Required External Dependency**: Fava must be running as a separate service. Configure `fava_url` and `fava_ledger_slug` in Castle settings (default: `http://localhost:3333` with slug `castle-accounting`). Castle will not function without Fava.
+
 **Pure Functional Core**: The `core/` directory contains pure accounting logic independent of the database layer:
-- `core/balance.py` - Balance calculation from journal entries
-- `core/inventory.py` - Multi-currency position tracking (similar to Beancount's Inventory)
 - `core/validation.py` - Entry validation rules
 
 **Account Hierarchy**: Beancount-style hierarchical naming with `:` separators:
@@ -23,7 +25,13 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable
 - `Liabilities:Payable:User-af983632`
 - `Expenses:Food:Supplies`
 
-**Metadata System**: Each `entry_line` stores JSON metadata preserving original fiat amounts. Critical: fiat balances are calculated by summing `fiat_amount` from metadata, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records.
+**Amount Format**: Recent architecture change uses string-based amounts with currency codes:
+- SATS amounts: `"200000 SATS"`
+- Fiat amounts: `"100.00 EUR"` or `"250.00 USD"`
+- Cost basis notation: `"200000 SATS {100.00 EUR}"` (200k sats acquired at 100 EUR)
+- Parsing handles both formats via `parse_amount_string()` in views_api.py
+
+**Metadata System**: Beancount metadata format stores original fiat amounts and exchange rates as key-value pairs. Critical: fiat balances are calculated by summing fiat amounts from journal entries, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records.
 
 ### Key Files
 
@@ -33,31 +41,27 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable
 - `views.py` - Web interface routing
 - `services.py` - Settings management layer
 - `migrations.py` - Database schema migrations
-- `tasks.py` - Background tasks (daily reconciliation checks)
+- `tasks.py` - Background tasks (invoice payment monitoring)
 - `account_utils.py` - Hierarchical account naming utilities
+- `fava_client.py` - HTTP client for Fava REST API (add_entry, query, balance_sheet)
+- `beancount_format.py` - Converts Castle entries to Beancount transaction format
+- `core/validation.py` - Pure validation functions for accounting rules
 
 ### Database Schema
 
-**accounts**: Chart of accounts with hierarchical names
+**Note**: With Fava integration, Castle maintains a local cache of some data but delegates authoritative balance calculations to Beancount/Fava.
-- `user_id` field for per-user accounts (Receivable, Payable, Equity)
-- Indexed on `user_id` and `account_type`
 
-**journal_entries**: Transaction headers
+**journal_entries**: Transaction headers stored locally and synced to Fava
 - `flag` field: `*` (cleared), `!` (pending), `#` (flagged), `x` (void)
 - `meta` field: JSON storing source, tags, audit info
 - `reference` field: Links to payment_hash, invoice numbers, etc.
-
+- Enriched with `username` field when retrieved via API (added from LNbits user data)
-**entry_lines**: Individual debit/credit lines
-- Always balanced (sum of debits = sum of credits per entry)
-- `metadata` field stores fiat currency info as JSON
-- Indexed on `journal_entry_id` and `account_id`
-
-**balance_assertions**: Reconciliation checkpoints (Beancount-style)
-- Assert expected balance at a date
-- Status: pending, passed, failed
-- Used for daily reconciliation checks
 
 **extension_settings**: Castle wallet configuration (admin-only)
+- `castle_wallet_id` - The LNbits wallet used for Castle operations
+- `fava_url` - Fava service URL (default: http://localhost:3333)
+- `fava_ledger_slug` - Ledger identifier in Fava (default: castle-accounting)
+- `fava_timeout` - API request timeout in seconds
 
 **user_wallet_settings**: Per-user wallet configuration
 
@@ -96,16 +100,18 @@ DR Liabilities:Payable:User-af983632   39,669 sats
 
 ## Balance Calculation Logic
 
-**User Balance**:
+**User Balance** (calculated by Beancount via Fava):
 - Positive = Castle owes user (LIABILITY accounts have credit balance)
 - Negative = User owes Castle (ASSET accounts have debit balance)
-- Calculated from sum of all entry lines across user's accounts
+- Calculated by querying Fava for sum of all postings across user's accounts
-- Fiat balances summed from metadata, NOT converted from sats
+- Fiat balances calculated by Beancount from cost basis annotations, NOT converted from current sats
 
 **Perspective-Based UI**:
 - **User View**: Green = Castle owes them, Red = They owe Castle
 - **Castle Admin View**: Green = User owes Castle, Red = Castle owes user
 
+**Balance Retrieval**: Use `GET /api/v1/balance` which queries Fava's balance sheet or account reports for accurate, Beancount-calculated balances.
+
 ## API Endpoints
 
 ### Accounts
@@ -169,34 +175,61 @@ Use `get_or_create_user_account()` in crud.py to ensure consistency.
 
 ### Currency Handling
 
-**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`. Fiat amounts are stored in metadata as strings to preserve precision:
+**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`.
-```python
-from decimal import Decimal
 
+**New Amount String Format** (recent architecture change):
+- Input format: `"100.00 EUR"` or `"200000 SATS"`
+- Cost basis format: `"200000 SATS {100.00 EUR}"` (for recording acquisition cost)
+- Parse using `parse_amount_string(amount_str)` in views_api.py
+- Returns tuple: `(amount: Decimal, currency: str, cost_basis: Optional[tuple])`
+
+**Beancount Metadata Format**:
+```python
+# Metadata attached to individual postings (legs of a transaction)
 metadata = {
     "fiat_currency": "EUR",
-    "fiat_amount": str(Decimal("250.00")),
+    "fiat_amount": "250.00",  # String for precision
-    "fiat_rate": str(Decimal("1074.192")),
+    "fiat_rate": "1074.192",   # Sats per fiat unit
-    "btc_rate": str(Decimal("0.000931"))
 }
 ```
 
-When reading: `fiat_amount = Decimal(metadata["fiat_amount"])`
+**Important**: When creating entries to submit to Fava, use `beancount_format.format_transaction()` to ensure proper Beancount syntax.
 
-### Balance Assertions for Reconciliation
+### Fava Integration Patterns
 
-Create balance assertions to verify accounting accuracy:
+**Adding a Transaction**:
 ```python
-await create_balance_assertion(
+from .fava_client import get_fava_client
-    account_id="lightning_account_id",
+from .beancount_format import format_transaction
-    expected_balance_sats=1000000,
+from datetime import date
-    expected_balance_fiat=Decimal("500.00"),
+
-    fiat_currency="EUR",
+# Format as Beancount transaction
-    tolerance_sats=100
+entry = format_transaction(
+    date_val=date.today(),
+    flag="*",
+    narration="Groceries purchase",
+    postings=[
+        {"account": "Expenses:Food", "amount": "50000 SATS {46.50 EUR}"},
+        {"account": "Liabilities:Payable:User-abc123", "amount": "-50000 SATS"}
+    ],
+    tags=["groceries"],
+    links=["castle-entry-123"]
+)
+
+# Submit to Fava
+client = get_fava_client()
+result = await client.add_entry(entry)
+```
+
+**Querying Balances**:
+```python
+# Query user balance from Fava
+balance_result = await client.query(
+    f"SELECT sum(position) WHERE account ~ 'User-{user_id_short}'"
 )
 ```
 
-Run `POST /api/v1/tasks/daily-reconciliation` to check all assertions.
+**Important**: Always use `sanitize_link()` from beancount_format.py when creating links to ensure Beancount compatibility (only A-Z, a-z, 0-9, -, _, /, . allowed).
 
 ### Permission Model
 
@@ -213,62 +246,134 @@ This extension follows LNbits extension structure:
 - Templates in `templates/castle/`
 - Database accessed via `db = Database("ext_castle")`
 
+**Startup Requirements**:
+- `castle_start()` initializes Fava client on extension load
+- Background task `wait_for_paid_invoices()` monitors Lightning invoice payments
+- Fava service MUST be running before starting LNbits with Castle extension
+
 ## Common Tasks
 
-### Add New Expense Account
+### Add New Account in Fava
 ```python
-await create_account(CreateAccount(
+from .fava_client import get_fava_client
-    name="Expenses:Internet",
+from datetime import date
-    account_type=AccountType.EXPENSE,
+
-    description="Internet service costs"
+# Create Open directive for new account
-))
+client = get_fava_client()
+entry = {
+    "t": "Open",
+    "date": str(date.today()),
+    "account": "Expenses:Internet",
+    "currencies": ["SATS", "EUR"]
+}
+await client.add_entry(entry)
 ```
 
-### Manually Record Cash Payment
+### Record Transaction to Fava
 ```python
-await create_journal_entry(CreateJournalEntry(
+from .beancount_format import format_transaction
-    description="Cash payment for groceries",
+
-    lines=[
+entry = format_transaction(
-        CreateEntryLine(account_id=expense_account_id, amount=50000),  # Positive = debit (expense increase)
+    date_val=date.today(),
-        CreateEntryLine(account_id=cash_account_id, amount=-50000)    # Negative = credit (asset decrease)
+    flag="*",
+    narration="Internet bill payment",
+    postings=[
+        {"account": "Expenses:Internet", "amount": "50000 SATS {46.50 EUR}"},
+        {"account": "Assets:Lightning:Balance", "amount": "-50000 SATS"}
     ],
-    flag=JournalEntryFlag.CLEARED,
+    tags=["utilities"],
-    meta={"source": "manual", "payment_method": "cash"}
+    links=["castle-tx-123"]
-))
+)
+
+client = get_fava_client()
+await client.add_entry(entry)
 ```
 
-### Check User Balance
+### Query User Balance from Fava
 ```python
-balance = await get_user_balance(user_id)
+client = get_fava_client()
-print(f"Sats: {balance.balance}")  # Positive = Castle owes user
-print(f"Fiat: {balance.fiat_balances}")  # {"EUR": Decimal("36.93")}
-```
 
-### Export to Beancount (Future)
+# Query all accounts for a user
-Follow patterns in `docs/BEANCOUNT_PATTERNS.md` for implementing Beancount export. Use hierarchical account names and preserve metadata in Beancount comments.
+user_short = user_id[:8]
+query = f"SELECT account, sum(position) WHERE account ~ 'User-{user_short}' GROUP BY account"
+result = await client.query(query)
+
+# Parse result to calculate net balance
+# (sum of all user accounts across Assets, Liabilities, Equity)
+```
 
 ## Data Integrity
 
 **Critical Invariants**:
-1. Every journal entry MUST have balanced debits and credits
+1. Every transaction submitted to Fava MUST have balanced debits and credits (Beancount enforces this)
-2. Fiat balances calculated from metadata, not from converting sats
+2. Fiat amounts tracked via cost basis notation: `"AMOUNT SATS {COST FIAT}"`
 3. User accounts use `user_id` (NOT `wallet_id`) for consistency
-4. Balance assertions checked daily via background task
+4. All accounting calculations delegated to Beancount/Fava
 
 **Validation** is performed in `core/validation.py`:
-- `validate_journal_entry()` - Checks balance, minimum lines
+- Pure validation functions for entry correctness before submitting to Fava
-- `validate_balance()` - Verifies account balance calculation
-- `validate_receivable_entry()` - Ensures receivable entries are valid
-- `validate_expense_entry()` - Ensures expense entries are valid
 
-## Known Issues & Future Work
+**Beancount String Sanitization**:
+- Links must match pattern: `[A-Za-z0-9\-_/.]`
+- Use `sanitize_link()` from beancount_format.py for all links and tags
 
-See `docs/DOCUMENTATION.md` for comprehensive list. Key items:
+## Recent Architecture Changes
-- No journal entry editing/deletion (use reversing entries)
+
-- No date range filtering on list endpoints (hardcoded limit of 100)
+**Migration to Fava/Beancount** (2025):
-- No batch operations for bulk imports
+- Removed local balance calculation logic (now handled by Beancount)
-- Plugin system architecture designed but not implemented
+- Removed local `accounts` and `entry_lines` tables (Fava is source of truth)
-- Beancount export endpoint not yet implemented
+- Added `fava_client.py` and `beancount_format.py` modules
+- Changed amount format to string-based with currency codes
+- Username enrichment added to journal entries for UI display
+
+**Key Breaking Changes**:
+- All balance queries now go through Fava API
+- Account creation must use Fava's Open directive
+- Transaction format must follow Beancount syntax
+- Cost basis notation required for multi-currency tracking
+
+## Development Setup
+
+### Prerequisites
+
+1. **LNbits**: This extension must be installed in the `lnbits/extensions/` directory
+2. **Fava Service**: Must be running before starting LNbits with Castle enabled
+   ```bash
+   # Install Fava
+   pip install fava
+
+   # Create a basic Beancount file
+   touch castle-ledger.beancount
+
+   # Start Fava (default: http://localhost:3333)
+   fava castle-ledger.beancount
+   ```
+3. **Configure Castle Settings**: Set `fava_url` and `fava_ledger_slug` via settings API or UI
+
+### Running Castle Extension
+
+Castle is loaded as part of LNbits. No separate build or test commands are needed for the extension itself. Development workflow:
+
+1. Modify code in `lnbits/extensions/castle/`
+2. Restart LNbits
+3. Extension hot-reloads are supported by LNbits in development mode
+
+### Testing Transactions
+
+Use the web UI or API endpoints to create test transactions. For API testing:
+
+```bash
+# Create expense (user owes Castle)
+curl -X POST http://localhost:5000/castle/api/v1/entries/expense \
+  -H "X-Api-Key: YOUR_INVOICE_KEY" \
+  -d '{"description": "Test expense", "amount": "100.00 EUR", "account_name": "Expenses:Test"}'
+
+# Check user balance
+curl http://localhost:5000/castle/api/v1/balance \
+  -H "X-Api-Key: YOUR_INVOICE_KEY"
+```
+
+**Debugging Fava Connection**: Check logs for "Fava client initialized" message on startup. If missing, verify Fava is running and settings are correct.
 
 ## Related Documentation
 

templates/castle/index.html

@@ -81,8 +81,8 @@
               <q-item-label caption>
                 {% raw %}{{ formatDate(entry.entry_date) }}{% endraw %}
               </q-item-label>
-              <q-item-label caption v-if="entry.meta && entry.meta.user_id">
+              <q-item-label caption v-if="entry.username">
-                User: {% raw %}{{ getUserName(entry.meta.user_id) }}{% endraw %}
+                User: {% raw %}{{ entry.username }}{% endraw %}
               </q-item-label>
               <q-item-label caption v-if="entry.reference" class="text-grey">
                 Ref: {% raw %}{{ entry.reference }}{% endraw %}

views_api.py

@@ -464,8 +464,8 @@ async def api_get_user_entries(
                         reference = link_clean
                         break
 
-        # Get username from user ID (first 8 chars for display)
+        # Look up actual username using helper function
-        username = f"User-{user_id_match[:8]}" if user_id_match else None
+        username = await _get_username_from_user_id(user_id_match) if user_id_match else None
 
         entry_data = {
             "id": entry_id or e.get("entry_hash", "unknown"),
@@ -506,6 +506,107 @@ async def api_get_user_entries(
     }
 
 
+async def _get_username_from_user_id(user_id: str) -> str:
+    """
+    Helper function to get username from user_id, handling various formats.
+
+    Supports:
+    - Full UUID with dashes (36 chars): "375ec158-686c-4a21-b44d-a51cc90ef07d"
+    - Dashless UUID (32 chars): "375ec158686c4a21b44da51cc90ef07d"
+    - Partial ID (8 chars from account names): "375ec158"
+
+    Returns username or formatted fallback.
+    """
+    from lnbits.core.crud.users import get_user
+
+    logger.info(f"[USERNAME] Called with: '{user_id}' (len={len(user_id) if user_id else 0})")
+
+    if not user_id:
+        return None
+
+    # Case 1: Already in standard UUID format (36 chars with dashes)
+    if len(user_id) == 36 and user_id.count('-') == 4:
+        logger.info(f"[USERNAME] Case 1: Full UUID format")
+        user = await get_user(user_id)
+        result = user.username if user and user.username else f"User-{user_id[:8]}"
+        logger.info(f"[USERNAME] Case 1 result: '{result}'")
+        return result
+
+    # Case 2: Dashless 32-char UUID - lookup via Castle user settings
+    elif len(user_id) == 32 and '-' not in user_id:
+        logger.info(f"[USERNAME] Case 2: Dashless UUID format - looking up in Castle user settings")
+        try:
+            # Get all Castle users (which have full user_ids)
+            user_settings = await get_all_user_wallet_settings()
+
+            # Convert dashless to dashed format for comparison
+            user_id_with_dashes = f"{user_id[0:8]}-{user_id[8:12]}-{user_id[12:16]}-{user_id[16:20]}-{user_id[20:32]}"
+            logger.info(f"[USERNAME] Converted to dashed format: {user_id_with_dashes}")
+
+            # Find matching user
+            for setting in user_settings:
+                if setting.id == user_id_with_dashes:
+                    logger.info(f"[USERNAME] Found matching user in Castle settings")
+                    # Get username from LNbits
+                    user = await get_user(setting.id)
+                    result = user.username if user and user.username else f"User-{user_id[:8]}"
+                    logger.info(f"[USERNAME] Case 2 result (found): '{result}'")
+                    return result
+
+            # No matching user found
+            logger.info(f"[USERNAME] No matching user found in Castle settings")
+            result = f"User-{user_id[:8]}"
+            logger.info(f"[USERNAME] Case 2 result (not found): '{result}'")
+            return result
+
+        except Exception as e:
+            logger.error(f"Error looking up user by dashless UUID {user_id}: {e}")
+            result = f"User-{user_id[:8]}"
+            return result
+
+    # Case 3: Partial ID (8 chars from account name) - lookup via Castle user settings
+    elif len(user_id) == 8:
+        logger.info(f"[USERNAME] Case 3: Partial ID format - looking up in Castle user settings")
+        try:
+            # Get all Castle users (which have full user_ids)
+            user_settings = await get_all_user_wallet_settings()
+
+            # Find matching user by first 8 chars
+            for setting in user_settings:
+                if setting.id.startswith(user_id):
+                    logger.info(f"[USERNAME] Found full user_id: {setting.id}")
+                    # Now get username from LNbits with full ID
+                    user = await get_user(setting.id)
+                    result = user.username if user and user.username else f"User-{user_id}"
+                    logger.info(f"[USERNAME] Case 3 result (found): '{result}'")
+                    return result
+
+            # No matching user found in Castle settings
+            logger.info(f"[USERNAME] No matching user found in Castle settings")
+            result = f"User-{user_id}"
+            logger.info(f"[USERNAME] Case 3 result (not found): '{result}'")
+            return result
+
+        except Exception as e:
+            logger.error(f"Error looking up user by partial ID {user_id}: {e}")
+            result = f"User-{user_id}"
+            return result
+
+    # Case 4: Unknown format - try as-is and fall back
+    else:
+        logger.info(f"[USERNAME] Case 4: Unknown format - trying as-is")
+        try:
+            user = await get_user(user_id)
+            result = user.username if user and user.username else f"User-{user_id[:8]}"
+            logger.info(f"[USERNAME] Case 4 result: '{result}'")
+            return result
+        except Exception as e:
+            logger.info(f"[USERNAME] Case 4 exception: {e}")
+            result = f"User-{user_id[:8]}"
+            logger.info(f"[USERNAME] Case 4 fallback result: '{result}'")
+            return result
+
+
 @castle_api_router.get("/api/v1/entries/pending")
 async def api_get_pending_entries(
     wallet: WalletTypeInfo = Depends(require_admin_key),
@@ -516,7 +617,6 @@ async def api_get_pending_entries(
     Returns transactions with flag='!' from Fava/Beancount.
     """
     from lnbits.settings import settings as lnbits_settings
-    from lnbits.core.crud.users import get_user
     from .fava_client import get_fava_client
 
     if wallet.wallet.user != lnbits_settings.super_user:
@@ -552,9 +652,13 @@ async def api_get_pending_entries(
             # Extract user ID from metadata or account names
             user_id = None
             entry_meta = e.get("meta", {})
+            logger.info(f"[EXTRACT] Entry metadata keys: {list(entry_meta.keys())}")
+            logger.info(f"[EXTRACT] Entry metadata: {entry_meta}")
             if "user-id" in entry_meta:
                 user_id = entry_meta["user-id"]
+                logger.info(f"[EXTRACT] Found user-id in metadata: {user_id}")
             else:
+                logger.info(f"[EXTRACT] No user-id in metadata, checking account names")
                 # Try to extract from account names in postings
                 for posting in e.get("postings", []):
                     account = posting.get("account", "")
@@ -563,13 +667,11 @@ async def api_get_pending_entries(
                         parts = account.split("User-")
                         if len(parts) > 1:
                             user_id = parts[1]  # Short ID after User-
+                            logger.info(f"[EXTRACT] Extracted user_id from account name: {user_id}")
                             break
 
-            # Look up username
+            # Look up username using helper function
-            username = None
+            username = await _get_username_from_user_id(user_id) if user_id else None
-            if user_id:
-                user = await get_user(user_id)
-                username = user.username if user and user.username else f"User-{user_id[:8]}"
 
             # Extract amount from postings (sum of absolute values / 2)
             amount_sats = 0
@@ -1243,17 +1345,15 @@ async def api_get_all_balances(
     wallet: WalletTypeInfo = Depends(require_admin_key),
 ) -> list[dict]:
     """Get all user balances (admin/super user only) from Fava/Beancount"""
-    from lnbits.core.crud.users import get_user
     from .fava_client import get_fava_client
 
     fava = get_fava_client()
     balances = await fava.get_all_user_balances()
 
-    # Enrich with username information
+    # Enrich with username information using helper function
     result = []
     for balance in balances:
-        user = await get_user(balance["user_id"])
+        username = await _get_username_from_user_id(balance["user_id"])
-        username = user.username if user and user.username else balance["user_id"][:16] + "..."
 
         result.append({
             "user_id": balance["user_id"],