diff --git a/CLAUDE.md b/CLAUDE.md index a80a185..3086441 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,9 +12,11 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable **Double-Entry Accounting**: Every transaction affects at least two accounts. Debits must equal credits. Five account types: Assets, Liabilities, Equity, Revenue (Income), Expenses. +**Fava/Beancount Backend**: Castle now uses [Fava](https://github.com/beancount/fava) as the primary accounting engine. Fava is a web interface for Beancount that provides a REST API for ledger operations. All accounting calculations (balance sheets, trial balances, account reports) are delegated to Fava/Beancount. Castle formats transactions as Beancount entries and submits them via Fava's API. + +**Required External Dependency**: Fava must be running as a separate service. Configure `fava_url` and `fava_ledger_slug` in Castle settings (default: `http://localhost:3333` with slug `castle-accounting`). Castle will not function without Fava. + **Pure Functional Core**: The `core/` directory contains pure accounting logic independent of the database layer: -- `core/balance.py` - Balance calculation from journal entries -- `core/inventory.py` - Multi-currency position tracking (similar to Beancount's Inventory) - `core/validation.py` - Entry validation rules **Account Hierarchy**: Beancount-style hierarchical naming with `:` separators: @@ -23,7 +25,13 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable - `Liabilities:Payable:User-af983632` - `Expenses:Food:Supplies` -**Metadata System**: Each `entry_line` stores JSON metadata preserving original fiat amounts. Critical: fiat balances are calculated by summing `fiat_amount` from metadata, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records. +**Amount Format**: Recent architecture change uses string-based amounts with currency codes: +- SATS amounts: `"200000 SATS"` +- Fiat amounts: `"100.00 EUR"` or `"250.00 USD"` +- Cost basis notation: `"200000 SATS {100.00 EUR}"` (200k sats acquired at 100 EUR) +- Parsing handles both formats via `parse_amount_string()` in views_api.py + +**Metadata System**: Beancount metadata format stores original fiat amounts and exchange rates as key-value pairs. Critical: fiat balances are calculated by summing fiat amounts from journal entries, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records. ### Key Files @@ -33,31 +41,27 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable - `views.py` - Web interface routing - `services.py` - Settings management layer - `migrations.py` - Database schema migrations -- `tasks.py` - Background tasks (daily reconciliation checks) +- `tasks.py` - Background tasks (invoice payment monitoring) - `account_utils.py` - Hierarchical account naming utilities +- `fava_client.py` - HTTP client for Fava REST API (add_entry, query, balance_sheet) +- `beancount_format.py` - Converts Castle entries to Beancount transaction format +- `core/validation.py` - Pure validation functions for accounting rules ### Database Schema -**accounts**: Chart of accounts with hierarchical names -- `user_id` field for per-user accounts (Receivable, Payable, Equity) -- Indexed on `user_id` and `account_type` +**Note**: With Fava integration, Castle maintains a local cache of some data but delegates authoritative balance calculations to Beancount/Fava. -**journal_entries**: Transaction headers +**journal_entries**: Transaction headers stored locally and synced to Fava - `flag` field: `*` (cleared), `!` (pending), `#` (flagged), `x` (void) - `meta` field: JSON storing source, tags, audit info - `reference` field: Links to payment_hash, invoice numbers, etc. - -**entry_lines**: Individual debit/credit lines -- Always balanced (sum of debits = sum of credits per entry) -- `metadata` field stores fiat currency info as JSON -- Indexed on `journal_entry_id` and `account_id` - -**balance_assertions**: Reconciliation checkpoints (Beancount-style) -- Assert expected balance at a date -- Status: pending, passed, failed -- Used for daily reconciliation checks +- Enriched with `username` field when retrieved via API (added from LNbits user data) **extension_settings**: Castle wallet configuration (admin-only) +- `castle_wallet_id` - The LNbits wallet used for Castle operations +- `fava_url` - Fava service URL (default: http://localhost:3333) +- `fava_ledger_slug` - Ledger identifier in Fava (default: castle-accounting) +- `fava_timeout` - API request timeout in seconds **user_wallet_settings**: Per-user wallet configuration @@ -96,16 +100,18 @@ DR Liabilities:Payable:User-af983632 39,669 sats ## Balance Calculation Logic -**User Balance**: +**User Balance** (calculated by Beancount via Fava): - Positive = Castle owes user (LIABILITY accounts have credit balance) - Negative = User owes Castle (ASSET accounts have debit balance) -- Calculated from sum of all entry lines across user's accounts -- Fiat balances summed from metadata, NOT converted from sats +- Calculated by querying Fava for sum of all postings across user's accounts +- Fiat balances calculated by Beancount from cost basis annotations, NOT converted from current sats **Perspective-Based UI**: - **User View**: Green = Castle owes them, Red = They owe Castle - **Castle Admin View**: Green = User owes Castle, Red = Castle owes user +**Balance Retrieval**: Use `GET /api/v1/balance` which queries Fava's balance sheet or account reports for accurate, Beancount-calculated balances. + ## API Endpoints ### Accounts @@ -169,34 +175,61 @@ Use `get_or_create_user_account()` in crud.py to ensure consistency. ### Currency Handling -**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`. Fiat amounts are stored in metadata as strings to preserve precision: -```python -from decimal import Decimal +**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`. +**New Amount String Format** (recent architecture change): +- Input format: `"100.00 EUR"` or `"200000 SATS"` +- Cost basis format: `"200000 SATS {100.00 EUR}"` (for recording acquisition cost) +- Parse using `parse_amount_string(amount_str)` in views_api.py +- Returns tuple: `(amount: Decimal, currency: str, cost_basis: Optional[tuple])` + +**Beancount Metadata Format**: +```python +# Metadata attached to individual postings (legs of a transaction) metadata = { "fiat_currency": "EUR", - "fiat_amount": str(Decimal("250.00")), - "fiat_rate": str(Decimal("1074.192")), - "btc_rate": str(Decimal("0.000931")) + "fiat_amount": "250.00", # String for precision + "fiat_rate": "1074.192", # Sats per fiat unit } ``` -When reading: `fiat_amount = Decimal(metadata["fiat_amount"])` +**Important**: When creating entries to submit to Fava, use `beancount_format.format_transaction()` to ensure proper Beancount syntax. -### Balance Assertions for Reconciliation +### Fava Integration Patterns -Create balance assertions to verify accounting accuracy: +**Adding a Transaction**: ```python -await create_balance_assertion( - account_id="lightning_account_id", - expected_balance_sats=1000000, - expected_balance_fiat=Decimal("500.00"), - fiat_currency="EUR", - tolerance_sats=100 +from .fava_client import get_fava_client +from .beancount_format import format_transaction +from datetime import date + +# Format as Beancount transaction +entry = format_transaction( + date_val=date.today(), + flag="*", + narration="Groceries purchase", + postings=[ + {"account": "Expenses:Food", "amount": "50000 SATS {46.50 EUR}"}, + {"account": "Liabilities:Payable:User-abc123", "amount": "-50000 SATS"} + ], + tags=["groceries"], + links=["castle-entry-123"] +) + +# Submit to Fava +client = get_fava_client() +result = await client.add_entry(entry) +``` + +**Querying Balances**: +```python +# Query user balance from Fava +balance_result = await client.query( + f"SELECT sum(position) WHERE account ~ 'User-{user_id_short}'" ) ``` -Run `POST /api/v1/tasks/daily-reconciliation` to check all assertions. +**Important**: Always use `sanitize_link()` from beancount_format.py when creating links to ensure Beancount compatibility (only A-Z, a-z, 0-9, -, _, /, . allowed). ### Permission Model @@ -213,62 +246,134 @@ This extension follows LNbits extension structure: - Templates in `templates/castle/` - Database accessed via `db = Database("ext_castle")` +**Startup Requirements**: +- `castle_start()` initializes Fava client on extension load +- Background task `wait_for_paid_invoices()` monitors Lightning invoice payments +- Fava service MUST be running before starting LNbits with Castle extension + ## Common Tasks -### Add New Expense Account +### Add New Account in Fava ```python -await create_account(CreateAccount( - name="Expenses:Internet", - account_type=AccountType.EXPENSE, - description="Internet service costs" -)) +from .fava_client import get_fava_client +from datetime import date + +# Create Open directive for new account +client = get_fava_client() +entry = { + "t": "Open", + "date": str(date.today()), + "account": "Expenses:Internet", + "currencies": ["SATS", "EUR"] +} +await client.add_entry(entry) ``` -### Manually Record Cash Payment +### Record Transaction to Fava ```python -await create_journal_entry(CreateJournalEntry( - description="Cash payment for groceries", - lines=[ - CreateEntryLine(account_id=expense_account_id, amount=50000), # Positive = debit (expense increase) - CreateEntryLine(account_id=cash_account_id, amount=-50000) # Negative = credit (asset decrease) +from .beancount_format import format_transaction + +entry = format_transaction( + date_val=date.today(), + flag="*", + narration="Internet bill payment", + postings=[ + {"account": "Expenses:Internet", "amount": "50000 SATS {46.50 EUR}"}, + {"account": "Assets:Lightning:Balance", "amount": "-50000 SATS"} ], - flag=JournalEntryFlag.CLEARED, - meta={"source": "manual", "payment_method": "cash"} -)) + tags=["utilities"], + links=["castle-tx-123"] +) + +client = get_fava_client() +await client.add_entry(entry) ``` -### Check User Balance +### Query User Balance from Fava ```python -balance = await get_user_balance(user_id) -print(f"Sats: {balance.balance}") # Positive = Castle owes user -print(f"Fiat: {balance.fiat_balances}") # {"EUR": Decimal("36.93")} -``` +client = get_fava_client() -### Export to Beancount (Future) -Follow patterns in `docs/BEANCOUNT_PATTERNS.md` for implementing Beancount export. Use hierarchical account names and preserve metadata in Beancount comments. +# Query all accounts for a user +user_short = user_id[:8] +query = f"SELECT account, sum(position) WHERE account ~ 'User-{user_short}' GROUP BY account" +result = await client.query(query) + +# Parse result to calculate net balance +# (sum of all user accounts across Assets, Liabilities, Equity) +``` ## Data Integrity **Critical Invariants**: -1. Every journal entry MUST have balanced debits and credits -2. Fiat balances calculated from metadata, not from converting sats +1. Every transaction submitted to Fava MUST have balanced debits and credits (Beancount enforces this) +2. Fiat amounts tracked via cost basis notation: `"AMOUNT SATS {COST FIAT}"` 3. User accounts use `user_id` (NOT `wallet_id`) for consistency -4. Balance assertions checked daily via background task +4. All accounting calculations delegated to Beancount/Fava **Validation** is performed in `core/validation.py`: -- `validate_journal_entry()` - Checks balance, minimum lines -- `validate_balance()` - Verifies account balance calculation -- `validate_receivable_entry()` - Ensures receivable entries are valid -- `validate_expense_entry()` - Ensures expense entries are valid +- Pure validation functions for entry correctness before submitting to Fava -## Known Issues & Future Work +**Beancount String Sanitization**: +- Links must match pattern: `[A-Za-z0-9\-_/.]` +- Use `sanitize_link()` from beancount_format.py for all links and tags -See `docs/DOCUMENTATION.md` for comprehensive list. Key items: -- No journal entry editing/deletion (use reversing entries) -- No date range filtering on list endpoints (hardcoded limit of 100) -- No batch operations for bulk imports -- Plugin system architecture designed but not implemented -- Beancount export endpoint not yet implemented +## Recent Architecture Changes + +**Migration to Fava/Beancount** (2025): +- Removed local balance calculation logic (now handled by Beancount) +- Removed local `accounts` and `entry_lines` tables (Fava is source of truth) +- Added `fava_client.py` and `beancount_format.py` modules +- Changed amount format to string-based with currency codes +- Username enrichment added to journal entries for UI display + +**Key Breaking Changes**: +- All balance queries now go through Fava API +- Account creation must use Fava's Open directive +- Transaction format must follow Beancount syntax +- Cost basis notation required for multi-currency tracking + +## Development Setup + +### Prerequisites + +1. **LNbits**: This extension must be installed in the `lnbits/extensions/` directory +2. **Fava Service**: Must be running before starting LNbits with Castle enabled + ```bash + # Install Fava + pip install fava + + # Create a basic Beancount file + touch castle-ledger.beancount + + # Start Fava (default: http://localhost:3333) + fava castle-ledger.beancount + ``` +3. **Configure Castle Settings**: Set `fava_url` and `fava_ledger_slug` via settings API or UI + +### Running Castle Extension + +Castle is loaded as part of LNbits. No separate build or test commands are needed for the extension itself. Development workflow: + +1. Modify code in `lnbits/extensions/castle/` +2. Restart LNbits +3. Extension hot-reloads are supported by LNbits in development mode + +### Testing Transactions + +Use the web UI or API endpoints to create test transactions. For API testing: + +```bash +# Create expense (user owes Castle) +curl -X POST http://localhost:5000/castle/api/v1/entries/expense \ + -H "X-Api-Key: YOUR_INVOICE_KEY" \ + -d '{"description": "Test expense", "amount": "100.00 EUR", "account_name": "Expenses:Test"}' + +# Check user balance +curl http://localhost:5000/castle/api/v1/balance \ + -H "X-Api-Key: YOUR_INVOICE_KEY" +``` + +**Debugging Fava Connection**: Check logs for "Fava client initialized" message on startup. If missing, verify Fava is running and settings are correct. ## Related Documentation diff --git a/templates/castle/index.html b/templates/castle/index.html index 6aff4f5..367f54e 100644 --- a/templates/castle/index.html +++ b/templates/castle/index.html @@ -81,8 +81,8 @@ {% raw %}{{ formatDate(entry.entry_date) }}{% endraw %} - - User: {% raw %}{{ getUserName(entry.meta.user_id) }}{% endraw %} + + User: {% raw %}{{ entry.username }}{% endraw %} Ref: {% raw %}{{ entry.reference }}{% endraw %} diff --git a/views_api.py b/views_api.py index a020cfa..548daac 100644 --- a/views_api.py +++ b/views_api.py @@ -464,8 +464,8 @@ async def api_get_user_entries( reference = link_clean break - # Get username from user ID (first 8 chars for display) - username = f"User-{user_id_match[:8]}" if user_id_match else None + # Look up actual username using helper function + username = await _get_username_from_user_id(user_id_match) if user_id_match else None entry_data = { "id": entry_id or e.get("entry_hash", "unknown"), @@ -506,6 +506,107 @@ async def api_get_user_entries( } +async def _get_username_from_user_id(user_id: str) -> str: + """ + Helper function to get username from user_id, handling various formats. + + Supports: + - Full UUID with dashes (36 chars): "375ec158-686c-4a21-b44d-a51cc90ef07d" + - Dashless UUID (32 chars): "375ec158686c4a21b44da51cc90ef07d" + - Partial ID (8 chars from account names): "375ec158" + + Returns username or formatted fallback. + """ + from lnbits.core.crud.users import get_user + + logger.info(f"[USERNAME] Called with: '{user_id}' (len={len(user_id) if user_id else 0})") + + if not user_id: + return None + + # Case 1: Already in standard UUID format (36 chars with dashes) + if len(user_id) == 36 and user_id.count('-') == 4: + logger.info(f"[USERNAME] Case 1: Full UUID format") + user = await get_user(user_id) + result = user.username if user and user.username else f"User-{user_id[:8]}" + logger.info(f"[USERNAME] Case 1 result: '{result}'") + return result + + # Case 2: Dashless 32-char UUID - lookup via Castle user settings + elif len(user_id) == 32 and '-' not in user_id: + logger.info(f"[USERNAME] Case 2: Dashless UUID format - looking up in Castle user settings") + try: + # Get all Castle users (which have full user_ids) + user_settings = await get_all_user_wallet_settings() + + # Convert dashless to dashed format for comparison + user_id_with_dashes = f"{user_id[0:8]}-{user_id[8:12]}-{user_id[12:16]}-{user_id[16:20]}-{user_id[20:32]}" + logger.info(f"[USERNAME] Converted to dashed format: {user_id_with_dashes}") + + # Find matching user + for setting in user_settings: + if setting.id == user_id_with_dashes: + logger.info(f"[USERNAME] Found matching user in Castle settings") + # Get username from LNbits + user = await get_user(setting.id) + result = user.username if user and user.username else f"User-{user_id[:8]}" + logger.info(f"[USERNAME] Case 2 result (found): '{result}'") + return result + + # No matching user found + logger.info(f"[USERNAME] No matching user found in Castle settings") + result = f"User-{user_id[:8]}" + logger.info(f"[USERNAME] Case 2 result (not found): '{result}'") + return result + + except Exception as e: + logger.error(f"Error looking up user by dashless UUID {user_id}: {e}") + result = f"User-{user_id[:8]}" + return result + + # Case 3: Partial ID (8 chars from account name) - lookup via Castle user settings + elif len(user_id) == 8: + logger.info(f"[USERNAME] Case 3: Partial ID format - looking up in Castle user settings") + try: + # Get all Castle users (which have full user_ids) + user_settings = await get_all_user_wallet_settings() + + # Find matching user by first 8 chars + for setting in user_settings: + if setting.id.startswith(user_id): + logger.info(f"[USERNAME] Found full user_id: {setting.id}") + # Now get username from LNbits with full ID + user = await get_user(setting.id) + result = user.username if user and user.username else f"User-{user_id}" + logger.info(f"[USERNAME] Case 3 result (found): '{result}'") + return result + + # No matching user found in Castle settings + logger.info(f"[USERNAME] No matching user found in Castle settings") + result = f"User-{user_id}" + logger.info(f"[USERNAME] Case 3 result (not found): '{result}'") + return result + + except Exception as e: + logger.error(f"Error looking up user by partial ID {user_id}: {e}") + result = f"User-{user_id}" + return result + + # Case 4: Unknown format - try as-is and fall back + else: + logger.info(f"[USERNAME] Case 4: Unknown format - trying as-is") + try: + user = await get_user(user_id) + result = user.username if user and user.username else f"User-{user_id[:8]}" + logger.info(f"[USERNAME] Case 4 result: '{result}'") + return result + except Exception as e: + logger.info(f"[USERNAME] Case 4 exception: {e}") + result = f"User-{user_id[:8]}" + logger.info(f"[USERNAME] Case 4 fallback result: '{result}'") + return result + + @castle_api_router.get("/api/v1/entries/pending") async def api_get_pending_entries( wallet: WalletTypeInfo = Depends(require_admin_key), @@ -516,7 +617,6 @@ async def api_get_pending_entries( Returns transactions with flag='!' from Fava/Beancount. """ from lnbits.settings import settings as lnbits_settings - from lnbits.core.crud.users import get_user from .fava_client import get_fava_client if wallet.wallet.user != lnbits_settings.super_user: @@ -552,9 +652,13 @@ async def api_get_pending_entries( # Extract user ID from metadata or account names user_id = None entry_meta = e.get("meta", {}) + logger.info(f"[EXTRACT] Entry metadata keys: {list(entry_meta.keys())}") + logger.info(f"[EXTRACT] Entry metadata: {entry_meta}") if "user-id" in entry_meta: user_id = entry_meta["user-id"] + logger.info(f"[EXTRACT] Found user-id in metadata: {user_id}") else: + logger.info(f"[EXTRACT] No user-id in metadata, checking account names") # Try to extract from account names in postings for posting in e.get("postings", []): account = posting.get("account", "") @@ -563,13 +667,11 @@ async def api_get_pending_entries( parts = account.split("User-") if len(parts) > 1: user_id = parts[1] # Short ID after User- + logger.info(f"[EXTRACT] Extracted user_id from account name: {user_id}") break - # Look up username - username = None - if user_id: - user = await get_user(user_id) - username = user.username if user and user.username else f"User-{user_id[:8]}" + # Look up username using helper function + username = await _get_username_from_user_id(user_id) if user_id else None # Extract amount from postings (sum of absolute values / 2) amount_sats = 0 @@ -1243,17 +1345,15 @@ async def api_get_all_balances( wallet: WalletTypeInfo = Depends(require_admin_key), ) -> list[dict]: """Get all user balances (admin/super user only) from Fava/Beancount""" - from lnbits.core.crud.users import get_user from .fava_client import get_fava_client fava = get_fava_client() balances = await fava.get_all_user_balances() - # Enrich with username information + # Enrich with username information using helper function result = [] for balance in balances: - user = await get_user(balance["user_id"]) - username = user.username if user and user.username else balance["user_id"][:16] + "..." + username = await _get_username_from_user_id(balance["user_id"]) result.append({ "user_id": balance["user_id"],