diff --git a/CLAUDE.md b/CLAUDE.md index 3086441..6376629 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,11 +12,9 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable **Double-Entry Accounting**: Every transaction affects at least two accounts. Debits must equal credits. Five account types: Assets, Liabilities, Equity, Revenue (Income), Expenses. -**Fava/Beancount Backend**: Castle now uses [Fava](https://github.com/beancount/fava) as the primary accounting engine. Fava is a web interface for Beancount that provides a REST API for ledger operations. All accounting calculations (balance sheets, trial balances, account reports) are delegated to Fava/Beancount. Castle formats transactions as Beancount entries and submits them via Fava's API. - -**Required External Dependency**: Fava must be running as a separate service. Configure `fava_url` and `fava_ledger_slug` in Castle settings (default: `http://localhost:3333` with slug `castle-accounting`). Castle will not function without Fava. - **Pure Functional Core**: The `core/` directory contains pure accounting logic independent of the database layer: +- `core/balance.py` - Balance calculation from journal entries +- `core/inventory.py` - Multi-currency position tracking (similar to Beancount's Inventory) - `core/validation.py` - Entry validation rules **Account Hierarchy**: Beancount-style hierarchical naming with `:` separators: @@ -25,13 +23,7 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable - `Liabilities:Payable:User-af983632` - `Expenses:Food:Supplies` -**Amount Format**: Recent architecture change uses string-based amounts with currency codes: -- SATS amounts: `"200000 SATS"` -- Fiat amounts: `"100.00 EUR"` or `"250.00 USD"` -- Cost basis notation: `"200000 SATS {100.00 EUR}"` (200k sats acquired at 100 EUR) -- Parsing handles both formats via `parse_amount_string()` in views_api.py - -**Metadata System**: Beancount metadata format stores original fiat amounts and exchange rates as key-value pairs. Critical: fiat balances are calculated by summing fiat amounts from journal entries, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records. +**Metadata System**: Each `entry_line` stores JSON metadata preserving original fiat amounts. Critical: fiat balances are calculated by summing `fiat_amount` from metadata, NOT by converting current satoshi balances. This prevents exchange rate fluctuations from affecting historical records. ### Key Files @@ -41,27 +33,31 @@ Castle Accounting is a double-entry bookkeeping extension for LNbits that enable - `views.py` - Web interface routing - `services.py` - Settings management layer - `migrations.py` - Database schema migrations -- `tasks.py` - Background tasks (invoice payment monitoring) +- `tasks.py` - Background tasks (daily reconciliation checks) - `account_utils.py` - Hierarchical account naming utilities -- `fava_client.py` - HTTP client for Fava REST API (add_entry, query, balance_sheet) -- `beancount_format.py` - Converts Castle entries to Beancount transaction format -- `core/validation.py` - Pure validation functions for accounting rules ### Database Schema -**Note**: With Fava integration, Castle maintains a local cache of some data but delegates authoritative balance calculations to Beancount/Fava. +**accounts**: Chart of accounts with hierarchical names +- `user_id` field for per-user accounts (Receivable, Payable, Equity) +- Indexed on `user_id` and `account_type` -**journal_entries**: Transaction headers stored locally and synced to Fava +**journal_entries**: Transaction headers - `flag` field: `*` (cleared), `!` (pending), `#` (flagged), `x` (void) - `meta` field: JSON storing source, tags, audit info - `reference` field: Links to payment_hash, invoice numbers, etc. -- Enriched with `username` field when retrieved via API (added from LNbits user data) + +**entry_lines**: Individual debit/credit lines +- Always balanced (sum of debits = sum of credits per entry) +- `metadata` field stores fiat currency info as JSON +- Indexed on `journal_entry_id` and `account_id` + +**balance_assertions**: Reconciliation checkpoints (Beancount-style) +- Assert expected balance at a date +- Status: pending, passed, failed +- Used for daily reconciliation checks **extension_settings**: Castle wallet configuration (admin-only) -- `castle_wallet_id` - The LNbits wallet used for Castle operations -- `fava_url` - Fava service URL (default: http://localhost:3333) -- `fava_ledger_slug` - Ledger identifier in Fava (default: castle-accounting) -- `fava_timeout` - API request timeout in seconds **user_wallet_settings**: Per-user wallet configuration @@ -100,18 +96,16 @@ DR Liabilities:Payable:User-af983632 39,669 sats ## Balance Calculation Logic -**User Balance** (calculated by Beancount via Fava): +**User Balance**: - Positive = Castle owes user (LIABILITY accounts have credit balance) - Negative = User owes Castle (ASSET accounts have debit balance) -- Calculated by querying Fava for sum of all postings across user's accounts -- Fiat balances calculated by Beancount from cost basis annotations, NOT converted from current sats +- Calculated from sum of all entry lines across user's accounts +- Fiat balances summed from metadata, NOT converted from sats **Perspective-Based UI**: - **User View**: Green = Castle owes them, Red = They owe Castle - **Castle Admin View**: Green = User owes Castle, Red = Castle owes user -**Balance Retrieval**: Use `GET /api/v1/balance` which queries Fava's balance sheet or account reports for accurate, Beancount-calculated balances. - ## API Endpoints ### Accounts @@ -175,61 +169,34 @@ Use `get_or_create_user_account()` in crud.py to ensure consistency. ### Currency Handling -**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`. - -**New Amount String Format** (recent architecture change): -- Input format: `"100.00 EUR"` or `"200000 SATS"` -- Cost basis format: `"200000 SATS {100.00 EUR}"` (for recording acquisition cost) -- Parse using `parse_amount_string(amount_str)` in views_api.py -- Returns tuple: `(amount: Decimal, currency: str, cost_basis: Optional[tuple])` - -**Beancount Metadata Format**: +**CRITICAL**: Use `Decimal` for all fiat amounts, never `float`. Fiat amounts are stored in metadata as strings to preserve precision: ```python -# Metadata attached to individual postings (legs of a transaction) +from decimal import Decimal + metadata = { "fiat_currency": "EUR", - "fiat_amount": "250.00", # String for precision - "fiat_rate": "1074.192", # Sats per fiat unit + "fiat_amount": str(Decimal("250.00")), + "fiat_rate": str(Decimal("1074.192")), + "btc_rate": str(Decimal("0.000931")) } ``` -**Important**: When creating entries to submit to Fava, use `beancount_format.format_transaction()` to ensure proper Beancount syntax. +When reading: `fiat_amount = Decimal(metadata["fiat_amount"])` -### Fava Integration Patterns +### Balance Assertions for Reconciliation -**Adding a Transaction**: +Create balance assertions to verify accounting accuracy: ```python -from .fava_client import get_fava_client -from .beancount_format import format_transaction -from datetime import date - -# Format as Beancount transaction -entry = format_transaction( - date_val=date.today(), - flag="*", - narration="Groceries purchase", - postings=[ - {"account": "Expenses:Food", "amount": "50000 SATS {46.50 EUR}"}, - {"account": "Liabilities:Payable:User-abc123", "amount": "-50000 SATS"} - ], - tags=["groceries"], - links=["castle-entry-123"] -) - -# Submit to Fava -client = get_fava_client() -result = await client.add_entry(entry) -``` - -**Querying Balances**: -```python -# Query user balance from Fava -balance_result = await client.query( - f"SELECT sum(position) WHERE account ~ 'User-{user_id_short}'" +await create_balance_assertion( + account_id="lightning_account_id", + expected_balance_sats=1000000, + expected_balance_fiat=Decimal("500.00"), + fiat_currency="EUR", + tolerance_sats=100 ) ``` -**Important**: Always use `sanitize_link()` from beancount_format.py when creating links to ensure Beancount compatibility (only A-Z, a-z, 0-9, -, _, /, . allowed). +Run `POST /api/v1/tasks/daily-reconciliation` to check all assertions. ### Permission Model @@ -246,134 +213,62 @@ This extension follows LNbits extension structure: - Templates in `templates/castle/` - Database accessed via `db = Database("ext_castle")` -**Startup Requirements**: -- `castle_start()` initializes Fava client on extension load -- Background task `wait_for_paid_invoices()` monitors Lightning invoice payments -- Fava service MUST be running before starting LNbits with Castle extension - ## Common Tasks -### Add New Account in Fava +### Add New Expense Account ```python -from .fava_client import get_fava_client -from datetime import date - -# Create Open directive for new account -client = get_fava_client() -entry = { - "t": "Open", - "date": str(date.today()), - "account": "Expenses:Internet", - "currencies": ["SATS", "EUR"] -} -await client.add_entry(entry) +await create_account(CreateAccount( + name="Expenses:Internet", + account_type=AccountType.EXPENSE, + description="Internet service costs" +)) ``` -### Record Transaction to Fava +### Manually Record Cash Payment ```python -from .beancount_format import format_transaction - -entry = format_transaction( - date_val=date.today(), - flag="*", - narration="Internet bill payment", - postings=[ - {"account": "Expenses:Internet", "amount": "50000 SATS {46.50 EUR}"}, - {"account": "Assets:Lightning:Balance", "amount": "-50000 SATS"} +await create_journal_entry(CreateJournalEntry( + description="Cash payment for groceries", + lines=[ + CreateEntryLine(account_id=expense_account_id, debit=50000), + CreateEntryLine(account_id=cash_account_id, credit=50000) ], - tags=["utilities"], - links=["castle-tx-123"] -) - -client = get_fava_client() -await client.add_entry(entry) + flag=JournalEntryFlag.CLEARED, + meta={"source": "manual", "payment_method": "cash"} +)) ``` -### Query User Balance from Fava +### Check User Balance ```python -client = get_fava_client() - -# Query all accounts for a user -user_short = user_id[:8] -query = f"SELECT account, sum(position) WHERE account ~ 'User-{user_short}' GROUP BY account" -result = await client.query(query) - -# Parse result to calculate net balance -# (sum of all user accounts across Assets, Liabilities, Equity) +balance = await get_user_balance(user_id) +print(f"Sats: {balance.balance}") # Positive = Castle owes user +print(f"Fiat: {balance.fiat_balances}") # {"EUR": Decimal("36.93")} ``` +### Export to Beancount (Future) +Follow patterns in `docs/BEANCOUNT_PATTERNS.md` for implementing Beancount export. Use hierarchical account names and preserve metadata in Beancount comments. + ## Data Integrity **Critical Invariants**: -1. Every transaction submitted to Fava MUST have balanced debits and credits (Beancount enforces this) -2. Fiat amounts tracked via cost basis notation: `"AMOUNT SATS {COST FIAT}"` +1. Every journal entry MUST have balanced debits and credits +2. Fiat balances calculated from metadata, not from converting sats 3. User accounts use `user_id` (NOT `wallet_id`) for consistency -4. All accounting calculations delegated to Beancount/Fava +4. Balance assertions checked daily via background task **Validation** is performed in `core/validation.py`: -- Pure validation functions for entry correctness before submitting to Fava +- `validate_journal_entry()` - Checks balance, minimum lines +- `validate_balance()` - Verifies account balance calculation +- `validate_receivable_entry()` - Ensures receivable entries are valid +- `validate_expense_entry()` - Ensures expense entries are valid -**Beancount String Sanitization**: -- Links must match pattern: `[A-Za-z0-9\-_/.]` -- Use `sanitize_link()` from beancount_format.py for all links and tags +## Known Issues & Future Work -## Recent Architecture Changes - -**Migration to Fava/Beancount** (2025): -- Removed local balance calculation logic (now handled by Beancount) -- Removed local `accounts` and `entry_lines` tables (Fava is source of truth) -- Added `fava_client.py` and `beancount_format.py` modules -- Changed amount format to string-based with currency codes -- Username enrichment added to journal entries for UI display - -**Key Breaking Changes**: -- All balance queries now go through Fava API -- Account creation must use Fava's Open directive -- Transaction format must follow Beancount syntax -- Cost basis notation required for multi-currency tracking - -## Development Setup - -### Prerequisites - -1. **LNbits**: This extension must be installed in the `lnbits/extensions/` directory -2. **Fava Service**: Must be running before starting LNbits with Castle enabled - ```bash - # Install Fava - pip install fava - - # Create a basic Beancount file - touch castle-ledger.beancount - - # Start Fava (default: http://localhost:3333) - fava castle-ledger.beancount - ``` -3. **Configure Castle Settings**: Set `fava_url` and `fava_ledger_slug` via settings API or UI - -### Running Castle Extension - -Castle is loaded as part of LNbits. No separate build or test commands are needed for the extension itself. Development workflow: - -1. Modify code in `lnbits/extensions/castle/` -2. Restart LNbits -3. Extension hot-reloads are supported by LNbits in development mode - -### Testing Transactions - -Use the web UI or API endpoints to create test transactions. For API testing: - -```bash -# Create expense (user owes Castle) -curl -X POST http://localhost:5000/castle/api/v1/entries/expense \ - -H "X-Api-Key: YOUR_INVOICE_KEY" \ - -d '{"description": "Test expense", "amount": "100.00 EUR", "account_name": "Expenses:Test"}' - -# Check user balance -curl http://localhost:5000/castle/api/v1/balance \ - -H "X-Api-Key: YOUR_INVOICE_KEY" -``` - -**Debugging Fava Connection**: Check logs for "Fava client initialized" message on startup. If missing, verify Fava is running and settings are correct. +See `docs/DOCUMENTATION.md` for comprehensive list. Key items: +- No journal entry editing/deletion (use reversing entries) +- No date range filtering on list endpoints (hardcoded limit of 100) +- No batch operations for bulk imports +- Plugin system architecture designed but not implemented +- Beancount export endpoint not yet implemented ## Related Documentation diff --git a/MIGRATION_SQUASH_SUMMARY.md b/MIGRATION_SQUASH_SUMMARY.md deleted file mode 100644 index 8d86b9b..0000000 --- a/MIGRATION_SQUASH_SUMMARY.md +++ /dev/null @@ -1,218 +0,0 @@ -# Castle Migration Squash Summary - -**Date:** November 10, 2025 -**Action:** Squashed 16 incremental migrations into a single clean initial migration - -## Overview - -The Castle extension had accumulated 16 migrations (m001-m016) during development. Since the software has not been released yet, we safely squashed all migrations into a single clean `m001_initial` migration. - -## Files Changed - -- **migrations.py** - Replaced with squashed single migration (651 → 327 lines) -- **migrations_old.py.bak** - Backup of original 16 migrations for reference - -## Final Database Schema - -The squashed migration creates **7 tables**: - -### 1. castle_accounts -- Core chart of accounts with hierarchical Beancount-style names -- Examples: "Assets:Bitcoin:Lightning", "Expenses:Food:Groceries" -- User-specific accounts: "Assets:Receivable:User-af983632" -- Includes comprehensive default account set (40+ accounts) - -### 2. castle_extension_settings -- Castle-wide configuration -- Stores castle_wallet_id for Lightning payments - -### 3. castle_user_wallet_settings -- Per-user wallet configuration -- Allows users to have separate wallet preferences - -### 4. castle_manual_payment_requests -- User-submitted payment requests to Castle -- Reviewed by admins before processing -- Includes notes field for additional context - -### 5. castle_balance_assertions -- Reconciliation and balance checking at specific dates -- Multi-currency support (satoshis + fiat) -- Tolerance checking for small discrepancies -- Includes notes field for reconciliation comments - -### 6. castle_user_equity_status -- Manages equity contribution eligibility -- Equity-eligible users can convert expenses to equity -- Creates dynamic user-specific equity accounts: Equity:User-{user_id} - -### 7. castle_account_permissions -- Granular access control for accounts -- Permission types: read, submit_expense, manage -- Supports hierarchical inheritance (parent permissions cascade) -- Time-based expiration support - -## What Was Removed - -The following tables were **intentionally NOT included** in the final schema (they were dropped in m016): - -- **castle_journal_entries** - Journal entries now managed by Fava/Beancount (external source of truth) -- **castle_entry_lines** - Entry lines now managed by Fava/Beancount - -Castle now uses Fava as the single source of truth for accounting data. Journal operations: -- **Write:** Submit to Fava via FavaClient.add_entry() -- **Read:** Query Fava via FavaClient.get_entries() - -## Key Schema Decisions - -1. **Hierarchical Account Names** - Beancount-style colon-separated hierarchy (e.g., "Assets:Bitcoin:Lightning") -2. **No Journal Tables** - Fava/Beancount is the source of truth for journal entries -3. **Dynamic User Accounts** - User-specific accounts created on-demand (Assets:Receivable:User-xxx, Equity:User-xxx) -4. **No Parent-Only Accounts** - Hierarchy is implicit in names (no "Assets:Bitcoin" parent account needed) -5. **Multi-Currency Support** - Balance assertions support both satoshis and fiat currencies -6. **Notes Fields** - Added notes to balance_assertions and manual_payment_requests for better documentation - -## Migration History (Original 16 Migrations) - -For reference, the original migration sequence (preserved in migrations_old.py.bak): - -1. **m001** - Initial accounts, journal_entries, entry_lines tables -2. **m002** - Extension settings table -3. **m003** - User wallet settings table -4. **m004** - Manual payment requests table -5. **m005** - Added flag/meta columns to journal_entries -6. **m006** - Migrated to hierarchical account names -7. **m007** - Balance assertions table -8. **m008** - Renamed Lightning account (Assets:Lightning:Balance → Assets:Bitcoin:Lightning) -9. **m009** - Added OnChain Bitcoin account (Assets:Bitcoin:OnChain) -10. **m010** - User equity status table -11. **m011** - Account permissions table -12. **m012** - Updated default accounts with detailed hierarchy (40+ accounts) -13. **m013** - Removed parent-only accounts (Assets:Bitcoin, Equity) -14. **m014** - Removed legacy equity accounts (MemberEquity, RetainedEarnings) -15. **m015** - Converted entry_lines from debit/credit to single amount field -16. **m016** - Dropped journal_entries and entry_lines tables (Fava integration) - -## Benefits of Squashing - -1. **Cleaner Codebase** - Single 327-line migration vs 651 lines across 16 functions -2. **Easier to Understand** - New developers see final schema immediately -3. **Faster Fresh Installs** - One migration run instead of 16 -4. **Better Documentation** - Comprehensive comments explain design decisions -5. **No Migration Artifacts** - No intermediate states, data conversions, or temporary columns - -## Fresh Install Process - -For new installations: - -```bash -# Castle's migration system will run m001_initial automatically -# No manual intervention needed -``` - -The migration will: -1. Create all 7 tables with proper indexes and foreign keys -2. Insert 40+ default accounts with hierarchical names -3. Set up proper constraints and defaults -4. Complete in a single transaction - -## Default Accounts Created - -The migration automatically creates a comprehensive chart of accounts: - -**Assets (12 accounts):** -- Assets:Bank -- Assets:Bitcoin:Lightning -- Assets:Bitcoin:OnChain -- Assets:Cash -- Assets:FixedAssets:Equipment -- Assets:FixedAssets:FarmEquipment -- Assets:FixedAssets:Network -- Assets:FixedAssets:ProductionFacility -- Assets:Inventory -- Assets:Livestock -- Assets:Receivable -- Assets:Tools - -**Liabilities (1 account):** -- Liabilities:Payable - -**Income (3 accounts):** -- Income:Accommodation:Guests -- Income:Service -- Income:Other - -**Expenses (24 accounts):** -- Expenses:Administrative -- Expenses:Construction:Materials -- Expenses:Furniture -- Expenses:Garden -- Expenses:Gas:Kitchen -- Expenses:Gas:Vehicle -- Expenses:Groceries -- Expenses:Hardware -- Expenses:Housewares -- Expenses:Insurance -- Expenses:Kitchen -- Expenses:Maintenance:Car -- Expenses:Maintenance:Garden -- Expenses:Maintenance:Property -- Expenses:Membership -- Expenses:Supplies -- Expenses:Tools -- Expenses:Utilities:Electric -- Expenses:Utilities:Internet -- Expenses:WebHosting:Domain -- Expenses:WebHosting:Wix - -**Equity:** -- Created dynamically as Equity:User-{user_id} when granting equity eligibility - -## Testing - -After squashing, verify the migration works: - -```bash -# 1. Backup existing database (if any) -cp castle.sqlite3 castle.sqlite3.backup - -# 2. Drop and recreate database to test fresh install -rm castle.sqlite3 - -# 3. Start LNbits - migration should run automatically -poetry run lnbits - -# 4. Verify tables created -sqlite3 castle.sqlite3 ".tables" -# Should show: castle_accounts, castle_extension_settings, etc. - -# 5. Verify default accounts -sqlite3 castle.sqlite3 "SELECT COUNT(*) FROM castle_accounts;" -# Should show: 40 (default accounts) -``` - -## Rollback Plan - -If issues are discovered: - -```bash -# Restore original migrations -cp migrations_old.py.bak migrations.py - -# Restore database -cp castle.sqlite3.backup castle.sqlite3 -``` - -## Notes - -- This squash is safe because Castle has not been released yet -- No existing production databases need migration -- Historical migrations preserved in migrations_old.py.bak -- All functionality preserved in final schema -- No data loss concerns (no production data exists) - ---- - -**Signed off by:** Claude Code -**Reviewed by:** Human operator -**Status:** Complete diff --git a/__init__.py b/__init__.py index 6209e9d..014ffec 100644 --- a/__init__.py +++ b/__init__.py @@ -34,32 +34,9 @@ def castle_stop(): def castle_start(): """Initialize Castle extension background tasks""" from lnbits.tasks import create_permanent_unique_task - from .fava_client import init_fava_client - from .models import CastleSettings - from .tasks import wait_for_account_sync - # Initialize Fava client with default settings - # (Will be re-initialized if admin updates settings) - defaults = CastleSettings() - try: - init_fava_client( - fava_url=defaults.fava_url, - ledger_slug=defaults.fava_ledger_slug, - timeout=defaults.fava_timeout - ) - logger.info(f"Fava client initialized: {defaults.fava_url}/{defaults.fava_ledger_slug}") - except Exception as e: - logger.error(f"Failed to initialize Fava client: {e}") - logger.warning("Castle will not function without Fava. Please configure Fava settings.") - - # Start background tasks task = create_permanent_unique_task("ext_castle", wait_for_paid_invoices) scheduled_tasks.append(task) - # Start account sync task (runs hourly) - sync_task = create_permanent_unique_task("ext_castle_account_sync", wait_for_account_sync) - scheduled_tasks.append(sync_task) - logger.info("Castle account sync task started (runs hourly)") - __all__ = ["castle_ext", "castle_static_files", "db", "castle_start", "castle_stop"] diff --git a/account_sync.py b/account_sync.py deleted file mode 100644 index 95fe41c..0000000 --- a/account_sync.py +++ /dev/null @@ -1,405 +0,0 @@ -""" -Account Synchronization Module - -Syncs accounts from Beancount (source of truth) to Castle DB (metadata store). - -This implements the hybrid approach: -- Beancount owns account existence (Open directives) -- Castle DB stores permissions and user associations -- Background sync keeps them in sync - -Related: ACCOUNTS-TABLE-REMOVAL-FEASIBILITY.md - Phase 2 implementation -""" - -from datetime import datetime -from typing import Optional -from loguru import logger - -from .crud import ( - create_account, - get_account_by_name, - get_all_accounts, - update_account_is_active, -) -from .fava_client import get_fava_client -from .models import AccountType, CreateAccount - - -def infer_account_type_from_name(account_name: str) -> AccountType: - """ - Infer Beancount account type from hierarchical name. - - Args: - account_name: Hierarchical account name (e.g., "Expenses:Food:Groceries") - - Returns: - AccountType enum value - - Examples: - "Assets:Cash" → AccountType.ASSET - "Liabilities:PayPal" → AccountType.LIABILITY - "Expenses:Food" → AccountType.EXPENSE - "Income:Services" → AccountType.REVENUE - "Equity:Opening-Balances" → AccountType.EQUITY - """ - root = account_name.split(":")[0] - - type_map = { - "Assets": AccountType.ASSET, - "Liabilities": AccountType.LIABILITY, - "Expenses": AccountType.EXPENSE, - "Income": AccountType.REVENUE, - "Equity": AccountType.EQUITY, - } - - # Default to ASSET if unknown (shouldn't happen with valid Beancount) - return type_map.get(root, AccountType.ASSET) - - -def extract_user_id_from_account_name(account_name: str) -> Optional[str]: - """ - Extract user ID from account name if it's a user-specific account. - - Args: - account_name: Hierarchical account name - - Returns: - User ID if found, None otherwise - - Examples: - "Assets:Receivable:User-abc123def" → "abc123def456ghi789" - "Liabilities:Payable:User-abc123" → "abc123def456ghi789" - "Expenses:Food" → None - """ - if ":User-" not in account_name: - return None - - # Extract the part after "User-" - parts = account_name.split(":User-") - if len(parts) < 2: - return None - - # First 8 characters are the user ID prefix - user_id_prefix = parts[1] - - # For now, return the prefix (could look up full user ID from DB if needed) - # Note: get_or_create_user_account() uses 8-char prefix in account names - return user_id_prefix - - -async def sync_accounts_from_beancount(force_full_sync: bool = False) -> dict: - """ - Sync accounts from Beancount to Castle DB. - - This ensures Castle DB has metadata entries for all accounts that exist - in Beancount, enabling permissions and user associations to work properly. - - New behavior (soft delete + virtual parents): - - Accounts in Beancount but not in Castle DB: Added as active - - Accounts in Castle DB but not in Beancount: Marked as inactive (soft delete) - - Inactive accounts that return to Beancount: Reactivated - - Missing intermediate parents: Auto-created as virtual accounts - - Virtual parent auto-generation example: - Beancount has: "Expenses:Supplies:Food" - Missing parent: "Expenses:Supplies" (doesn't exist in Beancount) - → Auto-create "Expenses:Supplies" as virtual account - → Enables granting permission on "Expenses:Supplies" to cover all Supplies:* children - - Args: - force_full_sync: If True, re-check all accounts. If False, only add new ones. - - Returns: - dict with sync statistics: - { - "total_beancount_accounts": 150, - "total_castle_accounts": 148, - "accounts_added": 2, - "accounts_updated": 0, - "accounts_skipped": 148, - "accounts_deactivated": 5, - "accounts_reactivated": 1, - "virtual_parents_created": 3, - "errors": [] - } - """ - logger.info("Starting account sync from Beancount to Castle DB") - - fava = get_fava_client() - - # Get all accounts from Beancount - try: - beancount_accounts = await fava.get_all_accounts() - except Exception as e: - logger.error(f"Failed to fetch accounts from Beancount: {e}") - return { - "total_beancount_accounts": 0, - "total_castle_accounts": 0, - "accounts_added": 0, - "accounts_updated": 0, - "accounts_skipped": 0, - "accounts_deactivated": 0, - "accounts_reactivated": 0, - "errors": [str(e)], - } - - # Get all accounts from Castle DB (including inactive ones for sync) - castle_accounts = await get_all_accounts(include_inactive=True) - - # Build lookup maps - beancount_account_names = {acc["account"] for acc in beancount_accounts} - castle_accounts_by_name = {acc.name: acc for acc in castle_accounts} - - stats = { - "total_beancount_accounts": len(beancount_accounts), - "total_castle_accounts": len(castle_accounts), - "accounts_added": 0, - "accounts_updated": 0, - "accounts_skipped": 0, - "accounts_deactivated": 0, - "accounts_reactivated": 0, - "virtual_parents_created": 0, - "errors": [], - } - - # Step 1: Sync accounts from Beancount to Castle DB - for bc_account in beancount_accounts: - account_name = bc_account["account"] - - try: - existing = castle_accounts_by_name.get(account_name) - - if existing: - # Account exists in Castle DB - # Check if it needs to be reactivated - if not existing.is_active: - await update_account_is_active(existing.id, True) - stats["accounts_reactivated"] += 1 - logger.info(f"Reactivated account: {account_name}") - else: - stats["accounts_skipped"] += 1 - logger.debug(f"Account already active: {account_name}") - continue - - # Create new account in Castle DB - account_type = infer_account_type_from_name(account_name) - user_id = extract_user_id_from_account_name(account_name) - - # Get description from Beancount metadata if available - description = None - if "meta" in bc_account and isinstance(bc_account["meta"], dict): - description = bc_account["meta"].get("description") - - await create_account( - CreateAccount( - name=account_name, - account_type=account_type, - description=description, - user_id=user_id, - ) - ) - - stats["accounts_added"] += 1 - logger.info(f"Added account from Beancount: {account_name}") - - except Exception as e: - error_msg = f"Failed to sync account {account_name}: {e}" - logger.error(error_msg) - stats["errors"].append(error_msg) - - # Step 2: Mark orphaned accounts (in Castle DB but not in Beancount) as inactive - # SKIP virtual accounts (they're intentionally metadata-only) - for castle_account in castle_accounts: - if castle_account.is_virtual: - # Virtual accounts are metadata-only, never deactivate them - continue - - if castle_account.name not in beancount_account_names: - # Account no longer exists in Beancount - if castle_account.is_active: - try: - await update_account_is_active(castle_account.id, False) - stats["accounts_deactivated"] += 1 - logger.info( - f"Deactivated orphaned account: {castle_account.name}" - ) - except Exception as e: - error_msg = ( - f"Failed to deactivate account {castle_account.name}: {e}" - ) - logger.error(error_msg) - stats["errors"].append(error_msg) - - # Step 3: Auto-generate virtual intermediate parent accounts - # For each account in Beancount, check if all parent levels exist - # If not, create them as virtual accounts - - # IMPORTANT: Re-fetch accounts from DB after Step 1 added new accounts - # Otherwise we'll be checking against stale data and miss newly synced children - current_castle_accounts = await get_all_accounts(include_inactive=True) - all_account_names = {acc.name for acc in current_castle_accounts} - - for bc_account in beancount_accounts: - account_name = bc_account["account"] - parts = account_name.split(":") - - # Check each parent level (e.g., for "Expenses:Supplies:Food", check "Expenses:Supplies") - for i in range(1, len(parts)): - parent_name = ":".join(parts[:i]) - - # Skip if parent already exists - if parent_name in all_account_names: - continue - - # Create virtual parent account - try: - parent_type = infer_account_type_from_name(parent_name) - await create_account( - CreateAccount( - name=parent_name, - account_type=parent_type, - description=f"Auto-generated virtual parent for {parent_name}:* accounts", - is_virtual=True, - ) - ) - - stats["virtual_parents_created"] += 1 - all_account_names.add(parent_name) # Track so we don't create duplicates - logger.info(f"Created virtual parent account: {parent_name}") - - except Exception as e: - error_msg = f"Failed to create virtual parent {parent_name}: {e}" - logger.error(error_msg) - stats["errors"].append(error_msg) - - logger.info( - f"Account sync complete: " - f"{stats['accounts_added']} added, " - f"{stats['accounts_reactivated']} reactivated, " - f"{stats['accounts_deactivated']} deactivated, " - f"{stats['virtual_parents_created']} virtual parents created, " - f"{stats['accounts_skipped']} skipped, " - f"{len(stats['errors'])} errors" - ) - - return stats - - -async def sync_single_account_from_beancount(account_name: str) -> bool: - """ - Sync a single account from Beancount to Castle DB. - - Useful for ensuring a specific account exists in Castle DB before - granting permissions on it. - - Args: - account_name: Hierarchical account name (e.g., "Expenses:Food") - - Returns: - True if account was created/updated, False if it already existed or failed - """ - logger.debug(f"Syncing single account: {account_name}") - - # Check if already exists - existing = await get_account_by_name(account_name) - if existing: - logger.debug(f"Account already exists: {account_name}") - return False - - # Get from Beancount - fava = get_fava_client() - try: - all_accounts = await fava.get_all_accounts() - bc_account = next( - (acc for acc in all_accounts if acc["account"] == account_name), None - ) - - if not bc_account: - logger.error(f"Account not found in Beancount: {account_name}") - return False - - # Create in Castle DB - account_type = infer_account_type_from_name(account_name) - user_id = extract_user_id_from_account_name(account_name) - - description = None - if "meta" in bc_account and isinstance(bc_account["meta"], dict): - description = bc_account["meta"].get("description") - - await create_account( - CreateAccount( - name=account_name, - account_type=account_type, - description=description, - user_id=user_id, - ) - ) - - logger.info(f"Created account from Beancount: {account_name}") - return True - - except Exception as e: - logger.error(f"Failed to sync account {account_name}: {e}") - return False - - -async def ensure_account_exists_in_castle(account_name: str) -> bool: - """ - Ensure account exists in Castle DB, creating from Beancount if needed. - - This is the recommended function to call before granting permissions. - - Args: - account_name: Hierarchical account name - - Returns: - True if account exists (or was created), False if failed - """ - # Check Castle DB first - existing = await get_account_by_name(account_name) - if existing: - return True - - # Try to sync from Beancount - return await sync_single_account_from_beancount(account_name) - - -# Background sync task (can be scheduled with cron or async scheduler) -async def scheduled_account_sync(): - """ - Scheduled task to sync accounts from Beancount to Castle DB. - - Run this periodically (e.g., every hour) to keep Castle DB in sync with Beancount. - - Example with APScheduler: - from apscheduler.schedulers.asyncio import AsyncIOScheduler - - scheduler = AsyncIOScheduler() - scheduler.add_job( - scheduled_account_sync, - 'interval', - hours=1, # Run every hour - id='account_sync' - ) - scheduler.start() - """ - logger.info("Running scheduled account sync") - - try: - stats = await sync_accounts_from_beancount(force_full_sync=False) - - if stats["accounts_added"] > 0: - logger.info( - f"Scheduled sync: Added {stats['accounts_added']} new accounts" - ) - - if stats["errors"]: - logger.warning( - f"Scheduled sync: {len(stats['errors'])} errors encountered" - ) - - return stats - - except Exception as e: - logger.error(f"Scheduled account sync failed: {e}") - raise diff --git a/account_utils.py b/account_utils.py index 46db327..ed781c9 100644 --- a/account_utils.py +++ b/account_utils.py @@ -190,66 +190,26 @@ def migrate_account_name(old_name: str, account_type: AccountType) -> str: # Default chart of accounts with hierarchical names DEFAULT_HIERARCHICAL_ACCOUNTS = [ # Assets - ("Assets:Bank", AccountType.ASSET, "Bank account"), - ("Assets:Bitcoin:Lightning", AccountType.ASSET, "Lightning Network balance"), - ("Assets:Bitcoin:OnChain", AccountType.ASSET, "On-chain Bitcoin wallet"), ("Assets:Cash", AccountType.ASSET, "Cash on hand"), - ("Assets:FixedAssets:Equipment", AccountType.ASSET, "Equipment and machinery"), - ("Assets:FixedAssets:FarmEquipment", AccountType.ASSET, "Farm equipment"), - ("Assets:FixedAssets:Network", AccountType.ASSET, "Network infrastructure"), - ("Assets:FixedAssets:ProductionFacility", AccountType.ASSET, "Production facilities"), - ("Assets:Inventory", AccountType.ASSET, "Inventory and stock"), - ("Assets:Livestock", AccountType.ASSET, "Livestock and animals"), + ("Assets:Bank", AccountType.ASSET, "Bank account"), + ("Assets:Lightning:Balance", AccountType.ASSET, "Lightning Network balance"), ("Assets:Receivable", AccountType.ASSET, "Money owed to the Castle"), - ("Assets:Tools", AccountType.ASSET, "Tools and hand equipment"), # Liabilities ("Liabilities:Payable", AccountType.LIABILITY, "Money owed by the Castle"), - # Equity - User equity accounts created dynamically as Equity:User-{user_id} - # No parent "Equity" account needed - hierarchy is implicit in the name + # Equity + ("Equity:MemberEquity", AccountType.EQUITY, "Member contributions"), + ("Equity:RetainedEarnings", AccountType.EQUITY, "Accumulated profits"), # Revenue (Income in Beancount terminology) - ("Income:Accommodation:Guests", AccountType.REVENUE, "Revenue from guest accommodation"), + ("Income:Accommodation", AccountType.REVENUE, "Revenue from stays"), ("Income:Service", AccountType.REVENUE, "Revenue from services"), ("Income:Other", AccountType.REVENUE, "Other revenue"), - # Expenses - SUPPLIES (consumables - things you buy regularly) - ("Expenses:Supplies:Food", AccountType.EXPENSE, "Food & groceries"), - ("Expenses:Supplies:Kitchen", AccountType.EXPENSE, "Kitchen supplies"), - ("Expenses:Supplies:Office", AccountType.EXPENSE, "Office supplies"), - ("Expenses:Supplies:Garden", AccountType.EXPENSE, "Garden supplies"), - ("Expenses:Supplies:Paint", AccountType.EXPENSE, "Paint & painting supplies"), - ("Expenses:Supplies:Cleaning", AccountType.EXPENSE, "Cleaning supplies"), - ("Expenses:Supplies:Other", AccountType.EXPENSE, "Other consumables"), - - # Expenses - MATERIALS (construction/building materials) - ("Expenses:Materials:Construction", AccountType.EXPENSE, "Building materials"), - ("Expenses:Materials:Hardware", AccountType.EXPENSE, "Hardware (nails, screws, fasteners)"), - - # Expenses - EQUIPMENT (durable goods that last) - ("Expenses:Equipment:Tools", AccountType.EXPENSE, "Tools"), - ("Expenses:Equipment:Furniture", AccountType.EXPENSE, "Furniture"), - ("Expenses:Equipment:Housewares", AccountType.EXPENSE, "Housewares & appliances"), - - # Expenses - UTILITIES (ongoing services with bills) - ("Expenses:Utilities:Electric", AccountType.EXPENSE, "Electricity"), - ("Expenses:Utilities:Internet", AccountType.EXPENSE, "Internet service"), - ("Expenses:Utilities:Gas:Kitchen", AccountType.EXPENSE, "Kitchen gas"), - ("Expenses:Utilities:Gas:Vehicle", AccountType.EXPENSE, "Vehicle fuel"), - ("Expenses:Utilities:Water", AccountType.EXPENSE, "Water"), - - # Expenses - MAINTENANCE (repairs & upkeep) - ("Expenses:Maintenance:Property", AccountType.EXPENSE, "Building/property repairs"), - ("Expenses:Maintenance:Vehicle", AccountType.EXPENSE, "Car maintenance & repairs"), - ("Expenses:Maintenance:Garden", AccountType.EXPENSE, "Garden maintenance"), - ("Expenses:Maintenance:Equipment", AccountType.EXPENSE, "Equipment repairs"), - - # Expenses - SERVICES (professional services & subscriptions) - ("Expenses:Services:Insurance", AccountType.EXPENSE, "Insurance premiums"), - ("Expenses:Services:Membership", AccountType.EXPENSE, "Membership fees"), - ("Expenses:Services:WebHosting:Domain", AccountType.EXPENSE, "Domain registration"), - ("Expenses:Services:WebHosting:Wix", AccountType.EXPENSE, "Wix hosting service"), - ("Expenses:Services:Administrative", AccountType.EXPENSE, "Administrative services"), - ("Expenses:Services:Other", AccountType.EXPENSE, "Other services"), + # Expenses + ("Expenses:Utilities", AccountType.EXPENSE, "Electricity, water, internet"), + ("Expenses:Food:Supplies", AccountType.EXPENSE, "Food and supplies"), + ("Expenses:Maintenance", AccountType.EXPENSE, "Repairs and maintenance"), + ("Expenses:Other", AccountType.EXPENSE, "Miscellaneous expenses"), ] diff --git a/beancount_format.py b/beancount_format.py deleted file mode 100644 index 2ffb01b..0000000 --- a/beancount_format.py +++ /dev/null @@ -1,868 +0,0 @@ -""" -Format Castle entries as Beancount transactions for Fava API. - -All entries submitted to Fava must follow Beancount syntax. -This module converts Castle data models to Fava API format. - -Key concepts: -- Amounts are strings: "200000 SATS" or "100.00 EUR" -- Cost basis syntax: "200000 SATS {100.00 EUR}" -- Flags: "*" (cleared), "!" (pending), "#" (flagged), "?" (unknown) -- Entry type: "t": "Transaction" (required by Fava) -""" - -from datetime import date, datetime -from decimal import Decimal -from typing import Any, Dict, List, Optional -import re - - -def sanitize_link(text: str) -> str: - """ - Sanitize a string to make it valid for Beancount links. - - Beancount links can only contain: A-Z, a-z, 0-9, -, _, /, . - All other characters are replaced with hyphens. - - Examples: - >>> sanitize_link("Test (pending)") - 'Test-pending' - >>> sanitize_link("Invoice #123") - 'Invoice-123' - >>> sanitize_link("castle-abc123") - 'castle-abc123' - """ - # Replace any character that's not alphanumeric, dash, underscore, slash, or period with a hyphen - sanitized = re.sub(r'[^A-Za-z0-9\-_/.]', '-', text) - # Remove consecutive hyphens - sanitized = re.sub(r'-+', '-', sanitized) - # Remove leading/trailing hyphens - sanitized = sanitized.strip('-') - return sanitized - - -def format_transaction( - date_val: date, - flag: str, - narration: str, - postings: List[Dict[str, Any]], - payee: str = "", - tags: Optional[List[str]] = None, - links: Optional[List[str]] = None, - meta: Optional[Dict[str, Any]] = None -) -> Dict[str, Any]: - """ - Format a transaction for Fava's add_entries API. - - Args: - date_val: Transaction date - flag: Beancount flag (* = cleared, ! = pending, # = flagged) - narration: Description - postings: List of posting dicts (formatted by format_posting) - payee: Optional payee - tags: Optional tags (e.g., ["expense-entry", "approved"]) - links: Optional links (e.g., ["castle-abc123", "^invoice-xyz"]) - meta: Optional transaction metadata - - Returns: - Fava API entry dict - - Example: - entry = format_transaction( - date_val=date.today(), - flag="*", - narration="Grocery shopping", - postings=[ - format_posting_with_cost( - account="Expenses:Food", - amount_sats=36930, - fiat_currency="EUR", - fiat_amount=Decimal("36.93") - ), - format_posting_with_cost( - account="Liabilities:Payable:User-abc", - amount_sats=-36930, - fiat_currency="EUR", - fiat_amount=Decimal("36.93") - ) - ], - tags=["expense-entry"], - links=["castle-abc123"], - meta={"user-id": "abc123", "source": "castle-expense-entry"} - ) - """ - return { - "t": "Transaction", # REQUIRED by Fava API - "date": str(date_val), - "flag": flag, - "payee": payee or "", # Empty string, not None - "narration": narration, - "tags": tags or [], - "links": links or [], - "postings": postings, - "meta": meta or {} - } - - -def format_balance( - date_val: date, - account: str, - amount: int, - currency: str = "SATS" -) -> str: - """ - Format a balance assertion directive for Beancount. - - Balance assertions verify that an account has an expected balance on a specific date. - They are checked automatically by Beancount when the file is loaded. - - Args: - date_val: Date of the balance assertion - account: Account name (e.g., "Assets:Bitcoin:Lightning") - amount: Expected balance amount - currency: Currency code (default: "SATS") - - Returns: - Beancount balance directive as a string - - Example: - >>> format_balance(date(2025, 11, 10), "Assets:Bitcoin:Lightning", 1500000, "SATS") - '2025-11-10 balance Assets:Bitcoin:Lightning 1500000 SATS' - """ - date_str = date_val.strftime('%Y-%m-%d') - # Two spaces between account and amount (Beancount convention) - return f"{date_str} balance {account} {amount} {currency}" - - -def format_posting_with_cost( - account: str, - amount_sats: int, - fiat_currency: Optional[str] = None, - fiat_amount: Optional[Decimal] = None, - metadata: Optional[Dict[str, Any]] = None -) -> Dict[str, Any]: - """ - Format a posting with cost basis for Fava API. - - This is the RECOMMENDED format for all Castle transactions. - Uses Beancount's cost basis syntax to preserve exchange rates. - - IMPORTANT: Beancount cost syntax uses PER-UNIT cost, not total cost. - This function calculates per-unit cost automatically. - - Args: - account: Account name (e.g., "Expenses:Food:Groceries") - amount_sats: Amount in satoshis (signed: positive = debit, negative = credit) - fiat_currency: Fiat currency (EUR, USD, etc.) - fiat_amount: Fiat amount TOTAL (Decimal, unsigned) - will be converted to per-unit - metadata: Optional posting metadata - - Returns: - Fava API posting dict - - Example: - posting = format_posting_with_cost( - account="Expenses:Food", - amount_sats=200000, - fiat_currency="EUR", - fiat_amount=Decimal("100.00") # Total cost - ) - # Calculates per-unit: 100.00 / 200000 = 0.0005 EUR per SAT - # Returns: { - # "account": "Expenses:Food", - # "amount": "200000 SATS {0.0005 EUR}", - # "meta": {} - # } - """ - # Build amount string with cost basis - if fiat_currency and fiat_amount and fiat_amount > 0 and amount_sats != 0: - # Calculate per-unit cost (Beancount requires per-unit, not total) - # Example: 1000.00 EUR / 1097994 SATS = 0.000911268 EUR per SAT - amount_sats_abs = abs(amount_sats) - per_unit_cost = abs(fiat_amount) / Decimal(str(amount_sats_abs)) - - # Use high precision for per-unit cost (8 decimal places) - # Cost basis syntax: "200000 SATS {0.00050000 EUR}" - # Sign is on the sats amount, cost is always positive per-unit value - amount_str = f"{amount_sats} SATS {{{per_unit_cost:.8f} {fiat_currency}}}" - else: - # No cost basis: "200000 SATS" - amount_str = f"{amount_sats} SATS" - - # Build metadata - include total fiat amount to avoid rounding errors in balance calculations - posting_meta = metadata or {} - if fiat_currency and fiat_amount and fiat_amount > 0: - # Store the exact total fiat amount as metadata - # This preserves the original amount exactly, avoiding rounding errors from per-unit calculations - posting_meta["fiat-amount-total"] = f"{abs(fiat_amount):.2f}" - posting_meta["fiat-currency"] = fiat_currency - - return { - "account": account, - "amount": amount_str, - "meta": posting_meta - } - - -def format_posting_at_average_cost( - account: str, - amount_sats: int, - cost_currency: Optional[str] = None, - metadata: Optional[Dict[str, Any]] = None -) -> Dict[str, Any]: - """ - Format a posting to reduce at average cost for Fava API. - - Use this for payments/settlements to reduce positions at average cost. - Specifying the cost currency tells Beancount which lots to reduce. - - Args: - account: Account name - amount_sats: Amount in satoshis (signed) - cost_currency: Currency of the original cost basis (e.g., "EUR") - metadata: Optional posting metadata - - Returns: - Fava API posting dict - - Example: - posting = format_posting_at_average_cost( - account="Assets:Receivable:User-abc", - amount_sats=-996896, - cost_currency="EUR" - ) - # Returns: { - # "account": "Assets:Receivable:User-abc", - # "amount": "-996896 SATS {EUR}", - # "meta": {} - # } - # Beancount will automatically reduce EUR balance proportionally - """ - # Cost currency specification: "996896 SATS {EUR}" - # This reduces positions with EUR cost at average cost - from loguru import logger - - if cost_currency: - amount_str = f"{amount_sats} SATS {{{cost_currency}}}" - logger.info(f"format_posting_at_average_cost: Generated amount_str='{amount_str}' with cost_currency='{cost_currency}'") - else: - # No cost - amount_str = f"{amount_sats} SATS {{}}" - logger.warning(f"format_posting_at_average_cost: cost_currency is None, using empty cost basis") - - posting_meta = metadata or {} - - return { - "account": account, - "amount": amount_str, - "meta": posting_meta - } - - -def format_posting_simple( - account: str, - amount_sats: int, - metadata: Optional[Dict[str, Any]] = None -) -> Dict[str, Any]: - """ - Format a simple posting (SATS only, no cost basis). - - Use this for: - - Lightning payments (no fiat conversion) - - SATS-only transactions - - Internal transfers - - Args: - account: Account name - amount_sats: Amount in satoshis (signed) - metadata: Optional posting metadata - - Returns: - Fava API posting dict - - Example: - posting = format_posting_simple( - account="Assets:Bitcoin:Lightning", - amount_sats=200000 - ) - # Returns: { - # "account": "Assets:Bitcoin:Lightning", - # "amount": "200000 SATS", - # "meta": {} - # } - """ - return { - "account": account, - "amount": f"{amount_sats} SATS", - "meta": metadata or {} - } - - -def format_expense_entry( - user_id: str, - expense_account: str, - user_account: str, - amount_sats: int, - description: str, - entry_date: date, - is_equity: bool = False, - fiat_currency: Optional[str] = None, - fiat_amount: Optional[Decimal] = None, - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format an expense entry for submission to Fava. - - Creates a pending transaction (flag="!") that requires admin approval. - - Stores payables in EUR (or other fiat) as this is the actual debt amount. - SATS amount stored as metadata for reference. - - Args: - user_id: User ID - expense_account: Expense account name (e.g., "Expenses:Food:Groceries") - user_account: User's liability/equity account name - amount_sats: Amount in satoshis (for reference/metadata) - description: Entry description - entry_date: Date of entry - is_equity: Whether this is an equity contribution - fiat_currency: Fiat currency (EUR, USD) - REQUIRED - fiat_amount: Fiat amount (unsigned) - REQUIRED - reference: Optional reference (invoice ID, etc.) - - Returns: - Fava API entry dict - - Example: - entry = format_expense_entry( - user_id="abc123", - expense_account="Expenses:Food:Groceries", - user_account="Liabilities:Payable:User-abc123", - amount_sats=200000, - description="Grocery shopping", - entry_date=date.today(), - fiat_currency="EUR", - fiat_amount=Decimal("100.00") - ) - """ - fiat_amount_abs = abs(fiat_amount) if fiat_amount else None - - if not fiat_currency or not fiat_amount_abs: - raise ValueError("fiat_currency and fiat_amount are required for expense entries") - - # Build narration - narration = description - narration += f" ({fiat_amount_abs:.2f} {fiat_currency})" - - # Build postings in EUR (debts are in operating currency) - postings = [ - { - "account": expense_account, - "amount": f"{fiat_amount_abs:.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - }, - { - "account": user_account, - "amount": f"-{fiat_amount_abs:.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - } - ] - - # Build entry metadata - entry_meta = { - "user-id": user_id, - "source": "castle-api", - "sats-amount": str(abs(amount_sats)) - } - - # Build links - links = [] - if reference: - links.append(reference) - - # Build tags - tags = ["expense-entry"] - if is_equity: - tags.append("equity-contribution") - - return format_transaction( - date_val=entry_date, - flag="!", # Pending approval - narration=narration, - postings=postings, - tags=tags, - links=links, - meta=entry_meta - ) - - -def format_receivable_entry( - user_id: str, - revenue_account: str, - receivable_account: str, - amount_sats: int, - description: str, - entry_date: date, - fiat_currency: Optional[str] = None, - fiat_amount: Optional[Decimal] = None, - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format a receivable entry (user owes castle). - - Creates a pending transaction that starts as receivable. - - Args: - user_id: User ID - revenue_account: Revenue account name - receivable_account: User's receivable account name (Assets:Receivable:User-{id}) - amount_sats: Amount in satoshis (unsigned) - description: Entry description - entry_date: Date of entry - fiat_currency: Optional fiat currency - fiat_amount: Optional fiat amount (unsigned) - reference: Optional reference - - Returns: - Fava API entry dict - """ - fiat_amount_abs = abs(fiat_amount) if fiat_amount else None - - if not fiat_currency or not fiat_amount_abs: - raise ValueError("fiat_currency and fiat_amount are required for receivable entries") - - narration = description - narration += f" ({fiat_amount_abs:.2f} {fiat_currency})" - - # Build postings in EUR (debts are in operating currency) - postings = [ - { - "account": receivable_account, - "amount": f"{fiat_amount_abs:.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - }, - { - "account": revenue_account, - "amount": f"-{fiat_amount_abs:.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - } - ] - - entry_meta = { - "user-id": user_id, - "source": "castle-api", - "sats-amount": str(abs(amount_sats)) - } - - links = [] - if reference: - links.append(reference) - - return format_transaction( - date_val=entry_date, - flag="*", # Receivables are immediately cleared (approved) - narration=narration, - postings=postings, - tags=["receivable-entry"], - links=links, - meta=entry_meta - ) - - -def format_payment_entry( - user_id: str, - payment_account: str, - payable_or_receivable_account: str, - amount_sats: int, - description: str, - entry_date: date, - is_payable: bool = True, - fiat_currency: Optional[str] = None, - fiat_amount: Optional[Decimal] = None, - payment_hash: Optional[str] = None, - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format a payment entry (Lightning payment recorded). - - Creates a cleared transaction (flag="*") since payment already happened. - - Args: - user_id: User ID - payment_account: Payment method account (e.g., "Assets:Bitcoin:Lightning") - payable_or_receivable_account: User's account being settled - amount_sats: Amount in satoshis (unsigned) - description: Payment description - entry_date: Date of payment - is_payable: True if castle paying user (payable), False if user paying castle (receivable) - fiat_currency: Optional fiat currency - fiat_amount: Optional fiat amount (unsigned) - payment_hash: Lightning payment hash - reference: Optional reference - - Returns: - Fava API entry dict - """ - amount_sats_abs = abs(amount_sats) - fiat_amount_abs = abs(fiat_amount) if fiat_amount else None - - # For payment settlements with fiat tracking, use cost syntax with per-unit cost - # This allows Beancount to match against existing lots and reduce them - # The per-unit cost is calculated from: fiat_amount / sats_amount - # Example: 908.44 EUR / 996896 SATS = 0.000911268 EUR/SAT (matches original receivable rate) - if fiat_currency and fiat_amount_abs and amount_sats_abs > 0: - if is_payable: - # Castle paying user: DR Payable, CR Lightning - postings = [ - format_posting_with_cost( - account=payable_or_receivable_account, - amount_sats=amount_sats_abs, - fiat_currency=fiat_currency, - fiat_amount=fiat_amount_abs # Will be converted to per-unit cost - ), - format_posting_simple( - account=payment_account, - amount_sats=-amount_sats_abs, - metadata={"payment-hash": payment_hash} if payment_hash else None - ) - ] - else: - # User paying castle: DR Lightning, CR Receivable - postings = [ - format_posting_simple( - account=payment_account, - amount_sats=amount_sats_abs, - metadata={"payment-hash": payment_hash} if payment_hash else None - ), - format_posting_with_cost( - account=payable_or_receivable_account, - amount_sats=-amount_sats_abs, - fiat_currency=fiat_currency, - fiat_amount=fiat_amount_abs # Will be converted to per-unit cost - ) - ] - else: - # No fiat tracking, use simple postings - if is_payable: - postings = [ - format_posting_simple(account=payable_or_receivable_account, amount_sats=amount_sats_abs), - format_posting_simple(account=payment_account, amount_sats=-amount_sats_abs, - metadata={"payment-hash": payment_hash} if payment_hash else None) - ] - else: - postings = [ - format_posting_simple(account=payment_account, amount_sats=amount_sats_abs, - metadata={"payment-hash": payment_hash} if payment_hash else None), - format_posting_simple(account=payable_or_receivable_account, amount_sats=-amount_sats_abs) - ] - - # Note: created-via is redundant with #lightning-payment tag - # Note: payer/payee can be inferred from transaction direction and accounts - entry_meta = { - "user-id": user_id, - "source": "lightning_payment" - } - - if payment_hash: - entry_meta["payment-hash"] = payment_hash - - links = [] - if reference: - links.append(reference) - if payment_hash: - links.append(f"ln-{payment_hash[:16]}") - - return format_transaction( - date_val=entry_date, - flag="*", # Cleared (payment already happened) - narration=description, - postings=postings, - tags=["lightning-payment"], - links=links, - meta=entry_meta - ) - - -def format_fiat_settlement_entry( - user_id: str, - payment_account: str, - payable_or_receivable_account: str, - fiat_amount: Decimal, - fiat_currency: str, - amount_sats: int, - description: str, - entry_date: date, - is_payable: bool = True, - payment_method: str = "cash", - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format a fiat (cash/bank) settlement entry. - - Unlike Lightning payments, fiat settlements use fiat currency as the primary amount - with SATS stored as metadata for reference. - - Args: - user_id: User ID - payment_account: Payment method account (e.g., "Assets:Cash", "Assets:Bank") - payable_or_receivable_account: User's account being settled - fiat_amount: Amount in fiat currency (unsigned) - fiat_currency: Fiat currency code (EUR, USD, etc.) - amount_sats: Equivalent amount in satoshis (for metadata only) - description: Payment description - entry_date: Date of settlement - is_payable: True if castle paying user (payable), False if user paying castle (receivable) - payment_method: Payment method (cash, bank_transfer, check, etc.) - reference: Optional reference - - Returns: - Fava API entry dict - """ - fiat_amount_abs = abs(fiat_amount) - amount_sats_abs = abs(amount_sats) - - if is_payable: - # Castle paying user: DR Payable, CR Cash/Bank - postings = [ - { - "account": payable_or_receivable_account, - "amount": f"{fiat_amount_abs:.2f} {fiat_currency}", - "meta": { - "sats-equivalent": str(amount_sats_abs) - } - }, - { - "account": payment_account, - "amount": f"-{fiat_amount_abs:.2f} {fiat_currency}", - "meta": { - "sats-equivalent": str(amount_sats_abs) - } - } - ] - else: - # User paying castle: DR Cash/Bank, CR Receivable - postings = [ - { - "account": payment_account, - "amount": f"{fiat_amount_abs:.2f} {fiat_currency}", - "meta": { - "sats-equivalent": str(amount_sats_abs) - } - }, - { - "account": payable_or_receivable_account, - "amount": f"-{fiat_amount_abs:.2f} {fiat_currency}", - "meta": { - "sats-equivalent": str(amount_sats_abs) - } - } - ] - - # Map payment method to appropriate source and tag - payment_method_map = { - "cash": ("cash_settlement", "cash-payment"), - "bank_transfer": ("bank_settlement", "bank-transfer"), - "check": ("check_settlement", "check-payment"), - "btc_onchain": ("onchain_settlement", "onchain-payment"), - "other": ("manual_settlement", "manual-payment") - } - - source, tag = payment_method_map.get(payment_method.lower(), ("manual_settlement", "manual-payment")) - - entry_meta = { - "user-id": user_id, - "source": source - } - - links = [] - if reference: - links.append(reference) - - return format_transaction( - date_val=entry_date, - flag="*", # Cleared (payment already happened) - narration=description, - postings=postings, - tags=[tag], - links=links, - meta=entry_meta - ) - - -def format_net_settlement_entry( - user_id: str, - payment_account: str, - receivable_account: str, - payable_account: str, - amount_sats: int, - net_fiat_amount: Decimal, - total_receivable_fiat: Decimal, - total_payable_fiat: Decimal, - fiat_currency: str, - description: str, - entry_date: date, - payment_hash: Optional[str] = None, - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format a net settlement payment entry (user paying net balance). - - Creates a three-posting transaction: - 1. Lightning payment in SATS with @@ total price notation - 2. Clear receivables in EUR - 3. Clear payables in EUR - - Example: - Assets:Bitcoin:Lightning 565251 SATS @@ 517.00 EUR - Assets:Receivable:User -555.00 EUR - Liabilities:Payable:User 38.00 EUR - = 517 - 555 + 38 = 0 ✓ - - Args: - user_id: User ID - payment_account: Payment account (e.g., "Assets:Bitcoin:Lightning") - receivable_account: User's receivable account - payable_account: User's payable account - amount_sats: SATS amount paid - net_fiat_amount: Net fiat amount (receivable - payable) - total_receivable_fiat: Total receivables to clear - total_payable_fiat: Total payables to clear - fiat_currency: Currency (EUR, USD) - description: Payment description - entry_date: Date of payment - payment_hash: Lightning payment hash - reference: Optional reference - - Returns: - Fava API entry dict - """ - # Build postings for net settlement - # Note: We use @@ (total price) syntax for cleaner formatting, but Fava's API - # will convert this to @ (per-unit price) with a long decimal when writing to file. - # This is Fava's internal normalization behavior and cannot be changed via API. - # The accounting is still 100% correct, just not as visually clean. - postings = [ - { - "account": payment_account, - "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}", - "meta": {"payment-hash": payment_hash} if payment_hash else {} - }, - { - "account": receivable_account, - "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - }, - { - "account": payable_account, - "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}", - "meta": {} - } - ] - - entry_meta = { - "user-id": user_id, - "source": "lightning_payment", - "payment-type": "net-settlement" - } - - if payment_hash: - entry_meta["payment-hash"] = payment_hash - - links = [] - if reference: - links.append(reference) - if payment_hash: - links.append(f"ln-{payment_hash[:16]}") - - return format_transaction( - date_val=entry_date, - flag="*", # Cleared (payment already happened) - narration=description, - postings=postings, - tags=["lightning-payment", "net-settlement"], - links=links, - meta=entry_meta - ) - - -def format_revenue_entry( - payment_account: str, - revenue_account: str, - amount_sats: int, - description: str, - entry_date: date, - fiat_currency: Optional[str] = None, - fiat_amount: Optional[Decimal] = None, - reference: Optional[str] = None -) -> Dict[str, Any]: - """ - Format a revenue entry (castle receives payment directly). - - Creates a cleared transaction (flag="*") since payment was received. - - Example: Cash sale, Lightning payment received, bank transfer received. - - Args: - payment_account: Payment method account (e.g., "Assets:Bitcoin:Lightning", "Assets:Cash") - revenue_account: Revenue account name (e.g., "Income:Sales", "Income:Services") - amount_sats: Amount in satoshis (unsigned) - description: Entry description - entry_date: Date of payment - fiat_currency: Optional fiat currency - fiat_amount: Optional fiat amount (unsigned) - reference: Optional reference - - Returns: - Fava API entry dict - - Example: - entry = format_revenue_entry( - payment_account="Assets:Cash", - revenue_account="Income:Sales", - amount_sats=100000, - description="Product sale", - entry_date=date.today(), - fiat_currency="EUR", - fiat_amount=Decimal("50.00") - ) - """ - amount_sats_abs = abs(amount_sats) - fiat_amount_abs = abs(fiat_amount) if fiat_amount else None - - narration = description - if fiat_currency and fiat_amount_abs: - narration += f" ({fiat_amount_abs:.2f} {fiat_currency})" - - postings = [ - format_posting_with_cost( - account=payment_account, - amount_sats=amount_sats_abs, # Positive = debit (asset increase) - fiat_currency=fiat_currency, - fiat_amount=fiat_amount_abs - ), - format_posting_with_cost( - account=revenue_account, - amount_sats=-amount_sats_abs, # Negative = credit (revenue increase) - fiat_currency=fiat_currency, - fiat_amount=fiat_amount_abs - ) - ] - - # Note: created-via is redundant with #revenue-entry tag - entry_meta = { - "source": "castle-api" - } - - links = [] - if reference: - links.append(reference) - - return format_transaction( - date_val=entry_date, - flag="*", # Cleared (payment received) - narration=narration, - postings=postings, - tags=["revenue-entry"], - links=links, - meta=entry_meta - ) diff --git a/core/__init__.py b/core/__init__.py index 662bb20..9b4cf2b 100644 --- a/core/__init__.py +++ b/core/__init__.py @@ -4,6 +4,8 @@ Castle Core Module - Pure accounting logic separated from database operations. This module contains the core business logic for double-entry accounting, following Beancount patterns for clean architecture: +- inventory.py: Position tracking across currencies +- balance.py: Balance calculation logic - validation.py: Comprehensive validation rules Benefits: @@ -11,14 +13,16 @@ Benefits: - Reusable across different storage backends - Clear separation of concerns - Easier to audit and verify - -Note: Balance calculation and inventory tracking have been migrated to Fava/Beancount. -All accounting calculations are now performed via Fava's query API. """ +from .inventory import CastleInventory, CastlePosition +from .balance import BalanceCalculator from .validation import ValidationError, validate_journal_entry, validate_balance __all__ = [ + "CastleInventory", + "CastlePosition", + "BalanceCalculator", "ValidationError", "validate_journal_entry", "validate_balance", diff --git a/core/balance.py b/core/balance.py new file mode 100644 index 0000000..1c4a03c --- /dev/null +++ b/core/balance.py @@ -0,0 +1,228 @@ +""" +Balance calculation logic for Castle accounting. + +Pure functions for calculating account and user balances from journal entries, +following double-entry accounting principles. +""" + +from decimal import Decimal +from typing import Any, Dict, List, Optional +from enum import Enum + +from .inventory import CastleInventory, CastlePosition + + +class AccountType(str, Enum): + """Account types in double-entry accounting""" + ASSET = "asset" + LIABILITY = "liability" + EQUITY = "equity" + REVENUE = "revenue" + EXPENSE = "expense" + + +class BalanceCalculator: + """ + Pure logic for calculating balances from journal entries. + + This class contains no database access - it operates on data structures + passed to it, making it easy to test and reuse. + """ + + @staticmethod + def calculate_account_balance( + total_debit: int, + total_credit: int, + account_type: AccountType + ) -> int: + """ + Calculate account balance based on account type. + + Normal balances: + - Assets and Expenses: Debit balance (debit - credit) + - Liabilities, Equity, and Revenue: Credit balance (credit - debit) + + Args: + total_debit: Sum of all debits in satoshis + total_credit: Sum of all credits in satoshis + account_type: Type of account + + Returns: + Balance in satoshis + """ + if account_type in [AccountType.ASSET, AccountType.EXPENSE]: + return total_debit - total_credit + else: + return total_credit - total_debit + + @staticmethod + def build_inventory_from_entry_lines( + entry_lines: List[Dict[str, Any]], + account_type: AccountType + ) -> CastleInventory: + """ + Build a CastleInventory from journal entry lines. + + Args: + entry_lines: List of entry line dictionaries with keys: + - debit: int (satoshis) + - credit: int (satoshis) + - metadata: str (JSON string with optional fiat_currency, fiat_amount) + account_type: Type of account (affects sign of amounts) + + Returns: + CastleInventory with positions for sats and fiat currencies + """ + import json + + inventory = CastleInventory() + + for line in entry_lines: + # Parse metadata + metadata = json.loads(line.get("metadata", "{}")) if line.get("metadata") else {} + fiat_currency = metadata.get("fiat_currency") + fiat_amount_raw = metadata.get("fiat_amount") + + # Convert fiat amount to Decimal + fiat_amount = Decimal(str(fiat_amount_raw)) if fiat_amount_raw else None + + # Calculate amount based on debit/credit and account type + debit = line.get("debit", 0) + credit = line.get("credit", 0) + + if debit > 0: + sats_amount = Decimal(debit) + # For liability accounts: debit decreases balance (negative) + # For asset accounts: debit increases balance (positive) + if account_type == AccountType.LIABILITY: + sats_amount = -sats_amount + fiat_amount = -fiat_amount if fiat_amount else None + + inventory.add_position( + CastlePosition( + currency="SATS", + amount=sats_amount, + cost_currency=fiat_currency, + cost_amount=fiat_amount, + metadata=metadata, + ) + ) + + if credit > 0: + sats_amount = Decimal(credit) + # For liability accounts: credit increases balance (positive) + # For asset accounts: credit decreases balance (negative) + if account_type == AccountType.ASSET: + sats_amount = -sats_amount + fiat_amount = -fiat_amount if fiat_amount else None + + inventory.add_position( + CastlePosition( + currency="SATS", + amount=sats_amount, + cost_currency=fiat_currency, + cost_amount=fiat_amount, + metadata=metadata, + ) + ) + + return inventory + + @staticmethod + def calculate_user_balance( + accounts: List[Dict[str, Any]], + account_balances: Dict[str, int], + account_inventories: Dict[str, CastleInventory] + ) -> Dict[str, Any]: + """ + Calculate user's total balance across all their accounts. + + User balance represents what the Castle owes the user: + - Positive: Castle owes user + - Negative: User owes Castle + + Args: + accounts: List of account dictionaries with keys: + - id: str + - account_type: str (asset/liability/equity) + account_balances: Dict mapping account_id to balance in sats + account_inventories: Dict mapping account_id to CastleInventory + + Returns: + Dictionary with: + - balance: int (total sats, positive = castle owes user) + - fiat_balances: Dict[str, Decimal] (fiat balances by currency) + """ + total_balance = 0 + combined_inventory = CastleInventory() + + for account in accounts: + account_id = account["id"] + account_type = AccountType(account["account_type"]) + balance = account_balances.get(account_id, 0) + inventory = account_inventories.get(account_id, CastleInventory()) + + # Add sats balance based on account type + if account_type == AccountType.LIABILITY: + # Liability: positive balance means castle owes user + total_balance += balance + elif account_type == AccountType.ASSET: + # Asset (receivable): positive balance means user owes castle (negative for user) + total_balance -= balance + # Equity contributions don't affect what castle owes + + # Merge inventories for fiat tracking + for position in inventory.positions.values(): + # Adjust sign based on account type + if account_type == AccountType.ASSET: + # For receivables, negate the position + combined_inventory.add_position(position.negate()) + else: + combined_inventory.add_position(position) + + fiat_balances = combined_inventory.get_all_fiat_balances() + + return { + "balance": total_balance, + "fiat_balances": fiat_balances, + } + + @staticmethod + def check_balance_matches( + actual_balance_sats: int, + expected_balance_sats: int, + tolerance_sats: int = 0 + ) -> bool: + """ + Check if actual balance matches expected within tolerance. + + Args: + actual_balance_sats: Actual calculated balance + expected_balance_sats: Expected balance from assertion + tolerance_sats: Allowed difference (±) + + Returns: + True if balances match within tolerance + """ + difference = abs(actual_balance_sats - expected_balance_sats) + return difference <= tolerance_sats + + @staticmethod + def check_fiat_balance_matches( + actual_balance_fiat: Decimal, + expected_balance_fiat: Decimal, + tolerance_fiat: Decimal = Decimal(0) + ) -> bool: + """ + Check if actual fiat balance matches expected within tolerance. + + Args: + actual_balance_fiat: Actual calculated fiat balance + expected_balance_fiat: Expected fiat balance from assertion + tolerance_fiat: Allowed difference (±) + + Returns: + True if balances match within tolerance + """ + difference = abs(actual_balance_fiat - expected_balance_fiat) + return difference <= tolerance_fiat diff --git a/core/inventory.py b/core/inventory.py new file mode 100644 index 0000000..858ff43 --- /dev/null +++ b/core/inventory.py @@ -0,0 +1,203 @@ +""" +Inventory system for position tracking. + +Similar to Beancount's Inventory class, this module provides position tracking +across multiple currencies with cost basis information. +""" + +from dataclasses import dataclass, field +from datetime import datetime +from decimal import Decimal +from typing import Any, Dict, Optional, Tuple + + +@dataclass(frozen=True) +class CastlePosition: + """ + A position in the Castle inventory. + + Represents an amount in a specific currency, optionally with cost basis + information for tracking currency conversions. + + Examples: + # Simple sats position + CastlePosition(currency="SATS", amount=Decimal("100000")) + + # Sats with EUR cost basis + CastlePosition( + currency="SATS", + amount=Decimal("100000"), + cost_currency="EUR", + cost_amount=Decimal("50.00") + ) + """ + + currency: str # "SATS", "EUR", "USD", etc. + amount: Decimal + + # Cost basis (for tracking conversions) + cost_currency: Optional[str] = None # Original currency if converted + cost_amount: Optional[Decimal] = None # Original amount + + # Metadata + date: Optional[datetime] = None + metadata: Dict[str, Any] = field(default_factory=dict) + + def __post_init__(self): + """Validate position data""" + if not isinstance(self.amount, Decimal): + object.__setattr__(self, "amount", Decimal(str(self.amount))) + + if self.cost_amount is not None and not isinstance(self.cost_amount, Decimal): + object.__setattr__( + self, "cost_amount", Decimal(str(self.cost_amount)) + ) + + def __add__(self, other: "CastlePosition") -> "CastlePosition": + """Add two positions (must be same currency and cost_currency)""" + if self.currency != other.currency: + raise ValueError(f"Cannot add positions with different currencies: {self.currency} != {other.currency}") + + if self.cost_currency != other.cost_currency: + raise ValueError(f"Cannot add positions with different cost currencies: {self.cost_currency} != {other.cost_currency}") + + return CastlePosition( + currency=self.currency, + amount=self.amount + other.amount, + cost_currency=self.cost_currency, + cost_amount=( + (self.cost_amount or Decimal(0)) + (other.cost_amount or Decimal(0)) + if self.cost_amount is not None or other.cost_amount is not None + else None + ), + date=other.date, # Use most recent date + metadata={**self.metadata, **other.metadata}, + ) + + def negate(self) -> "CastlePosition": + """Return a position with negated amount""" + return CastlePosition( + currency=self.currency, + amount=-self.amount, + cost_currency=self.cost_currency, + cost_amount=-self.cost_amount if self.cost_amount else None, + date=self.date, + metadata=self.metadata, + ) + + +class CastleInventory: + """ + Track balances across multiple currencies with conversion tracking. + + Similar to Beancount's Inventory but optimized for Castle's use case. + Positions are keyed by (currency, cost_currency) to track different + cost bases separately. + + Examples: + inv = CastleInventory() + inv.add_position(CastlePosition("SATS", Decimal("100000"))) + inv.add_position(CastlePosition("SATS", Decimal("50000"), "EUR", Decimal("25"))) + + inv.get_balance_sats() # Returns: Decimal("150000") + inv.get_balance_fiat("EUR") # Returns: Decimal("25") + """ + + def __init__(self): + self.positions: Dict[Tuple[str, Optional[str]], CastlePosition] = {} + + def add_position(self, position: CastlePosition): + """ + Add or merge a position into the inventory. + + Positions with the same (currency, cost_currency) key are merged. + """ + key = (position.currency, position.cost_currency) + + if key in self.positions: + self.positions[key] = self.positions[key] + position + else: + self.positions[key] = position + + def get_balance_sats(self) -> Decimal: + """Get total balance in satoshis""" + return sum( + pos.amount + for (curr, _), pos in self.positions.items() + if curr == "SATS" + ) + + def get_balance_fiat(self, currency: str) -> Decimal: + """ + Get balance in specific fiat currency from cost metadata. + + This sums up all cost_amount values for positions that have + the specified cost_currency. + """ + return sum( + pos.cost_amount or Decimal(0) + for (_, cost_curr), pos in self.positions.items() + if cost_curr == currency + ) + + def get_all_fiat_balances(self) -> Dict[str, Decimal]: + """Get balances for all fiat currencies present in the inventory""" + fiat_currencies = set( + cost_curr + for _, cost_curr in self.positions.keys() + if cost_curr + ) + + return { + curr: self.get_balance_fiat(curr) + for curr in fiat_currencies + } + + def is_empty(self) -> bool: + """Check if inventory has no positions""" + return len(self.positions) == 0 + + def is_zero(self) -> bool: + """ + Check if all positions sum to zero. + + Returns True if the inventory has positions but they all sum to zero. + """ + return all( + pos.amount == Decimal(0) + for pos in self.positions.values() + ) + + def to_dict(self) -> dict: + """ + Export inventory to dictionary format. + + Returns: + { + "sats": 100000, + "fiat": { + "EUR": 50.00, + "USD": 60.00 + } + } + """ + fiat_balances = self.get_all_fiat_balances() + + return { + "sats": int(self.get_balance_sats()), + "fiat": { + curr: float(amount) + for curr, amount in fiat_balances.items() + }, + } + + def __repr__(self) -> str: + """String representation for debugging""" + if self.is_empty(): + return "CastleInventory(empty)" + + positions_str = ", ".join( + f"{curr}: {pos.amount}" + for (curr, _), pos in self.positions.items() + ) + return f"CastleInventory({positions_str})" diff --git a/core/validation.py b/core/validation.py index d2372b8..75cec02 100644 --- a/core/validation.py +++ b/core/validation.py @@ -23,13 +23,13 @@ def validate_journal_entry( entry_lines: List[Dict[str, Any]] ) -> None: """ - Validate a journal entry and its lines (Beancount-style with single amount field). + Validate a journal entry and its lines. Checks: 1. Entry must have at least 2 lines (double-entry requirement) - 2. Entry must be balanced (sum of amounts = 0) - 3. All lines must have account_id - 4. No line should have amount = 0 (would serve no purpose) + 2. Entry must be balanced (sum of debits = sum of credits) + 3. All lines must have valid amounts (non-negative) + 4. All lines must have account_id Args: entry: Journal entry dict with keys: @@ -38,7 +38,8 @@ def validate_journal_entry( - entry_date: datetime entry_lines: List of entry line dicts with keys: - account_id: str - - amount: int (positive = debit, negative = credit) + - debit: int + - credit: int Raises: ValidationError: If validation fails @@ -65,30 +66,64 @@ def validate_journal_entry( } ) - # Get amount (Beancount-style: positive = debit, negative = credit) - amount = line.get("amount", 0) + # Check amounts are non-negative + debit = line.get("debit", 0) + credit = line.get("credit", 0) - # Check that amount is non-zero (zero amounts serve no purpose) - if amount == 0: + if debit < 0: raise ValidationError( - f"Entry line {i + 1} has amount = 0 (serves no purpose)", + f"Entry line {i + 1} has negative debit: {debit}", + { + "entry_id": entry.get("id"), + "line_index": i, + "debit": debit, + } + ) + + if credit < 0: + raise ValidationError( + f"Entry line {i + 1} has negative credit: {credit}", + { + "entry_id": entry.get("id"), + "line_index": i, + "credit": credit, + } + ) + + # Check that a line doesn't have both debit and credit + if debit > 0 and credit > 0: + raise ValidationError( + f"Entry line {i + 1} has both debit and credit", + { + "entry_id": entry.get("id"), + "line_index": i, + "debit": debit, + "credit": credit, + } + ) + + # Check that a line has at least one non-zero amount + if debit == 0 and credit == 0: + raise ValidationError( + f"Entry line {i + 1} has both debit and credit as zero", { "entry_id": entry.get("id"), "line_index": i, } ) - # Check entry is balanced (sum of amounts must equal 0) - # Beancount-style: positive amounts cancel out negative amounts - total_amount = sum(line.get("amount", 0) for line in entry_lines) + # Check entry is balanced + total_debits = sum(line.get("debit", 0) for line in entry_lines) + total_credits = sum(line.get("credit", 0) for line in entry_lines) - if total_amount != 0: + if total_debits != total_credits: raise ValidationError( - "Journal entry is not balanced (sum of amounts must equal 0)", + "Journal entry is not balanced", { "entry_id": entry.get("id"), - "total_amount": total_amount, - "line_count": len(entry_lines), + "total_debits": total_debits, + "total_credits": total_credits, + "difference": total_debits - total_credits, } ) diff --git a/crud.py b/crud.py index 0976e72..81e601a 100644 --- a/crud.py +++ b/crud.py @@ -2,44 +2,30 @@ import json from datetime import datetime from typing import Optional -import httpx from lnbits.db import Database from lnbits.helpers import urlsafe_short_hash -from lnbits.utils.cache import Cache from .models import ( Account, - AccountPermission, AccountType, AssertionStatus, - AssignUserRole, BalanceAssertion, CastleSettings, CreateAccount, - CreateAccountPermission, CreateBalanceAssertion, CreateEntryLine, CreateJournalEntry, - CreateRole, - CreateRolePermission, - CreateUserEquityStatus, EntryLine, JournalEntry, - PermissionType, - Role, - RolePermission, - RoleWithPermissions, StoredUserWalletSettings, - UpdateRole, UserBalance, UserCastleSettings, - UserEquityStatus, - UserRole, UserWalletSettings, - UserWithRoles, ) # Import core accounting logic +from .core.balance import BalanceCalculator, AccountType as CoreAccountType +from .core.inventory import CastleInventory, CastlePosition from .core.validation import ( ValidationError, validate_journal_entry, @@ -51,17 +37,6 @@ from .core.validation import ( db = Database("ext_castle") -# ===== CACHING ===== -# Cache for account and permission lookups to reduce DB queries -# TTLs: accounts=300s (5min), permissions=60s (1min) - -account_cache = Cache() # 5 minutes for accounts (rarely change) -permission_cache = Cache() # 1 minute for permissions (may change frequently) - -# Cache TTLs -ACCOUNT_CACHE_TTL = 300 # 5 minutes -PERMISSION_CACHE_TTL = 60 # 1 minute - # ===== ACCOUNT OPERATIONS ===== @@ -74,120 +49,42 @@ async def create_account(data: CreateAccount) -> Account: account_type=data.account_type, description=data.description, user_id=data.user_id, - is_virtual=data.is_virtual, created_at=datetime.now(), ) await db.insert("accounts", account) - - # Invalidate cache for this account (Cache class doesn't have delete method, use pop) - account_cache._values.pop(f"account:id:{account_id}", None) - account_cache._values.pop(f"account:name:{data.name}", None) - return account async def get_account(account_id: str) -> Optional[Account]: - """Get account by ID with caching""" - cache_key = f"account:id:{account_id}" - - # Try cache first - cached = account_cache.get(cache_key) - if cached is not None: - return cached - - # Query DB - account = await db.fetchone( + return await db.fetchone( "SELECT * FROM accounts WHERE id = :id", {"id": account_id}, Account, ) - # Cache result (even if None) - account_cache.set(cache_key, account, ACCOUNT_CACHE_TTL) - - return account - async def get_account_by_name(name: str) -> Optional[Account]: - """Get account by name (hierarchical format) with caching""" - cache_key = f"account:name:{name}" - - # Try cache first - cached = account_cache.get(cache_key) - if cached is not None: - return cached - - # Query DB - account = await db.fetchone( + """Get account by name (hierarchical format)""" + return await db.fetchone( "SELECT * FROM accounts WHERE name = :name", {"name": name}, Account, ) - # Cache result (even if None) - account_cache.set(cache_key, account, ACCOUNT_CACHE_TTL) - return account - - -async def get_all_accounts(include_inactive: bool = False) -> list[Account]: - """ - Get all accounts, optionally including inactive ones. - - Args: - include_inactive: If True, include inactive accounts. Default False. - - Returns: - List of Account objects - """ - if include_inactive: - query = "SELECT * FROM accounts ORDER BY account_type, name" - else: - query = "SELECT * FROM accounts WHERE is_active = TRUE ORDER BY account_type, name" - - return await db.fetchall(query, model=Account) - - -async def get_accounts_by_type( - account_type: AccountType, include_inactive: bool = False -) -> list[Account]: - """ - Get accounts by type, optionally including inactive ones. - - Args: - account_type: The account type to filter by - include_inactive: If True, include inactive accounts. Default False. - - Returns: - List of Account objects - """ - if include_inactive: - query = "SELECT * FROM accounts WHERE account_type = :type ORDER BY name" - else: - query = "SELECT * FROM accounts WHERE account_type = :type AND is_active = TRUE ORDER BY name" - - return await db.fetchall(query, {"type": account_type.value}, Account) - - -async def update_account_is_active(account_id: str, is_active: bool) -> None: - """ - Update the is_active status of an account (soft delete/reactivate). - - Args: - account_id: Account ID to update - is_active: True to activate, False to deactivate - """ - await db.execute( - """ - UPDATE accounts - SET is_active = :is_active - WHERE id = :account_id - """, - {"account_id": account_id, "is_active": is_active}, +async def get_all_accounts() -> list[Account]: + return await db.fetchall( + "SELECT * FROM accounts ORDER BY account_type, name", + model=Account, ) - # Invalidate cache - account_cache._values.pop(f"account:id:{account_id}", None) + +async def get_accounts_by_type(account_type: AccountType) -> list[Account]: + return await db.fetchall( + "SELECT * FROM accounts WHERE account_type = :type ORDER BY name", + {"type": account_type.value}, + Account, + ) async def get_or_create_user_account( @@ -196,10 +93,6 @@ async def get_or_create_user_account( """ Get or create a user-specific account with hierarchical naming. - This function checks if the account exists in Fava/Beancount and creates it - if it doesn't exist. The account is also registered in Castle's database for - metadata tracking (permissions, descriptions, etc.). - Examples: get_or_create_user_account("af983632", AccountType.ASSET, "Accounts Receivable") → "Assets:Receivable:User-af983632" @@ -208,13 +101,11 @@ async def get_or_create_user_account( → "Liabilities:Payable:User-af983632" """ from .account_utils import format_hierarchical_account_name - from .fava_client import get_fava_client - from loguru import logger # Generate hierarchical account name account_name = format_hierarchical_account_name(account_type, base_name, user_id) - # Try to find existing account with this hierarchical name in Castle DB + # Try to find existing account with this hierarchical name account = await db.fetchone( """ SELECT * FROM accounts @@ -224,152 +115,396 @@ async def get_or_create_user_account( Account, ) - logger.info(f"[ACCOUNT CHECK] User {user_id[:8]}, Account: {account_name}, In Castle DB: {account is not None}") - - # Always check/create in Fava, even if account exists in Castle DB - # This ensures Beancount has the Open directive - fava_account_exists = False - if True: # Always check Fava - # Check if account exists in Fava/Beancount - fava = get_fava_client() - try: - # Query Fava for this account - query = f"SELECT account WHERE account = '{account_name}'" - async with httpx.AsyncClient(timeout=5.0) as client: - response = await client.get( - f"{fava.base_url}/query", - params={"query_string": query} - ) - response.raise_for_status() - result = response.json() - - # Check if account exists in Fava - fava_account_exists = len(result.get("data", {}).get("rows", [])) > 0 - logger.info(f"[FAVA CHECK] Account {account_name} exists in Fava: {fava_account_exists}") - - if not fava_account_exists: - # Create account in Fava/Beancount via Open directive - logger.info(f"[FAVA CREATE] Creating account in Fava: {account_name}") - await fava.add_account( - account_name=account_name, - currencies=["EUR", "SATS", "USD"], # Support common currencies - metadata={ - "user_id": user_id, - "description": f"User-specific {account_type.value} account" - } - ) - logger.info(f"[FAVA CREATE] Successfully created account in Fava: {account_name}") - - except Exception as e: - logger.error(f"[FAVA ERROR] Could not check/create account in Fava: {e}", exc_info=True) - # Continue anyway - account creation in Castle DB is still useful for metadata - - # Ensure account exists in Castle DB (sync from Beancount if needed) - # This uses the account sync module for consistency if not account: - logger.info(f"[CASTLE DB] Syncing account from Beancount to Castle DB: {account_name}") - from .account_sync import sync_single_account_from_beancount - - # Sync from Beancount to Castle DB - created = await sync_single_account_from_beancount(account_name) - - if created: - logger.info(f"[CASTLE DB] Account synced from Beancount: {account_name}") - else: - logger.warning(f"[CASTLE DB] Failed to sync account from Beancount: {account_name}") - - # Fetch the account from Castle DB - account = await db.fetchone( - """ - SELECT * FROM accounts - WHERE user_id = :user_id AND account_type = :type AND name = :name - """, - {"user_id": user_id, "type": account_type.value, "name": account_name}, - Account, + # Create new account with hierarchical name + account = await create_account( + CreateAccount( + name=account_name, + account_type=account_type, + description=f"User-specific {account_type.value} account", + user_id=user_id, + ) ) - if not account: - logger.error(f"[CASTLE DB] Account still not found after sync: {account_name}") - # Fallback: create directly in Castle DB if sync failed - logger.info(f"[CASTLE DB] Creating account directly in Castle DB: {account_name}") - try: - account = await create_account( - CreateAccount( - name=account_name, - account_type=account_type, - description=f"User-specific {account_type.value} account", - user_id=user_id, - ) - ) - except Exception as e: - # Handle UNIQUE constraint error - account already exists - if "UNIQUE constraint failed" in str(e) and "accounts.name" in str(e): - logger.warning(f"[CASTLE DB] Account already exists (UNIQUE constraint), fetching by name: {account_name}") - # Fetch existing account by name only (ignore user_id in query) - account = await db.fetchone( - """ - SELECT * FROM accounts - WHERE name = :name - """, - {"name": account_name}, - Account, - ) - if account: - logger.info(f"[CASTLE DB] Found existing account: {account_name} (user_id: {account.user_id})") - # Update user_id if it's NULL or different - if account.user_id != user_id: - logger.info(f"[CASTLE DB] Updating account user_id from {account.user_id} to {user_id}") - await db.execute( - """ - UPDATE accounts - SET user_id = :user_id - WHERE name = :name - """, - {"user_id": user_id, "name": account_name} - ) - # Refresh account from DB - account = await db.fetchone( - """ - SELECT * FROM accounts - WHERE name = :name - """, - {"name": account_name}, - Account, - ) - else: - # Re-raise if it's a different error - raise - else: - logger.info(f"[CASTLE DB] Account already exists in Castle DB: {account_name}") - return account # ===== JOURNAL ENTRY OPERATIONS ===== -# ===== JOURNAL ENTRY OPERATIONS (REMOVED) ===== -# -# All journal entry operations have been moved to Fava/Beancount. -# Castle no longer maintains its own journal_entries and entry_lines tables. -# -# For journal entry operations, see: -# - views_api.py: api_create_journal_entry() - writes to Fava via FavaClient -# - views_api.py: API endpoints query Fava via FavaClient for reading entries -# -# Migration: m016_drop_obsolete_journal_tables -# Removed functions: -# - create_journal_entry() -# - get_journal_entry() -# - get_journal_entry_by_reference() -# - get_entry_lines() -# - get_all_journal_entries() -# - get_journal_entries_by_user() -# - count_all_journal_entries() -# - count_journal_entries_by_user() -# - get_journal_entries_by_user_and_account_type() -# - count_journal_entries_by_user_and_account_type() -# - get_account_transactions() +async def create_journal_entry( + data: CreateJournalEntry, created_by: str +) -> JournalEntry: + entry_id = urlsafe_short_hash() + + # Validate that debits equal credits + total_debits = sum(line.debit for line in data.lines) + total_credits = sum(line.credit for line in data.lines) + + if total_debits != total_credits: + raise ValueError( + f"Journal entry must balance: debits={total_debits}, credits={total_credits}" + ) + + entry_date = data.entry_date or datetime.now() + + journal_entry = JournalEntry( + id=entry_id, + description=data.description, + entry_date=entry_date, + created_by=created_by, + created_at=datetime.now(), + reference=data.reference, + lines=[], + flag=data.flag, + meta=data.meta, + ) + + # Insert journal entry without the lines field (lines are stored in entry_lines table) + await db.execute( + """ + INSERT INTO journal_entries (id, description, entry_date, created_by, created_at, reference, flag, meta) + VALUES (:id, :description, :entry_date, :created_by, :created_at, :reference, :flag, :meta) + """, + { + "id": journal_entry.id, + "description": journal_entry.description, + "entry_date": journal_entry.entry_date, + "created_by": journal_entry.created_by, + "created_at": journal_entry.created_at, + "reference": journal_entry.reference, + "flag": journal_entry.flag.value, + "meta": json.dumps(journal_entry.meta), + }, + ) + + # Create entry lines + lines = [] + for line_data in data.lines: + line_id = urlsafe_short_hash() + line = EntryLine( + id=line_id, + journal_entry_id=entry_id, + account_id=line_data.account_id, + debit=line_data.debit, + credit=line_data.credit, + description=line_data.description, + metadata=line_data.metadata, + ) + # Insert with metadata as JSON string + await db.execute( + """ + INSERT INTO entry_lines (id, journal_entry_id, account_id, debit, credit, description, metadata) + VALUES (:id, :journal_entry_id, :account_id, :debit, :credit, :description, :metadata) + """, + { + "id": line.id, + "journal_entry_id": line.journal_entry_id, + "account_id": line.account_id, + "debit": line.debit, + "credit": line.credit, + "description": line.description, + "metadata": json.dumps(line.metadata), + }, + ) + lines.append(line) + + journal_entry.lines = lines + return journal_entry + + +async def get_journal_entry(entry_id: str) -> Optional[JournalEntry]: + entry = await db.fetchone( + "SELECT * FROM journal_entries WHERE id = :id", + {"id": entry_id}, + JournalEntry, + ) + + if entry: + entry.lines = await get_entry_lines(entry_id) + + return entry + + +async def get_journal_entry_by_reference(reference: str) -> Optional[JournalEntry]: + """Get a journal entry by its reference field (e.g., payment_hash)""" + entry = await db.fetchone( + "SELECT * FROM journal_entries WHERE reference = :reference", + {"reference": reference}, + JournalEntry, + ) + + if entry: + entry.lines = await get_entry_lines(entry.id) + + return entry + + +async def get_entry_lines(journal_entry_id: str) -> list[EntryLine]: + rows = await db.fetchall( + "SELECT * FROM entry_lines WHERE journal_entry_id = :id", + {"id": journal_entry_id}, + ) + + lines = [] + for row in rows: + # Parse metadata from JSON string + metadata = json.loads(row.metadata) if row.metadata else {} + line = EntryLine( + id=row.id, + journal_entry_id=row.journal_entry_id, + account_id=row.account_id, + debit=row.debit, + credit=row.credit, + description=row.description, + metadata=metadata, + ) + lines.append(line) + + return lines + + +async def get_all_journal_entries(limit: int = 100) -> list[JournalEntry]: + entries_data = await db.fetchall( + """ + SELECT * FROM journal_entries + ORDER BY entry_date DESC, created_at DESC + LIMIT :limit + """, + {"limit": limit}, + ) + + entries = [] + for entry_data in entries_data: + # Parse flag and meta from database + from .models import JournalEntryFlag + flag = JournalEntryFlag(entry_data.get("flag", "*")) + meta = json.loads(entry_data.get("meta", "{}")) if entry_data.get("meta") else {} + + entry = JournalEntry( + id=entry_data["id"], + description=entry_data["description"], + entry_date=entry_data["entry_date"], + created_by=entry_data["created_by"], + created_at=entry_data["created_at"], + reference=entry_data["reference"], + flag=flag, + meta=meta, + lines=[], + ) + entry.lines = await get_entry_lines(entry.id) + entries.append(entry) + + return entries + + +async def get_journal_entries_by_user( + user_id: str, limit: int = 100 +) -> list[JournalEntry]: + """Get journal entries that affect the user's accounts""" + # Get all user-specific accounts + user_accounts = await db.fetchall( + "SELECT id FROM accounts WHERE user_id = :user_id", + {"user_id": user_id}, + ) + + if not user_accounts: + return [] + + account_ids = [acc["id"] for acc in user_accounts] + + # Get all journal entries that have lines affecting these accounts + # Build the IN clause with named parameters + placeholders = ','.join([f":account_{i}" for i in range(len(account_ids))]) + params = {f"account_{i}": acc_id for i, acc_id in enumerate(account_ids)} + params["limit"] = limit + + entries_data = await db.fetchall( + f""" + SELECT DISTINCT je.* + FROM journal_entries je + JOIN entry_lines el ON je.id = el.journal_entry_id + WHERE el.account_id IN ({placeholders}) + ORDER BY je.entry_date DESC, je.created_at DESC + LIMIT :limit + """, + params, + ) + + entries = [] + for entry_data in entries_data: + # Parse flag and meta from database + from .models import JournalEntryFlag + flag = JournalEntryFlag(entry_data.get("flag", "*")) + meta = json.loads(entry_data.get("meta", "{}")) if entry_data.get("meta") else {} + + entry = JournalEntry( + id=entry_data["id"], + description=entry_data["description"], + entry_date=entry_data["entry_date"], + created_by=entry_data["created_by"], + created_at=entry_data["created_at"], + reference=entry_data["reference"], + flag=flag, + meta=meta, + lines=[], + ) + entry.lines = await get_entry_lines(entry.id) + entries.append(entry) + + return entries + + +# ===== BALANCE AND REPORTING ===== + + +async def get_account_balance(account_id: str) -> int: + """Calculate account balance (debits - credits for assets/expenses, credits - debits for liabilities/equity/revenue) + Only includes entries that are cleared (flag='*'), excludes pending/flagged/voided entries.""" + result = await db.fetchone( + """ + SELECT + COALESCE(SUM(el.debit), 0) as total_debit, + COALESCE(SUM(el.credit), 0) as total_credit + FROM entry_lines el + JOIN journal_entries je ON el.journal_entry_id = je.id + WHERE el.account_id = :id + AND je.flag = '*' + """, + {"id": account_id}, + ) + + if not result: + return 0 + + account = await get_account(account_id) + if not account: + return 0 + + total_debit = result["total_debit"] + total_credit = result["total_credit"] + + # Use core BalanceCalculator for consistent logic + core_account_type = CoreAccountType(account.account_type.value) + return BalanceCalculator.calculate_account_balance( + total_debit, total_credit, core_account_type + ) + + +async def get_user_balance(user_id: str) -> UserBalance: + """Get user's balance with the Castle (positive = castle owes user, negative = user owes castle)""" + # Get all user-specific accounts + user_accounts = await db.fetchall( + "SELECT * FROM accounts WHERE user_id = :user_id", + {"user_id": user_id}, + Account, + ) + + # Calculate balances for each account + account_balances = {} + account_inventories = {} + + for account in user_accounts: + # Get satoshi balance + balance = await get_account_balance(account.id) + account_balances[account.id] = balance + + # Get all entry lines for this account to build inventory + # Only include cleared entries (exclude pending/flagged/voided) + entry_lines = await db.fetchall( + """ + SELECT el.* + FROM entry_lines el + JOIN journal_entries je ON el.journal_entry_id = je.id + WHERE el.account_id = :account_id + AND je.flag = '*' + """, + {"account_id": account.id}, + ) + + # Use BalanceCalculator to build inventory from entry lines + core_account_type = CoreAccountType(account.account_type.value) + inventory = BalanceCalculator.build_inventory_from_entry_lines( + [dict(line) for line in entry_lines], + core_account_type + ) + account_inventories[account.id] = inventory + + # Use BalanceCalculator to calculate total user balance + accounts_list = [ + {"id": acc.id, "account_type": acc.account_type.value} + for acc in user_accounts + ] + balance_result = BalanceCalculator.calculate_user_balance( + accounts_list, + account_balances, + account_inventories + ) + + return UserBalance( + user_id=user_id, + balance=balance_result["balance"], + accounts=user_accounts, + fiat_balances=balance_result["fiat_balances"], + ) + + +async def get_all_user_balances() -> list[UserBalance]: + """Get balances for all users (used by castle to see who they owe)""" + # Get all user-specific accounts + all_accounts = await db.fetchall( + "SELECT * FROM accounts WHERE user_id IS NOT NULL", + {}, + Account, + ) + + # Get unique user IDs + user_ids = set(account.user_id for account in all_accounts if account.user_id) + + # Calculate balance for each user using the refactored function + user_balances = [] + for user_id in user_ids: + balance = await get_user_balance(user_id) + + # Include users with non-zero balance or fiat balances + if balance.balance != 0 or balance.fiat_balances: + user_balances.append(balance) + + return user_balances + + +async def get_account_transactions( + account_id: str, limit: int = 100 +) -> list[tuple[JournalEntry, EntryLine]]: + """Get all transactions affecting a specific account""" + rows = await db.fetchall( + """ + SELECT * FROM entry_lines + WHERE account_id = :id + ORDER BY id DESC + LIMIT :limit + """, + {"id": account_id, "limit": limit}, + ) + + transactions = [] + for row in rows: + # Parse metadata from JSON string + metadata = json.loads(row.metadata) if row.metadata else {} + line = EntryLine( + id=row.id, + journal_entry_id=row.journal_entry_id, + account_id=row.account_id, + debit=row.debit, + credit=row.credit, + description=row.description, + metadata=metadata, + ) + entry = await get_journal_entry(line.journal_entry_id) + if entry: + transactions.append((entry, line)) + + return transactions # ===== SETTINGS ===== @@ -746,31 +881,26 @@ async def check_balance_assertion(assertion_id: str) -> BalanceAssertion: """ Check a balance assertion by comparing expected vs actual balance. Updates the assertion with the check results. - Uses Fava/Beancount for balance queries. """ from decimal import Decimal - from .fava_client import get_fava_client assertion = await get_balance_assertion(assertion_id) if not assertion: raise ValueError(f"Balance assertion {assertion_id} not found") - # Get actual account balance from Fava + # Get actual account balance account = await get_account(assertion.account_id) if not account: raise ValueError(f"Account {assertion.account_id} not found") - fava = get_fava_client() - - # Get balance from Fava - balance_data = await fava.get_account_balance(account.name) - actual_balance = balance_data["sats"] + # Calculate balance at the assertion date + actual_balance = await get_account_balance(assertion.account_id) # Get fiat balance if needed actual_fiat_balance = None if assertion.fiat_currency and account.user_id: - user_balance_data = await fava.get_user_balance(account.user_id) - actual_fiat_balance = user_balance_data["fiat_balances"].get(assertion.fiat_currency, Decimal("0")) + user_balance = await get_user_balance(account.user_id) + actual_fiat_balance = user_balance.fiat_balances.get(assertion.fiat_currency, Decimal("0")) # Check sats balance difference_sats = actual_balance - assertion.expected_balance_sats @@ -819,856 +949,3 @@ async def delete_balance_assertion(assertion_id: str) -> None: "DELETE FROM balance_assertions WHERE id = :id", {"id": assertion_id}, ) - - -# User Equity Status CRUD operations - - -async def get_user_equity_status(user_id: str) -> Optional["UserEquityStatus"]: - """Get user's equity eligibility status""" - from .models import UserEquityStatus - - row = await db.fetchone( - """ - SELECT * FROM user_equity_status - WHERE user_id = :user_id - """, - {"user_id": user_id}, - ) - - return UserEquityStatus(**row) if row else None - - -async def create_or_update_user_equity_status( - data: "CreateUserEquityStatus", granted_by: str -) -> "UserEquityStatus": - """Create or update user equity eligibility status""" - from datetime import datetime - from .models import UserEquityStatus, AccountType - import uuid - - # Auto-create user-specific equity account if granting eligibility - if data.is_equity_eligible: - # Generate equity account name: Equity:User-{user_id} - equity_account_name = f"Equity:User-{data.user_id[:8]}" - - # Check if the equity account already exists - equity_account = await get_account_by_name(equity_account_name) - - if not equity_account: - # Create the user-specific equity account - await db.execute( - """ - INSERT INTO accounts (id, name, account_type, description, user_id, created_at) - VALUES (:id, :name, :type, :description, :user_id, :created_at) - """, - { - "id": str(uuid.uuid4()), - "name": equity_account_name, - "type": AccountType.EQUITY.value, - "description": f"Equity contributions for user {data.user_id[:8]}", - "user_id": data.user_id, - "created_at": datetime.now(), - }, - ) - - # Auto-populate equity_account_name in the data - data.equity_account_name = equity_account_name - - # Check if user already has equity status - existing = await get_user_equity_status(data.user_id) - - if existing: - # Update existing record - await db.execute( - """ - UPDATE user_equity_status - SET is_equity_eligible = :is_equity_eligible, - equity_account_name = :equity_account_name, - notes = :notes, - granted_by = :granted_by, - granted_at = :granted_at, - revoked_at = :revoked_at - WHERE user_id = :user_id - """, - { - "user_id": data.user_id, - "is_equity_eligible": data.is_equity_eligible, - "equity_account_name": data.equity_account_name, - "notes": data.notes, - "granted_by": granted_by, - "granted_at": datetime.now(), - "revoked_at": None if data.is_equity_eligible else datetime.now(), - }, - ) - else: - # Create new record - await db.execute( - """ - INSERT INTO user_equity_status ( - user_id, is_equity_eligible, equity_account_name, - notes, granted_by, granted_at - ) - VALUES ( - :user_id, :is_equity_eligible, :equity_account_name, - :notes, :granted_by, :granted_at - ) - """, - { - "user_id": data.user_id, - "is_equity_eligible": data.is_equity_eligible, - "equity_account_name": data.equity_account_name, - "notes": data.notes, - "granted_by": granted_by, - "granted_at": datetime.now(), - }, - ) - - # Return the created/updated record - result = await get_user_equity_status(data.user_id) - if not result: - raise ValueError(f"Failed to create/update equity status for user {data.user_id}") - return result - - -async def revoke_user_equity_eligibility(user_id: str) -> Optional["UserEquityStatus"]: - """Revoke user's equity contribution eligibility""" - from datetime import datetime - - await db.execute( - """ - UPDATE user_equity_status - SET is_equity_eligible = FALSE, - revoked_at = :revoked_at - WHERE user_id = :user_id - """, - {"user_id": user_id, "revoked_at": datetime.now()}, - ) - - return await get_user_equity_status(user_id) - - -async def get_all_equity_eligible_users() -> list["UserEquityStatus"]: - """Get all equity-eligible users""" - from .models import UserEquityStatus - - rows = await db.fetchall( - """ - SELECT * FROM user_equity_status - WHERE is_equity_eligible = TRUE - ORDER BY granted_at DESC - """ - ) - - return [UserEquityStatus(**row) for row in rows] - - -# ===== ACCOUNT PERMISSION OPERATIONS ===== - - -async def create_account_permission( - data: "CreateAccountPermission", granted_by: str -) -> "AccountPermission": - """ - Create a new account permission. - - Raises: - ValueError: If account is inactive or doesn't exist - """ - from .models import AccountPermission - - # Validate account exists and is active - account = await get_account(data.account_id) - if not account: - raise ValueError(f"Account {data.account_id} not found") - if not account.is_active: - raise ValueError( - f"Cannot grant permission on inactive account: {account.name}" - ) - - permission_id = urlsafe_short_hash() - permission = AccountPermission( - id=permission_id, - user_id=data.user_id, - account_id=data.account_id, - permission_type=data.permission_type, - granted_by=granted_by, - granted_at=datetime.now(), - expires_at=data.expires_at, - notes=data.notes, - ) - - await db.execute( - """ - INSERT INTO account_permissions ( - id, user_id, account_id, permission_type, granted_by, - granted_at, expires_at, notes - ) - VALUES ( - :id, :user_id, :account_id, :permission_type, :granted_by, - :granted_at, :expires_at, :notes - ) - """, - { - "id": permission.id, - "user_id": permission.user_id, - "account_id": permission.account_id, - "permission_type": permission.permission_type.value, - "granted_by": permission.granted_by, - "granted_at": permission.granted_at, - "expires_at": permission.expires_at, - "notes": permission.notes, - }, - ) - - # Invalidate permission cache for this user (Cache class doesn't have delete method, use pop) - permission_cache._values.pop(f"permissions:user:{permission.user_id}", None) - permission_cache._values.pop(f"permissions:user:{permission.user_id}:{permission.permission_type.value}", None) - - return permission - - -async def get_account_permission(permission_id: str) -> Optional["AccountPermission"]: - """Get account permission by ID""" - from .models import AccountPermission, PermissionType - - row = await db.fetchone( - "SELECT * FROM account_permissions WHERE id = :id", - {"id": permission_id}, - ) - - if not row: - return None - - return AccountPermission( - id=row["id"], - user_id=row["user_id"], - account_id=row["account_id"], - permission_type=PermissionType(row["permission_type"]), - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - - -async def get_user_permissions( - user_id: str, permission_type: Optional["PermissionType"] = None -) -> list["AccountPermission"]: - """Get all permissions for a specific user with caching""" - from .models import AccountPermission, PermissionType - - # Build cache key - cache_key = f"permissions:user:{user_id}" - if permission_type: - cache_key += f":{permission_type.value}" - - # Try cache first - cached = permission_cache.get(cache_key) - if cached is not None: - return cached - - # Query DB - if permission_type: - rows = await db.fetchall( - """ - SELECT * FROM account_permissions - WHERE user_id = :user_id - AND permission_type = :permission_type - AND (expires_at IS NULL OR expires_at > :now) - ORDER BY granted_at DESC - """, - { - "user_id": user_id, - "permission_type": permission_type.value, - "now": datetime.now(), - }, - ) - else: - rows = await db.fetchall( - """ - SELECT * FROM account_permissions - WHERE user_id = :user_id - AND (expires_at IS NULL OR expires_at > :now) - ORDER BY granted_at DESC - """, - {"user_id": user_id, "now": datetime.now()}, - ) - - permissions = [ - AccountPermission( - id=row["id"], - user_id=row["user_id"], - account_id=row["account_id"], - permission_type=PermissionType(row["permission_type"]), - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - for row in rows - ] - - # Cache result - permission_cache.set(cache_key, permissions, PERMISSION_CACHE_TTL) - - return permissions - - -async def get_account_permissions(account_id: str) -> list["AccountPermission"]: - """Get all permissions for a specific account""" - from .models import AccountPermission, PermissionType - - rows = await db.fetchall( - """ - SELECT * FROM account_permissions - WHERE account_id = :account_id - AND (expires_at IS NULL OR expires_at > :now) - ORDER BY granted_at DESC - """, - {"account_id": account_id, "now": datetime.now()}, - ) - - return [ - AccountPermission( - id=row["id"], - user_id=row["user_id"], - account_id=row["account_id"], - permission_type=PermissionType(row["permission_type"]), - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - for row in rows - ] - - -async def delete_account_permission(permission_id: str) -> None: - """Delete (revoke) an account permission""" - # Get permission first to invalidate cache - permission = await get_account_permission(permission_id) - - await db.execute( - "DELETE FROM account_permissions WHERE id = :id", - {"id": permission_id}, - ) - - # Invalidate permission cache for this user (Cache class doesn't have delete method, use pop) - if permission: - permission_cache._values.pop(f"permissions:user:{permission.user_id}", None) - permission_cache._values.pop(f"permissions:user:{permission.user_id}:{permission.permission_type.value}", None) - - -async def check_user_has_permission( - user_id: str, account_id: str, permission_type: "PermissionType" -) -> bool: - """Check if user has a specific permission on an account (direct permission only, no inheritance)""" - row = await db.fetchone( - """ - SELECT id FROM account_permissions - WHERE user_id = :user_id - AND account_id = :account_id - AND permission_type = :permission_type - AND (expires_at IS NULL OR expires_at > :now) - """, - { - "user_id": user_id, - "account_id": account_id, - "permission_type": permission_type.value, - "now": datetime.now(), - }, - ) - - return row is not None - - -async def get_user_permissions_with_inheritance( - user_id: str, account_name: str, permission_type: "PermissionType" -) -> list[tuple["AccountPermission", Optional[str]]]: - """ - Get all permissions for a user on an account, including inherited permissions from parent accounts. - Includes both direct permissions AND role-based permissions. - Returns list of tuples: (permission, parent_account_name or None) - - Example: - If user has permission on "Expenses:Food", they also have permission on "Expenses:Food:Groceries" - Returns: [(permission_on_food, "Expenses:Food")] - """ - from .models import AccountPermission, PermissionType - - # Get direct user permissions of this type - direct_permissions = await get_user_permissions(user_id, permission_type) - - # Get role-based permissions of this type - role_permissions_list = await get_user_permissions_from_roles(user_id) - role_perms = [] - for role, perms in role_permissions_list: - # Filter for the specific permission type - role_perms.extend([p for p in perms if p.permission_type == permission_type]) - - # Combine direct and role-based permissions - all_permissions = list(direct_permissions) + role_perms - - # Find which permissions apply to this account (direct or inherited) - applicable_permissions = [] - - for perm in all_permissions: - # Get the account for this permission - account = await get_account(perm.account_id) - if not account: - continue - - # Check if this account is a parent of the target account - # Parent accounts are indicated by hierarchical names (colon-separated) - # e.g., "Expenses:Food" is parent of "Expenses:Food:Groceries" - if account_name == account.name: - # Direct permission - applicable_permissions.append((perm, None)) - elif account_name.startswith(account.name + ":"): - # Inherited permission from parent account - applicable_permissions.append((perm, account.name)) - - return applicable_permissions - - -# ===== ROLE-BASED ACCESS CONTROL (RBAC) OPERATIONS ===== - - -async def create_role(data: CreateRole, created_by: str) -> Role: - """Create a new role""" - role_id = urlsafe_short_hash() - role = Role( - id=role_id, - name=data.name, - description=data.description, - is_default=data.is_default, - created_by=created_by, - created_at=datetime.now(), - ) - - await db.execute( - """ - INSERT INTO roles (id, name, description, is_default, created_by, created_at) - VALUES (:id, :name, :description, :is_default, :created_by, :created_at) - """, - { - "id": role.id, - "name": role.name, - "description": role.description, - "is_default": role.is_default, - "created_by": role.created_by, - "created_at": role.created_at, - }, - ) - - return role - - -async def get_role(role_id: str) -> Optional[Role]: - """Get role by ID""" - row = await db.fetchone( - "SELECT * FROM roles WHERE id = :id", - {"id": role_id}, - ) - - if not row: - return None - - return Role( - id=row["id"], - name=row["name"], - description=row["description"], - is_default=row["is_default"], - created_by=row["created_by"], - created_at=row["created_at"], - ) - - -async def get_role_by_name(name: str) -> Optional[Role]: - """Get role by name""" - row = await db.fetchone( - "SELECT * FROM roles WHERE name = :name", - {"name": name}, - ) - - if not row: - return None - - return Role( - id=row["id"], - name=row["name"], - description=row["description"], - is_default=row["is_default"], - created_by=row["created_by"], - created_at=row["created_at"], - ) - - -async def get_all_roles() -> list[Role]: - """Get all roles""" - rows = await db.fetchall( - "SELECT * FROM roles ORDER BY name", - ) - - return [ - Role( - id=row["id"], - name=row["name"], - description=row["description"], - is_default=row["is_default"], - created_by=row["created_by"], - created_at=row["created_at"], - ) - for row in rows - ] - - -async def get_default_role() -> Optional[Role]: - """Get the default role that is auto-assigned to new users""" - row = await db.fetchone( - "SELECT * FROM roles WHERE is_default = TRUE LIMIT 1", - ) - - if not row: - return None - - return Role( - id=row["id"], - name=row["name"], - description=row["description"], - is_default=row["is_default"], - created_by=row["created_by"], - created_at=row["created_at"], - ) - - -async def update_role(role_id: str, data: UpdateRole) -> Optional[Role]: - """Update a role""" - # If setting this role as default, unset any other default roles - if data.is_default is True: - await db.execute( - "UPDATE roles SET is_default = FALSE WHERE id != :id", - {"id": role_id}, - ) - - # Build update statement dynamically based on provided fields - updates = [] - params = {"id": role_id} - - if data.name is not None: - updates.append("name = :name") - params["name"] = data.name - - if data.description is not None: - updates.append("description = :description") - params["description"] = data.description - - if data.is_default is not None: - updates.append("is_default = :is_default") - params["is_default"] = data.is_default - - if not updates: - return await get_role(role_id) - - await db.execute( - f"UPDATE roles SET {', '.join(updates)} WHERE id = :id", - params, - ) - - return await get_role(role_id) - - -async def delete_role(role_id: str) -> None: - """Delete a role (cascade deletes role_permissions and user_roles)""" - await db.execute( - "DELETE FROM roles WHERE id = :id", - {"id": role_id}, - ) - - -# ===== ROLE PERMISSION OPERATIONS ===== - - -async def create_role_permission(data: CreateRolePermission) -> RolePermission: - """Create a permission for a role""" - permission_id = urlsafe_short_hash() - permission = RolePermission( - id=permission_id, - role_id=data.role_id, - account_id=data.account_id, - permission_type=data.permission_type, - notes=data.notes, - created_at=datetime.now(), - ) - - await db.execute( - """ - INSERT INTO role_permissions (id, role_id, account_id, permission_type, notes, created_at) - VALUES (:id, :role_id, :account_id, :permission_type, :notes, :created_at) - """, - { - "id": permission.id, - "role_id": permission.role_id, - "account_id": permission.account_id, - "permission_type": permission.permission_type.value, - "notes": permission.notes, - "created_at": permission.created_at, - }, - ) - - return permission - - -async def get_role_permissions(role_id: str) -> list[RolePermission]: - """Get all permissions for a specific role""" - rows = await db.fetchall( - """ - SELECT * FROM role_permissions - WHERE role_id = :role_id - ORDER BY created_at DESC - """, - {"role_id": role_id}, - ) - - return [ - RolePermission( - id=row["id"], - role_id=row["role_id"], - account_id=row["account_id"], - permission_type=PermissionType(row["permission_type"]), - notes=row["notes"], - created_at=row["created_at"], - ) - for row in rows - ] - - -async def delete_role_permission(permission_id: str) -> None: - """Delete a role permission""" - await db.execute( - "DELETE FROM role_permissions WHERE id = :id", - {"id": permission_id}, - ) - - -# ===== USER ROLE OPERATIONS ===== - - -async def assign_user_role(data: AssignUserRole, granted_by: str) -> UserRole: - """Assign a user to a role""" - user_role_id = urlsafe_short_hash() - user_role = UserRole( - id=user_role_id, - user_id=data.user_id, - role_id=data.role_id, - granted_by=granted_by, - granted_at=datetime.now(), - expires_at=data.expires_at, - notes=data.notes, - ) - - await db.execute( - """ - INSERT INTO user_roles (id, user_id, role_id, granted_by, granted_at, expires_at, notes) - VALUES (:id, :user_id, :role_id, :granted_by, :granted_at, :expires_at, :notes) - """, - { - "id": user_role.id, - "user_id": user_role.user_id, - "role_id": user_role.role_id, - "granted_by": user_role.granted_by, - "granted_at": user_role.granted_at, - "expires_at": user_role.expires_at, - "notes": user_role.notes, - }, - ) - - return user_role - - -async def get_user_roles(user_id: str) -> list[UserRole]: - """Get all active roles for a user""" - rows = await db.fetchall( - """ - SELECT * FROM user_roles - WHERE user_id = :user_id - AND (expires_at IS NULL OR expires_at > :now) - ORDER BY granted_at DESC - """, - {"user_id": user_id, "now": datetime.now()}, - ) - - return [ - UserRole( - id=row["id"], - user_id=row["user_id"], - role_id=row["role_id"], - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - for row in rows - ] - - -async def get_all_user_roles() -> list[UserRole]: - """Get all active user role assignments""" - rows = await db.fetchall( - """ - SELECT * FROM user_roles - WHERE (expires_at IS NULL OR expires_at > :now) - ORDER BY user_id, granted_at DESC - """, - {"now": datetime.now()}, - ) - - return [ - UserRole( - id=row["id"], - user_id=row["user_id"], - role_id=row["role_id"], - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - for row in rows - ] - - -async def get_role_users(role_id: str) -> list[UserRole]: - """Get all users assigned to a role""" - rows = await db.fetchall( - """ - SELECT * FROM user_roles - WHERE role_id = :role_id - AND (expires_at IS NULL OR expires_at > :now) - ORDER BY granted_at DESC - """, - {"role_id": role_id, "now": datetime.now()}, - ) - - return [ - UserRole( - id=row["id"], - user_id=row["user_id"], - role_id=row["role_id"], - granted_by=row["granted_by"], - granted_at=row["granted_at"], - expires_at=row["expires_at"], - notes=row["notes"], - ) - for row in rows - ] - - -async def revoke_user_role(user_role_id: str) -> None: - """Revoke a user's role assignment""" - await db.execute( - "DELETE FROM user_roles WHERE id = :id", - {"id": user_role_id}, - ) - - -async def get_role_count_for_user(user_id: str) -> int: - """Get count of active roles for a user""" - row = await db.fetchone( - """ - SELECT COUNT(*) as count FROM user_roles - WHERE user_id = :user_id - AND (expires_at IS NULL OR expires_at > :now) - """, - {"user_id": user_id, "now": datetime.now()}, - ) - - return row["count"] if row else 0 - - -async def auto_assign_default_role(user_id: str, assigned_by: str) -> UserRole | None: - """ - Auto-assign the default role to a user if they don't have any roles yet. - Returns the created UserRole if assigned, None if user already has roles or no default role exists. - """ - from loguru import logger - - logger.info(f"[AUTO-ASSIGN] Checking auto-assignment for user {user_id}") - - # Check if user already has any roles - user_role_count = await get_role_count_for_user(user_id) - logger.info(f"[AUTO-ASSIGN] User {user_id} has {user_role_count} roles") - if user_role_count > 0: - logger.info(f"[AUTO-ASSIGN] User {user_id} already has roles, skipping auto-assignment") - return None - - # Find the default role - default_role = await get_default_role() - if not default_role: - logger.warning(f"[AUTO-ASSIGN] No default role found, cannot auto-assign for user {user_id}") - return None - - logger.info(f"[AUTO-ASSIGN] Found default role: {default_role.name} (id: {default_role.id})") - - # Assign the default role - data = AssignUserRole( - user_id=user_id, - role_id=default_role.id, - notes="Auto-assigned default role on first access", - ) - result = await assign_user_role(data, assigned_by) - logger.info(f"[AUTO-ASSIGN] Successfully assigned role {default_role.name} to user {user_id}") - return result - - -async def get_user_count_for_role(role_id: str) -> int: - """Get count of users assigned to a role""" - row = await db.fetchone( - """ - SELECT COUNT(*) as count FROM user_roles - WHERE role_id = :role_id - AND (expires_at IS NULL OR expires_at > :now) - """, - {"role_id": role_id, "now": datetime.now()}, - ) - - return row["count"] if row else 0 - - -# ===== RBAC HELPER FUNCTIONS ===== - - -async def get_user_permissions_from_roles( - user_id: str, -) -> list[tuple[Role, list[RolePermission]]]: - """ - Get all permissions a user has through their role assignments. - Returns list of tuples: (role, list of permissions from that role) - """ - # Get user's active roles - user_roles = await get_user_roles(user_id) - - result = [] - for user_role in user_roles: - role = await get_role(user_role.role_id) - if role: - permissions = await get_role_permissions(role.id) - result.append((role, permissions)) - - return result - - -async def check_user_has_role_permission( - user_id: str, account_id: str, permission_type: PermissionType -) -> bool: - """Check if user has a specific permission through any of their roles""" - # Get all permissions from user's roles - role_permissions = await get_user_permissions_from_roles(user_id) - - # Check if any role grants the required permission on this account - for role, permissions in role_permissions: - for perm in permissions: - if perm.account_id == account_id and perm.permission_type == permission_type: - return True - - return False diff --git a/docs/ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md b/docs/ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md deleted file mode 100644 index f5528f5..0000000 --- a/docs/ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md +++ /dev/null @@ -1,850 +0,0 @@ -# Account Sync & Permission Management Improvements - -**Date**: November 10, 2025 -**Status**: ✅ **Implemented** -**Related**: PERMISSIONS-SYSTEM.md, ACCOUNTS-TABLE-REMOVAL-FEASIBILITY.md - ---- - -## Summary - -Implemented two major improvements for Castle administration: - -1. **Account Synchronization** - Automatically sync accounts from Beancount → Castle DB -2. **Bulk Permission Management** - Tools for managing permissions at scale - -**Total Implementation Time**: ~4 hours -**Lines of Code Added**: ~750 lines -**Immediate Benefits**: 50-70% reduction in admin time - ---- - -## Part 1: Account Synchronization - -### Problem Solved - -**Before**: Accounts existed in both Beancount and Castle DB, with manual sync required. -**After**: Automatic sync keeps Castle DB in sync with Beancount (source of truth). - -### Implementation - -**New Module**: `castle/account_sync.py` - -**Core Functions**: - -```python -# 1. Full sync from Beancount to Castle -stats = await sync_accounts_from_beancount(force_full_sync=False) - -# 2. Sync single account -success = await sync_single_account_from_beancount("Expenses:Food") - -# 3. Ensure account exists (recommended before granting permissions) -exists = await ensure_account_exists_in_castle("Expenses:Marketing") - -# 4. Scheduled background sync (run hourly) -stats = await scheduled_account_sync() -``` - -### Key Features - -✅ **Automatic Type Inference**: -```python -"Assets:Cash" → AccountType.ASSET -"Expenses:Food" → AccountType.EXPENSE -"Income:Services" → AccountType.REVENUE -``` - -✅ **User ID Extraction**: -```python -"Assets:Receivable:User-abc123def" → user_id: "abc123def" -"Liabilities:Payable:User-xyz789" → user_id: "xyz789" -``` - -✅ **Metadata Preservation**: -- Imports descriptions from Beancount metadata -- Preserves user associations -- Tracks which accounts were synced - -✅ **Comprehensive Error Handling**: -- Continues on individual account failures -- Returns detailed statistics -- Logs all errors for debugging - -### Usage Examples - -#### Manual Sync (Admin Operation) - -```python -# Sync all accounts from Beancount -from castle.account_sync import sync_accounts_from_beancount - -stats = await sync_accounts_from_beancount() - -print(f"Added: {stats['accounts_added']}") -print(f"Skipped: {stats['accounts_skipped']}") -print(f"Errors: {len(stats['errors'])}") -``` - -**Output**: -``` -Added: 12 -Skipped: 138 -Errors: 0 -``` - -#### Before Granting Permission (Best Practice) - -```python -from castle.account_sync import ensure_account_exists_in_castle -from castle.crud import create_account_permission - -# Ensure account exists in Castle DB first -account_exists = await ensure_account_exists_in_castle("Expenses:Marketing") - -if account_exists: - # Now safe to grant permission - await create_account_permission( - user_id="alice", - account_name="Expenses:Marketing", # Now guaranteed to exist - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin" - ) -``` - -#### Scheduled Background Sync - -```python -# Add to your scheduler (cron, APScheduler, etc.) -from castle.account_sync import scheduled_account_sync - -# Run every hour to keep Castle DB in sync -scheduler.add_job( - scheduled_account_sync, - 'interval', - hours=1, - id='account_sync' -) -``` - -### API Endpoint (Admin Only) - -```http -POST /api/v1/admin/sync-accounts -Authorization: Bearer {admin_key} - -{ - "force_full_sync": false -} -``` - -**Response**: -```json -{ - "total_beancount_accounts": 150, - "total_castle_accounts": 150, - "accounts_added": 2, - "accounts_updated": 0, - "accounts_skipped": 148, - "errors": [] -} -``` - -### Benefits - -1. **Beancount as Source of Truth**: Castle DB automatically reflects Beancount state -2. **Reduced Manual Work**: No more manual account creation in Castle -3. **Prevents Permission Errors**: Cannot grant permission on non-existent account -4. **Audit Trail**: Tracks which accounts were synced and when -5. **Safe Operations**: Continues on errors, never deletes accounts - ---- - -## Part 2: Bulk Permission Management - -### Problem Solved - -**Before**: Granting permissions one-by-one was tedious for large teams. -**After**: Bulk operations for common admin tasks. - -### Implementation - -**New Module**: `castle/permission_management.py` - -**Core Functions**: - -```python -# 1. Grant to multiple users -result = await bulk_grant_permission( - user_ids=["alice", "bob", "charlie"], - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin" -) - -# 2. Revoke all user permissions (offboarding) -result = await revoke_all_user_permissions("departed_user") - -# 3. Revoke all permissions on account (project closure) -result = await revoke_all_permissions_on_account("old_project_id") - -# 4. Copy permissions from one user to another (templating) -result = await copy_permissions( - from_user_id="experienced_coordinator", - to_user_id="new_coordinator", - granted_by="admin" -) - -# 5. Get permission analytics (dashboard) -stats = await get_permission_analytics() - -# 6. Cleanup expired permissions (maintenance) -result = await cleanup_expired_permissions(days_old=30) -``` - -### Feature Highlights - -#### 1. Bulk Grant Permission - -**Use Case**: Onboard entire team at once - -```python -# Grant submit_expense to all food team members -await bulk_grant_permission( - user_ids=["alice", "bob", "charlie", "dave", "eve"], - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin", - expires_at=datetime(2025, 12, 31), - notes="Q4 food team members" -) -``` - -**Result**: -```json -{ - "granted": 5, - "failed": 0, - "errors": [], - "permissions": [...] -} -``` - -#### 2. User Offboarding - -**Use Case**: Remove all access when user leaves - -```python -# Revoke ALL permissions for departed user -await revoke_all_user_permissions("departed_user_id") -``` - -**Result**: -```json -{ - "revoked": 8, - "failed": 0, - "errors": [], - "permission_types_removed": ["read", "submit_expense", "manage"] -} -``` - -#### 3. Permission Templates - -**Use Case**: Copy permissions from experienced user to new hire - -```python -# Copy all SUBMIT_EXPENSE permissions from Alice to Bob -await copy_permissions( - from_user_id="alice", - to_user_id="bob", - granted_by="admin", - permission_types=[PermissionType.SUBMIT_EXPENSE], - notes="Copied from Alice - new food coordinator" -) -``` - -**Result**: -```json -{ - "copied": 5, - "failed": 0, - "errors": [], - "permissions": [...] -} -``` - -#### 4. Permission Analytics - -**Use Case**: Admin dashboard showing permission usage - -```python -stats = await get_permission_analytics() -``` - -**Result**: -```json -{ - "total_permissions": 150, - "by_type": { - "read": 50, - "submit_expense": 80, - "manage": 20 - }, - "expiring_soon": [ - { - "user_id": "alice", - "account_name": "Expenses:Food", - "permission_type": "submit_expense", - "expires_at": "2025-11-15T00:00:00" - } - ], - "users_with_permissions": 45, - "most_permissioned_accounts": [ - { - "account": "Expenses:Food", - "permission_count": 25 - } - ] -} -``` - -### API Endpoints (Admin Only) - -#### Bulk Grant -```http -POST /api/v1/admin/permissions/bulk-grant -Authorization: Bearer {admin_key} - -{ - "user_ids": ["alice", "bob", "charlie"], - "account_id": "acc123", - "permission_type": "submit_expense", - "expires_at": "2025-12-31T23:59:59", - "notes": "Q4 team" -} -``` - -#### User Offboarding -```http -DELETE /api/v1/admin/permissions/user/{user_id} -Authorization: Bearer {admin_key} -``` - -#### Account Closure -```http -DELETE /api/v1/admin/permissions/account/{account_id} -Authorization: Bearer {admin_key} -``` - -#### Copy Permissions -```http -POST /api/v1/admin/permissions/copy -Authorization: Bearer {admin_key} - -{ - "from_user_id": "alice", - "to_user_id": "bob", - "permission_types": ["submit_expense"], - "notes": "New coordinator onboarding" -} -``` - -#### Analytics -```http -GET /api/v1/admin/permissions/analytics -Authorization: Bearer {admin_key} -``` - -#### Cleanup -```http -POST /api/v1/admin/permissions/cleanup -Authorization: Bearer {admin_key} - -{ - "days_old": 30 -} -``` - ---- - -## Recommended Admin Workflows - -### Workflow 1: Onboarding New Team Member - -**Before** (Manual, ~10 minutes): -1. Manually create 5 permissions (one by one) -2. Hope you didn't miss any -3. Remember to set expiration dates - -**After** (Automated, ~1 minute): -```python -# Option A: Copy from experienced team member -await copy_permissions( - from_user_id="experienced_member", - to_user_id="new_member", - granted_by="admin", - notes="New food coordinator" -) - -# Option B: Bulk grant with template -await bulk_grant_permission( - user_ids=["new_member"], - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin", - expires_at=contract_end_date -) -``` - -### Workflow 2: Quarterly Access Review - -**Before** (Manual, ~2 hours): -1. Export all permissions to spreadsheet -2. Manually review each one -3. Delete expired ones individually -4. Update expiration dates one by one - -**After** (Automated, ~5 minutes): -```python -# 1. Get analytics -stats = await get_permission_analytics() - -# 2. Review expiring soon -print(f"Permissions expiring in 7 days: {len(stats['expiring_soon'])}") - -# 3. Cleanup old expired ones -cleanup = await cleanup_expired_permissions(days_old=30) -print(f"Cleaned up {cleanup['deleted']} expired permissions") - -# 4. Review most-permissioned accounts -print("Top 10 accounts by permission count:") -for account in stats['most_permissioned_accounts'][:10]: - print(f" {account['account']}: {account['permission_count']} permissions") -``` - -### Workflow 3: Project/Event Permission Management - -**Before** (Manual, ~15 minutes per event): -1. Grant permissions to 10 volunteers individually -2. Remember to revoke after event ends -3. Hope you didn't miss anyone - -**After** (Automated, ~2 minutes): -```python -# Before event: Bulk grant -await bulk_grant_permission( - user_ids=volunteer_ids, - account_id="expenses_event_summer_festival_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin", - expires_at=event_end_date, # Auto-expires - notes="Summer Festival 2025 volunteers" -) - -# After event: Revoke all (if needed before expiration) -await revoke_all_permissions_on_account("expenses_event_summer_festival_id") -``` - -### Workflow 4: User Offboarding - -**Before** (Manual, ~5 minutes): -1. Find all permissions for user -2. Delete each one individually -3. Hope you didn't miss any - -**After** (Automated, ~10 seconds): -```python -# One command removes all access -result = await revoke_all_user_permissions("departed_user") -print(f"Revoked {result['revoked']} permissions") -print(f"Permission types removed: {result['permission_types_removed']}") -``` - ---- - -## Integration with Existing Code - -### Updated Permission Creation Flow - -```python -# OLD: Manual permission creation (risky) -await create_account_permission( - user_id="alice", - account_id="acc123", # What if account doesn't exist in Castle DB? - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin" -) - -# NEW: Safe permission creation with account sync -from castle.account_sync import ensure_account_exists_in_castle - -# Ensure account exists first -account_exists = await ensure_account_exists_in_castle("Expenses:Marketing") - -if account_exists: - # Now safe - account guaranteed to be in Castle DB - await create_account_permission( - user_id="alice", - account_id=account_id, - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin" - ) -else: - raise HTTPException(404, "Account not found in Beancount") -``` - -### Scheduler Integration - -```python -# Add to your Castle extension startup -from apscheduler.schedulers.asyncio import AsyncIOScheduler -from castle.account_sync import scheduled_account_sync -from castle.permission_management import cleanup_expired_permissions - -scheduler = AsyncIOScheduler() - -# Sync accounts from Beancount every hour -scheduler.add_job( - scheduled_account_sync, - 'interval', - hours=1, - id='account_sync' -) - -# Cleanup expired permissions daily at 2 AM -scheduler.add_job( - cleanup_expired_permissions, - 'cron', - hour=2, - minute=0, - id='permission_cleanup', - kwargs={'days_old': 30} -) - -scheduler.start() -``` - ---- - -## Performance Impact - -### Account Sync - -**Metrics** (150 accounts): -- First sync: ~2 seconds (150 accounts) -- Incremental sync: ~0.1 seconds (0-5 new accounts) -- Memory usage: Negligible (~1MB) - -**Caching Strategy**: -- Account lookups already cached (5min TTL) -- Fava client reuses HTTP connection -- Minimal DB overhead - -### Bulk Permission Management - -**Metrics** (100 users): -- Bulk grant: ~0.5 seconds (vs 30 seconds individually) -- User offboarding: ~0.2 seconds (vs 10 seconds manually) -- Permission copy: ~0.3 seconds (vs 20 seconds manually) -- Analytics: ~0.1 seconds (cached) - -**Performance Improvement**: -- 60x faster for bulk grants -- 50x faster for offboarding -- 66x faster for permission templating - ---- - -## Testing - -### Unit Tests Needed - -```python -# test_account_sync.py -async def test_sync_accounts_from_beancount(): - """Test full account sync""" - stats = await sync_accounts_from_beancount() - assert stats['accounts_added'] >= 0 - assert stats['total_beancount_accounts'] > 0 - -async def test_infer_account_type(): - """Test account type inference""" - assert infer_account_type_from_name("Assets:Cash") == AccountType.ASSET - assert infer_account_type_from_name("Expenses:Food") == AccountType.EXPENSE - -async def test_extract_user_id(): - """Test user ID extraction""" - user_id = extract_user_id_from_account_name("Assets:Receivable:User-abc123") - assert user_id == "abc123" - -# test_permission_management.py -async def test_bulk_grant_permission(): - """Test bulk permission grant""" - result = await bulk_grant_permission( - user_ids=["user1", "user2", "user3"], - account_id="acc123", - permission_type=PermissionType.READ, - granted_by="admin" - ) - assert result['granted'] == 3 - assert result['failed'] == 0 - -async def test_copy_permissions(): - """Test permission templating""" - # Grant permission to source user - await create_account_permission(...) - - # Copy to target user - result = await copy_permissions( - from_user_id="source", - to_user_id="target", - granted_by="admin" - ) - assert result['copied'] > 0 -``` - -### Integration Tests - -```python -async def test_onboarding_workflow(): - """Test complete onboarding workflow""" - # 1. Sync account - await ensure_account_exists_in_castle("Expenses:Food") - - # 2. Copy permissions from template user - result = await copy_permissions( - from_user_id="template_user", - to_user_id="new_user", - granted_by="admin" - ) - - assert result['copied'] > 0 - - # 3. Verify permissions - perms = await get_user_permissions("new_user") - assert len(perms) > 0 - -async def test_offboarding_workflow(): - """Test complete offboarding workflow""" - # 1. Grant some permissions - await create_account_permission(...) - - # 2. Offboard user - result = await revoke_all_user_permissions("departed_user") - - assert result['revoked'] > 0 - - # 3. Verify all revoked - perms = await get_user_permissions("departed_user") - assert len(perms) == 0 -``` - ---- - -## Security Considerations - -### Account Sync - -✅ **Read-only from Beancount**: Never modifies Beancount, only reads -✅ **Admin-only operation**: Sync endpoints require admin key -✅ **Error isolation**: Single account failure doesn't stop entire sync -✅ **Audit trail**: All operations logged - -⚠️ **Considerations**: -- Syncing from compromised Beancount could create unwanted accounts -- Mitigation: Validate Beancount file integrity before sync - -### Bulk Permissions - -✅ **Admin-only**: All bulk operations require admin key -✅ **Atomic operations**: Each permission grant/revoke is atomic -✅ **Detailed logging**: All operations logged with admin ID -✅ **No permission escalation**: Cannot grant higher permissions than you have - -⚠️ **Considerations**: -- Bulk operations powerful - ensure admin keys are secure -- Consider adding approval workflow for bulk grants >10 users -- Monitor analytics for unusual permission patterns - ---- - -## Monitoring & Alerts - -### Recommended Alerts - -```python -# Alert on large bulk operations -async def on_bulk_grant(result): - if result['granted'] > 50: - await send_admin_alert( - f"Large bulk grant: {result['granted']} permissions granted" - ) - -# Alert on permission analytics anomalies -async def check_permission_health(): - stats = await get_permission_analytics() - - # Alert if permissions spike - if stats['total_permissions'] > 1000: - await send_admin_alert( - f"Permission count high: {stats['total_permissions']}" - ) - - # Alert if many expiring soon - if len(stats['expiring_soon']) > 20: - await send_admin_alert( - f"{len(stats['expiring_soon'])} permissions expiring in 7 days" - ) -``` - -### Logging - -```python -# All operations log with context -logger.info(f"Account sync complete: {stats['accounts_added']} added") -logger.info(f"Bulk grant: {result['granted']} permissions to {len(user_ids)} users") -logger.warning(f"Permission copy failed: {result['failed']} failures") -logger.error(f"Account sync error: {error}") -``` - ---- - -## Future Enhancements - -### Phase 2 (Next 2 weeks) - -1. **Permission Groups/Roles** (Recommended) - - Define standard permission sets - - Grant entire roles at once - - Easier onboarding - -2. **Permission Request Workflow** - - Users request permissions - - Admins approve/deny - - Self-service access - -3. **Advanced Analytics** - - Permission usage tracking - - Access pattern analysis - - Security monitoring - -### Phase 3 (Next month) - -4. **Automated Access Reviews** - - Periodic permission review prompts - - Auto-revoke unused permissions - - Compliance reporting - -5. **Permission Templates by Role** - - Pre-defined role templates - - Org-specific customization - - Version-controlled templates - ---- - -## Migration Guide - -### For Existing Castle Installations - -**Step 1: Deploy New Modules** -```bash -# Copy new files to Castle extension -cp account_sync.py /path/to/castle/ -cp permission_management.py /path/to/castle/ -``` - -**Step 2: Initial Account Sync** -```python -# Run once to sync existing accounts -from castle.account_sync import sync_accounts_from_beancount - -stats = await sync_accounts_from_beancount(force_full_sync=True) -print(f"Synced {stats['accounts_added']} accounts") -``` - -**Step 3: Add Scheduled Sync** (Optional) -```python -# Add to your startup code -scheduler.add_job( - scheduled_account_sync, - 'interval', - hours=1 -) -``` - -**Step 4: Start Using Bulk Operations** -```python -# No migration needed - start using immediately -await bulk_grant_permission(...) -``` - ---- - -## Documentation Updates - -**New files created**: -- ✅ `castle/account_sync.py` (230 lines) -- ✅ `castle/permission_management.py` (400 lines) -- ✅ `docs/PERMISSIONS-SYSTEM.md` (full permission system docs) -- ✅ `docs/ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md` (this file) - -**Files to update**: -- `castle/views_api.py` - Add new admin endpoints -- `castle/README.md` - Document new features -- `tests/` - Add comprehensive tests - ---- - -## Summary - -### What Was Built - -1. **Account Sync Module** (230 lines) - - Automatic sync from Beancount → Castle DB - - Type inference and user ID extraction - - Background scheduling support - -2. **Permission Management Module** (400 lines) - - Bulk grant/revoke operations - - Permission templating - - Analytics dashboard - - Automated cleanup - -3. **Documentation** (600+ lines) - - Complete permission system guide - - Admin workflow examples - - API reference - - Security best practices - -### Impact - -**Time Savings**: -- Onboarding: 10 min → 1 min (90% reduction) -- Offboarding: 5 min → 10 sec (97% reduction) -- Access review: 2 hours → 5 min (96% reduction) -- Permission grant: 30 sec/user → 0.5 sec/user (98% reduction) - -**Total Admin Time Saved**: ~50-70% per month - -**Code Quality**: -- Well-documented (inline + separate docs) -- Error handling throughout -- Comprehensive logging -- Type hints included -- Ready for testing - -### Next Steps - -1. ✅ **Completed**: Core implementation -2. ⏳ **In Progress**: Documentation -3. 🔲 **Next**: Add API endpoints to views_api.py -4. 🔲 **Next**: Write comprehensive tests -5. 🔲 **Next**: Add monitoring/alerts -6. 🔲 **Future**: Permission groups/roles - ---- - -**Implementation By**: Claude Code -**Date**: November 10, 2025 -**Status**: ✅ **Core Complete - Ready for API Integration** diff --git a/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.html b/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.html deleted file mode 100644 index d0e9bfe..0000000 --- a/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.html +++ /dev/null @@ -1,953 +0,0 @@ - - - - - - - ACCOUNTING-ANALYSIS-NET-SETTLEMENT - - - - - -

Accounting -Analysis: Net Settlement Entry Pattern

-

Date: 2025-01-12 Prepared By: -Senior Accounting Review Subject: Castle Extension - -Lightning Payment Settlement Entries Status: Technical -Review

-
-

Executive Summary

-

This document provides a professional accounting assessment of -Castle’s net settlement entry pattern used for recording Lightning -Network payments that settle fiat-denominated receivables. The analysis -identifies areas where the implementation deviates from traditional -accounting best practices and provides specific recommendations for -improvement.

-

Key Findings: - ✅ Double-entry integrity maintained -- ✅ Functional for intended purpose - ❌ Zero-amount postings violate -accounting principles - ❌ Redundant satoshi tracking - ❌ No exchange -gain/loss recognition - ⚠️ Mixed currency approach lacks clear -hierarchy

-
-

Background: The Technical -Challenge

-

Castle operates as a Lightning Network-integrated accounting system -for collectives (co-living spaces, makerspaces). It faces a unique -accounting challenge:

-

Scenario: User creates a receivable in EUR (e.g., -€200 for room rent), then pays via Lightning Network in satoshis -(225,033 sats).

-

Challenge: Record the payment while: 1. Clearing the -exact EUR receivable amount 2. Recording the exact satoshi amount -received 3. Handling cases where users have both receivables (owe -Castle) and payables (Castle owes them) 4. Maintaining Beancount -double-entry balance

-
-

Current Implementation

-

Transaction Example

-
; Step 1: Receivable Created
-2025-11-12 * "room (200.00 EUR)" #receivable-entry
-  user-id: "375ec158"
-  source: "castle-api"
-  sats-amount: "225033"
-  Assets:Receivable:User-375ec158     200.00 EUR
-    sats-equivalent: "225033"
-  Income:Accommodation:Guests        -200.00 EUR
-    sats-equivalent: "225033"
-
-; Step 2: Lightning Payment Received
-2025-11-12 * "Lightning payment settlement from user 375ec158"
-  #lightning-payment #net-settlement
-  user-id: "375ec158"
-  source: "lightning_payment"
-  payment-type: "net-settlement"
-  payment-hash: "8d080ec4cc4301715535004156085dd50c159185..."
-  Assets:Bitcoin:Lightning            225033 SATS @ 0.0008887585... EUR
-    payment-hash: "8d080ec4cc4301715535004156085dd50c159185..."
-  Assets:Receivable:User-375ec158    -200.00 EUR
-    sats-equivalent: "225033"
-  Liabilities:Payable:User-375ec158     0.00 EUR
-

Code Implementation

-

Location: -beancount_format.py:739-760

-
# Build postings for net settlement
-postings = [
-    {
-        "account": payment_account,
-        "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}",
-        "meta": {"payment-hash": payment_hash} if payment_hash else {}
-    },
-    {
-        "account": receivable_account,
-        "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}",
-        "meta": {"sats-equivalent": str(abs(amount_sats))}
-    },
-    {
-        "account": payable_account,
-        "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}",
-        "meta": {}
-    }
-]
-

Three-Posting Structure: 1. Lightning -Account: Records SATS received with @@ total price -notation 2. Receivable Account: Clears EUR receivable -with sats-equivalent metadata 3. Payable Account: -Clears any outstanding EUR payables (often 0.00)

-
-

Accounting Issues Identified

-

Issue 1: Zero-Amount Postings

-

Problem: The third posting often records -0.00 EUR when no payable exists.

-
Liabilities:Payable:User-375ec158     0.00 EUR
-

Why This Is Wrong: - Zero-amount postings have no -economic substance - Clutters the journal with non-events - Violates the -principle of materiality (GAAP Concept Statement 2) - Makes auditing -more difficult (reviewers must verify why zero amounts exist)

-

Accounting Principle Violated: > “Transactions -should only include postings that represent actual economic events or -changes in account balances.”

-

Impact: Low severity, but unprofessional -presentation

-

Recommendation:

-
# Make payable posting conditional
-postings = [
-    {"account": payment_account, "amount": ...},
-    {"account": receivable_account, "amount": ...}
-]
-
-# Only add payable posting if there's actually a payable
-if total_payable_fiat > 0:
-    postings.append({
-        "account": payable_account,
-        "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}",
-        "meta": {}
-    })
-
-

Issue 2: Redundant Satoshi -Tracking

-

Problem: Satoshis are tracked in TWO places in the -same transaction:

-
    -

  1. Position Amount (via @@ -notation):

    -
    Assets:Bitcoin:Lightning  225033 SATS @@ 200.00 EUR
    2. -

  2. Metadata (sats-equivalent):

    -
    Assets:Receivable:User-375ec158  -200.00 EUR
-  sats-equivalent: "225033"
    3. -
-

Why This Is Problematic: - The @@ -notation already records the exact satoshi amount - Beancount’s price -database stores this relationship - Metadata becomes redundant for this -specific posting - Increases storage and potential for inconsistency

-

Technical Detail:

-

The @@ notation means “total price” and Beancount -converts it to per-unit price:

-
; You write:
-Assets:Bitcoin:Lightning  225033 SATS @@ 200.00 EUR
-
-; Beancount stores:
-Assets:Bitcoin:Lightning  225033 SATS @ 0.0008887585... EUR
-; (where 200.00 / 225033 = 0.0008887585...)
-

Beancount can query this:

-
SELECT account, sum(convert(position, SATS))
-WHERE account = 'Assets:Bitcoin:Lightning'
-

Recommendation:

-

Choose ONE approach consistently:

-

Option A - Use @ notation (Beancount standard):

-
Assets:Bitcoin:Lightning           225033 SATS @@ 200.00 EUR
-  payment-hash: "8d080ec4..."
-Assets:Receivable:User-375ec158   -200.00 EUR
-  ; No sats-equivalent needed here
-

Option B - Use EUR positions with metadata (Castle’s -current approach):

-
Assets:Bitcoin:Lightning           200.00 EUR
-  sats-received: "225033"
-  payment-hash: "8d080ec4..."
-Assets:Receivable:User-375ec158   -200.00 EUR
-  sats-cleared: "225033"
-

Don’t: Mix both in the same transaction (current -implementation)

-
-

Issue 3: No Exchange -Gain/Loss Recognition

-

Problem: When receivables are denominated in one -currency (EUR) and paid in another (SATS), exchange rate fluctuations -create gains or losses that should be recognized.

-

Example Scenario:

-
Day 1 - Receivable Created:
-  200 EUR = 225,033 SATS (rate: 1,125.165 sats/EUR)
-
-Day 5 - Payment Received:
-  225,033 SATS = 199.50 EUR (rate: 1,127.682 sats/EUR)
-  Exchange rate moved unfavorably
-
-Economic Reality: 0.50 EUR LOSS
-

Current Implementation: Forces balance by -calculating the @ rate to make it exactly 200 EUR:

-
Assets:Bitcoin:Lightning  225033 SATS @ 0.000888... EUR  ; = exactly 200.00 EUR
-

This hides the exchange variance by treating the -payment as if it was worth exactly the receivable amount.

-

GAAP/IFRS Requirement:

-

Under both US GAAP (ASC 830) and IFRS (IAS 21), exchange gains and -losses on monetary items (like receivables) should be recognized in the -period they occur.

-

Proper Accounting Treatment:

-
2025-11-12 * "Lightning payment with exchange loss"
-  Assets:Bitcoin:Lightning           225033 SATS @ 0.000886... EUR
-    ; Market rate at payment time = 199.50 EUR
-  Expenses:Foreign-Exchange-Loss     0.50 EUR
-  Assets:Receivable:User-375ec158   -200.00 EUR
-

Impact: Moderate severity - affects financial -statement accuracy

-

Why This Matters: - Tax reporting may require -exchange gain/loss recognition - Financial statements misstate true -economic results - Auditors would flag this as a compliance issue - -Cannot accurately calculate ROI or performance metrics

-
-

Issue 4: Semantic -Misuse of Price Notation

-

Problem: The @ notation in Beancount -represents acquisition cost, not settlement -value.

-

Current Usage:

-
Assets:Bitcoin:Lightning  225033 SATS @ 0.000888... EUR
-

What this notation means in accounting: “We -purchased 225,033 satoshis at a cost of 0.000888 EUR -per satoshi”

-

What actually happened: “We -received 225,033 satoshis as payment for a debt”

-

Economic Difference: - Purchase: -You exchange cash for an asset (buying Bitcoin) - Payment -Receipt: You receive an asset in settlement of a receivable

-

Accounting Substance vs. Form: - -Form: The transaction looks like a Bitcoin purchase - -Substance: The transaction is actually a receivable -collection

-

GAAP Principle (ASC 105-10-05): > “Accounting -should reflect the economic substance of transactions, not merely their -legal form.”

-

Why This Creates Issues:

-
    -
  1. Cost Basis Tracking: For tax purposes, the “cost” -of Bitcoin received as payment should be its fair market value at -receipt, not the receivable amount
    2. -
  2. Price Database Pollution: Beancount’s price -database now contains “prices” that aren’t real market prices
    3. -
  3. Auditor Confusion: An auditor reviewing this would -question why purchase prices don’t match market rates
    4. -
-

Proper Accounting Approach:

-
; Approach 1: Record at fair market value
-Assets:Bitcoin:Lightning           225033 SATS @ 0.000886... EUR
-  ; Using actual market price at time of receipt
-  acquisition-type: "payment-received"
-Revenue:Exchange-Gain              0.50 EUR
-Assets:Receivable:User-375ec158   -200.00 EUR
-
-; Approach 2: Don't use @ notation at all
-Assets:Bitcoin:Lightning           200.00 EUR
-  sats-received: "225033"
-  fmv-at-receipt: "199.50 EUR"
-Assets:Receivable:User-375ec158   -200.00 EUR
-
-

Issue 5: Misnamed -Function and Incorrect Usage

-

Problem: Function is called -format_net_settlement_entry, but it’s used for simple -payments that aren’t true net settlements.

-

Example from User’s Transaction: - Receivable: -200.00 EUR - Payable: 0.00 EUR - Net: 200.00 EUR (this is just a -payment, not a settlement)

-

Accounting Terminology:

- -

When Net Settlement is Appropriate:

-
User owes Castle:    555.00 EUR (receivable)
-Castle owes User:     38.00 EUR (payable)
-Net amount due:      517.00 EUR (true settlement)
-

Proper three-posting entry:

-
Assets:Bitcoin:Lightning           565251 SATS @@ 517.00 EUR
-Assets:Receivable:User            -555.00 EUR
-Liabilities:Payable:User            38.00 EUR
-; Net: 517.00 = -555.00 + 38.00 ✓
-

When Two Postings Suffice:

-
User owes Castle:    200.00 EUR (receivable)
-Castle owes User:      0.00 EUR (no payable)
-Amount due:          200.00 EUR (simple payment)
-

Simpler two-posting entry:

-
Assets:Bitcoin:Lightning           225033 SATS @@ 200.00 EUR
-Assets:Receivable:User            -200.00 EUR
-

Best Practice: Use the simplest journal entry -structure that accurately represents the transaction.

-

Recommendation: 1. Rename function to -format_payment_entry or -format_receivable_payment_entry 2. Create separate -format_net_settlement_entry for true netting scenarios 3. -Use conditional logic to choose 2-posting vs 3-posting based on whether -both receivables AND payables exist

-
-

Traditional Accounting -Approaches

-

Approach -1: Record Bitcoin at Fair Market Value (Tax Compliant)

-
2025-11-12 * "Bitcoin payment from user 375ec158"
-  Assets:Bitcoin:Lightning           199.50 EUR
-    sats-received: "225033"
-    fmv-per-sat: "0.000886 EUR"
-    cost-basis: "199.50 EUR"
-    payment-hash: "8d080ec4..."
-  Revenue:Exchange-Gain              0.50 EUR
-    source: "cryptocurrency-receipt"
-  Assets:Receivable:User-375ec158   -200.00 EUR
-

Pros: - ✅ Tax compliant (establishes cost basis) - -✅ Recognizes exchange gain/loss - ✅ Uses actual market rates - ✅ -Audit trail for cryptocurrency receipts

-

Cons: - ❌ Requires real-time price feeds - ❌ -Creates taxable events

-
-

Approach 2: -Simplified EUR-Only Ledger (No SATS Positions)

-
2025-11-12 * "Bitcoin payment from user 375ec158"
-  Assets:Bitcoin:Lightning           200.00 EUR
-    sats-received: "225033"
-    sats-rate: "1125.165"
-    payment-hash: "8d080ec4..."
-  Assets:Receivable:User-375ec158   -200.00 EUR
-

Pros: - ✅ Simple and clean - ✅ EUR positions match -accounting reality - ✅ SATS tracked in metadata for reference - ✅ No -artificial price notation

-

Cons: - ❌ SATS not queryable via Beancount -positions - ❌ Requires metadata parsing for SATS balances

-
-

Approach -3: True Net Settlement (When Both Obligations Exist)

-
2025-11-12 * "Net settlement via Lightning"
-  ; User owes 555 EUR, Castle owes 38 EUR, net: 517 EUR
-  Assets:Bitcoin:Lightning           517.00 EUR
-    sats-received: "565251"
-  Assets:Receivable:User-375ec158   -555.00 EUR
-  Liabilities:Payable:User-375ec158   38.00 EUR
-

When to Use: Only when both -receivables and payables exist and you’re truly netting them.

-
-

Recommendations

-

Priority 1: Immediate -Fixes (Easy Wins)

-

1.1 Remove Zero-Amount -Postings

-

File: beancount_format.py:739-760

-

Current Code:

-
postings = [
-    {...},  # Lightning
-    {...},  # Receivable
-    {       # Payable (always included, even if 0.00)
-        "account": payable_account,
-        "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}",
-        "meta": {}
-    }
-]
-

Fixed Code:

-
postings = [
-    {
-        "account": payment_account,
-        "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}",
-        "meta": {"payment-hash": payment_hash} if payment_hash else {}
-    },
-    {
-        "account": receivable_account,
-        "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}",
-        "meta": {"sats-equivalent": str(abs(amount_sats))}
-    }
-]
-
-# Only add payable posting if there's actually a payable to clear
-if total_payable_fiat > 0:
-    postings.append({
-        "account": payable_account,
-        "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}",
-        "meta": {}
-    })
-

Impact: Cleaner journal, professional presentation, -easier auditing

-
-

1.2 Choose One SATS Tracking -Method

-

Decision Required: Select either position-based OR -metadata-based satoshi tracking.

-

Option A - Keep Metadata Approach (recommended for -Castle):

-
# In format_net_settlement_entry()
-postings = [
-    {
-        "account": payment_account,
-        "amount": f"{abs(net_fiat_amount):.2f} {fiat_currency}",  # EUR only
-        "meta": {
-            "sats-received": str(abs(amount_sats)),
-            "payment-hash": payment_hash
-        }
-    },
-    {
-        "account": receivable_account,
-        "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}",
-        "meta": {"sats-cleared": str(abs(amount_sats))}
-    }
-]
-

Option B - Use Position-Based Tracking:

-
# Remove sats-equivalent metadata entirely
-postings = [
-    {
-        "account": payment_account,
-        "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}",
-        "meta": {"payment-hash": payment_hash}
-    },
-    {
-        "account": receivable_account,
-        "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}",
-        # No sats-equivalent needed - queryable via price database
-    }
-]
-

Recommendation: Choose Option A (metadata) for -consistency with Castle’s architecture.

-
-

1.3 Rename Function for -Clarity

-

File: beancount_format.py

-

Current: -format_net_settlement_entry()

-

New: format_receivable_payment_entry() -or format_payment_settlement_entry()

-

Rationale: More accurately describes what the -function does (processes payments, not always net settlements)

-
-

Priority 2: -Medium-Term Improvements (Compliance)

-

2.1 Add Exchange Gain/Loss -Tracking

-

File: tasks.py:259-276 (get balance and -calculate settlement)

-

New Logic:

-
# Get user's current balance
-balance = await fava.get_user_balance(user_id)
-fiat_balances = balance.get("fiat_balances", {})
-total_fiat_balance = fiat_balances.get(fiat_currency, Decimal(0))
-
-# Calculate expected fiat value of SATS payment at current market rate
-market_rate = await get_current_sats_eur_rate()  # New function needed
-market_value = Decimal(amount_sats) * market_rate
-
-# Calculate exchange variance
-receivable_amount = abs(total_fiat_balance) if total_fiat_balance > 0 else Decimal(0)
-exchange_variance = market_value - receivable_amount
-
-# If variance is material (> 1 cent), create exchange gain/loss posting
-if abs(exchange_variance) > Decimal("0.01"):
-    # Add exchange gain/loss to postings
-    if exchange_variance > 0:
-        # Gain: payment worth more than receivable
-        exchange_account = "Revenue:Foreign-Exchange-Gain"
-    else:
-        # Loss: payment worth less than receivable
-        exchange_account = "Expenses:Foreign-Exchange-Loss"
-
-    # Include in entry creation
-    exchange_posting = {
-        "account": exchange_account,
-        "amount": f"{abs(exchange_variance):.2f} {fiat_currency}",
-        "meta": {
-            "sats-amount": str(amount_sats),
-            "market-rate": str(market_rate),
-            "receivable-amount": str(receivable_amount)
-        }
-    }
-

Benefits: - ✅ Tax compliance - ✅ Accurate -financial reporting - ✅ Audit trail for cryptocurrency gains/losses - -✅ Regulatory compliance (GAAP/IFRS)

-
-

2.2 -Implement True Net Settlement vs. Simple Payment Logic

-

File: tasks.py or new -payment_logic.py

-
async def create_payment_entry(
-    user_id: str,
-    amount_sats: int,
-    fiat_amount: Decimal,
-    fiat_currency: str,
-    payment_hash: str
-):
-    """
-    Create appropriate payment entry based on user's balance situation.
-    Uses 2-posting for simple payments, 3-posting for net settlements.
-    """
-    # Get user balance
-    balance = await fava.get_user_balance(user_id)
-    fiat_balances = balance.get("fiat_balances", {})
-    total_balance = fiat_balances.get(fiat_currency, Decimal(0))
-
-    receivable_amount = Decimal(0)
-    payable_amount = Decimal(0)
-
-    if total_balance > 0:
-        receivable_amount = total_balance
-    elif total_balance < 0:
-        payable_amount = abs(total_balance)
-
-    # Determine entry type
-    if receivable_amount > 0 and payable_amount > 0:
-        # TRUE NET SETTLEMENT: Both obligations exist
-        return await format_net_settlement_entry(
-            user_id=user_id,
-            amount_sats=amount_sats,
-            receivable_amount=receivable_amount,
-            payable_amount=payable_amount,
-            fiat_amount=fiat_amount,
-            fiat_currency=fiat_currency,
-            payment_hash=payment_hash
-        )
-    elif receivable_amount > 0:
-        # SIMPLE RECEIVABLE PAYMENT: Only receivable exists
-        return await format_receivable_payment_entry(
-            user_id=user_id,
-            amount_sats=amount_sats,
-            receivable_amount=receivable_amount,
-            fiat_amount=fiat_amount,
-            fiat_currency=fiat_currency,
-            payment_hash=payment_hash
-        )
-    else:
-        # PAYABLE PAYMENT: Castle paying user (different flow)
-        return await format_payable_payment_entry(...)
-
-

Priority 3: -Long-Term Architectural Decisions

-

3.1 Establish Primary -Currency Hierarchy

-

Current Issue: Mixed approach (EUR positions with -SATS metadata, but also SATS positions with @ notation)

-

Decision Required: Choose ONE of the following -architectures:

-

Architecture A - EUR Primary, SATS Secondary -(recommended):

-
; All positions in EUR, SATS in metadata
-2025-11-12 * "Payment"
-  Assets:Bitcoin:Lightning           200.00 EUR
-    sats-received: "225033"
-  Assets:Receivable:User            -200.00 EUR
-    sats-cleared: "225033"
-

Architecture B - SATS Primary, EUR Secondary:

-
; All positions in SATS, EUR in metadata
-2025-11-12 * "Payment"
-  Assets:Bitcoin:Lightning           225033 SATS
-    eur-value: "200.00"
-  Assets:Receivable:User            -225033 SATS
-    eur-cleared: "200.00"
-

Recommendation: Architecture A (EUR primary) -because: 1. Most receivables created in EUR 2. Financial reporting -requirements typically in fiat 3. Tax obligations calculated in fiat 4. -Aligns with current Castle metadata approach

-
-

3.2 -Consider Separate Ledger for Cryptocurrency Holdings

-

Advanced Approach: Separate cryptocurrency movements -from fiat accounting

-

Main Ledger (EUR-denominated):

-
2025-11-12 * "Payment received from user"
-  Assets:Bitcoin-Custody:User-375ec158  200.00 EUR
-  Assets:Receivable:User-375ec158      -200.00 EUR
-

Cryptocurrency Sub-Ledger (SATS-denominated):

-
2025-11-12 * "Lightning payment received"
-  Assets:Bitcoin:Lightning:Castle    225033 SATS
-  Assets:Bitcoin:Custody:User-375ec  225033 SATS
-

Benefits: - ✅ Clean separation of concerns - ✅ -Cryptocurrency movements tracked independently - ✅ Fiat accounting -unaffected by Bitcoin volatility - ✅ Can generate separate financial -statements

-

Drawbacks: - ❌ Increased complexity - ❌ -Reconciliation between ledgers required - ❌ Two sets of books to -maintain

-
-

Code Files Requiring Changes

-

High Priority (Immediate -Fixes)

-
    -
  1. beancount_format.py:739-760 -
    2. -
  2. beancount_format.py:692 -
    3. -
-

Medium Priority (Compliance)

-
    -
  1. tasks.py:235-310 -
    2. -
  2. New file: exchange_rates.py -
    3. -
  3. beancount_format.py -
    4. -
-
-

Testing Requirements

-

Test Case 1: -Simple Receivable Payment (No Payable)

-

Setup: - User has receivable: 200.00 EUR - User has -payable: 0.00 EUR - User pays: 225,033 SATS

-

Expected Entry (after fixes):

-
2025-11-12 * "Lightning payment from user"
-  Assets:Bitcoin:Lightning           200.00 EUR
-    sats-received: "225033"
-    payment-hash: "8d080ec4..."
-  Assets:Receivable:User            -200.00 EUR
-    sats-cleared: "225033"
-

Verify: - ✅ Only 2 postings (no zero-amount -payable) - ✅ Entry balances - ✅ SATS tracked in metadata - ✅ User -balance becomes 0 (both EUR and SATS)

-
-

Test Case 2: True Net -Settlement

-

Setup: - User has receivable: 555.00 EUR - User has -payable: 38.00 EUR - Net owed: 517.00 EUR - User pays: 565,251 SATS -(worth 517.00 EUR)

-

Expected Entry:

-
2025-11-12 * "Net settlement via Lightning"
-  Assets:Bitcoin:Lightning           517.00 EUR
-    sats-received: "565251"
-    payment-hash: "abc123..."
-  Assets:Receivable:User            -555.00 EUR
-    sats-portion: "565251"
-  Liabilities:Payable:User            38.00 EUR
-

Verify: - ✅ 3 postings (receivable + payable -cleared) - ✅ Net amount = receivable - payable - ✅ Both balances -become 0 - ✅ Mathematically balanced

-
-

Test Case 3: Exchange -Gain/Loss (Future)

-

Setup: - User has receivable: 200.00 EUR (created at -1,125 sats/EUR) - User pays: 225,033 SATS (now worth 199.50 EUR at -market) - Exchange loss: 0.50 EUR

-

Expected Entry (with exchange tracking):

-
2025-11-12 * "Lightning payment with exchange loss"
-  Assets:Bitcoin:Lightning           199.50 EUR
-    sats-received: "225033"
-    market-rate: "0.000886"
-  Expenses:Foreign-Exchange-Loss     0.50 EUR
-  Assets:Receivable:User            -200.00 EUR
-

Verify: - ✅ Bitcoin recorded at fair market value - -✅ Exchange loss recognized - ✅ Receivable cleared at book value - ✅ -Entry balances

-
-

Conclusion

-

Summary of Issues

- ------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IssueSeverityAccounting ImpactRecommended Action
Zero-amount postingsLowPresentation onlyRemove immediately
Redundant SATS trackingLowStorage/efficiencyChoose one method
No exchange gain/lossHighFinancial accuracyImplement for compliance
Semantic misuse of @MediumAudit clarityConsider EUR-only positions
Misnamed functionLowCode clarityRename function
-

Professional Assessment

-

Is this “best practice” accounting? -No, this implementation deviates from traditional -accounting standards in several ways.

-

Is it acceptable for Castle’s use case? Yes, -with modifications, it’s a reasonable pragmatic solution for a -novel problem (cryptocurrency payments of fiat debts).

-

Critical improvements needed: 1. ✅ Remove -zero-amount postings (easy fix, professional presentation) 2. ✅ -Implement exchange gain/loss tracking (required for compliance) 3. ✅ -Separate payment vs. settlement logic (accuracy and clarity)

-

The fundamental challenge: Traditional accounting -wasn’t designed for this scenario. There is no established “standard” -for recording cryptocurrency payments of fiat-denominated receivables. -Castle’s approach is functional, but should be refined to align better -with accounting principles where possible.

-

Next Steps

-
    -
  1. Week 1: Implement Priority 1 fixes (remove zero -postings, rename function)
    2. -
  2. Week 2-3: Design and implement exchange gain/loss -tracking
    3. -
  3. Week 4: Add payment vs. settlement logic
    4. -
  4. Ongoing: Monitor regulatory guidance on -cryptocurrency accounting
    5. -
-
-

References

- -
-

Document Version: 1.0 Last Updated: -2025-01-12 Next Review: After Priority 1 fixes -implemented

-
-

This analysis was prepared for internal review and development -planning. It represents a professional accounting assessment of the -current implementation and should be used to guide improvements to -Castle’s payment recording system.

- - diff --git a/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.md b/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.md deleted file mode 100644 index b145128..0000000 --- a/docs/ACCOUNTING-ANALYSIS-NET-SETTLEMENT.md +++ /dev/null @@ -1,861 +0,0 @@ -# Accounting Analysis: Net Settlement Entry Pattern - -**Date**: 2025-01-12 -**Prepared By**: Senior Accounting Review -**Subject**: Castle Extension - Lightning Payment Settlement Entries -**Status**: Technical Review - ---- - -## Executive Summary - -This document provides a professional accounting assessment of Castle's net settlement entry pattern used for recording Lightning Network payments that settle fiat-denominated receivables. The analysis identifies areas where the implementation deviates from traditional accounting best practices and provides specific recommendations for improvement. - -**Key Findings**: -- ✅ Double-entry integrity maintained -- ✅ Functional for intended purpose -- ❌ Zero-amount postings violate accounting principles -- ❌ Redundant satoshi tracking -- ❌ No exchange gain/loss recognition -- ⚠️ Mixed currency approach lacks clear hierarchy - ---- - -## Background: The Technical Challenge - -Castle operates as a Lightning Network-integrated accounting system for collectives (co-living spaces, makerspaces). It faces a unique accounting challenge: - -**Scenario**: User creates a receivable in EUR (e.g., €200 for room rent), then pays via Lightning Network in satoshis (225,033 sats). - -**Challenge**: Record the payment while: -1. Clearing the exact EUR receivable amount -2. Recording the exact satoshi amount received -3. Handling cases where users have both receivables (owe Castle) and payables (Castle owes them) -4. Maintaining Beancount double-entry balance - ---- - -## Current Implementation - -### Transaction Example - -```beancount -; Step 1: Receivable Created -2025-11-12 * "room (200.00 EUR)" #receivable-entry - user-id: "375ec158" - source: "castle-api" - sats-amount: "225033" - Assets:Receivable:User-375ec158 200.00 EUR - sats-equivalent: "225033" - Income:Accommodation:Guests -200.00 EUR - sats-equivalent: "225033" - -; Step 2: Lightning Payment Received -2025-11-12 * "Lightning payment settlement from user 375ec158" - #lightning-payment #net-settlement - user-id: "375ec158" - source: "lightning_payment" - payment-type: "net-settlement" - payment-hash: "8d080ec4cc4301715535004156085dd50c159185..." - Assets:Bitcoin:Lightning 225033 SATS @ 0.0008887585... EUR - payment-hash: "8d080ec4cc4301715535004156085dd50c159185..." - Assets:Receivable:User-375ec158 -200.00 EUR - sats-equivalent: "225033" - Liabilities:Payable:User-375ec158 0.00 EUR -``` - -### Code Implementation - -**Location**: `beancount_format.py:739-760` - -```python -# Build postings for net settlement -postings = [ - { - "account": payment_account, - "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}", - "meta": {"payment-hash": payment_hash} if payment_hash else {} - }, - { - "account": receivable_account, - "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - }, - { - "account": payable_account, - "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}", - "meta": {} - } -] -``` - -**Three-Posting Structure**: -1. **Lightning Account**: Records SATS received with `@@` total price notation -2. **Receivable Account**: Clears EUR receivable with sats-equivalent metadata -3. **Payable Account**: Clears any outstanding EUR payables (often 0.00) - ---- - -## Accounting Issues Identified - -### Issue 1: Zero-Amount Postings - -**Problem**: The third posting often records `0.00 EUR` when no payable exists. - -```beancount -Liabilities:Payable:User-375ec158 0.00 EUR -``` - -**Why This Is Wrong**: -- Zero-amount postings have no economic substance -- Clutters the journal with non-events -- Violates the principle of materiality (GAAP Concept Statement 2) -- Makes auditing more difficult (reviewers must verify why zero amounts exist) - -**Accounting Principle Violated**: -> "Transactions should only include postings that represent actual economic events or changes in account balances." - -**Impact**: Low severity, but unprofessional presentation - -**Recommendation**: -```python -# Make payable posting conditional -postings = [ - {"account": payment_account, "amount": ...}, - {"account": receivable_account, "amount": ...} -] - -# Only add payable posting if there's actually a payable -if total_payable_fiat > 0: - postings.append({ - "account": payable_account, - "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}", - "meta": {} - }) -``` - ---- - -### Issue 2: Redundant Satoshi Tracking - -**Problem**: Satoshis are tracked in TWO places in the same transaction: - -1. **Position Amount** (via `@@` notation): - ```beancount - Assets:Bitcoin:Lightning 225033 SATS @@ 200.00 EUR - ``` - -2. **Metadata** (sats-equivalent): - ```beancount - Assets:Receivable:User-375ec158 -200.00 EUR - sats-equivalent: "225033" - ``` - -**Why This Is Problematic**: -- The `@@` notation already records the exact satoshi amount -- Beancount's price database stores this relationship -- Metadata becomes redundant for this specific posting -- Increases storage and potential for inconsistency - -**Technical Detail**: - -The `@@` notation means "total price" and Beancount converts it to per-unit price: -```beancount -; You write: -Assets:Bitcoin:Lightning 225033 SATS @@ 200.00 EUR - -; Beancount stores: -Assets:Bitcoin:Lightning 225033 SATS @ 0.0008887585... EUR -; (where 200.00 / 225033 = 0.0008887585...) -``` - -Beancount can query this: -```sql -SELECT account, sum(convert(position, SATS)) -WHERE account = 'Assets:Bitcoin:Lightning' -``` - -**Recommendation**: - -Choose ONE approach consistently: - -**Option A - Use @ notation** (Beancount standard): -```beancount -Assets:Bitcoin:Lightning 225033 SATS @@ 200.00 EUR - payment-hash: "8d080ec4..." -Assets:Receivable:User-375ec158 -200.00 EUR - ; No sats-equivalent needed here -``` - -**Option B - Use EUR positions with metadata** (Castle's current approach): -```beancount -Assets:Bitcoin:Lightning 200.00 EUR - sats-received: "225033" - payment-hash: "8d080ec4..." -Assets:Receivable:User-375ec158 -200.00 EUR - sats-cleared: "225033" -``` - -**Don't**: Mix both in the same transaction (current implementation) - ---- - -### Issue 3: No Exchange Gain/Loss Recognition - -**Problem**: When receivables are denominated in one currency (EUR) and paid in another (SATS), exchange rate fluctuations create gains or losses that should be recognized. - -**Example Scenario**: - -``` -Day 1 - Receivable Created: - 200 EUR = 225,033 SATS (rate: 1,125.165 sats/EUR) - -Day 5 - Payment Received: - 225,033 SATS = 199.50 EUR (rate: 1,127.682 sats/EUR) - Exchange rate moved unfavorably - -Economic Reality: 0.50 EUR LOSS -``` - -**Current Implementation**: Forces balance by calculating the `@` rate to make it exactly 200 EUR: -```beancount -Assets:Bitcoin:Lightning 225033 SATS @ 0.000888... EUR ; = exactly 200.00 EUR -``` - -This **hides the exchange variance** by treating the payment as if it was worth exactly the receivable amount. - -**GAAP/IFRS Requirement**: - -Under both US GAAP (ASC 830) and IFRS (IAS 21), exchange gains and losses on monetary items (like receivables) should be recognized in the period they occur. - -**Proper Accounting Treatment**: - -```beancount -2025-11-12 * "Lightning payment with exchange loss" - Assets:Bitcoin:Lightning 225033 SATS @ 0.000886... EUR - ; Market rate at payment time = 199.50 EUR - Expenses:Foreign-Exchange-Loss 0.50 EUR - Assets:Receivable:User-375ec158 -200.00 EUR -``` - -**Impact**: Moderate severity - affects financial statement accuracy - -**Why This Matters**: -- Tax reporting may require exchange gain/loss recognition -- Financial statements misstate true economic results -- Auditors would flag this as a compliance issue -- Cannot accurately calculate ROI or performance metrics - ---- - -### Issue 4: Semantic Misuse of Price Notation - -**Problem**: The `@` notation in Beancount represents **acquisition cost**, not **settlement value**. - -**Current Usage**: -```beancount -Assets:Bitcoin:Lightning 225033 SATS @ 0.000888... EUR -``` - -**What this notation means in accounting**: "We **purchased** 225,033 satoshis at a cost of 0.000888 EUR per satoshi" - -**What actually happened**: "We **received** 225,033 satoshis as payment for a debt" - -**Economic Difference**: -- **Purchase**: You exchange cash for an asset (buying Bitcoin) -- **Payment Receipt**: You receive an asset in settlement of a receivable - -**Accounting Substance vs. Form**: -- **Form**: The transaction looks like a Bitcoin purchase -- **Substance**: The transaction is actually a receivable collection - -**GAAP Principle (ASC 105-10-05)**: -> "Accounting should reflect the economic substance of transactions, not merely their legal form." - -**Why This Creates Issues**: - -1. **Cost Basis Tracking**: For tax purposes, the "cost" of Bitcoin received as payment should be its fair market value at receipt, not the receivable amount -2. **Price Database Pollution**: Beancount's price database now contains "prices" that aren't real market prices -3. **Auditor Confusion**: An auditor reviewing this would question why purchase prices don't match market rates - -**Proper Accounting Approach**: - -```beancount -; Approach 1: Record at fair market value -Assets:Bitcoin:Lightning 225033 SATS @ 0.000886... EUR - ; Using actual market price at time of receipt - acquisition-type: "payment-received" -Revenue:Exchange-Gain 0.50 EUR -Assets:Receivable:User-375ec158 -200.00 EUR - -; Approach 2: Don't use @ notation at all -Assets:Bitcoin:Lightning 200.00 EUR - sats-received: "225033" - fmv-at-receipt: "199.50 EUR" -Assets:Receivable:User-375ec158 -200.00 EUR -``` - ---- - -### Issue 5: Misnamed Function and Incorrect Usage - -**Problem**: Function is called `format_net_settlement_entry`, but it's used for simple payments that aren't true net settlements. - -**Example from User's Transaction**: -- Receivable: 200.00 EUR -- Payable: 0.00 EUR -- Net: 200.00 EUR (this is just a **payment**, not a **settlement**) - -**Accounting Terminology**: - -- **Payment**: Settling a single obligation (receivable OR payable) -- **Net Settlement**: Offsetting multiple obligations (receivable AND payable) - -**When Net Settlement is Appropriate**: - -``` -User owes Castle: 555.00 EUR (receivable) -Castle owes User: 38.00 EUR (payable) -Net amount due: 517.00 EUR (true settlement) -``` - -Proper three-posting entry: -```beancount -Assets:Bitcoin:Lightning 565251 SATS @@ 517.00 EUR -Assets:Receivable:User -555.00 EUR -Liabilities:Payable:User 38.00 EUR -; Net: 517.00 = -555.00 + 38.00 ✓ -``` - -**When Two Postings Suffice**: - -``` -User owes Castle: 200.00 EUR (receivable) -Castle owes User: 0.00 EUR (no payable) -Amount due: 200.00 EUR (simple payment) -``` - -Simpler two-posting entry: -```beancount -Assets:Bitcoin:Lightning 225033 SATS @@ 200.00 EUR -Assets:Receivable:User -200.00 EUR -``` - -**Best Practice**: Use the simplest journal entry structure that accurately represents the transaction. - -**Recommendation**: -1. Rename function to `format_payment_entry` or `format_receivable_payment_entry` -2. Create separate `format_net_settlement_entry` for true netting scenarios -3. Use conditional logic to choose 2-posting vs 3-posting based on whether both receivables AND payables exist - ---- - -## Traditional Accounting Approaches - -### Approach 1: Record Bitcoin at Fair Market Value (Tax Compliant) - -```beancount -2025-11-12 * "Bitcoin payment from user 375ec158" - Assets:Bitcoin:Lightning 199.50 EUR - sats-received: "225033" - fmv-per-sat: "0.000886 EUR" - cost-basis: "199.50 EUR" - payment-hash: "8d080ec4..." - Revenue:Exchange-Gain 0.50 EUR - source: "cryptocurrency-receipt" - Assets:Receivable:User-375ec158 -200.00 EUR -``` - -**Pros**: -- ✅ Tax compliant (establishes cost basis) -- ✅ Recognizes exchange gain/loss -- ✅ Uses actual market rates -- ✅ Audit trail for cryptocurrency receipts - -**Cons**: -- ❌ Requires real-time price feeds -- ❌ Creates taxable events - ---- - -### Approach 2: Simplified EUR-Only Ledger (No SATS Positions) - -```beancount -2025-11-12 * "Bitcoin payment from user 375ec158" - Assets:Bitcoin:Lightning 200.00 EUR - sats-received: "225033" - sats-rate: "1125.165" - payment-hash: "8d080ec4..." - Assets:Receivable:User-375ec158 -200.00 EUR -``` - -**Pros**: -- ✅ Simple and clean -- ✅ EUR positions match accounting reality -- ✅ SATS tracked in metadata for reference -- ✅ No artificial price notation - -**Cons**: -- ❌ SATS not queryable via Beancount positions -- ❌ Requires metadata parsing for SATS balances - ---- - -### Approach 3: True Net Settlement (When Both Obligations Exist) - -```beancount -2025-11-12 * "Net settlement via Lightning" - ; User owes 555 EUR, Castle owes 38 EUR, net: 517 EUR - Assets:Bitcoin:Lightning 517.00 EUR - sats-received: "565251" - Assets:Receivable:User-375ec158 -555.00 EUR - Liabilities:Payable:User-375ec158 38.00 EUR -``` - -**When to Use**: Only when **both** receivables and payables exist and you're truly netting them. - ---- - -## Recommendations - -### Priority 1: Immediate Fixes (Easy Wins) - -#### 1.1 Remove Zero-Amount Postings - -**File**: `beancount_format.py:739-760` - -**Current Code**: -```python -postings = [ - {...}, # Lightning - {...}, # Receivable - { # Payable (always included, even if 0.00) - "account": payable_account, - "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}", - "meta": {} - } -] -``` - -**Fixed Code**: -```python -postings = [ - { - "account": payment_account, - "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}", - "meta": {"payment-hash": payment_hash} if payment_hash else {} - }, - { - "account": receivable_account, - "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}", - "meta": {"sats-equivalent": str(abs(amount_sats))} - } -] - -# Only add payable posting if there's actually a payable to clear -if total_payable_fiat > 0: - postings.append({ - "account": payable_account, - "amount": f"{abs(total_payable_fiat):.2f} {fiat_currency}", - "meta": {} - }) -``` - -**Impact**: Cleaner journal, professional presentation, easier auditing - ---- - -#### 1.2 Choose One SATS Tracking Method - -**Decision Required**: Select either position-based OR metadata-based satoshi tracking. - -**Option A - Keep Metadata Approach** (recommended for Castle): -```python -# In format_net_settlement_entry() -postings = [ - { - "account": payment_account, - "amount": f"{abs(net_fiat_amount):.2f} {fiat_currency}", # EUR only - "meta": { - "sats-received": str(abs(amount_sats)), - "payment-hash": payment_hash - } - }, - { - "account": receivable_account, - "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}", - "meta": {"sats-cleared": str(abs(amount_sats))} - } -] -``` - -**Option B - Use Position-Based Tracking**: -```python -# Remove sats-equivalent metadata entirely -postings = [ - { - "account": payment_account, - "amount": f"{abs(amount_sats)} SATS @@ {abs(net_fiat_amount):.2f} {fiat_currency}", - "meta": {"payment-hash": payment_hash} - }, - { - "account": receivable_account, - "amount": f"-{abs(total_receivable_fiat):.2f} {fiat_currency}", - # No sats-equivalent needed - queryable via price database - } -] -``` - -**Recommendation**: Choose Option A (metadata) for consistency with Castle's architecture. - ---- - -#### 1.3 Rename Function for Clarity - -**File**: `beancount_format.py` - -**Current**: `format_net_settlement_entry()` - -**New**: `format_receivable_payment_entry()` or `format_payment_settlement_entry()` - -**Rationale**: More accurately describes what the function does (processes payments, not always net settlements) - ---- - -### Priority 2: Medium-Term Improvements (Compliance) - -#### 2.1 Add Exchange Gain/Loss Tracking - -**File**: `tasks.py:259-276` (get balance and calculate settlement) - -**New Logic**: -```python -# Get user's current balance -balance = await fava.get_user_balance(user_id) -fiat_balances = balance.get("fiat_balances", {}) -total_fiat_balance = fiat_balances.get(fiat_currency, Decimal(0)) - -# Calculate expected fiat value of SATS payment at current market rate -market_rate = await get_current_sats_eur_rate() # New function needed -market_value = Decimal(amount_sats) * market_rate - -# Calculate exchange variance -receivable_amount = abs(total_fiat_balance) if total_fiat_balance > 0 else Decimal(0) -exchange_variance = market_value - receivable_amount - -# If variance is material (> 1 cent), create exchange gain/loss posting -if abs(exchange_variance) > Decimal("0.01"): - # Add exchange gain/loss to postings - if exchange_variance > 0: - # Gain: payment worth more than receivable - exchange_account = "Revenue:Foreign-Exchange-Gain" - else: - # Loss: payment worth less than receivable - exchange_account = "Expenses:Foreign-Exchange-Loss" - - # Include in entry creation - exchange_posting = { - "account": exchange_account, - "amount": f"{abs(exchange_variance):.2f} {fiat_currency}", - "meta": { - "sats-amount": str(amount_sats), - "market-rate": str(market_rate), - "receivable-amount": str(receivable_amount) - } - } -``` - -**Benefits**: -- ✅ Tax compliance -- ✅ Accurate financial reporting -- ✅ Audit trail for cryptocurrency gains/losses -- ✅ Regulatory compliance (GAAP/IFRS) - ---- - -#### 2.2 Implement True Net Settlement vs. Simple Payment Logic - -**File**: `tasks.py` or new `payment_logic.py` - -```python -async def create_payment_entry( - user_id: str, - amount_sats: int, - fiat_amount: Decimal, - fiat_currency: str, - payment_hash: str -): - """ - Create appropriate payment entry based on user's balance situation. - Uses 2-posting for simple payments, 3-posting for net settlements. - """ - # Get user balance - balance = await fava.get_user_balance(user_id) - fiat_balances = balance.get("fiat_balances", {}) - total_balance = fiat_balances.get(fiat_currency, Decimal(0)) - - receivable_amount = Decimal(0) - payable_amount = Decimal(0) - - if total_balance > 0: - receivable_amount = total_balance - elif total_balance < 0: - payable_amount = abs(total_balance) - - # Determine entry type - if receivable_amount > 0 and payable_amount > 0: - # TRUE NET SETTLEMENT: Both obligations exist - return await format_net_settlement_entry( - user_id=user_id, - amount_sats=amount_sats, - receivable_amount=receivable_amount, - payable_amount=payable_amount, - fiat_amount=fiat_amount, - fiat_currency=fiat_currency, - payment_hash=payment_hash - ) - elif receivable_amount > 0: - # SIMPLE RECEIVABLE PAYMENT: Only receivable exists - return await format_receivable_payment_entry( - user_id=user_id, - amount_sats=amount_sats, - receivable_amount=receivable_amount, - fiat_amount=fiat_amount, - fiat_currency=fiat_currency, - payment_hash=payment_hash - ) - else: - # PAYABLE PAYMENT: Castle paying user (different flow) - return await format_payable_payment_entry(...) -``` - ---- - -### Priority 3: Long-Term Architectural Decisions - -#### 3.1 Establish Primary Currency Hierarchy - -**Current Issue**: Mixed approach (EUR positions with SATS metadata, but also SATS positions with @ notation) - -**Decision Required**: Choose ONE of the following architectures: - -**Architecture A - EUR Primary, SATS Secondary** (recommended): -```beancount -; All positions in EUR, SATS in metadata -2025-11-12 * "Payment" - Assets:Bitcoin:Lightning 200.00 EUR - sats-received: "225033" - Assets:Receivable:User -200.00 EUR - sats-cleared: "225033" -``` - -**Architecture B - SATS Primary, EUR Secondary**: -```beancount -; All positions in SATS, EUR in metadata -2025-11-12 * "Payment" - Assets:Bitcoin:Lightning 225033 SATS - eur-value: "200.00" - Assets:Receivable:User -225033 SATS - eur-cleared: "200.00" -``` - -**Recommendation**: Architecture A (EUR primary) because: -1. Most receivables created in EUR -2. Financial reporting requirements typically in fiat -3. Tax obligations calculated in fiat -4. Aligns with current Castle metadata approach - ---- - -#### 3.2 Consider Separate Ledger for Cryptocurrency Holdings - -**Advanced Approach**: Separate cryptocurrency movements from fiat accounting - -**Main Ledger** (EUR-denominated): -```beancount -2025-11-12 * "Payment received from user" - Assets:Bitcoin-Custody:User-375ec158 200.00 EUR - Assets:Receivable:User-375ec158 -200.00 EUR -``` - -**Cryptocurrency Sub-Ledger** (SATS-denominated): -```beancount -2025-11-12 * "Lightning payment received" - Assets:Bitcoin:Lightning:Castle 225033 SATS - Assets:Bitcoin:Custody:User-375ec 225033 SATS -``` - -**Benefits**: -- ✅ Clean separation of concerns -- ✅ Cryptocurrency movements tracked independently -- ✅ Fiat accounting unaffected by Bitcoin volatility -- ✅ Can generate separate financial statements - -**Drawbacks**: -- ❌ Increased complexity -- ❌ Reconciliation between ledgers required -- ❌ Two sets of books to maintain - ---- - -## Code Files Requiring Changes - -### High Priority (Immediate Fixes) - -1. **`beancount_format.py:739-760`** - - Remove zero-amount postings - - Make payable posting conditional - -2. **`beancount_format.py:692`** - - Rename function to `format_receivable_payment_entry` - -### Medium Priority (Compliance) - -3. **`tasks.py:235-310`** - - Add exchange gain/loss calculation - - Implement payment vs. settlement logic - -4. **New file: `exchange_rates.py`** - - Create `get_current_sats_eur_rate()` function - - Implement price feed integration - -5. **`beancount_format.py`** - - Create new `format_net_settlement_entry()` for true netting - - Create `format_receivable_payment_entry()` for simple payments - ---- - -## Testing Requirements - -### Test Case 1: Simple Receivable Payment (No Payable) - -**Setup**: -- User has receivable: 200.00 EUR -- User has payable: 0.00 EUR -- User pays: 225,033 SATS - -**Expected Entry** (after fixes): -```beancount -2025-11-12 * "Lightning payment from user" - Assets:Bitcoin:Lightning 200.00 EUR - sats-received: "225033" - payment-hash: "8d080ec4..." - Assets:Receivable:User -200.00 EUR - sats-cleared: "225033" -``` - -**Verify**: -- ✅ Only 2 postings (no zero-amount payable) -- ✅ Entry balances -- ✅ SATS tracked in metadata -- ✅ User balance becomes 0 (both EUR and SATS) - ---- - -### Test Case 2: True Net Settlement - -**Setup**: -- User has receivable: 555.00 EUR -- User has payable: 38.00 EUR -- Net owed: 517.00 EUR -- User pays: 565,251 SATS (worth 517.00 EUR) - -**Expected Entry**: -```beancount -2025-11-12 * "Net settlement via Lightning" - Assets:Bitcoin:Lightning 517.00 EUR - sats-received: "565251" - payment-hash: "abc123..." - Assets:Receivable:User -555.00 EUR - sats-portion: "565251" - Liabilities:Payable:User 38.00 EUR -``` - -**Verify**: -- ✅ 3 postings (receivable + payable cleared) -- ✅ Net amount = receivable - payable -- ✅ Both balances become 0 -- ✅ Mathematically balanced - ---- - -### Test Case 3: Exchange Gain/Loss (Future) - -**Setup**: -- User has receivable: 200.00 EUR (created at 1,125 sats/EUR) -- User pays: 225,033 SATS (now worth 199.50 EUR at market) -- Exchange loss: 0.50 EUR - -**Expected Entry** (with exchange tracking): -```beancount -2025-11-12 * "Lightning payment with exchange loss" - Assets:Bitcoin:Lightning 199.50 EUR - sats-received: "225033" - market-rate: "0.000886" - Expenses:Foreign-Exchange-Loss 0.50 EUR - Assets:Receivable:User -200.00 EUR -``` - -**Verify**: -- ✅ Bitcoin recorded at fair market value -- ✅ Exchange loss recognized -- ✅ Receivable cleared at book value -- ✅ Entry balances - ---- - -## Conclusion - -### Summary of Issues - -| Issue | Severity | Accounting Impact | Recommended Action | -|-------|----------|-------------------|-------------------| -| Zero-amount postings | Low | Presentation only | Remove immediately | -| Redundant SATS tracking | Low | Storage/efficiency | Choose one method | -| No exchange gain/loss | **High** | Financial accuracy | Implement for compliance | -| Semantic misuse of @ | Medium | Audit clarity | Consider EUR-only positions | -| Misnamed function | Low | Code clarity | Rename function | - -### Professional Assessment - -**Is this "best practice" accounting?** -**No**, this implementation deviates from traditional accounting standards in several ways. - -**Is it acceptable for Castle's use case?** -**Yes, with modifications**, it's a reasonable pragmatic solution for a novel problem (cryptocurrency payments of fiat debts). - -**Critical improvements needed**: -1. ✅ Remove zero-amount postings (easy fix, professional presentation) -2. ✅ Implement exchange gain/loss tracking (required for compliance) -3. ✅ Separate payment vs. settlement logic (accuracy and clarity) - -**The fundamental challenge**: Traditional accounting wasn't designed for this scenario. There is no established "standard" for recording cryptocurrency payments of fiat-denominated receivables. Castle's approach is functional, but should be refined to align better with accounting principles where possible. - -### Next Steps - -1. **Week 1**: Implement Priority 1 fixes (remove zero postings, rename function) -2. **Week 2-3**: Design and implement exchange gain/loss tracking -3. **Week 4**: Add payment vs. settlement logic -4. **Ongoing**: Monitor regulatory guidance on cryptocurrency accounting - ---- - -## References - -- **FASB ASC 830**: Foreign Currency Matters -- **IAS 21**: The Effects of Changes in Foreign Exchange Rates -- **FASB Concept Statement No. 2**: Qualitative Characteristics of Accounting Information -- **ASC 105-10-05**: Substance Over Form -- **Beancount Documentation**: http://furius.ca/beancount/doc/index -- **Castle Extension**: `docs/SATS-EQUIVALENT-METADATA.md` -- **BQL Analysis**: `docs/BQL-BALANCE-QUERIES.md` - ---- - -**Document Version**: 1.0 -**Last Updated**: 2025-01-12 -**Next Review**: After Priority 1 fixes implemented - ---- - -*This analysis was prepared for internal review and development planning. It represents a professional accounting assessment of the current implementation and should be used to guide improvements to Castle's payment recording system.* diff --git a/docs/BEANCOUNT_PATTERNS.md b/docs/BEANCOUNT_PATTERNS.md index 2124c92..907ebc6 100644 --- a/docs/BEANCOUNT_PATTERNS.md +++ b/docs/BEANCOUNT_PATTERNS.md @@ -61,7 +61,8 @@ class ImmutableEntryLine(NamedTuple): id: str journal_entry_id: str account_id: str - amount: int # Beancount-style: positive = debit, negative = credit + debit: int + credit: int description: Optional[str] metadata: dict[str, Any] flag: Optional[str] # Like Beancount: '!', '*', etc. @@ -144,14 +145,15 @@ class CastlePlugin(Protocol): __plugins__ = ('check_all_balanced',) def check_all_balanced(entries, settings, config): - """Verify all journal entries balance (sum of amounts = 0)""" + """Verify all journal entries have debits = credits""" errors = [] for entry in entries: - total_amount = sum(line.amount for line in entry.lines) - if total_amount != 0: + total_debits = sum(line.debit for line in entry.lines) + total_credits = sum(line.credit for line in entry.lines) + if total_debits != total_credits: errors.append({ 'entry_id': entry.id, - 'message': f'Unbalanced entry: sum of amounts={total_amount} (must equal 0)', + 'message': f'Unbalanced entry: debits={total_debits}, credits={total_credits}', 'severity': 'error' }) return entries, errors @@ -182,7 +184,7 @@ def check_receivable_limits(entries, settings, config): for line in entry.lines: if 'Accounts Receivable' in line.account_name: user_id = extract_user_from_account(line.account_name) - receivables[user_id] = receivables.get(user_id, 0) + line.amount + receivables[user_id] = receivables.get(user_id, 0) + line.debit - line.credit for user_id, amount in receivables.items(): if amount > max_per_user: @@ -365,15 +367,22 @@ async def get_user_inventory(user_id: str) -> CastleInventory: # Add as position metadata = json.loads(line.metadata) if line.metadata else {} - if line.amount != 0: - # Beancount-style: positive = debit, negative = credit - # Adjust sign for cost amount based on amount direction - cost_sign = 1 if line.amount > 0 else -1 + if line.debit > 0: inventory.add_position(CastlePosition( currency="SATS", - amount=Decimal(line.amount), + amount=Decimal(line.debit), cost_currency=metadata.get("fiat_currency"), - cost_amount=cost_sign * Decimal(metadata.get("fiat_amount", 0)), + cost_amount=Decimal(metadata.get("fiat_amount", 0)), + date=line.created_at, + metadata=metadata + )) + + if line.credit > 0: + inventory.add_position(CastlePosition( + currency="SATS", + amount=-Decimal(line.credit), + cost_currency=metadata.get("fiat_currency"), + cost_amount=-Decimal(metadata.get("fiat_amount", 0)), date=line.created_at, metadata=metadata )) @@ -831,16 +840,17 @@ class UnbalancedEntryError(NamedTuple): async def validate_journal_entry(entry: CreateJournalEntry) -> list[CastleError]: errors = [] - # Beancount-style: sum of amounts must equal 0 - total_amount = sum(line.amount for line in entry.lines) + total_debits = sum(line.debit for line in entry.lines) + total_credits = sum(line.credit for line in entry.lines) - if total_amount != 0: + if total_debits != total_credits: errors.append(UnbalancedEntryError( source={'created_by': entry.created_by}, - message=f"Entry does not balance: sum of amounts={total_amount} (must equal 0)", + message=f"Entry does not balance: debits={total_debits}, credits={total_credits}", entry=entry.dict(), - total_amount=total_amount, - difference=total_amount + total_debits=total_debits, + total_credits=total_credits, + difference=total_debits - total_credits )) return errors diff --git a/docs/BQL-BALANCE-QUERIES.md b/docs/BQL-BALANCE-QUERIES.md deleted file mode 100644 index d4997ab..0000000 --- a/docs/BQL-BALANCE-QUERIES.md +++ /dev/null @@ -1,643 +0,0 @@ -# BQL Balance Queries Implementation - -**Date**: November 10, 2025 -**Status**: In Progress -**Context**: Replace manual aggregation with Beancount Query Language (BQL) - ---- - -## Problem - -Current `get_user_balance()` implementation: -- **115 lines** of manual aggregation logic -- Fetches **ALL** journal entries (inefficient) -- Manual regex parsing of amounts -- Manual looping through entries/postings -- O(n) complexity for every balance query - -**Performance Impact**: -- Every balance check fetches entire ledger -- No database-level filtering -- CPU-intensive parsing and aggregation -- Scales poorly as ledger grows - ---- - -## Solution: Use Beancount Query Language (BQL) - -Beancount has a built-in query language that can efficiently: -- Filter accounts (regex patterns) -- Sum positions (balances) -- Exclude transactions by flag -- Group and aggregate -- All processing done by Beancount engine (optimized C code) - ---- - -## BQL Query Design - -### Query 1: Get User Balance (SATS + Fiat) - -```sql -SELECT - account, - sum(position) as balance -WHERE - account ~ ':User-{user_id[:8]}' - AND (account ~ 'Payable' OR account ~ 'Receivable') - AND flag != '!' -GROUP BY account -``` - -**What this does**: -- `account ~ ':User-abc12345'` - Match user's accounts (regex) -- `account ~ 'Payable' OR account ~ 'Receivable'` - Only payable/receivable accounts -- `flag != '!'` - Exclude pending transactions -- `sum(position)` - Aggregate balances -- `GROUP BY account` - Separate totals per account - -**Result Format** (from Fava API): -```json -{ - "data": { - "rows": [ - ["Liabilities:Payable:User-abc12345", {"SATS": "150000", "EUR": "145.50"}], - ["Assets:Receivable:User-abc12345", {"SATS": "50000", "EUR": "48.00"}] - ], - "types": [ - {"name": "account", "type": "str"}, - {"name": "balance", "type": "Position"} - ] - } -} -``` - -### Query 2: Get All User Balances (Admin View) - -```sql -SELECT - account, - sum(position) as balance -WHERE - (account ~ 'Payable:User-' OR account ~ 'Receivable:User-') - AND flag != '!' -GROUP BY account -``` - -**What this does**: -- Match ALL user accounts (not just one user) -- Aggregate balances per account -- Extract user_id from account name in post-processing - ---- - -## Implementation Plan - -### Step 1: Add General BQL Query Method - -Add to `fava_client.py`: - -```python -async def query_bql(self, query_string: str) -> Dict[str, Any]: - """ - Execute arbitrary Beancount Query Language (BQL) query. - - Args: - query_string: BQL query (e.g., "SELECT account, sum(position) WHERE ...") - - Returns: - { - "rows": [[col1, col2, ...], ...], - "types": [{"name": "col1", "type": "str"}, ...], - "column_names": ["col1", "col2", ...] - } - - Example: - result = await fava.query_bql("SELECT account, sum(position) WHERE account ~ 'User-abc'") - for row in result["rows"]: - account, balance = row - print(f"{account}: {balance}") - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get( - f"{self.base_url}/query", - params={"query_string": query_string} - ) - response.raise_for_status() - result = response.json() - - # Fava returns: {"data": {"rows": [...], "types": [...]}} - data = result.get("data", {}) - rows = data.get("rows", []) - types = data.get("types", []) - column_names = [t.get("name") for t in types] - - return { - "rows": rows, - "types": types, - "column_names": column_names - } - - except httpx.HTTPStatusError as e: - logger.error(f"BQL query error: {e.response.status_code} - {e.response.text}") - logger.error(f"Query was: {query_string}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise -``` - -### Step 2: Implement BQL-Based Balance Query - -Add to `fava_client.py`: - -```python -async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]: - """ - Get user balance using BQL (efficient, ~10 lines vs 115 lines manual). - - Args: - user_id: User ID - - Returns: - { - "balance": int (sats), - "fiat_balances": {"EUR": Decimal("100.50")}, - "accounts": [{"account": "...", "sats": 150000}] - } - """ - # Build BQL query for this user's Payable/Receivable accounts - user_id_prefix = user_id[:8] - query = f""" - SELECT account, sum(position) as balance - WHERE account ~ ':User-{user_id_prefix}' - AND (account ~ 'Payable' OR account ~ 'Receivable') - AND flag != '!' - GROUP BY account - """ - - result = await self.query_bql(query) - - # Process results - total_sats = 0 - fiat_balances = {} - accounts = [] - - for row in result["rows"]: - account_name, position = row - - # Position is a dict like {"SATS": "150000", "EUR": "145.50"} - # or a string for single-currency - - if isinstance(position, dict): - # Extract SATS - sats_str = position.get("SATS", "0") - sats_amount = int(sats_str) if sats_str else 0 - total_sats += sats_amount - - accounts.append({ - "account": account_name, - "sats": sats_amount - }) - - # Extract fiat currencies - for currency in ["EUR", "USD", "GBP"]: - if currency in position: - fiat_str = position[currency] - fiat_amount = Decimal(fiat_str) if fiat_str else Decimal(0) - - if currency not in fiat_balances: - fiat_balances[currency] = Decimal(0) - fiat_balances[currency] += fiat_amount - - elif isinstance(position, str): - # Single currency (parse "150000 SATS" or "145.50 EUR") - import re - sats_match = re.match(r'^(-?\d+)\s+SATS$', position) - if sats_match: - sats_amount = int(sats_match.group(1)) - total_sats += sats_amount - accounts.append({ - "account": account_name, - "sats": sats_amount - }) - else: - fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', position) - if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - fiat_amount = Decimal(fiat_match.group(1)) - currency = fiat_match.group(2) - - if currency not in fiat_balances: - fiat_balances[currency] = Decimal(0) - fiat_balances[currency] += fiat_amount - - logger.info(f"User {user_id[:8]} balance (BQL): {total_sats} sats, fiat: {dict(fiat_balances)}") - - return { - "balance": total_sats, - "fiat_balances": fiat_balances, - "accounts": accounts - } -``` - -### Step 3: Implement BQL-Based All Users Balance - -```python -async def get_all_user_balances_bql(self) -> List[Dict[str, Any]]: - """ - Get balances for all users using BQL (efficient admin view). - - Returns: - [ - { - "user_id": "abc123", - "balance": 100000, - "fiat_balances": {"EUR": Decimal("100.50")}, - "accounts": [...] - }, - ... - ] - """ - query = """ - SELECT account, sum(position) as balance - WHERE (account ~ 'Payable:User-' OR account ~ 'Receivable:User-') - AND flag != '!' - GROUP BY account - """ - - result = await self.query_bql(query) - - # Group by user_id - user_data = {} - - for row in result["rows"]: - account_name, position = row - - # Extract user_id from account name - # Format: "Liabilities:Payable:User-abc12345" or "Assets:Receivable:User-abc12345" - if ":User-" not in account_name: - continue - - user_id_with_prefix = account_name.split(":User-")[1] - # User ID is the first 8 chars (our standard) - user_id = user_id_with_prefix[:8] - - if user_id not in user_data: - user_data[user_id] = { - "user_id": user_id, - "balance": 0, - "fiat_balances": {}, - "accounts": [] - } - - # Process position (same logic as single-user query) - if isinstance(position, dict): - sats_str = position.get("SATS", "0") - sats_amount = int(sats_str) if sats_str else 0 - user_data[user_id]["balance"] += sats_amount - - user_data[user_id]["accounts"].append({ - "account": account_name, - "sats": sats_amount - }) - - for currency in ["EUR", "USD", "GBP"]: - if currency in position: - fiat_str = position[currency] - fiat_amount = Decimal(fiat_str) if fiat_str else Decimal(0) - - if currency not in user_data[user_id]["fiat_balances"]: - user_data[user_id]["fiat_balances"][currency] = Decimal(0) - user_data[user_id]["fiat_balances"][currency] += fiat_amount - - # (Handle string format similarly...) - - return list(user_data.values()) -``` - ---- - -## Testing Strategy - -### Unit Tests - -```python -# tests/test_fava_client_bql.py - -async def test_query_bql(): - """Test general BQL query method.""" - fava = get_fava_client() - - result = await fava.query_bql("SELECT account WHERE account ~ 'Assets'") - - assert "rows" in result - assert "column_names" in result - assert len(result["rows"]) > 0 - -async def test_get_user_balance_bql(): - """Test BQL-based user balance query.""" - fava = get_fava_client() - - balance = await fava.get_user_balance_bql("test_user_id") - - assert "balance" in balance - assert "fiat_balances" in balance - assert "accounts" in balance - assert isinstance(balance["balance"], int) - -async def test_bql_matches_manual(): - """Verify BQL results match manual aggregation (for migration).""" - fava = get_fava_client() - user_id = "test_user_id" - - # Get balance both ways - bql_balance = await fava.get_user_balance_bql(user_id) - manual_balance = await fava.get_user_balance(user_id) - - # Should match - assert bql_balance["balance"] == manual_balance["balance"] - assert bql_balance["fiat_balances"] == manual_balance["fiat_balances"] -``` - -### Integration Tests - -```python -async def test_bql_performance(): - """BQL should be significantly faster than manual aggregation.""" - import time - - fava = get_fava_client() - user_id = "test_user_id" - - # Time BQL approach - start = time.time() - bql_result = await fava.get_user_balance_bql(user_id) - bql_time = time.time() - start - - # Time manual approach - start = time.time() - manual_result = await fava.get_user_balance(user_id) - manual_time = time.time() - start - - logger.info(f"BQL: {bql_time:.3f}s, Manual: {manual_time:.3f}s") - - # BQL should be faster (or at least not slower) - # With large ledgers, BQL should be 2-10x faster - assert bql_time <= manual_time * 2 # Allow some variance -``` - ---- - -## Migration Strategy - -### Phase 1: Add BQL Methods (Non-Breaking) - -1. Add `query_bql()` method -2. Add `get_user_balance_bql()` method -3. Add `get_all_user_balances_bql()` method -4. Keep existing methods unchanged - -**Benefit**: Can test BQL in parallel without breaking existing code. - -### Phase 2: Switch to BQL (Breaking Change) - -1. Rename old methods: - - `get_user_balance()` → `get_user_balance_manual()` (deprecated) - - `get_all_user_balances()` → `get_all_user_balances_manual()` (deprecated) - -2. Rename new methods: - - `get_user_balance_bql()` → `get_user_balance()` - - `get_all_user_balances_bql()` → `get_all_user_balances()` - -3. Update all call sites - -4. Test thoroughly - -5. Remove deprecated manual methods after 1-2 sprints - ---- - -## Expected Performance Improvements - -### Before (Manual Aggregation) - -``` -User balance query: -- Fetch ALL entries: ~100-500ms (depends on ledger size) -- Manual parsing: ~50-200ms (CPU-bound) -- Total: 150-700ms -``` - -### After (BQL) - -``` -User balance query: -- BQL query (filtered at source): ~20-50ms -- Minimal parsing: ~5-10ms -- Total: 25-60ms - -Improvement: 5-10x faster -``` - -### Scalability - -**Manual approach**: -- O(n) where n = total number of entries -- Gets slower as ledger grows -- Fetches entire ledger every time - -**BQL approach**: -- O(log n) with indexing (Beancount internal optimization) -- Filtered at source (only user's accounts) -- Constant time as ledger grows (for single user) - ---- - -## Code Reduction - -- **Before**: `get_user_balance()` = 115 lines -- **After**: `get_user_balance_bql()` = ~60 lines (with comments and error handling) -- **Net reduction**: 55 lines (~48%) - -- **Before**: `get_all_user_balances()` = ~100 lines -- **After**: `get_all_user_balances_bql()` = ~70 lines -- **Net reduction**: 30 lines (~30%) - -**Total code reduction**: ~85 lines across balance query methods - ---- - -## Risks and Mitigation - -### Risk 1: BQL Query Syntax Errors - -**Mitigation**: -- Test queries manually in Fava UI first -- Add comprehensive error logging -- Validate query results format - -### Risk 2: Position Format Variations - -**Mitigation**: -- Handle both dict and string position formats -- Add fallback parsing -- Log unexpected formats for investigation - -### Risk 3: Regression in Balance Calculations - -**Mitigation**: -- Run both methods in parallel during transition -- Compare results and log discrepancies -- Comprehensive test suite - ---- - -## Test Results and Findings - -**Date**: November 10, 2025 -**Status**: ⚠️ **NOT FEASIBLE for Castle's Current Data Structure** - -### Implementation Completed - -1. ✅ Analyze current implementation -2. ✅ Design BQL queries -3. ✅ Implement `query_bql()` method (fava_client.py:494-547) -4. ✅ Implement `get_user_balance_bql()` method (fava_client.py:549-644) -5. ✅ Implement `get_all_user_balances_bql()` method (fava_client.py:646-747) -6. ✅ Test against real data - -### Test Results - -**✅ BQL query execution works perfectly:** -- Successfully queries Fava's `/query` endpoint -- Returns structured results (rows, types, column_names) -- Can filter accounts by regex patterns -- Can aggregate positions using `sum(position)` - -**❌ Cannot access SATS balances:** -- BQL returns EUR/USD positions correctly -- BQL **CANNOT** access posting metadata -- SATS values stored in `posting.meta["sats-equivalent"]` -- No BQL syntax to query metadata fields - -### Root Cause: Architecture Limitation - -**Current Castle Ledger Structure:** -``` -Posting format: - Amount: -360.00 EUR ← Position (BQL can query this) - Metadata: - sats-equivalent: 337096 ← Metadata (BQL CANNOT query this) -``` - -**Test Data:** -- User 375ec158 has 82 EUR postings -- ALL postings have `sats-equivalent` metadata -- ZERO postings have SATS as position amount -- Manual method: -7,694,356 sats (from metadata) -- BQL method: 0 sats (cannot access metadata) - -**BQL Limitation:** -```sql --- ✅ This works (queries position): -SELECT account, sum(position) WHERE account ~ 'User-' - --- ❌ This is NOT possible (metadata access): -SELECT account, sum(meta["sats-equivalent"]) WHERE account ~ 'User-' -``` - -### Why Manual Aggregation is Necessary - -1. **SATS are Castle's primary currency** for balance tracking -2. **SATS values are in metadata**, not positions -3. **BQL has no metadata query capability** -4. **Must iterate through postings** to read `meta["sats-equivalent"]` - -### Performance: Cache Optimization is the Solution - -**Phase 1 Caching (Already Implemented)** provides the performance boost: -- ✅ Account lookups cached (5min TTL) -- ✅ Permission lookups cached (1min TTL) -- ✅ 60-80% reduction in DB queries -- ✅ Addresses the actual bottleneck (database queries, not aggregation) - -**BQL would not improve performance** because: -- Still need to fetch all postings to read metadata -- Aggregation is not the bottleneck (it's fast) -- Database queries are the bottleneck (solved by caching) - ---- - -## Conclusion - -**Status**: ⚠️ **BQL Implementation Not Feasible** - -**Recommendation**: **Keep manual aggregation method with Phase 1 caching** - -**Rationale:** -1. ✅ Caching already provides 60-80% performance improvement -2. ✅ SATS metadata requires posting iteration regardless of query method -3. ✅ BQL cannot access the data we need (metadata) -4. ✅ Manual aggregation is well-tested and working correctly - -**BQL Methods Status**: -- ✅ Implemented and committed as reference code -- ⚠️ NOT used in production (cannot query SATS from metadata) -- 📝 Kept for future consideration if ledger format changes - ---- - -## Future Consideration: Ledger Format Change - -**If** Castle's ledger format changes to use SATS as position amounts: - -```beancount -; Current format (EUR position, SATS in metadata): -2025-11-10 * "Groceries" - Expenses:Food -360.00 EUR - sats-equivalent: 337096 - Liabilities:Payable:User-abc 360.00 EUR - sats-equivalent: 337096 - -; Hypothetical future format (SATS position, EUR as cost): -2025-11-10 * "Groceries" - Expenses:Food -337096 SATS {360.00 EUR} - Liabilities:Payable:User-abc 337096 SATS {360.00 EUR} -``` - -**Then** BQL would become feasible: -```sql --- Would work with SATS as position: -SELECT account, sum(position) as balance -WHERE account ~ 'User-' AND currency = 'SATS' -``` - -**Trade-offs of format change:** -- ✅ Would enable BQL optimization -- ✅ Aligns with "Bitcoin-first" philosophy -- ⚠️ Requires ledger migration -- ⚠️ Changes reporting currency (impacts existing workflows) -- ⚠️ Beancount cost syntax has precision limitations - -**Recommendation**: Consider during major version upgrade or architectural redesign. - ---- - -## Next Steps - -1. ✅ Analyze current implementation -2. ✅ Design BQL queries -3. ✅ Implement `query_bql()` method -4. ✅ Implement `get_user_balance_bql()` method -5. ✅ Test against real data -6. ✅ Implement `get_all_user_balances_bql()` method -7. ✅ Document findings and limitations -8. ❌ Update call sites (NOT APPLICABLE - BQL not feasible) -9. ❌ Remove manual methods (NOT APPLICABLE - manual method is correct approach) - ---- - -**Implementation By**: Claude Code -**Date**: November 10, 2025 -**Status**: ✅ **Tested and Documented** | ⚠️ **Not Feasible for Production Use** diff --git a/docs/BQL-PRICE-NOTATION-SOLUTION.md b/docs/BQL-PRICE-NOTATION-SOLUTION.md deleted file mode 100644 index 24cd073..0000000 --- a/docs/BQL-PRICE-NOTATION-SOLUTION.md +++ /dev/null @@ -1,529 +0,0 @@ -# BQL Price Notation Solution for SATS Tracking - -**Date**: 2025-01-12 -**Status**: Testing -**Context**: Explore price notation as alternative to metadata for SATS tracking - ---- - -## Problem Recap - -Current approach stores SATS in metadata: -```beancount -2025-11-10 * "Groceries" - Expenses:Food -360.00 EUR - sats-equivalent: 337096 - Liabilities:Payable:User-abc 360.00 EUR - sats-equivalent: 337096 -``` - -**Issue**: BQL cannot access metadata, so balance queries require manual aggregation. - ---- - -## Solution: Use Price Notation - -### Proposed Format - -Post in actual transaction currency (EUR) with SATS as price: - -```beancount -2025-11-10 * "Groceries" - Expenses:Food -360.00 EUR @@ 337096 SATS - Liabilities:Payable:User-abc 360.00 EUR @@ 337096 SATS -``` - -**What this means**: -- Primary amount: `-360.00 EUR` (the actual transaction currency) -- Total price: `337096 SATS` (the bitcoin equivalent value) -- Transaction integrity preserved (posted in EUR as it occurred) -- SATS tracked as price (queryable by BQL) - ---- - -## Price Notation Options - -### Option 1: Per-Unit Price (`@`) - -```beancount -Expenses:Food -360.00 EUR @ 936.38 SATS -``` - -**What it means**: Each EUR is worth 936.38 SATS -**Total calculation**: 360 × 936.38 = 337,096.8 SATS -**Precision**: May introduce rounding (336,696.8 vs 337,096) - -### Option 2: Total Price (`@@`) ✅ RECOMMENDED - -```beancount -Expenses:Food -360.00 EUR @@ 337096 SATS -``` - -**What it means**: Total transaction value is 337,096 SATS -**Total calculation**: Exact 337,096 SATS (no rounding) -**Precision**: Preserves exact SATS amount from original calculation - -**Why `@@` is better for Castle:** -- ✅ Preserves exact SATS amount (no rounding errors) -- ✅ Matches current metadata storage exactly -- ✅ Clearer intent: "this transaction equals X SATS total" - ---- - -## How BQL Handles Prices - -### Available Price Columns - -From BQL schema: -- `price_number` - The numeric price amount (Decimal) -- `price_currency` - The currency of the price (str) -- `position` - Full posting (includes price) -- `WEIGHT(position)` - Function that returns balance weight - -### BQL Query Capabilities - -**Test Query 1: Access price directly** -```sql -SELECT account, number, currency, price_number, price_currency -WHERE account ~ 'User-375ec158' - AND price_currency = 'SATS'; -``` - -**Expected Result** (if price notation works): -```json -{ - "rows": [ - ["Liabilities:Payable:User-abc", "360.00", "EUR", "337096", "SATS"] - ] -} -``` - -**Test Query 2: Aggregate SATS from prices** -```sql -SELECT account, - SUM(price_number) as total_sats -WHERE account ~ 'User-' - AND price_currency = 'SATS' - AND flag != '!' -GROUP BY account; -``` - -**Expected Result**: -```json -{ - "rows": [ - ["Liabilities:Payable:User-abc", "337096"] - ] -} -``` - ---- - -## Testing Plan - -### Step 1: Run Metadata Test - -```bash -cd /home/padreug/projects/castle-beancounter -./test_metadata_simple.sh -``` - -**What to look for**: -- Does `meta` column exist in response? -- Is `sats-equivalent` accessible in the data? - -**If YES**: Metadata IS accessible, simpler solution available -**If NO**: Proceed with price notation approach - -### Step 2: Test Current Data Structure - -```bash -./test_bql_metadata.sh -``` - -This runs 6 tests: -1. Check metadata column -2. Check price columns -3. Basic position query -4. Test WEIGHT function -5. Aggregate positions -6. Aggregate weights - -**What to look for**: -- Which columns are available? -- What does `position` return for entries with prices? -- Can we access `price_number` and `price_currency`? - -### Step 3: Create Test Ledger Entry - -Add one test entry to your ledger: - -```beancount -2025-01-12 * "TEST: Price notation test" - Expenses:Test:PriceNotation -100.00 EUR @@ 93600 SATS - Liabilities:Payable:User-TEST 100.00 EUR @@ 93600 SATS -``` - -Then query: -```bash -curl -s "http://localhost:3333/castle-ledger/api/query" \ - -G \ - --data-urlencode "query_string=SELECT account, position, price_number, price_currency WHERE account ~ 'TEST'" \ - | jq '.' -``` - -**Expected if working**: -```json -{ - "data": { - "rows": [ - ["Expenses:Test:PriceNotation", "-100.00 EUR @@ 93600 SATS", "93600", "SATS"], - ["Liabilities:Payable:User-TEST", "100.00 EUR @@ 93600 SATS", "93600", "SATS"] - ], - "types": [ - {"name": "account", "type": "str"}, - {"name": "position", "type": "Position"}, - {"name": "price_number", "type": "Decimal"}, - {"name": "price_currency", "type": "str"} - ] - } -} -``` - ---- - -## Migration Strategy (If Price Notation Works) - -### Phase 1: Test on Sample Data - -1. Create test ledger with mix of formats -2. Verify BQL can query price_number -3. Verify aggregation accuracy -4. Compare with manual method results - -### Phase 2: Write Migration Script - -```python -#!/usr/bin/env python3 -""" -Migrate metadata sats-equivalent to price notation. - -Converts: - Expenses:Food -360.00 EUR - sats-equivalent: 337096 - -To: - Expenses:Food -360.00 EUR @@ 337096 SATS -""" - -import re -from pathlib import Path - -def migrate_entry(entry_lines): - """Migrate a single transaction entry.""" - result = [] - current_posting = None - sats_value = None - - for line in entry_lines: - # Check if this is a posting line - if re.match(r'^\s{2,}\w+:', line): - # If we have pending sats from previous posting, add it - if current_posting and sats_value: - # Add @@ notation to posting - posting = current_posting.rstrip() - posting += f" @@ {sats_value} SATS\n" - result.append(posting) - current_posting = None - sats_value = None - else: - if current_posting: - result.append(current_posting) - current_posting = line - - # Check if this is sats-equivalent metadata - elif 'sats-equivalent:' in line: - match = re.search(r'sats-equivalent:\s*(-?\d+)', line) - if match: - sats_value = match.group(1) - # Don't include metadata line in result - - else: - # Other lines (date, narration, other metadata) - if current_posting and sats_value: - posting = current_posting.rstrip() - posting += f" @@ {sats_value} SATS\n" - result.append(posting) - current_posting = None - sats_value = None - elif current_posting: - result.append(current_posting) - current_posting = None - - result.append(line) - - # Handle last posting - if current_posting and sats_value: - posting = current_posting.rstrip() - posting += f" @@ {sats_value} SATS\n" - result.append(posting) - elif current_posting: - result.append(current_posting) - - return result - -def migrate_ledger(input_file, output_file): - """Migrate entire ledger file.""" - with open(input_file, 'r') as f: - lines = f.readlines() - - result = [] - current_entry = [] - in_transaction = False - - for line in lines: - # Transaction start - if re.match(r'^\d{4}-\d{2}-\d{2}\s+[*!]', line): - in_transaction = True - current_entry = [line] - - # Empty line ends transaction - elif in_transaction and line.strip() == '': - current_entry.append(line) - migrated = migrate_entry(current_entry) - result.extend(migrated) - current_entry = [] - in_transaction = False - - # Inside transaction - elif in_transaction: - current_entry.append(line) - - # Outside transaction - else: - result.append(line) - - # Handle last entry if file doesn't end with blank line - if current_entry: - migrated = migrate_entry(current_entry) - result.extend(migrated) - - with open(output_file, 'w') as f: - f.writelines(result) - -if __name__ == '__main__': - import sys - if len(sys.argv) != 3: - print("Usage: migrate_ledger.py ") - sys.exit(1) - - migrate_ledger(sys.argv[1], sys.argv[2]) - print(f"Migrated {sys.argv[1]} -> {sys.argv[2]}") -``` - -### Phase 3: Update Balance Query Methods - -Replace `get_user_balance_bql()` with price-based version: - -```python -async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]: - """ - Get user balance using price notation (SATS stored as @@ price). - - Returns: - { - "balance": int (sats from price_number), - "fiat_balances": {"EUR": Decimal("100.50")}, - "accounts": [{"account": "...", "sats": 150000}] - } - """ - user_id_prefix = user_id[:8] - - # Query: Get EUR positions with SATS prices - query = f""" - SELECT - account, - number as eur_amount, - price_number as sats_amount - WHERE account ~ ':User-{user_id_prefix}' - AND (account ~ 'Payable' OR account ~ 'Receivable') - AND flag != '!' - AND price_currency = 'SATS' - """ - - result = await self.query_bql(query) - - total_sats = 0 - fiat_balances = {} - accounts_map = {} - - for row in result["rows"]: - account_name, eur_amount, sats_amount = row - - # Parse amounts - sats = int(Decimal(sats_amount)) if sats_amount else 0 - eur = Decimal(eur_amount) if eur_amount else Decimal(0) - - total_sats += sats - - # Aggregate fiat - if eur != 0: - if "EUR" not in fiat_balances: - fiat_balances["EUR"] = Decimal(0) - fiat_balances["EUR"] += eur - - # Track per account - if account_name not in accounts_map: - accounts_map[account_name] = {"account": account_name, "sats": 0} - accounts_map[account_name]["sats"] += sats - - return { - "balance": total_sats, - "fiat_balances": fiat_balances, - "accounts": list(accounts_map.values()) - } -``` - -### Phase 4: Validation - -1. Run both methods in parallel -2. Compare results for all users -3. Log any discrepancies -4. Investigate and fix differences -5. Once validated, switch to BQL method - ---- - -## Advantages of Price Notation Approach - -### 1. BQL Compatibility ✅ -- `price_number` is a standard BQL column -- Can aggregate: `SUM(price_number)` -- Can filter: `WHERE price_currency = 'SATS'` - -### 2. Transaction Integrity ✅ -- Post in actual transaction currency (EUR) -- SATS as secondary value (price) -- Proper accounting: source currency preserved - -### 3. Beancount Features ✅ -- Price database automatically updated -- Can query historical EUR/SATS rates -- Reports can show both EUR and SATS values - -### 4. Performance ✅ -- BQL filters at source (no fetching all entries) -- Direct column access (no metadata parsing) -- Efficient aggregation (database-level) - -### 5. Reporting Flexibility ✅ -- Show EUR amounts in reports -- Show SATS equivalents alongside -- Filter by either currency -- Calculate gains/losses if SATS price changes - ---- - -## Potential Issues and Solutions - -### Issue 1: Price vs Cost Confusion - -**Problem**: Beancount distinguishes between `@` price and `{}` cost -**Solution**: Always use price (`@` or `@@`), never cost (`{}`) - -**Why**: -- Cost is for tracking cost basis (investments, capital gains) -- Price is for conversion rates (what we need) - -### Issue 2: Precision Loss with `@` - -**Problem**: Per-unit price may have rounding -```beancount -360.00 EUR @ 936.38 SATS = 336,696.8 SATS (not 337,096) -``` - -**Solution**: Always use `@@` total price -```beancount -360.00 EUR @@ 337096 SATS = 337,096 SATS (exact) -``` - -### Issue 3: Negative Numbers - -**Problem**: How to handle negative EUR with positive SATS? -```beancount --360.00 EUR @@ ??? SATS -``` - -**Solution**: Price is always positive (it's a rate, not an amount) -```beancount --360.00 EUR @@ 337096 SATS ✅ Correct -``` - -The sign applies to the position, price is the conversion factor. - -### Issue 4: Historical Data - -**Problem**: Existing entries have metadata, not prices - -**Solution**: Migration script (see Phase 2) -- One-time conversion -- Validate with checksums -- Keep backup of original - ---- - -## Testing Checklist - -- [ ] Run `test_metadata_simple.sh` - Check if metadata is accessible -- [ ] Run `test_bql_metadata.sh` - Full BQL capabilities test -- [ ] Add test entry with `@@` notation to ledger -- [ ] Query test entry with BQL to verify price_number access -- [ ] Compare aggregation: metadata vs price notation -- [ ] Test negative amounts with prices -- [ ] Test zero amounts -- [ ] Test multi-currency scenarios (EUR, USD with SATS prices) -- [ ] Verify price database is populated correctly -- [ ] Check that WEIGHT() function returns SATS value -- [ ] Validate balances match current manual method - ---- - -## Decision Matrix - -| Criteria | Metadata | Price Notation | Winner | -|----------|----------|----------------|--------| -| BQL Queryable | ❌ No | ✅ Yes | Price | -| Transaction Integrity | ✅ EUR first | ✅ EUR first | Tie | -| SATS Precision | ✅ Exact int | ✅ Exact (with @@) | Tie | -| Migration Effort | ✅ None | ⚠️ Script needed | Metadata | -| Performance | ❌ Manual loop | ✅ BQL optimized | Price | -| Beancount Standard | ⚠️ Non-standard | ✅ Standard feature | Price | -| Reporting Flexibility | ⚠️ Limited | ✅ Both currencies | Price | -| Future Proof | ⚠️ Custom | ✅ Standard | Price | - -**Recommendation**: **Price Notation** if tests confirm BQL can access `price_number` - ---- - -## Next Steps - -1. **Run tests** (test_metadata_simple.sh and test_bql_metadata.sh) -2. **Review results** - Can BQL access price_number? -3. **Add test entry** with @@ notation -4. **Query test entry** - Verify aggregation works -5. **If successful**: - - Write full migration script - - Test on copy of production ledger - - Validate balances match - - Schedule migration (maintenance window) - - Update balance query methods - - Deploy and monitor -6. **If unsuccessful**: - - Document why price notation doesn't work - - Consider Beancount plugin approach - - Or accept manual aggregation with caching - ---- - -**Document Status**: Awaiting test results -**Next Action**: Run test scripts and report findings diff --git a/docs/DOCUMENTATION.md b/docs/DOCUMENTATION.md index ac79f03..936802b 100644 --- a/docs/DOCUMENTATION.md +++ b/docs/DOCUMENTATION.md @@ -71,7 +71,8 @@ CREATE TABLE entry_lines ( id TEXT PRIMARY KEY, journal_entry_id TEXT NOT NULL, account_id TEXT NOT NULL, - amount INTEGER NOT NULL, -- Amount in satoshis (positive = debit, negative = credit) + debit INTEGER NOT NULL DEFAULT 0, -- Amount in satoshis + credit INTEGER NOT NULL DEFAULT 0, -- Amount in satoshis description TEXT, metadata TEXT DEFAULT '{}' -- JSON: {fiat_currency, fiat_amount, fiat_rate, btc_rate} ); @@ -313,20 +314,17 @@ for account in user_accounts: total_balance -= account_balance # Positive asset = User owes Castle, so negative balance # Calculate fiat balance from metadata - # Beancount-style: positive amount = debit, negative amount = credit for line in account_entry_lines: if line.metadata.fiat_currency and line.metadata.fiat_amount: if account.account_type == AccountType.LIABILITY: - # For liabilities, negative amounts (credits) increase what castle owes - if line.amount < 0: + if line.credit > 0: fiat_balances[currency] += fiat_amount # Castle owes more - else: + elif line.debit > 0: fiat_balances[currency] -= fiat_amount # Castle owes less elif account.account_type == AccountType.ASSET: - # For assets, positive amounts (debits) increase what user owes - if line.amount > 0: + if line.debit > 0: fiat_balances[currency] -= fiat_amount # User owes more (negative balance) - else: + elif line.credit > 0: fiat_balances[currency] += fiat_amount # User owes less ``` @@ -769,8 +767,10 @@ async def export_beancount( beancount_name = format_account_name(account.name, account.user_id) beancount_type = map_account_type(account.account_type) - # Beancount-style: amount is already signed (positive = debit, negative = credit) - amount = line.amount + if line.debit > 0: + amount = line.debit + else: + amount = -line.credit lines.append(f" {beancount_type}:{beancount_name} {amount} SATS") diff --git a/docs/EXPENSE_APPROVAL.md b/docs/EXPENSE_APPROVAL.md index 3123b32..b8b3261 100644 --- a/docs/EXPENSE_APPROVAL.md +++ b/docs/EXPENSE_APPROVAL.md @@ -41,7 +41,7 @@ Only entries with `flag='*'` (CLEARED) are included in balance calculations: ```sql -- Balance query excludes pending/flagged/voided entries -SELECT SUM(amount) +SELECT SUM(debit), SUM(credit) FROM entry_lines el JOIN journal_entries je ON el.journal_entry_id = je.id WHERE el.account_id = :account_id diff --git a/docs/PERMISSIONS-SYSTEM.md b/docs/PERMISSIONS-SYSTEM.md deleted file mode 100644 index c3c88b7..0000000 --- a/docs/PERMISSIONS-SYSTEM.md +++ /dev/null @@ -1,861 +0,0 @@ -# Castle Permissions System - Overview & Administration Guide - -**Date**: November 10, 2025 -**Status**: 📚 **Documentation** + 🔧 **Improvement Recommendations** - ---- - -## Executive Summary - -Castle implements a **granular, hierarchical permission system** that controls who can access which accounts and perform what actions. The system supports permission inheritance, making it easy to grant access to entire account hierarchies with a single permission. - -**Key Features:** -- ✅ **Three permission levels**: READ, SUBMIT_EXPENSE, MANAGE -- ✅ **Hierarchical inheritance**: Permission on parent → access to all children -- ✅ **Expiration support**: Time-limited permissions -- ✅ **Caching**: 1-minute TTL for performance -- ✅ **Audit trail**: Track who granted permissions and when - ---- - -## Permission Types - -### 1. READ -**Purpose**: View account balances and transaction history - -**Capabilities**: -- View account balance -- See transaction history for the account -- List sub-accounts (if hierarchical) - -**Use cases**: -- Transparency for community members -- Auditors reviewing finances -- Users checking their own balances - -**Example**: -```python -# Grant read access to view food expenses -await create_account_permission( - user_id="user123", - account_id="expenses_food_account_id", - permission_type=PermissionType.READ -) -``` - -### 2. SUBMIT_EXPENSE -**Purpose**: Submit expenses against an account - -**Capabilities**: -- Submit new expense entries -- Create transactions that debit the account -- Automatically creates user receivable/payable entries - -**Use cases**: -- Members submitting food expenses -- Workers logging accommodation costs -- Contributors recording service expenses - -**Example**: -```python -# Grant permission to submit food expenses -await create_account_permission( - user_id="user123", - account_id="expenses_food_account_id", - permission_type=PermissionType.SUBMIT_EXPENSE -) - -# User can now submit: -# Debit: Expenses:Food:Groceries 100 EUR -# Credit: Liabilities:Payable:User-user123 100 EUR -``` - -### 3. MANAGE -**Purpose**: Administrative control over an account - -**Capabilities**: -- Modify account settings -- Change account description/metadata -- Grant permissions to other users (delegated administration) -- Archive/close accounts - -**Use cases**: -- Department heads managing their budgets -- Admins delegating permission management -- Account owners controlling access - -**Example**: -```python -# Grant full management rights to department head -await create_account_permission( - user_id="dept_head", - account_id="expenses_marketing_account_id", - permission_type=PermissionType.MANAGE -) -``` - ---- - -## Hierarchical Inheritance - -### How It Works - -Permissions on **parent accounts automatically apply to all child accounts**. - -**Hierarchy Example:** -``` -Expenses:Food -├── Expenses:Food:Groceries -├── Expenses:Food:Restaurants -└── Expenses:Food:Cafeteria -``` - -**Permission on Parent:** -```python -# Grant SUBMIT_EXPENSE on "Expenses:Food" -await create_account_permission( - user_id="alice", - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE -) -``` - -**Result:** Alice can now submit expenses to: -- ✅ `Expenses:Food` -- ✅ `Expenses:Food:Groceries` (inherited) -- ✅ `Expenses:Food:Restaurants` (inherited) -- ✅ `Expenses:Food:Cafeteria` (inherited) - -### Implementation - -The `get_user_permissions_with_inheritance()` function checks for both direct and inherited permissions: - -```python -async def get_user_permissions_with_inheritance( - user_id: str, account_name: str, permission_type: PermissionType -) -> list[tuple[AccountPermission, Optional[str]]]: - """ - Returns: [(permission, parent_account_name or None)] - - Example: - Checking permission on "Expenses:Food:Groceries" - User has permission on "Expenses:Food" - - Returns: [(permission_obj, "Expenses:Food")] - """ - user_permissions = await get_user_permissions(user_id, permission_type) - - applicable_permissions = [] - for perm in user_permissions: - account = await get_account(perm.account_id) - - if account_name == account.name: - # Direct permission - applicable_permissions.append((perm, None)) - elif account_name.startswith(account.name + ":"): - # Inherited from parent - applicable_permissions.append((perm, account.name)) - - return applicable_permissions -``` - -**Benefits:** -- Grant one permission → access to entire subtree -- Easier administration (fewer permissions to manage) -- Natural organizational structure -- Can still override with specific permissions on children - ---- - -## Permission Lifecycle - -### 1. Granting Permission - -**Admin grants permission:** -```python -await create_account_permission( - data=CreateAccountPermission( - user_id="alice", - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - expires_at=None, # No expiration - notes="Food coordinator for Q1 2025" - ), - granted_by="admin_user_id" -) -``` - -**Result:** -- Permission stored in DB -- Cache invalidated for user -- Audit trail recorded (who, when) - -### 2. Checking Permission - -**Before allowing expense submission:** -```python -# Check if user can submit expense to account -permissions = await get_user_permissions_with_inheritance( - user_id="alice", - account_name="Expenses:Food:Groceries", - permission_type=PermissionType.SUBMIT_EXPENSE -) - -if not permissions: - raise HTTPException(403, "Permission denied") - -# Permission found - allow operation -``` - -**Performance:** First check hits DB, subsequent checks hit cache (1min TTL) - -### 3. Permission Expiration - -**Automatic expiration check:** -```python -# get_user_permissions() automatically filters expired permissions -SELECT * FROM account_permissions -WHERE user_id = :user_id -AND permission_type = :permission_type -AND (expires_at IS NULL OR expires_at > NOW()) ← Automatic filtering -``` - -**Time-limited permission example:** -```python -await create_account_permission( - data=CreateAccountPermission( - user_id="contractor", - account_id="expenses_temp_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - expires_at=datetime(2025, 12, 31), # Expires end of year - notes="Temporary contractor access" - ), - granted_by="admin" -) -``` - -### 4. Revoking Permission - -**Manual revocation:** -```python -await delete_account_permission(permission_id="perm123") -``` - -**Result:** -- Permission deleted from DB -- Cache invalidated for user -- User immediately loses access (after cache TTL) - ---- - -## Caching Strategy - -### Cache Configuration - -```python -# Cache for permission lookups -permission_cache = Cache(default_ttl=60) # 1 minute TTL - -# Cache keys: -# - "permissions:user:{user_id}" → All permissions for user -# - "permissions:user:{user_id}:{permission_type}" → Filtered by type -``` - -**Why 1 minute TTL?** -- Permissions may change frequently (grant/revoke) -- Security-sensitive data needs to be fresh -- Balance between performance and accuracy - -### Cache Invalidation - -**On permission creation:** -```python -# Invalidate both general and type-specific caches -permission_cache._values.pop(f"permissions:user:{user_id}", None) -permission_cache._values.pop(f"permissions:user:{user_id}:{permission_type.value}", None) -``` - -**On permission deletion:** -```python -# Get permission first to know which user's cache to clear -permission = await get_account_permission(permission_id) -await db.execute("DELETE FROM account_permissions WHERE id = :id", {"id": permission_id}) - -# Invalidate caches -permission_cache._values.pop(f"permissions:user:{permission.user_id}", None) -permission_cache._values.pop(f"permissions:user:{permission.user_id}:{permission.permission_type.value}", None) -``` - -**Performance Impact:** -- Cold cache: ~50ms (DB query) -- Warm cache: ~1ms (memory lookup) -- **Reduction**: 60-80% fewer DB queries - ---- - -## Administration Best Practices - -### 1. Use Hierarchical Permissions - -**❌ Don't do this:** -```python -# Granting 10 separate permissions (hard to manage) -await create_account_permission(user, "Expenses:Food:Groceries", SUBMIT_EXPENSE) -await create_account_permission(user, "Expenses:Food:Restaurants", SUBMIT_EXPENSE) -await create_account_permission(user, "Expenses:Food:Cafeteria", SUBMIT_EXPENSE) -await create_account_permission(user, "Expenses:Food:Snacks", SUBMIT_EXPENSE) -# ... 6 more -``` - -**✅ Do this instead:** -```python -# Single permission covers all children -await create_account_permission(user, "Expenses:Food", SUBMIT_EXPENSE) -``` - -**Benefits:** -- Fewer permissions to track -- Easier to revoke (one permission vs many) -- Automatically covers new sub-accounts -- Cleaner audit trail - -### 2. Use Expiration for Temporary Access - -**❌ Don't do this:** -```python -# Grant permanent access to temp worker -await create_account_permission(user, account, SUBMIT_EXPENSE) -# ... then forget to revoke when they leave -``` - -**✅ Do this instead:** -```python -# Auto-expiring permission -await create_account_permission( - user, - account, - SUBMIT_EXPENSE, - expires_at=contract_end_date, # Automatic cleanup - notes="Contractor until 2025-12-31" -) -``` - -**Benefits:** -- No manual cleanup needed -- Reduced security risk -- Self-documenting access period -- Admin can still revoke early if needed - -### 3. Use Notes for Audit Trail - -**❌ Don't do this:** -```python -# No context -await create_account_permission(user, account, SUBMIT_EXPENSE) -``` - -**✅ Do this instead:** -```python -# Clear documentation -await create_account_permission( - user, - account, - SUBMIT_EXPENSE, - notes="Food coordinator for Q1 2025 - approved in meeting 2025-01-05" -) -``` - -**Benefits:** -- Future admins understand why permission exists -- Audit trail for compliance -- Easier to review permissions -- Can reference approval process - -### 4. Principle of Least Privilege - -**Start with READ, escalate only if needed:** - -```python -# Initial access: READ only -await create_account_permission(user, account, PermissionType.READ) - -# If user needs to submit expenses, upgrade: -await create_account_permission(user, account, PermissionType.SUBMIT_EXPENSE) - -# Only grant MANAGE to trusted users: -await create_account_permission(dept_head, account, PermissionType.MANAGE) -``` - -**Security principle:** Grant minimum permissions needed for the task. - ---- - -## Current Implementation Strengths - -✅ **Well-designed features:** -1. **Hierarchical inheritance** - Reduces admin burden -2. **Type safety** - Enum-based permission types prevent typos -3. **Caching** - Good performance without sacrificing security -4. **Expiration support** - Automatic cleanup of temporary access -5. **Audit trail** - Tracks who granted permissions and when -6. **Foreign key constraints** - Cannot grant permission on non-existent account - ---- - -## Improvement Opportunities - -### 🔧 Opportunity 1: Permission Groups/Roles - -**Current limitation:** Must grant permissions individually - -**Proposed enhancement:** -```python -# Define reusable permission groups -ROLE_FOOD_COORDINATOR = [ - (PermissionType.READ, "Expenses:Food"), - (PermissionType.SUBMIT_EXPENSE, "Expenses:Food"), - (PermissionType.MANAGE, "Expenses:Food:Groceries"), -] - -# Grant entire role at once -await grant_role(user_id="alice", role=ROLE_FOOD_COORDINATOR) -``` - -**Benefits:** -- Standard permission sets -- Easier onboarding -- Consistent access patterns -- Bulk grant/revoke - -**Implementation effort:** 1-2 days - ---- - -### 🔧 Opportunity 2: Permission Templates - -**Current limitation:** No way to clone permissions from one user to another - -**Proposed enhancement:** -```python -# Copy all permissions from one user to another -await copy_permissions( - from_user="experienced_coordinator", - to_user="new_coordinator", - permission_types=[PermissionType.SUBMIT_EXPENSE], # Optional filter - notes="Copied from Alice - new food coordinator" -) -``` - -**Benefits:** -- Faster onboarding -- Consistency -- Reduces errors -- Preserves expiration patterns - -**Implementation effort:** 1 day - ---- - -### 🔧 Opportunity 3: Bulk Permission Management - -**Current limitation:** One permission at a time - -**Proposed enhancement:** -```python -# Grant same permission to multiple users -await bulk_grant_permission( - user_ids=["alice", "bob", "charlie"], - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - expires_at=datetime(2025, 12, 31), - notes="Q4 food team" -) - -# Revoke all permissions on an account -await revoke_all_permissions_on_account(account_id="old_project_id") - -# Revoke all permissions for a user (offboarding) -await revoke_all_user_permissions(user_id="departed_user") -``` - -**Benefits:** -- Faster administration -- Consistent permission sets -- Easy offboarding -- Bulk operations for events/projects - -**Implementation effort:** 2 days - ---- - -### 🔧 Opportunity 4: Permission Analytics Dashboard - -**Current limitation:** No visibility into permission usage - -**Proposed enhancement:** -```python -# Admin endpoint for permission analytics -@router.get("/api/v1/admin/permissions/analytics") -async def get_permission_analytics(): - return { - "total_permissions": 150, - "by_type": { - "READ": 50, - "SUBMIT_EXPENSE": 80, - "MANAGE": 20 - }, - "expiring_soon": [ - {"user_id": "alice", "account": "Expenses:Food", "expires": "2025-11-15"}, - # ... more - ], - "most_permissioned_accounts": [ - {"account": "Expenses:Food", "permission_count": 25}, - # ... more - ], - "users_without_permissions": ["bob", "charlie"], # Alert for review - "orphaned_permissions": [] # Permissions on deleted accounts - } -``` - -**Benefits:** -- Visibility into access patterns -- Proactive expiration management -- Security audit support -- Identify unused permissions - -**Implementation effort:** 2-3 days - ---- - -### 🔧 Opportunity 5: Permission Request Workflow - -**Current limitation:** Users must ask admin manually to grant permissions - -**Proposed enhancement:** -```python -# User requests permission -await request_permission( - user_id="alice", - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - justification="I'm the new food coordinator starting next week" -) - -# Admin reviews and approves -pending = await get_pending_permission_requests() -await approve_permission_request(request_id="req123", admin_user_id="admin") - -# Or deny with reason -await deny_permission_request( - request_id="req456", - admin_user_id="admin", - reason="Please request via department head first" -) -``` - -**Benefits:** -- Self-service permission requests -- Audit trail for approvals -- Reduces admin manual work -- Transparent process - -**Implementation effort:** 3-4 days - ---- - -### 🔧 Opportunity 6: Permission Monitoring & Alerts - -**Current limitation:** No alerts for security events - -**Proposed enhancement:** -```python -# Monitor and alert on permission changes -class PermissionMonitor: - async def on_permission_granted(self, permission): - # Alert if MANAGE permission granted - if permission.permission_type == PermissionType.MANAGE: - await send_admin_alert( - f"MANAGE permission granted to {permission.user_id} on {account.name}" - ) - - async def on_permission_expired(self, permission): - # Alert user their access is expiring - await send_user_notification( - user_id=permission.user_id, - message=f"Your access to {account.name} expires in 7 days" - ) - - async def on_suspicious_activity(self, user_id, account_id): - # Alert on unusual permission usage patterns - if failed_permission_checks > 5: - await send_admin_alert( - f"User {user_id} attempted access to {account_id} 5 times (denied)" - ) -``` - -**Benefits:** -- Security monitoring -- Proactive expiration management -- Detect permission issues early -- Compliance support - -**Implementation effort:** 2-3 days - ---- - -## Recommended Implementation Priority - -### Phase 1: Quick Wins (1 week) -1. **Bulk Permission Management** (2 days) - Immediate productivity boost -2. **Permission Templates** (1 day) - Easy onboarding -3. **Permission Analytics** (2 days) - Visibility and audit support - -**Total effort**: 5 days -**Impact**: High (reduces admin time by 50%) - -### Phase 2: Process Improvements (1 week) -4. **Permission Request Workflow** (3-4 days) - Self-service -5. **Permission Groups/Roles** (2 days) - Standardization - -**Total effort**: 5-6 days -**Impact**: Medium (better user experience) - -### Phase 3: Security & Compliance (1 week) -6. **Permission Monitoring & Alerts** (2-3 days) - Security -7. **Audit log enhancements** (2 days) - Compliance -8. **Permission review workflow** (2 days) - Periodic access review - -**Total effort**: 6-7 days -**Impact**: Medium (security & compliance) - ---- - -## API Reference - -### Grant Permission -```python -POST /api/v1/permissions -{ - "user_id": "alice", - "account_id": "acc123", - "permission_type": "submit_expense", - "expires_at": "2025-12-31T23:59:59", - "notes": "Food coordinator Q4" -} -``` - -### Get User Permissions -```python -GET /api/v1/permissions/user/{user_id} -GET /api/v1/permissions/user/{user_id}?type=submit_expense -``` - -### Get Account Permissions -```python -GET /api/v1/permissions/account/{account_id} -``` - -### Revoke Permission -```python -DELETE /api/v1/permissions/{permission_id} -``` - -### Check Permission (with inheritance) -```python -GET /api/v1/permissions/check?user_id=alice&account=Expenses:Food:Groceries&type=submit_expense -``` - ---- - -## Database Schema - -```sql -CREATE TABLE account_permissions ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - account_id TEXT NOT NULL, - permission_type TEXT NOT NULL, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL, - expires_at TIMESTAMP, - notes TEXT, - - FOREIGN KEY (account_id) REFERENCES castle_accounts (id) -); - -CREATE INDEX idx_account_permissions_user_id ON account_permissions (user_id); -CREATE INDEX idx_account_permissions_account_id ON account_permissions (account_id); -CREATE INDEX idx_account_permissions_expires_at ON account_permissions (expires_at); -``` - ---- - -## Security Considerations - -### 1. Permission Escalation Prevention - -**Risk:** User with MANAGE on child account tries to grant permissions on parent - -**Mitigation:** -```python -async def create_account_permission(data, granted_by): - # Check granter has MANAGE permission on account (or parent) - granter_permissions = await get_user_permissions_with_inheritance( - granted_by, account.name, PermissionType.MANAGE - ) - if not granter_permissions: - raise HTTPException(403, "You don't have permission to grant access to this account") -``` - -### 2. Cache Timing Attacks - -**Risk:** Stale cache shows old permissions after revocation - -**Mitigation:** -- Conservative 1-minute TTL -- Explicit cache invalidation on writes -- Admin can force cache clear if needed - -### 3. Expired Permission Cleanup - -**Current:** Expired permissions filtered at query time but remain in DB - -**Improvement:** Add background job to purge old permissions -```python -async def cleanup_expired_permissions(): - """Run daily to remove expired permissions""" - await db.execute( - "DELETE FROM account_permissions WHERE expires_at < NOW() - INTERVAL '30 days'" - ) -``` - ---- - -## Troubleshooting - -### Permission Denied Despite Valid Permission - -**Possible causes:** -1. Cache not invalidated after grant -2. Permission expired -3. Checking wrong account name (case sensitive) -4. Account ID mismatch - -**Solution:** -```python -# Clear cache and re-check -permission_cache._values.clear() - -# Verify permission exists -perms = await get_user_permissions(user_id) -logger.info(f"User {user_id} permissions: {perms}") - -# Check with inheritance -inherited = await get_user_permissions_with_inheritance(user_id, account_name, perm_type) -logger.info(f"Inherited permissions: {inherited}") -``` - -### Performance Issues - -**Symptom:** Slow permission checks - -**Causes:** -1. Cache not working -2. Too many permissions per user -3. Deep hierarchy causing many account lookups - -**Solution:** -```python -# Monitor cache hit rate -hits = len([v for v in permission_cache._values.values() if v is not None]) -logger.info(f"Permission cache: {hits} entries") - -# Optimize with account cache (implemented separately) -# Use account_cache to reduce DB queries for account lookups -``` - ---- - -## Testing Permissions - -### Unit Tests -```python -async def test_permission_inheritance(): - """Test that permission on parent grants access to child""" - # Grant on parent - await create_account_permission( - user="alice", - account="Expenses:Food", - permission_type=PermissionType.SUBMIT_EXPENSE - ) - - # Check child access - perms = await get_user_permissions_with_inheritance( - "alice", - "Expenses:Food:Groceries", - PermissionType.SUBMIT_EXPENSE - ) - - assert len(perms) == 1 - assert perms[0][1] == "Expenses:Food" # Inherited from parent - -async def test_permission_expiration(): - """Test that expired permissions are filtered""" - # Create expired permission - await create_account_permission( - user="bob", - account="acc123", - permission_type=PermissionType.READ, - expires_at=datetime.now() - timedelta(days=1) # Expired yesterday - ) - - # Should not be returned - perms = await get_user_permissions("bob") - assert len(perms) == 0 -``` - -### Integration Tests -```python -async def test_expense_submission_with_permission(): - """Test full flow: grant permission → submit expense""" - # 1. Grant permission - await create_account_permission(user, account, PermissionType.SUBMIT_EXPENSE) - - # 2. Submit expense - response = await api_create_expense_entry(ExpenseEntry(...)) - - # 3. Verify success - assert response.status_code == 200 - -async def test_expense_submission_without_permission(): - """Test that expense submission fails without permission""" - # Try to submit without permission - with pytest.raises(HTTPException) as exc: - await api_create_expense_entry(ExpenseEntry(...)) - - assert exc.value.status_code == 403 -``` - ---- - -## Summary - -The Castle permissions system is **well-designed** with strong features: -- Hierarchical inheritance reduces admin burden -- Caching provides good performance -- Expiration and audit trail support compliance -- Type-safe enums prevent errors - -**Recommended next steps:** -1. Implement **bulk permission management** (quick win) -2. Add **permission analytics dashboard** (visibility) -3. Consider **permission request workflow** (self-service) -4. Monitor cache performance and security events - -The system is production-ready and scales well for small-to-medium deployments. For larger deployments (1000+ users), consider implementing the permission groups/roles feature for easier management. - ---- - -**Document Version**: 1.0 -**Last Updated**: November 10, 2025 -**Status**: Complete + Improvement Recommendations diff --git a/docs/PHASE3_COMPLETE.md b/docs/PHASE3_COMPLETE.md index 1a3dbb6..bce9a76 100644 --- a/docs/PHASE3_COMPLETE.md +++ b/docs/PHASE3_COMPLETE.md @@ -276,8 +276,8 @@ balance = BalanceCalculator.calculate_account_balance( # Build inventory from entry lines entry_lines = [ - {"amount": 100000, "metadata": '{"fiat_currency": "EUR", "fiat_amount": "50.00"}'}, # Positive = debit - {"amount": -50000, "metadata": "{}"} # Negative = credit + {"debit": 100000, "credit": 0, "metadata": '{"fiat_currency": "EUR", "fiat_amount": "50.00"}'}, + {"debit": 0, "credit": 50000, "metadata": "{}"} ] inventory = BalanceCalculator.build_inventory_from_entry_lines( @@ -306,8 +306,8 @@ entry = { } entry_lines = [ - {"account_id": "acc1", "amount": 100000}, # Positive = debit - {"account_id": "acc2", "amount": -100000} # Negative = credit + {"account_id": "acc1", "debit": 100000, "credit": 0}, + {"account_id": "acc2", "debit": 0, "credit": 100000} ] try: diff --git a/docs/SATS-EQUIVALENT-METADATA.md b/docs/SATS-EQUIVALENT-METADATA.md deleted file mode 100644 index 48ab36c..0000000 --- a/docs/SATS-EQUIVALENT-METADATA.md +++ /dev/null @@ -1,386 +0,0 @@ -# SATS-Equivalent Metadata Field - -**Date**: 2025-01-12 -**Status**: Current Architecture -**Location**: Beancount posting metadata - ---- - -## Overview - -The `sats-equivalent` metadata field is Castle's solution for **dual-currency tracking** in a fiat-denominated ledger. It preserves Bitcoin (satoshi) amounts alongside fiat currency amounts without violating accounting principles or creating multi-currency complexity in position balances. - -### Quick Summary - -- **Purpose**: Track Bitcoin/Lightning amounts in a EUR-denominated ledger -- **Location**: Beancount posting metadata (not position amounts) -- **Format**: String containing absolute satoshi amount (e.g., `"337096"`) -- **Primary Use**: Calculate user balances in satoshis (Castle's primary currency) -- **Key Principle**: Satoshis are for reference; EUR is the actual transaction currency - ---- - -## The Problem: Dual-Currency Tracking - -Castle needs to track both: -1. **Fiat amounts** (EUR, USD) - The actual transaction currency -2. **Bitcoin amounts** (satoshis) - The Lightning Network settlement currency - -### Why Not Just Use SATS as Position Amounts? - -**Accounting Reality**: When a user pays €36.93 cash for groceries, the transaction is denominated in EUR, not Bitcoin. Recording it as Bitcoin would: -- ❌ Misrepresent the actual transaction -- ❌ Create exchange rate volatility issues -- ❌ Complicate traditional accounting reconciliation -- ❌ Make fiat-based reporting difficult - -**Castle's Philosophy**: Record transactions in their **actual currency**, with Bitcoin as supplementary data. - ---- - -## Architecture: EUR-Primary Format - -### Current Ledger Format - -```beancount -2025-11-10 * "Groceries (36.93 EUR)" #expense-entry - Expenses:Food:Supplies 36.93 EUR - sats-equivalent: "39669" - reference: "cash-payment-abc123" - Liabilities:Payable:User-5987ae95 -36.93 EUR - sats-equivalent: "39669" -``` - -**Key Components:** -- **Position Amount**: `36.93 EUR` - The actual transaction amount -- **Metadata**: `sats-equivalent: "39669"` - The Bitcoin equivalent at time of transaction -- **Sign**: The sign (debit/credit) is on the EUR amount; sats-equivalent is always absolute value - -### How It's Created - -In `views_api.py:839`: - -```python -# If fiat currency is provided, use EUR-based format -if fiat_currency and fiat_amount: - # EUR-based posting (current architecture) - posting_metadata["sats-equivalent"] = str(abs(line.amount)) - - # Apply the sign from line.amount to fiat_amount - signed_fiat_amount = fiat_amount if line.amount >= 0 else -fiat_amount - - posting = { - "account": account.name, - "amount": f"{signed_fiat_amount:.2f} {fiat_currency}", - "meta": posting_metadata if posting_metadata else None - } -``` - -**Critical Details:** -- `line.amount` is always in satoshis internally -- The sign (debit/credit) transfers to the fiat amount -- `sats-equivalent` stores the **absolute value** of the satoshi amount -- Sign interpretation depends on account type (Asset/Liability/etc.) - ---- - -## Usage: Balance Calculation - -### Primary Use Case: User Balances - -Castle's core function is tracking **who owes whom** in satoshis. The `sats-equivalent` metadata enables this. - -**Flow** (`fava_client.py:220-248`): - -```python -# Parse posting amount (EUR/USD) -fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', amount_str) -if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - fiat_amount = Decimal(fiat_match.group(1)) - fiat_currency = fiat_match.group(2) - - # Track fiat balance - fiat_balances[fiat_currency] += fiat_amount - - # Extract SATS equivalent from metadata - posting_meta = posting.get("meta", {}) - sats_equiv = posting_meta.get("sats-equivalent") - if sats_equiv: - # Apply the sign from fiat_amount to sats_equiv - sats_amount = int(sats_equiv) if fiat_amount > 0 else -int(sats_equiv) - total_sats += sats_amount -``` - -**Sign Interpretation:** -- EUR amount is `36.93` (positive/debit) → sats is `+39669` -- EUR amount is `-36.93` (negative/credit) → sats is `-39669` - -### Secondary Use: Journal Entry Display - -When displaying transactions to users (`views_api.py:747-751`): - -```python -# Extract sats equivalent from metadata -posting_meta = first_posting.get("meta", {}) -sats_equiv = posting_meta.get("sats-equivalent") -if sats_equiv: - amount_sats = abs(int(sats_equiv)) -``` - -This allows the UI to show both EUR and SATS amounts for each transaction. - ---- - -## Why Metadata Instead of Positions? - -### The BQL Limitation - -Beancount Query Language (BQL) **cannot access metadata**. This means: - -```sql --- ✅ This works (queries position amounts): -SELECT account, sum(position) WHERE account ~ 'User-5987ae95' --- Returns: EUR positions (not useful for satoshi balances) - --- ❌ This is NOT possible: -SELECT account, sum(meta["sats-equivalent"]) WHERE account ~ 'User-5987ae95' --- Error: BQL cannot access metadata -``` - -### Why Castle Accepts This Trade-off - -**Performance Analysis** (see `docs/BQL-BALANCE-QUERIES.md`): -1. **Caching solves the bottleneck**: 60-80% performance improvement from caching account/permission lookups -2. **Iteration is necessary anyway**: Even with BQL, we'd need to iterate postings to access metadata -3. **Manual aggregation is fast**: The actual summation is not the bottleneck -4. **Database queries are the bottleneck**: Solved by Phase 1 caching, not BQL - -**Architectural Correctness > Query Performance**: -- ✅ Transactions recorded in their actual currency -- ✅ No artificial multi-currency positions -- ✅ Clean accounting reconciliation -- ✅ Exchange rate changes don't affect historical records - ---- - -## Alternative Considered: Price Notation - -### Price Notation Format (Not Implemented) - -```beancount -2025-11-10 * "Groceries" - Expenses:Food -360.00 EUR @@ 337096 SATS - Liabilities:Payable:User-abc 360.00 EUR @@ 337096 SATS -``` - -**Pros:** -- ✅ BQL can query prices (enables BQL aggregation) -- ✅ Standard Beancount syntax -- ✅ SATS trackable via price database - -**Cons:** -- ❌ Semantically incorrect: `@@` means "total price paid", not "equivalent value" -- ❌ Implies currency conversion happened (it didn't) -- ❌ Confuses future readers about transaction nature -- ❌ Complicates Beancount's price database - -**Decision**: Metadata is more semantically correct for "reference value" than price notation. - -See `docs/BQL-PRICE-NOTATION-SOLUTION.md` for full analysis. - ---- - -## Data Flow Example - -### User Adds Expense - -**User Action**: "I paid €36.93 cash for groceries" - -**Castle's Internal Representation**: -```python -# User provides or Castle calculates: -fiat_amount = Decimal("36.93") # EUR -fiat_currency = "EUR" -amount_sats = 39669 # Calculated from exchange rate - -# Create journal entry line: -line = CreateEntryLine( - account_id=expense_account.id, - amount=amount_sats, # Internal: always satoshis - metadata={ - "fiat_currency": "EUR", - "fiat_amount": "36.93" - } -) -``` - -**Beancount Entry Created** (`views_api.py:835-849`): -```beancount -2025-11-10 * "Groceries (36.93 EUR)" #expense-entry - Expenses:Food:Supplies 36.93 EUR - sats-equivalent: "39669" - Liabilities:Payable:User-5987ae95 -36.93 EUR - sats-equivalent: "39669" -``` - -**Balance Calculation** (`fava_client.py:get_user_balance`): -```python -# Iterate all postings for user accounts -# For each posting: -# - Parse EUR amount: -36.93 EUR (credit to liability) -# - Extract sats-equivalent: "39669" -# - Apply sign: -36.93 is negative → sats = -39669 -# - Accumulate: user_balance_sats += -39669 - -# Result: negative balance = Castle owes user -``` - -**User Balance Response**: -```json -{ - "user_id": "5987ae95", - "balance": -39669, // Castle owes user 39,669 sats - "fiat_balances": { - "EUR": "-36.93" // Castle owes user €36.93 - } -} -``` - ---- - -## Implementation Details - -### Where It's Set - -**Primary Location**: `views_api.py:835-849` (Creating journal entries) - -All EUR-based postings get `sats-equivalent` metadata: -- Expense entries (user adds liability) -- Receivable entries (admin records what user owes) -- Revenue entries (direct income) -- Payment entries (settling balances) - -### Where It's Read - -**Primary Location**: `fava_client.py:239-247` (Balance calculation) - -Used in: -1. `get_user_balance()` - Calculate individual user balance -2. `get_all_user_balances()` - Calculate all user balances -3. `get_journal_entries()` - Display transaction amounts - -### Data Type and Format - -- **Type**: String (Beancount metadata values must be strings or numbers) -- **Format**: Absolute value, no sign, no decimal point -- **Examples**: - - ✅ `"39669"` (correct) - - ✅ `"1000000"` (1M sats) - - ❌ `"-39669"` (incorrect: sign goes on EUR amount) - - ❌ `"396.69"` (incorrect: satoshis are integers) - ---- - -## Key Principles - -### 1. Record in Transaction Currency - -```beancount -# ✅ CORRECT: User paid EUR, record in EUR -Expenses:Food 36.93 EUR - sats-equivalent: "39669" - -# ❌ WRONG: Recording Bitcoin when user paid cash -Expenses:Food 39669 SATS {36.93 EUR} -``` - -### 2. Preserve Historical Values - -The `sats-equivalent` is the **exact satoshi amount at transaction time**. It does NOT change when exchange rates change. - -**Example:** -- 2025-11-10: User pays €36.93 → 39,669 sats (rate: 1074.19 sats/EUR) -- 2025-11-15: Exchange rate changes to 1100 sats/EUR -- **Metadata stays**: `sats-equivalent: "39669"` ✅ -- **If we used current rate**: Would become 40,623 sats ❌ - -### 3. Separate Fiat and Sats Balances - -Castle tracks TWO independent balances: -- **Satoshi balance**: Sum of `sats-equivalent` metadata (primary) -- **Fiat balances**: Sum of EUR/USD position amounts (secondary) - -These are calculated independently and don't cross-convert. - -### 4. Absolute Values in Metadata - -The sign (debit/credit) lives on the position amount, NOT the metadata. - -```beancount -# Debit (expense increases): -Expenses:Food 36.93 EUR # Positive - sats-equivalent: "39669" # Absolute value - -# Credit (liability increases): -Liabilities:Payable -36.93 EUR # Negative - sats-equivalent: "39669" # Same absolute value -``` - ---- - -## Migration Path - -### Future: If We Change to SATS-Primary Format - -**Hypothetical future format:** -```beancount -; SATS as position, EUR as cost: -2025-11-10 * "Groceries" - Expenses:Food 39669 SATS {36.93 EUR} - Liabilities:Payable:User-abc -39669 SATS {36.93 EUR} -``` - -**Benefits:** -- ✅ BQL can query SATS directly -- ✅ No metadata parsing needed -- ✅ Standard Beancount cost accounting - -**Migration Script** (conceptual): -```python -# Read all postings with sats-equivalent metadata -# For each posting: -# - Extract sats from metadata -# - Extract EUR from position -# - Rewrite as: " SATS { EUR}" -``` - -**Decision**: Not implementing now because: -1. Current architecture is semantically correct -2. Performance is acceptable with caching -3. Migration would break existing tooling -4. EUR-primary aligns with accounting reality - ---- - -## Related Documentation - -- `docs/BQL-BALANCE-QUERIES.md` - Why BQL can't query metadata and performance analysis -- `docs/BQL-PRICE-NOTATION-SOLUTION.md` - Alternative using price notation (not implemented) -- `beancount_format.py` - Functions that create entries with sats-equivalent metadata -- `fava_client.py:get_user_balance()` - How metadata is parsed for balance calculation - ---- - -## Technical Summary - -**Field**: `sats-equivalent` -**Type**: Metadata (string) -**Location**: Beancount posting metadata -**Format**: Absolute satoshi amount as string (e.g., `"39669"`) -**Purpose**: Track Bitcoin equivalent of fiat transactions -**Primary Use**: Calculate user satoshi balances -**Sign Handling**: Inherits from position amount (EUR/USD) -**Queryable via BQL**: ❌ No (BQL cannot access metadata) -**Performance**: ✅ Acceptable with caching (60-80% improvement) -**Architectural Status**: ✅ Current production format -**Future Migration**: Possible to SATS-primary if needed diff --git a/docs/UI-IMPROVEMENTS-PLAN.md b/docs/UI-IMPROVEMENTS-PLAN.md deleted file mode 100644 index 97bc9d3..0000000 --- a/docs/UI-IMPROVEMENTS-PLAN.md +++ /dev/null @@ -1,734 +0,0 @@ -# Castle UI Improvements Plan - -**Date**: November 10, 2025 -**Status**: 📋 **Planning Document** -**Related**: ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md, PERMISSIONS-SYSTEM.md - ---- - -## Overview - -Enhance the Castle permissions UI to showcase new bulk permission management and account sync features, making admin tasks faster and more intuitive. - ---- - -## Current UI Assessment - -**What's Good:** -- ✅ Clean Quasar/Vue.js structure -- ✅ Three views: By User, By Account, Equity -- ✅ Basic grant/revoke functionality -- ✅ Good visual design with icons and colors -- ✅ Admin-only protection - -**What's Missing:** -- ❌ No bulk operations -- ❌ No permission analytics dashboard -- ❌ No permission templates/copying -- ❌ No account sync UI -- ❌ No user offboarding workflow -- ❌ No expiring permissions alerts - ---- - -## Proposed Enhancements - -### 1. Add "Analytics" Tab - -**Purpose**: Give admins visibility into permission usage - -**Features:** -- Total permissions count (by type) -- Permissions expiring soon (7 days) -- Most-permissioned accounts (top 10) -- Users with/without permissions -- Permission grant timeline chart - -**UI Mockup:** -``` -┌─────────────────────────────────────────────┐ -│ 📊 Permission Analytics │ -├─────────────────────────────────────────────┤ -│ │ -│ ┌──────────────┐ ┌──────────────┐ │ -│ │ Total │ │ Expiring │ │ -│ │ 150 │ │ 5 (7 days) │ │ -│ │ Permissions │ │ │ │ -│ └──────────────┘ └──────────────┘ │ -│ │ -│ Permission Distribution │ -│ ┌────────────────────────────────────┐ │ -│ │ READ ██████ 50 (33%) │ │ -│ │ SUBMIT_EXPENSE ████████ 80 (53%) │ │ -│ │ MANAGE ████ 20 (13%) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ ⚠️ Expiring Soon │ -│ ┌────────────────────────────────────┐ │ -│ │ alice on Expenses:Food (3 days) │ │ -│ │ bob on Income:Services (5 days) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ Top Accounts by Permissions │ -│ ┌────────────────────────────────────┐ │ -│ │ 1. Expenses:Food (25 permissions) │ │ -│ │ 2. Expenses:Accommodation (18) │ │ -│ │ 3. Income:Services (12) │ │ -│ └────────────────────────────────────┘ │ -└─────────────────────────────────────────────┘ -``` - -**Implementation:** -- New API endpoint: `GET /api/v1/admin/permissions/analytics` -- Client-side stats display with Quasar charts -- Auto-refresh every 30 seconds - ---- - -### 2. Bulk Permission Operations Menu - -**Purpose**: Enable admins to perform bulk operations efficiently - -**Features:** -- Bulk Grant (multiple users) -- Copy Permissions (template from user) -- Offboard User (revoke all) -- Close Account (revoke all on account) - -**UI Mockup:** -``` -┌─────────────────────────────────────────────┐ -│ 🔐 Permission Management │ -│ │ -│ [Grant Permission ▼] [Bulk Operations ▼] │ -│ │ -│ ┌──────────────────────────────────┐ │ -│ │ • Bulk Grant to Multiple Users │ │ -│ │ • Copy Permissions from User │ │ -│ │ • Offboard User (Revoke All) │ │ -│ │ • Close Account (Revoke All) │ │ -│ │ • Sync Accounts from Beancount │ │ -│ └──────────────────────────────────┘ │ -└─────────────────────────────────────────────┘ -``` - ---- - -### 3. Bulk Grant Dialog - -**UI Mockup:** -``` -┌───────────────────────────────────────────┐ -│ 👥 Bulk Grant Permission │ -├───────────────────────────────────────────┤ -│ │ -│ Select Users * │ -│ ┌────────────────────────────────────┐ │ -│ │ 🔍 Search users... │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ Selected: alice, bob, charlie (3 users) │ -│ │ -│ Select Account * │ -│ ┌────────────────────────────────────┐ │ -│ │ Expenses:Food │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ Permission Type * │ -│ ┌────────────────────────────────────┐ │ -│ │ Submit Expense │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ Expires (Optional) │ -│ [2025-12-31 23:59:59] │ -│ │ -│ Notes (Optional) │ -│ [Q4 2025 food team members] │ -│ │ -│ ℹ️ This will grant SUBMIT_EXPENSE │ -│ permission to 3 users on │ -│ Expenses:Food │ -│ │ -│ [Cancel] [Grant to 3 Users] │ -└───────────────────────────────────────────┘ -``` - -**Features:** -- Multi-select user dropdown -- Preview of operation before confirm -- Shows estimated time savings - ---- - -### 4. Copy Permissions Dialog - -**UI Mockup:** -``` -┌───────────────────────────────────────────┐ -│ 📋 Copy Permissions │ -├───────────────────────────────────────────┤ -│ │ -│ Copy From (Template User) * │ -│ ┌────────────────────────────────────┐ │ -│ │ alice (Experienced Coordinator) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ 📊 alice has 5 permissions: │ -│ • Expenses:Food (Submit Expense) │ -│ • Expenses:Food:Groceries (Submit) │ -│ • Income:Services (Read) │ -│ • Assets:Cash (Read) │ -│ • Expenses:Utilities (Submit) │ -│ │ -│ Copy To (New User) * │ -│ ┌────────────────────────────────────┐ │ -│ │ bob (New Hire) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ Filter by Permission Type (Optional) │ -│ ☑ Submit Expense ☐ Read ☐ Manage │ -│ │ -│ Notes │ -│ [Copied from Alice - new coordinator] │ -│ │ -│ ℹ️ This will copy 3 SUBMIT_EXPENSE │ -│ permissions from alice to bob │ -│ │ -│ [Cancel] [Copy Permissions] │ -└───────────────────────────────────────────┘ -``` - -**Features:** -- Shows source user's permissions -- Filter by permission type -- Preview before copying - ---- - -### 5. Offboard User Dialog - -**UI Mockup:** -``` -┌───────────────────────────────────────────┐ -│ 🚪 Offboard User │ -├───────────────────────────────────────────┤ -│ │ -│ Select User to Offboard * │ -│ ┌────────────────────────────────────┐ │ -│ │ charlie (Departed Employee) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ ⚠️ Current Permissions (8 total): │ -│ ┌────────────────────────────────────┐ │ -│ │ • Expenses:Food (Submit Expense) │ │ -│ │ • Expenses:Utilities (Submit) │ │ -│ │ • Income:Services (Read) │ │ -│ │ • Assets:Cash (Read) │ │ -│ │ • Expenses:Accommodation (Submit) │ │ -│ │ • ... 3 more │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ ⚠️ Warning: This will revoke ALL │ -│ permissions for this user. They will │ -│ immediately lose access to Castle. │ -│ │ -│ Reason for Offboarding │ -│ [Employee departure - last day] │ -│ │ -│ [Cancel] [Revoke All (8)] │ -└───────────────────────────────────────────┘ -``` - -**Features:** -- Shows all current permissions -- Requires confirmation -- Logs reason for audit - ---- - -### 6. Account Sync UI - -**Location**: Admin Settings or Bulk Operations menu - -**UI Mockup:** -``` -┌───────────────────────────────────────────┐ -│ 🔄 Sync Accounts from Beancount │ -├───────────────────────────────────────────┤ -│ │ -│ Sync accounts from your Beancount ledger │ -│ to Castle database for permission mgmt. │ -│ │ -│ Last Sync: 2 hours ago │ -│ Status: ✅ Up to date │ -│ │ -│ Accounts in Beancount: 150 │ -│ Accounts in Castle DB: 150 │ -│ │ -│ Options: │ -│ ☐ Force full sync (re-check all) │ -│ │ -│ [Sync Now] │ -│ │ -│ Recent Sync History: │ -│ ┌────────────────────────────────────┐ │ -│ │ Nov 10, 2:00 PM - Added 2 accounts │ │ -│ │ Nov 10, 12:00 PM - Up to date │ │ -│ │ Nov 10, 10:00 AM - Added 1 account │ │ -│ └────────────────────────────────────┘ │ -└───────────────────────────────────────────┘ -``` - -**Features:** -- Shows sync status -- Last sync timestamp -- Account counts -- Sync history - ---- - -### 7. Expiring Permissions Alert - -**Location**: Top of permissions page (if any expiring soon) - -**UI Mockup:** -``` -┌─────────────────────────────────────────────┐ -│ ⚠️ 5 Permissions Expiring Soon (Next 7 Days)│ -├─────────────────────────────────────────────┤ -│ • alice on Expenses:Food (3 days) │ -│ • bob on Income:Services (5 days) │ -│ • charlie on Assets:Cash (7 days) │ -│ │ -│ [View All] [Extend Expiration] [Dismiss] │ -└─────────────────────────────────────────────┘ -``` - -**Features:** -- Prominent alert banner -- Shows expiring in next 7 days -- Quick actions to extend - ---- - -### 8. Permission Templates (Future) - -**Concept**: Pre-defined permission sets for common roles - -**UI Mockup:** -``` -┌───────────────────────────────────────────┐ -│ 📝 Apply Permission Template │ -├───────────────────────────────────────────┤ -│ │ -│ Select User * │ -│ [bob] │ -│ │ -│ Select Template * │ -│ ┌────────────────────────────────────┐ │ -│ │ Food Coordinator (5 permissions) │ │ -│ │ • Expenses:Food (Submit) │ │ -│ │ • Expenses:Food:* (Submit) │ │ -│ │ • Income:Services (Read) │ │ -│ │ │ │ -│ │ Accommodation Manager (8 perms) │ │ -│ │ Finance Admin (15 perms) │ │ -│ │ Read-Only Auditor (25 perms) │ │ -│ └────────────────────────────────────┘ │ -│ │ -│ [Cancel] [Apply Template] │ -└───────────────────────────────────────────┘ -``` - ---- - -## Implementation Priority - -### Phase 1: Quick Wins (This Week) -**Effort**: 2-3 days - -1. **Analytics Tab** (1 day) - - Add new tab to permissions.html - - Call analytics API endpoint - - Display stats with Quasar components - -2. **Bulk Grant Dialog** (1 day) - - Add multi-select user dropdown - - Call bulk grant API - - Show success/failure results - -3. **Account Sync Button** (0.5 days) - - Add sync button to admin area - - Call sync API - - Show progress and results - -**Impact**: Immediate productivity boost for admins - ---- - -### Phase 2: Bulk Operations (Next Week) -**Effort**: 2-3 days - -4. **Copy Permissions Dialog** (1 day) - - Template selection UI - - Preview permissions - - Copy operation - -5. **Offboard User Dialog** (1 day) - - User selection with permission preview - - Confirmation with reason logging - - Bulk revoke operation - -6. **Expiring Permissions Alert** (0.5 days) - - Alert banner component - - Query expiring permissions - - Quick actions - -**Impact**: Major time savings for common workflows - ---- - -### Phase 3: Polish (Later) -**Effort**: 2-3 days - -7. **Permission Templates** (2 days) - - Template management UI - - Template CRUD operations - - Apply template workflow - -8. **Advanced Analytics** (1 day) - - Charts and graphs - - Permission history timeline - - Usage patterns - -**Impact**: Long-term ease of use - ---- - -## Technical Implementation - -### New API Endpoints Needed - -```javascript -// Analytics -GET /api/v1/admin/permissions/analytics - -// Bulk Operations -POST /api/v1/admin/permissions/bulk-grant -{ - user_ids: ["alice", "bob", "charlie"], - account_id: "acc123", - permission_type: "submit_expense", - expires_at: "2025-12-31T23:59:59", - notes: "Q4 team" -} - -POST /api/v1/admin/permissions/copy -{ - from_user_id: "alice", - to_user_id: "bob", - permission_types: ["submit_expense"], - notes: "New coordinator" -} - -DELETE /api/v1/admin/permissions/user/{user_id} - -DELETE /api/v1/admin/permissions/account/{account_id} - -// Account Sync -POST /api/v1/admin/sync-accounts -{ - force_full_sync: false -} - -GET /api/v1/admin/sync-accounts/status -``` - -### Vue.js Component Structure - -``` -permissions.html -├── Analytics Tab (new) -│ ├── Stats Cards -│ ├── Distribution Chart -│ ├── Expiring Soon List -│ └── Top Accounts List -│ -├── By User Tab (existing) -│ └── Enhanced with bulk operations -│ -├── By Account Tab (existing) -│ └── Enhanced with bulk operations -│ -├── Equity Tab (existing) -│ -└── Dialogs - ├── Bulk Grant Dialog (new) - ├── Copy Permissions Dialog (new) - ├── Offboard User Dialog (new) - ├── Account Sync Dialog (new) - ├── Grant Permission Dialog (existing) - └── Revoke Confirmation Dialog (existing) -``` - -### State Management - -```javascript -// Add to Vue app data -{ - // Analytics - analytics: { - total: 0, - byType: {}, - expiringSoon: [], - topAccounts: [] - }, - - // Bulk Operations - bulkGrantForm: { - user_ids: [], - account_id: null, - permission_type: null, - expires_at: null, - notes: '' - }, - - copyPermissionsForm: { - from_user_id: null, - to_user_id: null, - permission_types: [], - notes: '' - }, - - offboardForm: { - user_id: null, - reason: '' - }, - - // Account Sync - syncStatus: { - lastSync: null, - beancountAccounts: 0, - castleAccounts: 0, - status: 'idle' - } -} -``` - ---- - -## User Experience Flow - -### Onboarding New Team Member (Before vs After) - -**Before** (10 minutes): -1. Open permissions page -2. Click "Grant Permission" 5 times -3. Fill form each time (user, account, type) -4. Click grant, repeat -5. Hope you didn't forget any - -**After** (1 minute): -1. Click "Bulk Operations" → "Copy Permissions" -2. Select template user → Select new user -3. Click "Copy" -4. Done! ✨ - -**Time Saved**: 90% - ---- - -### Quarterly Access Review (Before vs After) - -**Before** (2 hours): -1. Export permissions to spreadsheet -2. Manually review each one -3. Delete expired individually -4. Update expiration dates one by one - -**After** (5 minutes): -1. Click "Analytics" tab -2. See "5 Expiring Soon" alert -3. Review list, click "Extend" on relevant ones -4. Done! ✨ - -**Time Saved**: 96% - ---- - -## Testing Plan - -### UI Testing - -```javascript -// Test bulk grant -async function testBulkGrant() { - // 1. Open bulk grant dialog - // 2. Select 3 users - // 3. Select account - // 4. Select permission type - // 5. Click grant - // 6. Verify success message - // 7. Verify permissions appear in UI -} - -// Test copy permissions -async function testCopyPermissions() { - // 1. Open copy dialog - // 2. Select source user with 5 permissions - // 3. Select target user - // 4. Filter to SUBMIT_EXPENSE only - // 5. Verify preview shows 3 permissions - // 6. Click copy - // 7. Verify target user has 3 new permissions -} - -// Test analytics -async function testAnalytics() { - // 1. Switch to analytics tab - // 2. Verify stats load - // 3. Verify charts display - // 4. Verify expiring permissions show - // 5. Click on expiring permission - // 6. Verify details dialog opens -} -``` - -### Integration Testing - -```python -# Test full workflow -async def test_onboarding_workflow(): - # 1. Admin syncs accounts - sync_result = await api.post("/admin/sync-accounts") - assert sync_result["accounts_added"] >= 0 - - # 2. Admin copies permissions from template user - copy_result = await api.post("/admin/permissions/copy", { - "from_user_id": "template", - "to_user_id": "new_user" - }) - assert copy_result["copied"] > 0 - - # 3. Verify new user has permissions in UI - perms = await api.get(f"/admin/permissions?user_id=new_user") - assert len(perms) > 0 - - # 4. Check analytics reflect new permissions - analytics = await api.get("/admin/permissions/analytics") - assert analytics["total_permissions"] increased -``` - ---- - -## Accessibility - -- ✅ Keyboard navigation support -- ✅ Screen reader friendly labels -- ✅ Color contrast compliance (WCAG AA) -- ✅ Focus indicators -- ✅ ARIA labels on interactive elements - ---- - -## Mobile Responsiveness - -- ✅ Analytics cards stack vertically on mobile -- ✅ Dialogs are full-screen on small devices -- ✅ Touch-friendly button sizes -- ✅ Swipe gestures for tabs - ---- - -## Error Handling - -**Bulk Grant Fails:** -``` -⚠️ Bulk Grant Results -✅ Granted to 3 users -❌ Failed for 2 users: - • bob: Already has permission - • charlie: Account not found - -[View Details] [Retry Failed] [Dismiss] -``` - -**Account Sync Fails:** -``` -❌ Account Sync Failed -Could not connect to Beancount service. - -Error: Connection timeout after 10s - -[Retry] [Check Settings] [Dismiss] -``` - ---- - -## Performance Considerations - -- **Pagination**: Load permissions in batches of 50 -- **Lazy Loading**: Load analytics only when tab is viewed -- **Debouncing**: Debounce search inputs (300ms) -- **Caching**: Cache analytics for 30 seconds -- **Optimistic UI**: Show loading state immediately - ---- - -## Security Considerations - -- ✅ All bulk operations require admin key -- ✅ Confirmation dialogs for destructive actions -- ✅ Audit log all bulk operations -- ✅ Rate limiting on API endpoints -- ✅ CSRF protection on forms - ---- - -## Documentation - -**User Guide** (to create): -1. How to bulk grant permissions -2. How to copy permissions (templating) -3. How to offboard a user -4. How to sync accounts -5. How to use analytics dashboard - -**Admin Guide** (to create): -1. When to use bulk operations -2. Best practices for permission templates -3. How to monitor permission usage -4. Troubleshooting sync issues - ---- - -## Success Metrics - -**Measure after deployment:** -- Time to onboard new user: 10min → 1min -- Time for access review: 2hr → 5min -- Admin satisfaction score: 6/10 → 9/10 -- Support tickets for permissions: -70% -- Permissions granted per month: +40% - ---- - -## Summary - -This UI improvement plan focuses on: - -1. **Quick Wins**: Analytics and bulk grant (2-3 days) -2. **Bulk Operations**: Copy, offboard, sync (2-3 days) -3. **Polish**: Templates and advanced features (later) - -**Total Time**: ~5-6 days for Phase 1 & 2 -**Impact**: 50-70% reduction in admin time -**ROI**: Immediate productivity boost - -The enhancements leverage the new backend features we built (account sync, bulk permission management) and make them accessible through an intuitive UI, significantly improving the admin experience. - ---- - -**Document Version**: 1.0 -**Last Updated**: November 10, 2025 -**Status**: Ready for Implementation diff --git a/fava_client.py b/fava_client.py deleted file mode 100644 index 4cde16b..0000000 --- a/fava_client.py +++ /dev/null @@ -1,1231 +0,0 @@ -""" -Fava API client for Castle. - -This module provides an async HTTP client for interacting with Fava's JSON API. -All accounting logic is delegated to Fava/Beancount. - -Fava provides a REST API for: -- Adding transactions (PUT /api/add_entries) -- Adding accounts via Open directives (PUT /api/add_entries) -- Querying balances (GET /api/query) -- Balance sheets (GET /api/balance_sheet) -- Account reports (GET /api/account_report) -- Updating/deleting entries (PUT/DELETE /api/source_slice) - -See: https://github.com/beancount/fava/blob/main/src/fava/json_api.py -""" - -import httpx -from typing import Any, Dict, List, Optional -from decimal import Decimal -from datetime import date, datetime -from loguru import logger - - -class FavaClient: - """ - Async client for Fava REST API. - - Fava runs as a separate web service and provides a JSON API - for adding entries and querying ledger data. - - All accounting calculations are performed by Beancount via Fava. - """ - - def __init__(self, fava_url: str, ledger_slug: str, timeout: float = 10.0): - """ - Initialize Fava client. - - Args: - fava_url: Base URL of Fava server (e.g., http://localhost:3333) - ledger_slug: URL-safe ledger identifier (e.g., castle-accounting) - timeout: Request timeout in seconds - """ - self.fava_url = fava_url.rstrip('/') - self.ledger_slug = ledger_slug - self.base_url = f"{self.fava_url}/{self.ledger_slug}/api" - self.timeout = timeout - - async def add_entry(self, entry: Dict[str, Any]) -> Dict[str, Any]: - """ - Submit a new journal entry to Fava. - - Args: - entry: Beancount entry dict (format per Fava API spec) - Must include: - - t: "Transaction" (required by Fava) - - date: "YYYY-MM-DD" - - flag: "*" (cleared) or "!" (pending) - - narration: str - - postings: list of posting dicts - - payee: str (empty string, not None) - - tags: list of str - - links: list of str - - meta: dict - - Returns: - Response from Fava ({"data": "Stored 1 entries.", "mtime": "..."}) - - Raises: - httpx.HTTPStatusError: If Fava returns an error - httpx.RequestError: If connection fails - - Example: - entry = { - "t": "Transaction", - "date": "2025-01-15", - "flag": "*", - "payee": "Store", - "narration": "Purchase", - "postings": [ - {"account": "Expenses:Food", "amount": "50.00 EUR"}, - {"account": "Assets:Cash", "amount": "-50.00 EUR"} - ], - "tags": [], - "links": [], - "meta": {"user_id": "abc123"} - } - result = await fava_client.add_entry(entry) - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.put( - f"{self.base_url}/add_entries", - json={"entries": [entry]}, - headers={"Content-Type": "application/json"} - ) - response.raise_for_status() - result = response.json() - - logger.info(f"Added entry to Fava: {result.get('data', 'Unknown')}") - return result - - except httpx.HTTPStatusError as e: - logger.error(f"Fava HTTP error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def get_account_balance(self, account_name: str) -> Dict[str, Any]: - """ - Get balance for a specific account (excluding pending transactions). - - Args: - account_name: Full account name (e.g., "Assets:Receivable:User-abc123") - - Returns: - Dict with: - - sats: int (balance in satoshis) - - positions: dict (currency → amount with cost basis) - - Note: - Excludes pending transactions (flag='!') from balance calculation. - Only cleared/completed transactions (flag='*') are included. - - Example: - balance = await fava_client.get_account_balance("Assets:Receivable:User-abc") - # Returns: { - # "sats": 200000, - # "positions": {"SATS": {"{100.00 EUR}": 200000}} - # } - """ - query = f"SELECT sum(position) WHERE account = '{account_name}' AND flag != '!'" - - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get( - f"{self.base_url}/query", - params={"query_string": query} - ) - response.raise_for_status() - data = response.json() - - if not data['data']['rows']: - return {"sats": 0, "positions": {}} - - # Fava returns: [[account, {"SATS": {cost: amount}}]] - positions = data['data']['rows'][0][1] if data['data']['rows'] else {} - - # Sum up all SATS positions - total_sats = 0 - if isinstance(positions, dict) and "SATS" in positions: - sats_positions = positions["SATS"] - if isinstance(sats_positions, dict): - # Sum all amounts (with different cost bases) - total_sats = sum(int(amount) for amount in sats_positions.values()) - elif isinstance(sats_positions, (int, float)): - # Simple number (no cost basis) - total_sats = int(sats_positions) - - return { - "sats": total_sats, - "positions": positions - } - - except httpx.HTTPStatusError as e: - logger.error(f"Fava query error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def get_user_balance(self, user_id: str) -> Dict[str, Any]: - """ - Get user's balance from castle's perspective. - - Aggregates: - - Liabilities:Payable:User-{user_id} (negative = castle owes user) - - Assets:Receivable:User-{user_id} (positive = user owes castle) - - Args: - user_id: User ID - - Returns: - { - "balance": int (sats, positive = user owes castle, negative = castle owes user), - "fiat_balances": {"EUR": Decimal("100.50")}, - "accounts": [list of account dicts with balances] - } - - Note: - Excludes pending transactions (flag='!') from balance calculation. - Only cleared/completed transactions (flag='*') are included. - """ - # Get all journal entries for this user - all_entries = await self.get_journal_entries() - - total_sats = 0 - fiat_balances = {} - accounts_dict = {} # Track balances per account - - for entry in all_entries: - # Skip non-transactions, pending (!), and voided - if entry.get("t") != "Transaction": - continue - if entry.get("flag") == "!": - continue - if "voided" in entry.get("tags", []): - continue - - # Process postings for this user - for posting in entry.get("postings", []): - account_name = posting.get("account", "") - - # Only process this user's accounts (account names use first 8 chars of user_id) - if f":User-{user_id[:8]}" not in account_name: - continue - if "Payable" not in account_name and "Receivable" not in account_name: - continue - - # Parse amount string: can be EUR, USD, or SATS - amount_str = posting.get("amount", "") - if not isinstance(amount_str, str) or not amount_str: - continue - - import re - # Try to extract EUR/USD amount first (new format) - fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', amount_str) - if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - # Direct EUR/USD amount (new approach) - fiat_amount = Decimal(fiat_match.group(1)) - fiat_currency = fiat_match.group(2) - - if fiat_currency not in fiat_balances: - fiat_balances[fiat_currency] = Decimal(0) - - fiat_balances[fiat_currency] += fiat_amount - - # Also track SATS equivalent from metadata if available - posting_meta = posting.get("meta", {}) - sats_equiv = posting_meta.get("sats-equivalent") - if sats_equiv: - sats_amount = int(sats_equiv) if fiat_amount > 0 else -int(sats_equiv) - total_sats += sats_amount - if account_name not in accounts_dict: - accounts_dict[account_name] = {"account": account_name, "sats": 0} - accounts_dict[account_name]["sats"] += sats_amount - - else: - # Old format: SATS with cost/price notation - extract SATS amount - sats_match = re.match(r'^(-?\d+)\s+SATS', amount_str) - if sats_match: - sats_amount = int(sats_match.group(1)) - total_sats += sats_amount - - # Track per account - if account_name not in accounts_dict: - accounts_dict[account_name] = {"account": account_name, "sats": 0} - accounts_dict[account_name]["sats"] += sats_amount - - # Try to extract fiat from metadata or cost syntax (backward compatibility) - posting_meta = posting.get("meta", {}) - fiat_amount_total_str = posting_meta.get("fiat-amount-total") - fiat_currency_meta = posting_meta.get("fiat-currency") - - if fiat_amount_total_str and fiat_currency_meta: - # Use exact total from metadata - fiat_total = Decimal(fiat_amount_total_str) - fiat_currency = fiat_currency_meta - - if fiat_currency not in fiat_balances: - fiat_balances[fiat_currency] = Decimal(0) - - # Apply the same sign as the SATS amount - if sats_match: - sats_amount_for_sign = int(sats_match.group(1)) - if sats_amount_for_sign < 0: - fiat_total = -fiat_total - - fiat_balances[fiat_currency] += fiat_total - - logger.info(f"User {user_id[:8]} balance: {total_sats} sats, fiat: {dict(fiat_balances)}") - return { - "balance": total_sats, - "fiat_balances": fiat_balances, - "accounts": list(accounts_dict.values()) - } - - async def get_all_user_balances(self) -> List[Dict[str, Any]]: - """ - Get balances for all users (admin view). - - Returns: - [ - { - "user_id": "abc123", - "balance": 100000, - "fiat_balances": {"EUR": Decimal("100.50")}, - "accounts": [...] - }, - ... - ] - - Note: - Excludes pending transactions (flag='!') and voided (tag #voided) from balance calculation. - Only cleared/completed transactions (flag='*') are included. - """ - # Get all journal entries and calculate balances from postings - all_entries = await self.get_journal_entries() - - # Group by user_id - user_data = {} - - for entry in all_entries: - # Skip non-transactions, pending (!), and voided - if entry.get("t") != "Transaction": - continue - if entry.get("flag") == "!": - continue - if "voided" in entry.get("tags", []): - continue - - # Process postings - for posting in entry.get("postings", []): - account_name = posting.get("account", "") - - # Only process user accounts (Payable or Receivable) - if ":User-" not in account_name: - continue - if "Payable" not in account_name and "Receivable" not in account_name: - continue - - # Extract user_id from account name - user_id = account_name.split(":User-")[1] - - if user_id not in user_data: - user_data[user_id] = { - "user_id": user_id, - "balance": 0, - "fiat_balances": {}, - "accounts": [] - } - - # Parse amount string: can be EUR/USD directly (new format) or "SATS {EUR}" (old format) - amount_str = posting.get("amount", "") - if not isinstance(amount_str, str) or not amount_str: - continue - - import re - # Try to extract EUR/USD amount first (new format) - fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', amount_str) - if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - # Direct EUR/USD amount (new approach) - fiat_amount = Decimal(fiat_match.group(1)) - fiat_currency = fiat_match.group(2) - - if fiat_currency not in user_data[user_id]["fiat_balances"]: - user_data[user_id]["fiat_balances"][fiat_currency] = Decimal(0) - - user_data[user_id]["fiat_balances"][fiat_currency] += fiat_amount - - # Also track SATS equivalent from metadata if available - posting_meta = posting.get("meta", {}) - sats_equiv = posting_meta.get("sats-equivalent") - if sats_equiv: - sats_amount = int(sats_equiv) if fiat_amount > 0 else -int(sats_equiv) - user_data[user_id]["balance"] += sats_amount - - else: - # Old format: SATS with cost/price notation - sats_match = re.match(r'^(-?\d+)\s+SATS', amount_str) - if sats_match: - sats_amount = int(sats_match.group(1)) - user_data[user_id]["balance"] += sats_amount - - # Extract fiat from cost syntax or metadata (backward compatibility) - posting_meta = posting.get("meta", {}) - fiat_amount_total_str = posting_meta.get("fiat-amount-total") - fiat_currency_meta = posting_meta.get("fiat-currency") - - if fiat_amount_total_str and fiat_currency_meta: - fiat_total = Decimal(fiat_amount_total_str) - fiat_currency = fiat_currency_meta - - if fiat_currency not in user_data[user_id]["fiat_balances"]: - user_data[user_id]["fiat_balances"][fiat_currency] = Decimal(0) - - # Apply the same sign as the SATS amount - if sats_match: - sats_amount_for_sign = int(sats_match.group(1)) - if sats_amount_for_sign < 0: - fiat_total = -fiat_total - - user_data[user_id]["fiat_balances"][fiat_currency] += fiat_total - - return list(user_data.values()) - - async def check_fava_health(self) -> bool: - """ - Check if Fava is running and accessible. - - Returns: - True if Fava responds, False otherwise - """ - try: - async with httpx.AsyncClient(timeout=2.0) as client: - response = await client.get( - f"{self.base_url}/changed" - ) - return response.status_code == 200 - except Exception as e: - logger.warning(f"Fava health check failed: {e}") - return False - - async def query_transactions( - self, - account_pattern: Optional[str] = None, - limit: int = 100, - include_pending: bool = True - ) -> List[Dict[str, Any]]: - """ - Query transactions from Fava/Beancount. - - Args: - account_pattern: Optional regex pattern to filter accounts (e.g., "User-abc123") - limit: Maximum number of transactions to return - include_pending: Include pending transactions (flag='!') - - Returns: - List of transaction dictionaries with date, description, postings, etc. - - Example: - # All transactions - txns = await fava.query_transactions() - - # User's transactions - txns = await fava.query_transactions(account_pattern="User-abc123") - - # Account transactions - txns = await fava.query_transactions(account_pattern="Assets:Receivable:User-abc") - """ - # Build Beancount query - if account_pattern: - query = f"SELECT * WHERE account ~ '{account_pattern}' ORDER BY date DESC LIMIT {limit}" - else: - query = f"SELECT * ORDER BY date DESC LIMIT {limit}" - - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get( - f"{self.base_url}/query", - params={"query_string": query} - ) - response.raise_for_status() - result = response.json() - - # Fava query API returns: {"data": {"rows": [...], "types": [...]}} - data = result.get("data", {}) - rows = data.get("rows", []) - types = data.get("types", []) - - # Build column name mapping - column_names = [t.get("name") for t in types] - - # Transform Fava's query result to transaction list - transactions = [] - for row in rows: - # Rows are arrays, convert to dict using column names - if isinstance(row, list) and len(row) == len(column_names): - txn = dict(zip(column_names, row)) - - # Filter by flag if needed - flag = txn.get("flag", "*") - if not include_pending and flag == "!": - continue - - transactions.append(txn) - elif isinstance(row, dict): - # Already a dict (shouldn't happen with BQL, but handle it) - flag = row.get("flag", "*") - if not include_pending and flag == "!": - continue - transactions.append(row) - - return transactions[:limit] - - except httpx.HTTPStatusError as e: - logger.error(f"Fava query error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def query_bql(self, query_string: str) -> Dict[str, Any]: - """ - Execute arbitrary Beancount Query Language (BQL) query. - - This is a general-purpose method for executing BQL queries against Fava/Beancount. - Use this for efficient aggregations, filtering, and data retrieval. - - ⚠️ LIMITATION: BQL can only query position amounts and transaction-level data. - It CANNOT access posting metadata (like 'sats-equivalent'). For Castle's current - ledger format where SATS are stored in metadata, manual aggregation is required. - - See: docs/BQL-BALANCE-QUERIES.md for detailed analysis and test results. - - FUTURE CONSIDERATION: If Castle's ledger format changes to use SATS as position - amounts (instead of metadata), BQL could provide significant performance benefits. - - Args: - query_string: BQL query (e.g., "SELECT account, sum(position) WHERE account ~ 'User-abc'") - - Returns: - { - "rows": [[col1, col2, ...], ...], - "types": [{"name": "col1", "type": "str"}, ...], - "column_names": ["col1", "col2", ...] - } - - Example: - result = await fava.query_bql("SELECT account, sum(position) WHERE account ~ 'User-abc'") - for row in result["rows"]: - account, balance = row - print(f"{account}: {balance}") - - See: - https://beancount.github.io/docs/beancount_query_language.html - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get( - f"{self.base_url}/query", - params={"query_string": query_string} - ) - response.raise_for_status() - result = response.json() - - # Fava returns: {"data": {"rows": [...], "types": [...]}} - data = result.get("data", {}) - rows = data.get("rows", []) - types = data.get("types", []) - column_names = [t.get("name") for t in types] - - return { - "rows": rows, - "types": types, - "column_names": column_names - } - - except httpx.HTTPStatusError as e: - logger.error(f"BQL query error: {e.response.status_code} - {e.response.text}") - logger.error(f"Query was: {query_string}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]: - """ - Get user balance using BQL (efficient, replaces 115-line manual aggregation). - - ⚠️ NOT CURRENTLY USED: This method cannot access SATS balances stored in posting - metadata. It only queries position amounts (EUR/USD). For Castle's current ledger - format, use get_user_balance() instead (manual aggregation with caching). - - This method uses Beancount Query Language for server-side filtering and aggregation, - which would provide 5-10x performance improvement IF SATS were stored as position - amounts instead of metadata. - - FUTURE CONSIDERATION: If Castle's ledger format changes to store SATS as position - amounts (e.g., "100000 SATS {100.00 EUR}"), this method would become feasible and - provide significant performance benefits. - - See: docs/BQL-BALANCE-QUERIES.md for detailed test results and analysis. - - Args: - user_id: User ID - - Returns: - { - "balance": int (sats), # Currently returns 0 (cannot access metadata) - "fiat_balances": {"EUR": Decimal("100.50"), ...}, # Works correctly - "accounts": [{"account": "...", "sats": 150000}, ...] - } - - Example: - balance = await fava.get_user_balance_bql("af983632") - print(f"Balance: {balance['balance']} sats") # Will be 0 with current ledger format - """ - from decimal import Decimal - import re - - # Build BQL query for this user's Payable/Receivable accounts - user_id_prefix = user_id[:8] - query = f""" - SELECT account, sum(position) as balance - WHERE account ~ ':User-{user_id_prefix}' - AND (account ~ 'Payable' OR account ~ 'Receivable') - AND flag != '!' - GROUP BY account - """ - - result = await self.query_bql(query) - - # Process results - total_sats = 0 - fiat_balances = {} - accounts = [] - - for row in result["rows"]: - account_name, position = row - - # Position can be: - # - Dict: {"SATS": "150000", "EUR": "145.50"} - # - String: "150000 SATS" or "145.50 EUR" - - if isinstance(position, dict): - # Extract SATS - sats_str = position.get("SATS", "0") - sats_amount = int(sats_str) if sats_str else 0 - total_sats += sats_amount - - accounts.append({ - "account": account_name, - "sats": sats_amount - }) - - # Extract fiat currencies - for currency in ["EUR", "USD", "GBP"]: - if currency in position: - fiat_str = position[currency] - fiat_amount = Decimal(fiat_str) if fiat_str else Decimal(0) - - if currency not in fiat_balances: - fiat_balances[currency] = Decimal(0) - fiat_balances[currency] += fiat_amount - - elif isinstance(position, str): - # Single currency (parse "150000 SATS" or "145.50 EUR") - sats_match = re.match(r'^(-?\d+)\s+SATS$', position) - if sats_match: - sats_amount = int(sats_match.group(1)) - total_sats += sats_amount - accounts.append({ - "account": account_name, - "sats": sats_amount - }) - else: - fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', position) - if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - fiat_amount = Decimal(fiat_match.group(1)) - currency = fiat_match.group(2) - - if currency not in fiat_balances: - fiat_balances[currency] = Decimal(0) - fiat_balances[currency] += fiat_amount - - logger.info(f"User {user_id[:8]} balance (BQL): {total_sats} sats, fiat: {dict(fiat_balances)}") - - return { - "balance": total_sats, - "fiat_balances": fiat_balances, - "accounts": accounts - } - - async def get_all_user_balances_bql(self) -> List[Dict[str, Any]]: - """ - Get balances for all users using BQL (efficient admin view). - - ⚠️ NOT CURRENTLY USED: This method cannot access SATS balances stored in posting - metadata. It only queries position amounts (EUR/USD). For Castle's current ledger - format, use get_all_user_balances() instead (manual aggregation with caching). - - This method uses Beancount Query Language to query all user balances - in a single efficient query, which would be faster than fetching all entries IF - SATS were stored as position amounts instead of metadata. - - FUTURE CONSIDERATION: If Castle's ledger format changes to store SATS as position - amounts, this method would provide significant performance benefits for admin views. - - See: docs/BQL-BALANCE-QUERIES.md for detailed test results and analysis. - - Returns: - [ - { - "user_id": "abc123", - "balance": 100000, # Currently 0 (cannot access metadata) - "fiat_balances": {"EUR": Decimal("100.50")}, # Works correctly - "accounts": [{"account": "...", "sats": 150000}, ...] - }, - ... - ] - - Example: - all_balances = await fava.get_all_user_balances_bql() - for user in all_balances: - print(f"{user['user_id']}: {user['balance']} sats") # Will be 0 with current format - """ - from decimal import Decimal - import re - - # BQL query for ALL user accounts - query = """ - SELECT account, sum(position) as balance - WHERE (account ~ 'Payable:User-' OR account ~ 'Receivable:User-') - AND flag != '!' - GROUP BY account - """ - - result = await self.query_bql(query) - - # Group by user_id - user_data = {} - - for row in result["rows"]: - account_name, position = row - - # Extract user_id from account name - # Format: "Liabilities:Payable:User-abc12345" or "Assets:Receivable:User-abc12345" - if ":User-" not in account_name: - continue - - user_id_with_prefix = account_name.split(":User-")[1] - # User ID is the first 8 chars (our standard) - user_id = user_id_with_prefix[:8] - - if user_id not in user_data: - user_data[user_id] = { - "user_id": user_id, - "balance": 0, - "fiat_balances": {}, - "accounts": [] - } - - # Process position (same logic as single-user query) - if isinstance(position, dict): - sats_str = position.get("SATS", "0") - sats_amount = int(sats_str) if sats_str else 0 - user_data[user_id]["balance"] += sats_amount - - user_data[user_id]["accounts"].append({ - "account": account_name, - "sats": sats_amount - }) - - for currency in ["EUR", "USD", "GBP"]: - if currency in position: - fiat_str = position[currency] - fiat_amount = Decimal(fiat_str) if fiat_str else Decimal(0) - - if currency not in user_data[user_id]["fiat_balances"]: - user_data[user_id]["fiat_balances"][currency] = Decimal(0) - user_data[user_id]["fiat_balances"][currency] += fiat_amount - - elif isinstance(position, str): - # Single currency (parse "150000 SATS" or "145.50 EUR") - sats_match = re.match(r'^(-?\d+)\s+SATS$', position) - if sats_match: - sats_amount = int(sats_match.group(1)) - user_data[user_id]["balance"] += sats_amount - user_data[user_id]["accounts"].append({ - "account": account_name, - "sats": sats_amount - }) - else: - fiat_match = re.match(r'^(-?[\d.]+)\s+([A-Z]{3})$', position) - if fiat_match and fiat_match.group(2) in ('EUR', 'USD', 'GBP'): - fiat_amount = Decimal(fiat_match.group(1)) - currency = fiat_match.group(2) - - if currency not in user_data[user_id]["fiat_balances"]: - user_data[user_id]["fiat_balances"][currency] = Decimal(0) - user_data[user_id]["fiat_balances"][currency] += fiat_amount - - logger.info(f"Fetched balances for {len(user_data)} users (BQL)") - - return list(user_data.values()) - - async def get_account_transactions( - self, - account_name: str, - limit: int = 100 - ) -> List[Dict[str, Any]]: - """ - Get all transactions affecting a specific account. - - Args: - account_name: Full account name (e.g., "Assets:Receivable:User-abc123") - limit: Maximum number of transactions - - Returns: - List of transactions affecting this account - """ - return await self.query_transactions( - account_pattern=account_name.replace(":", "\\:"), # Escape colons for regex - limit=limit - ) - - async def get_user_transactions( - self, - user_id: str, - limit: int = 100 - ) -> List[Dict[str, Any]]: - """ - Get all transactions affecting a user's accounts. - - Args: - user_id: User ID - limit: Maximum number of transactions - - Returns: - List of transactions affecting user's accounts - """ - return await self.query_transactions( - account_pattern=f"User-{user_id[:8]}", - limit=limit - ) - - async def get_all_accounts(self) -> List[Dict[str, Any]]: - """ - Get all accounts from Beancount/Fava using BQL query. - - Returns: - List of account dictionaries: - [ - {"account": "Assets:Cash", "meta": {...}}, - {"account": "Expenses:Food", "meta": {...}}, - ... - ] - - Example: - accounts = await fava.get_all_accounts() - for acc in accounts: - print(acc["account"]) # "Assets:Cash" - """ - try: - # Use BQL to get all unique accounts - query = "SELECT DISTINCT account" - result = await self.query_bql(query) - - # Convert BQL result to expected format - accounts = [] - for row in result["rows"]: - account_name = row[0] if isinstance(row, list) else row.get("account") - if account_name: - accounts.append({ - "account": account_name, - "meta": {} # BQL doesn't return metadata easily - }) - - logger.debug(f"Fava returned {len(accounts)} accounts via BQL") - return accounts - - except Exception as e: - logger.error(f"Failed to fetch accounts via BQL: {e}") - raise - - async def get_journal_entries( - self, - days: int = None, - start_date: str = None, - end_date: str = None - ) -> List[Dict[str, Any]]: - """ - Get journal entries from Fava (with entry hashes), optionally filtered by date. - - Args: - days: If provided, only return entries from the last N days. - If None, returns all entries (default behavior). - start_date: ISO format date string (YYYY-MM-DD). If provided with end_date, - filters entries between start_date and end_date (inclusive). - end_date: ISO format date string (YYYY-MM-DD). If provided with start_date, - filters entries between start_date and end_date (inclusive). - - Note: - If both days and start_date/end_date are provided, start_date/end_date takes precedence. - - Returns: - List of entries (transactions, opens, closes, etc.) with entry_hash field. - - Example: - # Get all entries - entries = await fava.get_journal_entries() - - # Get only last 30 days - recent = await fava.get_journal_entries(days=30) - - # Get entries in custom date range - custom = await fava.get_journal_entries(start_date="2024-01-01", end_date="2024-01-31") - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get(f"{self.base_url}/journal") - response.raise_for_status() - result = response.json() - entries = result.get("data", []) - logger.info(f"Fava /journal returned {len(entries)} entries") - - # Filter by date range or days - from datetime import datetime, timedelta - - # Use date range if both start_date and end_date are provided - if start_date and end_date: - try: - filter_start = datetime.strptime(start_date, "%Y-%m-%d").date() - filter_end = datetime.strptime(end_date, "%Y-%m-%d").date() - filtered_entries = [] - for e in entries: - entry_date_str = e.get("date") - if entry_date_str: - try: - entry_date = datetime.strptime(entry_date_str, "%Y-%m-%d").date() - if filter_start <= entry_date <= filter_end: - filtered_entries.append(e) - except (ValueError, TypeError): - # Include entries with invalid dates (shouldn't happen) - filtered_entries.append(e) - logger.info(f"Filtered to {len(filtered_entries)} entries between {start_date} and {end_date}") - entries = filtered_entries - except ValueError as e: - logger.error(f"Invalid date format: {e}") - # Return all entries if date parsing fails - - # Fall back to days filter if no date range provided - elif days is not None: - cutoff_date = (datetime.now() - timedelta(days=days)).date() - filtered_entries = [] - for e in entries: - entry_date_str = e.get("date") - if entry_date_str: - try: - entry_date = datetime.strptime(entry_date_str, "%Y-%m-%d").date() - if entry_date >= cutoff_date: - filtered_entries.append(e) - except (ValueError, TypeError): - # Include entries with invalid dates (shouldn't happen) - filtered_entries.append(e) - logger.info(f"Filtered to {len(filtered_entries)} entries from last {days} days (cutoff: {cutoff_date})") - entries = filtered_entries - - # Log transactions with "Lightning payment" in narration - lightning_entries = [e for e in entries if "Lightning payment" in e.get("narration", "")] - logger.info(f"Found {len(lightning_entries)} Lightning payment entries in journal") - - return entries - - except httpx.HTTPStatusError as e: - logger.error(f"Fava journal error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def get_entry_context(self, entry_hash: str) -> Dict[str, Any]: - """ - Get entry context including source text and sha256sum. - - Args: - entry_hash: Entry hash from get_journal_entries() - - Returns: - { - "entry": {...}, # Serialized entry - "slice": "2025-01-15 ! \"Description\"...", # Beancount source text - "sha256sum": "abc123...", # For concurrency control - "balances_before": {...}, - "balances_after": {...} - } - - Example: - context = await fava.get_entry_context("abc123") - source = context["slice"] - sha256sum = context["sha256sum"] - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.get( - f"{self.base_url}/context", - params={"entry_hash": entry_hash} - ) - response.raise_for_status() - result = response.json() - return result.get("data", {}) - - except httpx.HTTPStatusError as e: - logger.error(f"Fava context error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def update_entry_source(self, entry_hash: str, new_source: str, sha256sum: str) -> str: - """ - Update an entry's source text (e.g., change flag from ! to *). - - Args: - entry_hash: Entry hash - new_source: Modified Beancount source text - sha256sum: Current sha256sum from get_entry_context() for concurrency control - - Returns: - New sha256sum after update - - Example: - # Get context - context = await fava.get_entry_context("abc123") - source = context["slice"] - sha256 = context["sha256sum"] - - # Change flag - new_source = source.replace("2025-01-15 !", "2025-01-15 *") - - # Update - new_sha256 = await fava.update_entry_source("abc123", new_source, sha256) - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.put( - f"{self.base_url}/source_slice", - json={ - "entry_hash": entry_hash, - "source": new_source, - "sha256sum": sha256sum - } - ) - response.raise_for_status() - result = response.json() - return result.get("data", "") - - except httpx.HTTPStatusError as e: - logger.error(f"Fava update error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def delete_entry(self, entry_hash: str, sha256sum: str) -> str: - """ - Delete an entry from the Beancount file. - - Args: - entry_hash: Entry hash - sha256sum: Current sha256sum for concurrency control - - Returns: - Success message - - Example: - context = await fava.get_entry_context("abc123") - await fava.delete_entry("abc123", context["sha256sum"]) - """ - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - response = await client.delete( - f"{self.base_url}/source_slice", - params={ - "entry_hash": entry_hash, - "sha256sum": sha256sum - } - ) - response.raise_for_status() - result = response.json() - return result.get("data", "") - - except httpx.HTTPStatusError as e: - logger.error(f"Fava delete error: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - async def add_account( - self, - account_name: str, - currencies: list[str], - opening_date: Optional[date] = None, - metadata: Optional[Dict[str, Any]] = None - ) -> Dict[str, Any]: - """ - Add an account to the Beancount ledger via an Open directive. - - NOTE: Fava's /api/add_entries endpoint does NOT support Open directives. - This method uses /api/source to directly edit the Beancount file. - - Args: - account_name: Full account name (e.g., "Assets:Receivable:User-abc123") - currencies: List of currencies for this account (e.g., ["EUR", "SATS"]) - opening_date: Date to open the account (defaults to today) - metadata: Optional metadata for the account - - Returns: - Response from Fava ({"data": "new_sha256sum", "mtime": "..."}) - - Example: - # Add a user's receivable account - result = await fava.add_account( - account_name="Assets:Receivable:User-abc123", - currencies=["EUR", "SATS", "USD"], - metadata={"user_id": "abc123", "description": "User receivables"} - ) - - # Add a user's payable account - result = await fava.add_account( - account_name="Liabilities:Payable:User-abc123", - currencies=["EUR", "SATS"] - ) - """ - from datetime import date as date_type - - if opening_date is None: - opening_date = date_type.today() - - try: - async with httpx.AsyncClient(timeout=self.timeout) as client: - # Step 1: Get the main Beancount file path from Fava - options_response = await client.get(f"{self.base_url}/options") - options_response.raise_for_status() - options_data = options_response.json()["data"] - file_path = options_data["beancount_options"]["filename"] - - logger.debug(f"Fava main file: {file_path}") - - # Step 2: Get current source file - response = await client.get( - f"{self.base_url}/source", - params={"filename": file_path} - ) - response.raise_for_status() - source_data = response.json()["data"] - - sha256sum = source_data["sha256sum"] - source = source_data["source"] - - # Step 2: Check if account already exists - if f"open {account_name}" in source: - logger.info(f"Account {account_name} already exists in Beancount file") - return {"data": sha256sum, "mtime": source_data.get("mtime", "")} - - # Step 3: Find insertion point (after last Open directive AND its metadata) - lines = source.split('\n') - insert_index = 0 - for i, line in enumerate(lines): - if line.strip().startswith(('open ', f'{opening_date.year}-')) and 'open' in line: - # Found an Open directive, now skip over any metadata lines - insert_index = i + 1 - # Skip metadata lines (lines starting with whitespace) - while insert_index < len(lines) and lines[insert_index].startswith((' ', '\t')) and lines[insert_index].strip(): - insert_index += 1 - - # Step 4: Format Open directive as Beancount text - currencies_str = ", ".join(currencies) - open_lines = [ - "", - f"{opening_date.isoformat()} open {account_name} {currencies_str}" - ] - - # Add metadata if provided - if metadata: - for key, value in metadata.items(): - # Format metadata with proper indentation - if isinstance(value, str): - open_lines.append(f' {key}: "{value}"') - else: - open_lines.append(f' {key}: {value}') - - # Step 5: Insert into source - for i, line in enumerate(open_lines): - lines.insert(insert_index + i, line) - - new_source = '\n'.join(lines) - - # Step 6: Update source file via PUT /api/source - update_payload = { - "file_path": file_path, - "source": new_source, - "sha256sum": sha256sum - } - - response = await client.put( - f"{self.base_url}/source", - json=update_payload, - headers={"Content-Type": "application/json"} - ) - response.raise_for_status() - result = response.json() - - logger.info(f"Added account {account_name} to Beancount file with currencies {currencies}") - return result - - except httpx.HTTPStatusError as e: - logger.error(f"Fava HTTP error adding account: {e.response.status_code} - {e.response.text}") - raise - except httpx.RequestError as e: - logger.error(f"Fava connection error: {e}") - raise - - -# Singleton instance (configured from settings) -_fava_client: Optional[FavaClient] = None - - -def init_fava_client(fava_url: str, ledger_slug: str, timeout: float = 10.0): - """ - Initialize the global Fava client. - - Args: - fava_url: Base URL of Fava server - ledger_slug: Ledger identifier - timeout: Request timeout in seconds - """ - global _fava_client - _fava_client = FavaClient(fava_url, ledger_slug, timeout) - logger.info(f"Fava client initialized: {fava_url}/{ledger_slug}") - - -def get_fava_client() -> FavaClient: - """ - Get the configured Fava client. - - Returns: - FavaClient instance - - Raises: - RuntimeError: If client not initialized - """ - if _fava_client is None: - raise RuntimeError( - "Fava client not initialized. Call init_fava_client() first. " - "Castle requires Fava for all accounting operations." - ) - return _fava_client diff --git a/helper/README.md b/helper/README.md deleted file mode 100644 index 648b987..0000000 --- a/helper/README.md +++ /dev/null @@ -1,168 +0,0 @@ -# Castle Beancount Import Helper - -Import Beancount ledger transactions into Castle accounting extension. - -## 📁 Files - -- `import_beancount.py` - Main import script -- `btc_eur_rates.csv` - Daily BTC/EUR rates (create your own) -- `README.md` - This file - -## 🚀 Setup - -### 1. Create BTC/EUR Rates CSV - -Create `btc_eur_rates.csv` in this directory with your actual rates: - -```csv -date,btc_eur_rate -2025-07-01,86500 -2025-07-02,87200 -2025-07-03,87450 -``` - -### 2. Update User Mappings - -Edit `import_beancount.py` and update the `USER_MAPPINGS` dictionary: - -```python -USER_MAPPINGS = { - "Pat": "actual_wallet_id_for_pat", - "Alice": "actual_wallet_id_for_alice", - "Bob": "actual_wallet_id_for_bob", -} -``` - -**How to get wallet IDs:** -- Check your LNbits admin panel -- Or query: `curl -X GET http://localhost:5000/api/v1/wallet -H "X-Api-Key: user_invoice_key"` - -### 3. Set API Key - -```bash -export CASTLE_ADMIN_KEY="your_lnbits_admin_invoice_key" -export LNBITS_URL="http://localhost:5000" # Optional -``` - -## 📖 Usage - -```bash -cd /path/to/castle/helper - -# Test with dry run -python import_beancount.py ledger.beancount --dry-run - -# Actually import -python import_beancount.py ledger.beancount -``` - -## 📄 Beancount File Format - -Your Beancount transactions must have an `Equity:` account: - -```beancount -2025-07-06 * "Foix market" - Expenses:Groceries 69.40 EUR - Equity:Pat - -2025-07-07 * "Gas station" - Expenses:Transport 45.00 EUR - Equity:Alice -``` - -**Requirements:** -- Every transaction must have an `Equity:` account -- Account names must match exactly what's in Castle -- The name after `Equity:` must be in `USER_MAPPINGS` - -## 🔄 How It Works - -1. **Loads rates** from `btc_eur_rates.csv` -2. **Loads accounts** from Castle API automatically -3. **Maps users** - Extracts user name from `Equity:Name` accounts -4. **Parses** Beancount transactions -5. **Converts** EUR → sats using daily rate -6. **Uploads** to Castle with metadata - -## 📊 Example Output - -```bash -$ python import_beancount.py ledger.beancount -====================================================================== -🏰 Beancount to Castle Import Script -====================================================================== - -📊 Loaded 15 daily rates from btc_eur_rates.csv - Date range: 2025-07-01 to 2025-07-15 - -🏦 Loaded 28 accounts from Castle - -👥 User ID mappings: - - Pat → wallet_abc123 - - Alice → wallet_def456 - - Bob → wallet_ghi789 - -📄 Found 25 potential transactions in ledger.beancount - -✅ Transaction 1: 2025-07-06 - Foix market (User: Pat) (Rate: 87,891 EUR/BTC) -✅ Transaction 2: 2025-07-07 - Gas station (User: Alice) (Rate: 88,100 EUR/BTC) -✅ Transaction 3: 2025-07-08 - Restaurant (User: Bob) (Rate: 88,350 EUR/BTC) - -====================================================================== -📊 Summary: 25 succeeded, 0 failed, 0 skipped -====================================================================== - -✅ Successfully imported 25 transactions to Castle! -``` - -## ❓ Troubleshooting - -### "No account found in Castle" -**Error:** `No account found in Castle with name 'Expenses:XYZ'` - -**Solution:** Create the account in Castle first with that exact name. - -### "No user ID mapping found" -**Error:** `No user ID mapping found for 'Pat'` - -**Solution:** Add Pat to the `USER_MAPPINGS` dictionary in the script. - -### "No BTC/EUR rate found" -**Error:** `No BTC/EUR rate found for 2025-07-15` - -**Solution:** Add that date to `btc_eur_rates.csv`. - -### "Could not determine user ID" -**Error:** `Could not determine user ID for transaction` - -**Solution:** Every transaction needs an `Equity:` account (e.g., `Equity:Pat`). - -## 📝 Transaction Metadata - -Each imported transaction includes: - -```json -{ - "meta": { - "source": "beancount_import", - "imported_at": "2025-11-08T12:00:00", - "btc_eur_rate": 87891.0, - "user_id": "wallet_abc123" - } -} -``` - -And each line includes: - -```json -{ - "metadata": { - "fiat_currency": "EUR", - "fiat_amount": "69.400", - "fiat_rate": 1137.88, - "btc_rate": 87891.0 - } -} -``` - -This preserves the original EUR amount and exchange rate for auditing. diff --git a/helper/btc_eur_rates.csv b/helper/btc_eur_rates.csv deleted file mode 120000 index 559e863..0000000 --- a/helper/btc_eur_rates.csv +++ /dev/null @@ -1 +0,0 @@ -/home/padreug/projects/historical-bitcoin-data/bitcoin_daily_prices.csv \ No newline at end of file diff --git a/helper/import_beancount.py b/helper/import_beancount.py deleted file mode 100755 index 417d0fd..0000000 --- a/helper/import_beancount.py +++ /dev/null @@ -1,673 +0,0 @@ -#!/usr/bin/env python3 -""" -Beancount to Castle Import Script - -⚠️ NOTE: This script is for ONE-OFF MIGRATION purposes only. - - Now that Castle uses Fava/Beancount as the single source of truth, - the data flow is: Castle → Fava/Beancount (not the reverse). - - This script was used for initial data import from existing Beancount files. - - Future disposition: - - DELETE if no longer needed for migrations - - REPURPOSE for bidirectional sync if that becomes a requirement - - ARCHIVE to misc-docs/old-helpers/ if keeping for reference - -Imports Beancount ledger transactions into Castle accounting extension. -Reads daily BTC/EUR rates from btc_eur_rates.csv in the same directory. - -Usage: - python import_beancount.py [--dry-run] - -Example: - python import_beancount.py my_ledger.beancount --dry-run - python import_beancount.py my_ledger.beancount -""" -import requests -import csv -import os -from datetime import datetime, timedelta -from decimal import Decimal -from typing import Dict, Optional - -# ===== CONFIGURATION ===== - -# LNbits URL and API Key -LNBITS_URL = os.environ.get("LNBITS_URL", "http://localhost:5000") -ADMIN_API_KEY = os.environ.get("CASTLE_ADMIN_KEY", "48d787d862484a6c89d6a557b4d5be9d") - -# Rates CSV file (looks in same directory as this script) -SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) -RATES_CSV_FILE = os.path.join(SCRIPT_DIR, "btc_eur_rates.csv") - -# User ID mappings: Equity account name -> Castle user ID (wallet ID) -# TODO: Update these with your actual Castle user/wallet IDs -USER_MAPPINGS = { - "Pat": "75be145a42884b22b60bf97510ed46e3", - "Coco": "375ec158ceca46de86cf6561ca20f881", - "Charlie": "921340b802104c25901eae6c420b1ba1", -} - -# ===== RATE LOOKUP ===== - -class RateLookup: - """Load and lookup BTC/EUR rates from CSV file""" - - def __init__(self, csv_file: str): - self.rates = {} - self._load_csv(csv_file) - - def _load_csv(self, csv_file: str): - """Load rates from CSV file""" - if not os.path.exists(csv_file): - raise FileNotFoundError( - f"Rates CSV file not found: {csv_file}\n" - f"Please create btc_eur_rates.csv in the same directory as this script." - ) - - with open(csv_file, 'r', encoding='utf-8') as f: - reader = csv.DictReader(f) - for row in reader: - date = datetime.strptime(row['date'], '%Y-%m-%d').date() - # Handle comma as thousands separator - rate_str = row['btc_eur_rate'].replace(',', '').replace(' ', '') - rate = float(rate_str) - self.rates[date] = rate - - if not self.rates: - raise ValueError(f"No rates loaded from {csv_file}") - - print(f"📊 Loaded {len(self.rates)} daily rates from {os.path.basename(csv_file)}") - print(f" Date range: {min(self.rates.keys())} to {max(self.rates.keys())}") - - def get_rate(self, date: datetime.date, fallback_days: int = 7) -> Optional[float]: - """ - Get BTC/EUR rate for a specific date. - If exact date not found, tries nearby dates within fallback_days. - - Args: - date: Date to lookup - fallback_days: How many days to look back/forward if exact date missing - - Returns: - BTC/EUR rate or None if not found - """ - # Try exact date first - if date in self.rates: - return self.rates[date] - - # Try nearby dates (prefer earlier dates) - for days_offset in range(1, fallback_days + 1): - # Try earlier date first - earlier = date - timedelta(days=days_offset) - if earlier in self.rates: - print(f" ⚠️ Using rate from {earlier} for {date} (exact date not found)") - return self.rates[earlier] - - # Try later date - later = date + timedelta(days=days_offset) - if later in self.rates: - print(f" ⚠️ Using rate from {later} for {date} (exact date not found)") - return self.rates[later] - - return None - -# ===== ACCOUNT LOOKUP ===== - -class AccountLookup: - """Fetch and lookup Castle accounts from API""" - - def __init__(self, lnbits_url: str, api_key: str): - self.accounts = {} # name -> account_id - self.accounts_by_user = {} # user_id -> {account_type -> account_id} - self.account_details = [] # Full account objects - self._fetch_accounts(lnbits_url, api_key) - - def _fetch_accounts(self, lnbits_url: str, api_key: str): - """Fetch all accounts from Castle API""" - url = f"{lnbits_url}/castle/api/v1/accounts" - headers = {"X-Api-Key": api_key} - - try: - response = requests.get(url, headers=headers) - response.raise_for_status() - accounts_list = response.json() - - # Build mappings - for account in accounts_list: - name = account.get('name') - account_id = account.get('id') - user_id = account.get('user_id') - account_type = account.get('account_type') - - self.account_details.append(account) - - # Name -> ID mapping - if name and account_id: - self.accounts[name] = account_id - - # User -> Account Type -> ID mapping (for equity accounts) - if user_id and account_type: - if user_id not in self.accounts_by_user: - self.accounts_by_user[user_id] = {} - self.accounts_by_user[user_id][account_type] = account_id - - print(f"🏦 Loaded {len(self.accounts)} accounts from Castle") - - except requests.RequestException as e: - raise ConnectionError(f"Failed to fetch accounts from Castle API: {e}") - - def get_account_id(self, account_name: str) -> Optional[str]: - """ - Get Castle account ID for a Beancount account name. - - Special handling for user-specific accounts: - - "Liabilities:Payable:Pat" -> looks up Pat's user_id and finds their Castle payable account - - "Assets:Receivable:Pat" -> looks up Pat's user_id and finds their Castle receivable account - - "Equity:Pat" -> looks up Pat's user_id and finds their Castle equity account - - Args: - account_name: Beancount account name (e.g., "Expenses:Food:Supplies", "Liabilities:Payable:Pat", "Assets:Receivable:Pat", "Equity:Pat") - - Returns: - Castle account UUID or None if not found - """ - # Check if this is a Liabilities:Payable: account - # Map Beancount Liabilities:Payable:Pat to Castle Liabilities:Payable:User- - if account_name.startswith("Liabilities:Payable:"): - user_name = extract_user_from_user_account(account_name) - if user_name: - # Look up user's actual user_id - user_id = USER_MAPPINGS.get(user_name) - if user_id: - # Find this user's liability (payable) account - # This is the Liabilities:Payable:User- account in Castle - if user_id in self.accounts_by_user: - liability_account_id = self.accounts_by_user[user_id].get('liability') - if liability_account_id: - return liability_account_id - - # If not found, provide helpful error - raise ValueError( - f"User '{user_name}' (ID: {user_id}) does not have a payable account.\n" - f"This should have been created when they configured their wallet.\n" - f"Please configure the wallet for user ID: {user_id}" - ) - - # Check if this is an Assets:Receivable: account - # Map Beancount Assets:Receivable:Pat to Castle Assets:Receivable:User- - elif account_name.startswith("Assets:Receivable:"): - user_name = extract_user_from_user_account(account_name) - if user_name: - # Look up user's actual user_id - user_id = USER_MAPPINGS.get(user_name) - if user_id: - # Find this user's asset (receivable) account - # This is the Assets:Receivable:User- account in Castle - if user_id in self.accounts_by_user: - asset_account_id = self.accounts_by_user[user_id].get('asset') - if asset_account_id: - return asset_account_id - - # If not found, provide helpful error - raise ValueError( - f"User '{user_name}' (ID: {user_id}) does not have a receivable account.\n" - f"This should have been created when they configured their wallet.\n" - f"Please configure the wallet for user ID: {user_id}" - ) - - # Check if this is an Equity: account - # Map Beancount Equity:Pat to Castle Equity:User- - elif account_name.startswith("Equity:"): - user_name = extract_user_from_user_account(account_name) - if user_name: - # Look up user's actual user_id - user_id = USER_MAPPINGS.get(user_name) - if user_id: - # Find this user's equity account - # This is the Equity:User- account in Castle - if user_id in self.accounts_by_user: - equity_account_id = self.accounts_by_user[user_id].get('equity') - if equity_account_id: - return equity_account_id - - # If not found, provide helpful error - raise ValueError( - f"User '{user_name}' (ID: {user_id}) does not have an equity account.\n" - f"Equity eligibility must be enabled for this user in Castle.\n" - f"Please enable equity for user ID: {user_id}" - ) - - # Normal account lookup by name - return self.accounts.get(account_name) - - def list_accounts(self): - """Print all available accounts""" - print("\n📋 Available accounts:") - for name in sorted(self.accounts.keys()): - print(f" - {name}") - -# ===== CONVERSION FUNCTIONS ===== - -def sanitize_link(text: str) -> str: - """ - Sanitize a string to make it valid for Beancount links. - - Beancount links can only contain: A-Z, a-z, 0-9, -, _, /, . - All other characters are replaced with hyphens. - - Examples: - >>> sanitize_link("Test (pending)") - 'Test-pending' - >>> sanitize_link("Invoice #123") - 'Invoice-123' - >>> sanitize_link("import-20250623-Action Ressourcerie") - 'import-20250623-Action-Ressourcerie' - """ - import re - # Replace any character that's not alphanumeric, dash, underscore, slash, or period with a hyphen - sanitized = re.sub(r'[^A-Za-z0-9\-_/.]', '-', text) - # Remove consecutive hyphens - sanitized = re.sub(r'-+', '-', sanitized) - # Remove leading/trailing hyphens - sanitized = sanitized.strip('-') - return sanitized - -def eur_to_sats(eur_amount: Decimal, btc_eur_rate: float) -> int: - """Convert EUR to satoshis using BTC/EUR rate""" - btc_amount = eur_amount / Decimal(str(btc_eur_rate)) - sats = btc_amount * Decimal(100_000_000) - return int(sats.quantize(Decimal('1'))) - -def build_metadata(eur_amount: Decimal, btc_eur_rate: float) -> dict: - """ - Build metadata dict for Castle entry line. - - The API will extract fiat_currency and fiat_amount and use them - to create proper EUR-based postings with SATS in metadata. - """ - abs_eur = abs(eur_amount) - abs_sats = abs(eur_to_sats(abs_eur, btc_eur_rate)) - - return { - "fiat_currency": "EUR", - "fiat_amount": str(abs_eur.quantize(Decimal("0.01"))), # Store as string for JSON - "btc_rate": str(btc_eur_rate) # Store exchange rate for reference - } - -# ===== BEANCOUNT PARSER ===== - -def parse_beancount_transaction(txn_text: str) -> Optional[Dict]: - """ - Parse a Beancount transaction. - - Expected format: - 2025-07-06 * "Foix market" - Expenses:Groceries 69.40 EUR - Equity:Pat - """ - lines = txn_text.strip().split('\n') - if not lines: - return None - - # Skip leading comments to find the transaction header - header_line_idx = 0 - for i, line in enumerate(lines): - stripped = line.strip() - # Skip comments and empty lines - if not stripped or stripped.startswith(';'): - continue - # Found the first non-comment line - header_line_idx = i - break - - # Parse header line - header = lines[header_line_idx].strip() - - # Handle both * and ! flags - if '*' in header: - parts = header.split('*') - flag = '*' - elif '!' in header: - parts = header.split('!') - flag = '!' - else: - return None - - date_str = parts[0].strip() - description = parts[1].strip().strip('"') - - try: - date = datetime.strptime(date_str, '%Y-%m-%d') - except ValueError: - return None - - # Parse postings (start after the header line) - postings = [] - for line in lines[header_line_idx + 1:]: - line = line.strip() - - # Skip comments and empty lines - if not line or line.startswith(';'): - continue - - # Parse posting line - parts = line.split() - if not parts: - continue - - account = parts[0] - - # Check if amount is specified - if len(parts) >= 3 and parts[-1] == 'EUR': - # Strip commas from amount (e.g., "1,500.00" -> "1500.00") - amount_str = parts[-2].replace(',', '') - eur_amount = Decimal(amount_str) - else: - # No amount specified - will be calculated to balance - eur_amount = None - - postings.append({ - 'account': account, - 'eur_amount': eur_amount - }) - - # Calculate missing amounts (Beancount auto-balance) - # TODO: Support auto-balancing for transactions with >2 postings - # For now, only handles simple 2-posting transactions - if len(postings) == 2: - if postings[0]['eur_amount'] and not postings[1]['eur_amount']: - postings[1]['eur_amount'] = -postings[0]['eur_amount'] - elif postings[1]['eur_amount'] and not postings[0]['eur_amount']: - postings[0]['eur_amount'] = -postings[1]['eur_amount'] - - return { - 'date': date, - 'description': description, - 'postings': postings - } - -# ===== HELPER FUNCTIONS ===== - -def extract_user_from_user_account(account_name: str) -> Optional[str]: - """ - Extract user name from user-specific accounts (Payable, Receivable, or Equity). - - Examples: - "Liabilities:Payable:Pat" -> "Pat" - "Assets:Receivable:Alice" -> "Alice" - "Equity:Pat" -> "Pat" - "Expenses:Food" -> None - - Returns: - User name or None if not a user-specific account - """ - if account_name.startswith("Liabilities:Payable:"): - parts = account_name.split(":") - if len(parts) >= 3: - return parts[2] - elif account_name.startswith("Assets:Receivable:"): - parts = account_name.split(":") - if len(parts) >= 3: - return parts[2] - elif account_name.startswith("Equity:"): - parts = account_name.split(":") - if len(parts) >= 2: - return parts[1] - return None - -def determine_user_id(postings: list) -> Optional[str]: - """ - Determine which user ID to use for this transaction based on user-specific accounts. - - Args: - postings: List of posting dicts with 'account' key - - Returns: - User ID (wallet ID) from USER_MAPPINGS, or None if no user account found - """ - for posting in postings: - user_name = extract_user_from_user_account(posting['account']) - if user_name: - user_id = USER_MAPPINGS.get(user_name) - if not user_id: - raise ValueError( - f"No user ID mapping found for '{user_name}'.\n" - f"Please add '{user_name}' to USER_MAPPINGS in the script." - ) - return user_id - - # No user-specific account found - this shouldn't happen for typical transactions - return None - -# ===== CASTLE CONVERTER ===== - -def convert_to_castle_entry(parsed: dict, btc_eur_rate: float, account_lookup: AccountLookup) -> dict: - """ - Convert parsed Beancount transaction to Castle format. - - Sends SATS amounts with fiat metadata. The Castle API will automatically - convert to EUR-based postings with SATS stored in metadata. - """ - - # Determine which user this transaction is for (based on user-specific accounts) - user_id = determine_user_id(parsed['postings']) - if not user_id: - raise ValueError( - f"Could not determine user ID for transaction.\n" - f"Transactions must have a user-specific account:\n" - f" - Liabilities:Payable: (for payables)\n" - f" - Assets:Receivable: (for receivables)\n" - f" - Equity: (for equity)\n" - f"Examples: Liabilities:Payable:Pat, Assets:Receivable:Pat, Equity:Pat" - ) - - # Build entry lines - lines = [] - for posting in parsed['postings']: - account_id = account_lookup.get_account_id(posting['account']) - if not account_id: - raise ValueError( - f"No account found in Castle with name '{posting['account']}'.\n" - f"Please create this account in Castle first." - ) - - eur_amount = posting['eur_amount'] - if eur_amount is None: - raise ValueError(f"Could not determine amount for {posting['account']}") - - # Convert EUR to sats (amount sent to API) - sats = eur_to_sats(eur_amount, btc_eur_rate) - - # Build metadata (API will extract fiat_currency and fiat_amount) - metadata = build_metadata(eur_amount, btc_eur_rate) - - lines.append({ - "account_id": account_id, - "amount": sats, # Positive = debit, negative = credit - "description": posting['account'], - "metadata": metadata - }) - - # Create sanitized reference link - desc_part = sanitize_link(parsed['description'][:30]) - - return { - "description": parsed['description'], - "entry_date": parsed['date'].isoformat(), - "reference": f"import-{parsed['date'].strftime('%Y%m%d')}-{desc_part}", - "flag": "*", - "meta": { - "source": "beancount_import", - "imported_at": datetime.now().isoformat(), - "btc_eur_rate": str(btc_eur_rate), - "user_id": user_id # Track which user this transaction is for - }, - "lines": lines - } - -# ===== API UPLOAD ===== - -def upload_entry(entry: dict, api_key: str, dry_run: bool = False) -> dict: - """Upload journal entry to Castle API""" - if dry_run: - print(f"\n[DRY RUN] Entry preview:") - print(f" Description: {entry['description']}") - print(f" Date: {entry['entry_date']}") - print(f" BTC/EUR Rate: {entry['meta']['btc_eur_rate']:,.2f}") - total_sats = 0 - for line in entry['lines']: - sign = '+' if line['amount'] > 0 else '' - print(f" {line['description']}: {sign}{line['amount']:,} sats " - f"({line['metadata']['fiat_amount']} EUR)") - total_sats += line['amount'] - print(f" Balance check: {total_sats} (should be 0)") - return {"id": "dry-run"} - - url = f"{LNBITS_URL}/castle/api/v1/entries" - headers = { - "X-Api-Key": api_key, - "Content-Type": "application/json" - } - - try: - response = requests.post(url, json=entry, headers=headers) - response.raise_for_status() - return response.json() - except requests.exceptions.HTTPError as e: - print(f" ❌ HTTP Error: {e}") - if response.text: - print(f" Response: {response.text}") - raise - except Exception as e: - print(f" ❌ Error: {e}") - raise - -# ===== MAIN IMPORT FUNCTION ===== - -def import_beancount_file(beancount_file: str, dry_run: bool = False): - """Import transactions from Beancount file using rates from CSV""" - - # Validate configuration - if not ADMIN_API_KEY: - print("❌ Error: CASTLE_ADMIN_KEY not set!") - print(" Set it as environment variable or update ADMIN_API_KEY in the script.") - return - - # Load rates - try: - rate_lookup = RateLookup(RATES_CSV_FILE) - except (FileNotFoundError, ValueError) as e: - print(f"❌ Error loading rates: {e}") - return - - # Load accounts from Castle - try: - account_lookup = AccountLookup(LNBITS_URL, ADMIN_API_KEY) - except (ConnectionError, ValueError) as e: - print(f"❌ Error loading accounts: {e}") - return - - # Show user mappings and verify equity accounts exist - print(f"\n👥 User ID mappings and equity accounts:") - for name, user_id in USER_MAPPINGS.items(): - has_equity = user_id in account_lookup.accounts_by_user and 'equity' in account_lookup.accounts_by_user[user_id] - status = "✅" if has_equity else "❌" - print(f" {status} {name} → {user_id} {'(has equity account)' if has_equity else '(NO EQUITY ACCOUNT - create in Castle!)'}") - - # Read beancount file - if not os.path.exists(beancount_file): - print(f"❌ Error: Beancount file not found: {beancount_file}") - return - - with open(beancount_file, 'r', encoding='utf-8') as f: - content = f.read() - - # Split by blank lines to get individual transactions - transactions = [t.strip() for t in content.split('\n\n') if t.strip()] - - print(f"\n📄 Found {len(transactions)} potential transactions in {os.path.basename(beancount_file)}") - if dry_run: - print("🔍 [DRY RUN MODE] No changes will be made\n") - - success_count = 0 - error_count = 0 - skip_count = 0 - skipped_items = [] # Track what was skipped - - for i, txn_text in enumerate(transactions, 1): - try: - # Try to parse the transaction - parsed = parse_beancount_transaction(txn_text) - if not parsed: - # Not a valid transaction (likely a directive, option, or comment block) - skip_count += 1 - first_line = txn_text.split('\n')[0][:60] - skipped_items.append(f"Entry {i}: {first_line}... (not a transaction)") - continue - - # Look up rate for this transaction's date - btc_eur_rate = rate_lookup.get_rate(parsed['date'].date()) - if not btc_eur_rate: - raise ValueError(f"No BTC/EUR rate found for {parsed['date'].date()}") - - castle_entry = convert_to_castle_entry(parsed, btc_eur_rate, account_lookup) - result = upload_entry(castle_entry, ADMIN_API_KEY, dry_run) - - # Get user name for display - user_name = None - for posting in parsed['postings']: - user_name = extract_user_from_user_account(posting['account']) - if user_name: - break - - user_info = f" (User: {user_name})" if user_name else "" - print(f"✅ Transaction {i}: {parsed['date'].date()} - {parsed['description'][:35]}{user_info} " - f"(Rate: {btc_eur_rate:,.0f} EUR/BTC)") - success_count += 1 - - except Exception as e: - print(f"❌ Transaction {i} failed: {e}") - print(f" Content: {txn_text[:100]}...") - error_count += 1 - - print(f"\n{'='*70}") - print(f"📊 Summary: {success_count} succeeded, {error_count} failed, {skip_count} skipped") - print(f"{'='*70}") - - # Show details of skipped entries - if skipped_items: - print(f"\n⏭️ Skipped entries:") - for item in skipped_items: - print(f" {item}") - - if success_count > 0 and not dry_run: - print(f"\n✅ Successfully imported {success_count} transactions to Castle!") - print(f"\n💡 Note: Transactions are stored in EUR with SATS in metadata.") - print(f" Check Fava to see the imported entries.") - -# ===== MAIN ===== - -if __name__ == "__main__": - import sys - - print("=" * 70) - print("🏰 Beancount to Castle Import Script") - print("=" * 70) - - if len(sys.argv) < 2: - print("\nUsage: python import_beancount.py [--dry-run]") - print("\nExample:") - print(" python import_beancount.py my_ledger.beancount --dry-run") - print(" python import_beancount.py my_ledger.beancount") - print("\nConfiguration:") - print(f" LNBITS_URL: {LNBITS_URL}") - print(f" RATES_CSV: {RATES_CSV_FILE}") - print(f" API Key set: {'Yes' if ADMIN_API_KEY else 'No (set CASTLE_ADMIN_KEY env var)'}") - sys.exit(1) - - beancount_file = sys.argv[1] - dry_run = "--dry-run" in sys.argv - - import_beancount_file(beancount_file, dry_run) diff --git a/migrations.py b/migrations.py index c9a7e30..5efb00d 100644 --- a/migrations.py +++ b/migrations.py @@ -1,71 +1,13 @@ -""" -Castle Extension Database Migrations - -This file contains a single squashed migration that creates the complete -database schema for the Castle extension. - -MIGRATION HISTORY: -This is a squashed migration that combines m001-m016 from the original -incremental migration history. The complete historical migrations are -preserved in migrations_old.py.bak for reference. - -Key schema decisions reflected in this migration: -1. Hierarchical Beancount-style account names (e.g., "Assets:Bitcoin:Lightning") -2. No journal_entries/entry_lines tables (Fava is source of truth) -3. User-specific equity accounts created dynamically (Equity:User-{user_id}) -4. Parent-only accounts removed (hierarchy implicit in colon-separated names) -5. Multi-currency support via balance_assertions -6. Granular permission system via account_permissions - -Original migration sequence (Nov 2025): -- m001: Initial accounts, journal_entries, entry_lines tables -- m002: Extension settings -- m003: User wallet settings -- m004: Manual payment requests -- m005: Added flag/meta to journal entries -- m006: Migrated to hierarchical account names -- m007: Balance assertions -- m008: Renamed Lightning account -- m009: Added OnChain Bitcoin account -- m010: User equity status -- m011: Account permissions -- m012: Updated default accounts with detailed hierarchy -- m013: Removed parent-only accounts (Assets:Bitcoin, Equity) -- m014: Removed legacy equity accounts (MemberEquity, RetainedEarnings) -- m015: Converted entry_lines to single amount field -- m016: Dropped journal_entries and entry_lines tables (Fava integration) -""" - - async def m001_initial(db): """ - Initial Castle database schema (squashed from m001-m016). - - Creates complete database structure for Castle accounting extension: - - Accounts: Chart of accounts with hierarchical Beancount-style names - - Extension settings: Castle-wide configuration - - User wallet settings: Per-user wallet configuration - - Manual payment requests: User-submitted payment requests to Castle - - Balance assertions: Reconciliation and balance checking - - User equity status: Equity contribution eligibility - - Account permissions: Granular access control - - Note: Journal entries are managed by Fava/Beancount (external source of truth). - Castle submits entries to Fava and queries Fava for journal data. + Initial migration for Castle accounting extension. + Creates tables for double-entry bookkeeping system. """ - - # ========================================================================= - # ACCOUNTS TABLE - # ========================================================================= - # Core chart of accounts with hierarchical Beancount-style naming. - # Examples: "Assets:Bitcoin:Lightning", "Expenses:Food:Groceries" - # User-specific accounts: "Assets:Receivable:User-af983632" - await db.execute( f""" CREATE TABLE accounts ( id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, account_type TEXT NOT NULL, description TEXT, user_id TEXT, @@ -86,29 +28,113 @@ async def m001_initial(db): """ ) - # ========================================================================= - # EXTENSION SETTINGS TABLE - # ========================================================================= - # Castle-wide configuration settings + await db.execute( + f""" + CREATE TABLE journal_entries ( + id TEXT PRIMARY KEY, + description TEXT NOT NULL, + entry_date TIMESTAMP NOT NULL, + created_by TEXT NOT NULL, + created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, + reference TEXT + ); + """ + ) + await db.execute( + """ + CREATE INDEX idx_journal_entries_created_by ON journal_entries (created_by); + """ + ) + + await db.execute( + """ + CREATE INDEX idx_journal_entries_date ON journal_entries (entry_date); + """ + ) + + await db.execute( + f""" + CREATE TABLE entry_lines ( + id TEXT PRIMARY KEY, + journal_entry_id TEXT NOT NULL, + account_id TEXT NOT NULL, + debit INTEGER NOT NULL DEFAULT 0, + credit INTEGER NOT NULL DEFAULT 0, + description TEXT, + metadata TEXT DEFAULT '{{}}' + ); + """ + ) + + await db.execute( + """ + CREATE INDEX idx_entry_lines_journal_entry ON entry_lines (journal_entry_id); + """ + ) + + await db.execute( + """ + CREATE INDEX idx_entry_lines_account ON entry_lines (account_id); + """ + ) + + # Insert default chart of accounts + default_accounts = [ + # Assets + ("cash", "Cash", "asset", "Cash on hand"), + ("bank", "Bank Account", "asset", "Bank account"), + ("lightning", "Lightning Balance", "asset", "Lightning Network balance"), + ("accounts_receivable", "Accounts Receivable", "asset", "Money owed to the Castle"), + + # Liabilities + ("accounts_payable", "Accounts Payable", "liability", "Money owed by the Castle"), + + # Equity + ("member_equity", "Member Equity", "equity", "Member contributions"), + ("retained_earnings", "Retained Earnings", "equity", "Accumulated profits"), + + # Revenue + ("accommodation_revenue", "Accommodation Revenue", "revenue", "Revenue from stays"), + ("service_revenue", "Service Revenue", "revenue", "Revenue from services"), + ("other_revenue", "Other Revenue", "revenue", "Other revenue"), + + # Expenses + ("utilities", "Utilities", "expense", "Electricity, water, internet"), + ("food", "Food & Supplies", "expense", "Food and supplies"), + ("maintenance", "Maintenance", "expense", "Repairs and maintenance"), + ("other_expense", "Other Expenses", "expense", "Miscellaneous expenses"), + ] + + for acc_id, name, acc_type, desc in default_accounts: + await db.execute( + """ + INSERT INTO accounts (id, name, account_type, description) + VALUES (:id, :name, :type, :description) + """, + {"id": acc_id, "name": name, "type": acc_type, "description": desc} + ) + + +async def m002_extension_settings(db): + """ + Create extension_settings table for Castle configuration. + """ await db.execute( f""" CREATE TABLE extension_settings ( id TEXT NOT NULL PRIMARY KEY, castle_wallet_id TEXT, - fava_url TEXT NOT NULL DEFAULT 'http://localhost:3333', - fava_ledger_slug TEXT NOT NULL DEFAULT 'castle-ledger', - fava_timeout REAL NOT NULL DEFAULT 10.0, updated_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now} ); """ ) - # ========================================================================= - # USER WALLET SETTINGS TABLE - # ========================================================================= - # Per-user wallet configuration +async def m003_user_wallet_settings(db): + """ + Create user_wallet_settings table for per-user wallet configuration. + """ await db.execute( f""" CREATE TABLE user_wallet_settings ( @@ -119,11 +145,11 @@ async def m001_initial(db): """ ) - # ========================================================================= - # MANUAL PAYMENT REQUESTS TABLE - # ========================================================================= - # User-submitted payment requests to Castle (reviewed by admins) +async def m004_manual_payment_requests(db): + """ + Create manual_payment_requests table for user payment requests to Castle. + """ await db.execute( f""" CREATE TABLE manual_payment_requests ( @@ -131,7 +157,6 @@ async def m001_initial(db): user_id TEXT NOT NULL, amount INTEGER NOT NULL, description TEXT NOT NULL, - notes TEXT, status TEXT NOT NULL DEFAULT 'pending', created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, reviewed_at TIMESTAMP, @@ -143,24 +168,115 @@ async def m001_initial(db): await db.execute( """ - CREATE INDEX idx_manual_payment_requests_user_id - ON manual_payment_requests (user_id); + CREATE INDEX idx_manual_payment_requests_user_id ON manual_payment_requests (user_id); """ ) await db.execute( """ - CREATE INDEX idx_manual_payment_requests_status - ON manual_payment_requests (status); + CREATE INDEX idx_manual_payment_requests_status ON manual_payment_requests (status); """ ) - # ========================================================================= - # BALANCE ASSERTIONS TABLE - # ========================================================================= - # Reconciliation and balance checking at specific dates - # Supports multi-currency (satoshis + fiat) with tolerance checking +async def m005_add_flag_and_meta(db): + """ + Add flag and meta columns to journal_entries table. + - flag: Transaction status (* = cleared, ! = pending, # = flagged, x = void) + - meta: JSON metadata for audit trail (source, tags, links, notes) + """ + await db.execute( + """ + ALTER TABLE journal_entries ADD COLUMN flag TEXT DEFAULT '*'; + """ + ) + + await db.execute( + """ + ALTER TABLE journal_entries ADD COLUMN meta TEXT DEFAULT '{}'; + """ + ) + + +async def m006_hierarchical_account_names(db): + """ + Migrate account names to hierarchical Beancount-style format. + - "Cash" → "Assets:Cash" + - "Accounts Receivable" → "Assets:Receivable" + - "Food & Supplies" → "Expenses:Food:Supplies" + - "Accounts Receivable - af983632" → "Assets:Receivable:User-af983632" + """ + from .account_utils import migrate_account_name + from .models import AccountType + + # Get all existing accounts + accounts = await db.fetchall("SELECT * FROM accounts") + + # Mapping of old names to new names + name_mappings = { + # Assets + "cash": "Assets:Cash", + "bank": "Assets:Bank", + "lightning": "Assets:Bitcoin:Lightning", + "accounts_receivable": "Assets:Receivable", + + # Liabilities + "accounts_payable": "Liabilities:Payable", + + # Equity + "member_equity": "Equity:MemberEquity", + "retained_earnings": "Equity:RetainedEarnings", + + # Revenue → Income + "accommodation_revenue": "Income:Accommodation", + "service_revenue": "Income:Service", + "other_revenue": "Income:Other", + + # Expenses + "utilities": "Expenses:Utilities", + "food": "Expenses:Food:Supplies", + "maintenance": "Expenses:Maintenance", + "other_expense": "Expenses:Other", + } + + # Update default accounts using ID-based mapping + for old_id, new_name in name_mappings.items(): + await db.execute( + """ + UPDATE accounts + SET name = :new_name + WHERE id = :old_id + """, + {"new_name": new_name, "old_id": old_id} + ) + + # Update user-specific accounts (those with user_id set) + user_accounts = await db.fetchall( + "SELECT * FROM accounts WHERE user_id IS NOT NULL" + ) + + for account in user_accounts: + # Parse account type + account_type = AccountType(account["account_type"]) + + # Migrate name + new_name = migrate_account_name(account["name"], account_type) + + await db.execute( + """ + UPDATE accounts + SET name = :new_name + WHERE id = :id + """, + {"new_name": new_name, "id": account["id"]} + ) + + +async def m007_balance_assertions(db): + """ + Create balance_assertions table for reconciliation. + Allows admins to assert expected balances at specific dates. + """ await db.execute( f""" CREATE TABLE balance_assertions ( @@ -176,7 +292,6 @@ async def m001_initial(db): checked_balance_fiat TEXT, difference_sats INTEGER, difference_fiat TEXT, - notes TEXT, status TEXT NOT NULL DEFAULT 'pending', created_by TEXT NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, @@ -188,134 +303,54 @@ async def m001_initial(db): await db.execute( """ - CREATE INDEX idx_balance_assertions_account_id - ON balance_assertions (account_id); + CREATE INDEX idx_balance_assertions_account_id ON balance_assertions (account_id); """ ) await db.execute( """ - CREATE INDEX idx_balance_assertions_status - ON balance_assertions (status); + CREATE INDEX idx_balance_assertions_status ON balance_assertions (status); """ ) await db.execute( """ - CREATE INDEX idx_balance_assertions_date - ON balance_assertions (date); + CREATE INDEX idx_balance_assertions_date ON balance_assertions (date); """ ) - # ========================================================================= - # USER EQUITY STATUS TABLE - # ========================================================================= - # Manages equity contribution eligibility for users - # Equity-eligible users can convert expenses to equity contributions - # Creates dynamic user-specific equity accounts: Equity:User-{user_id} - - await db.execute( - f""" - CREATE TABLE user_equity_status ( - user_id TEXT PRIMARY KEY, - is_equity_eligible BOOLEAN NOT NULL DEFAULT FALSE, - equity_account_name TEXT, - notes TEXT, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - revoked_at TIMESTAMP - ); - """ - ) +async def m008_rename_lightning_account(db): + """ + Rename Lightning account from Assets:Lightning:Balance to Assets:Bitcoin:Lightning + for better naming consistency. + """ await db.execute( """ - CREATE INDEX idx_user_equity_status_eligible - ON user_equity_status (is_equity_eligible) - WHERE is_equity_eligible = TRUE; + UPDATE accounts + SET name = 'Assets:Bitcoin:Lightning' + WHERE name = 'Assets:Lightning:Balance' """ ) - # ========================================================================= - # ACCOUNT PERMISSIONS TABLE - # ========================================================================= - # Granular access control for accounts - # Permission types: read, submit_expense, manage - # Supports hierarchical inheritance (parent account permissions cascade) - - await db.execute( - f""" - CREATE TABLE account_permissions ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - account_id TEXT NOT NULL, - permission_type TEXT NOT NULL, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - expires_at TIMESTAMP, - notes TEXT, - FOREIGN KEY (account_id) REFERENCES accounts (id) - ); - """ - ) - - # Index for looking up permissions by user - await db.execute( - """ - CREATE INDEX idx_account_permissions_user_id - ON account_permissions (user_id); - """ - ) - - # Index for looking up permissions by account - await db.execute( - """ - CREATE INDEX idx_account_permissions_account_id - ON account_permissions (account_id); - """ - ) - - # Composite index for checking specific user+account permissions - await db.execute( - """ - CREATE INDEX idx_account_permissions_user_account - ON account_permissions (user_id, account_id); - """ - ) - - # Index for finding permissions by type - await db.execute( - """ - CREATE INDEX idx_account_permissions_type - ON account_permissions (permission_type); - """ - ) - - # Index for finding expired permissions - await db.execute( - """ - CREATE INDEX idx_account_permissions_expires - ON account_permissions (expires_at) - WHERE expires_at IS NOT NULL; - """ - ) - - # ========================================================================= - # DEFAULT CHART OF ACCOUNTS - # ========================================================================= - # Insert comprehensive default accounts with hierarchical names. - # These accounts cover common use cases and can be extended by admins. - # - # Note: User-specific accounts (e.g., Assets:Receivable:User-xxx) are - # created dynamically when users interact with the system. - # - # Note: Equity accounts (Equity:User-xxx) are created dynamically when - # admins grant equity eligibility to users. +async def m009_add_onchain_bitcoin_account(db): + """ + Add Assets:Bitcoin:OnChain account for on-chain Bitcoin transactions. + This allows tracking on-chain Bitcoin separately from Lightning Network payments. + """ import uuid - from .account_utils import DEFAULT_HIERARCHICAL_ACCOUNTS - for name, account_type, description in DEFAULT_HIERARCHICAL_ACCOUNTS: + # Check if the account already exists + existing = await db.fetchone( + """ + SELECT id FROM accounts + WHERE name = 'Assets:Bitcoin:OnChain' + """ + ) + + if not existing: + # Create the on-chain Bitcoin asset account await db.execute( f""" INSERT INTO accounts (id, name, account_type, description, created_at) @@ -323,275 +358,8 @@ async def m001_initial(db): """, { "id": str(uuid.uuid4()), - "name": name, - "type": account_type.value, - "description": description + "name": "Assets:Bitcoin:OnChain", + "type": "asset", + "description": "On-chain Bitcoin wallet" } ) - - -async def m002_add_account_is_active(db): - """ - Add is_active field to accounts table for soft delete functionality. - - This enables marking accounts as inactive when they're removed from Beancount - while preserving historical data and permissions. Inactive accounts: - - Cannot have new permissions granted - - Are filtered out of default queries - - Can be reactivated if account is re-added to Beancount - - Default: All existing accounts are marked as active (TRUE). - """ - await db.execute( - """ - ALTER TABLE accounts - ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT TRUE - """ - ) - - # Create index for faster queries filtering by is_active - await db.execute( - """ - CREATE INDEX idx_accounts_is_active ON accounts (is_active) - """ - ) - - -async def m003_add_account_is_virtual(db): - """ - Add is_virtual field to accounts table for virtual parent accounts. - - Virtual parent accounts: - - Exist only in Castle DB (metadata-only, not in Beancount) - - Used solely for permission inheritance - - Allow granting permissions on top-level accounts like "Expenses", "Assets" - - Are not synced to/from Beancount - - Cannot be deactivated by account sync (they're intentionally metadata-only) - - Use case: Grant permission on "Expenses" → user gets access to all Expenses:* children - - Default: All existing accounts are real (is_virtual = FALSE). - """ - await db.execute( - """ - ALTER TABLE accounts - ADD COLUMN is_virtual BOOLEAN NOT NULL DEFAULT FALSE - """ - ) - - # Create index for faster queries filtering by is_virtual - await db.execute( - """ - CREATE INDEX idx_accounts_is_virtual ON accounts (is_virtual) - """ - ) - - # Insert default virtual parent accounts for permission management - import uuid - - virtual_parents = [ - ("Assets", "asset", "All asset accounts"), - ("Liabilities", "liability", "All liability accounts"), - ("Equity", "equity", "All equity accounts"), - ("Income", "revenue", "All income accounts"), - ("Expenses", "expense", "All expense accounts"), - ] - - for name, account_type, description in virtual_parents: - await db.execute( - f""" - INSERT INTO accounts (id, name, account_type, description, is_active, is_virtual, created_at) - VALUES (:id, :name, :type, :description, TRUE, TRUE, {db.timestamp_now}) - """, - { - "id": str(uuid.uuid4()), - "name": name, - "type": account_type, - "description": description, - }, - ) - - -async def m004_add_rbac_tables(db): - """ - Add Role-Based Access Control (RBAC) tables. - - This migration introduces a flexible RBAC system that complements - the existing individual permission grants: - - - Roles: Named bundles of permissions (Employee, Contractor, Admin, etc.) - - Role Permissions: Define what accounts each role can access - - User Roles: Assign users to roles - - Default Role: Auto-assign new users to a default role - - Permission Resolution Order: - 1. Individual account_permissions (exceptions/overrides) - 2. Role-based permissions via user_roles - 3. Inherited permissions (hierarchical account names) - 4. Deny by default - """ - - # ========================================================================= - # ROLES TABLE - # ========================================================================= - # Define named roles (Employee, Contractor, Admin, etc.) - - await db.execute( - f""" - CREATE TABLE roles ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - description TEXT, - is_default BOOLEAN NOT NULL DEFAULT FALSE, - created_by TEXT NOT NULL, - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now} - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_roles_name ON roles (name); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_roles_is_default ON roles (is_default) - WHERE is_default = TRUE; - """ - ) - - # ========================================================================= - # ROLE PERMISSIONS TABLE - # ========================================================================= - # Define which accounts each role can access and with what permission type - - await db.execute( - f""" - CREATE TABLE role_permissions ( - id TEXT PRIMARY KEY, - role_id TEXT NOT NULL, - account_id TEXT NOT NULL, - permission_type TEXT NOT NULL, - notes TEXT, - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - FOREIGN KEY (role_id) REFERENCES roles (id) ON DELETE CASCADE, - FOREIGN KEY (account_id) REFERENCES accounts (id) ON DELETE CASCADE - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_role_permissions_role_id ON role_permissions (role_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_role_permissions_account_id ON role_permissions (account_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_role_permissions_type ON role_permissions (permission_type); - """ - ) - - # ========================================================================= - # USER ROLES TABLE - # ========================================================================= - # Assign users to roles - - await db.execute( - f""" - CREATE TABLE user_roles ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - role_id TEXT NOT NULL, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - expires_at TIMESTAMP, - notes TEXT, - FOREIGN KEY (role_id) REFERENCES roles (id) ON DELETE CASCADE - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_user_roles_user_id ON user_roles (user_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_user_roles_role_id ON user_roles (role_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_user_roles_expires ON user_roles (expires_at) - WHERE expires_at IS NOT NULL; - """ - ) - - # Composite index for checking specific user+role assignments - await db.execute( - """ - CREATE INDEX idx_user_roles_user_role ON user_roles (user_id, role_id); - """ - ) - - # ========================================================================= - # CREATE DEFAULT ROLES - # ========================================================================= - # Insert standard roles that most organizations will use - - import uuid - - # Define default roles and their descriptions - default_roles = [ - ( - "employee", - "Employee", - "Standard employee role with access to common expense accounts", - True, # This is the default role for new users - ), - ( - "contractor", - "Contractor", - "External contractor with limited expense account access", - False, - ), - ( - "accountant", - "Accountant", - "Accounting staff with read access to financial accounts", - False, - ), - ( - "manager", - "Manager", - "Management role with broader expense approval and account access", - False, - ), - ] - - for slug, name, description, is_default in default_roles: - await db.execute( - f""" - INSERT INTO roles (id, name, description, is_default, created_by, created_at) - VALUES (:id, :name, :description, :is_default, :created_by, {db.timestamp_now}) - """, - { - "id": str(uuid.uuid4()), - "name": name, - "description": description, - "is_default": is_default, - "created_by": "system", # System-created default roles - }, - ) diff --git a/migrations_old.py.bak b/migrations_old.py.bak deleted file mode 100644 index a412e3e..0000000 --- a/migrations_old.py.bak +++ /dev/null @@ -1,651 +0,0 @@ -async def m001_initial(db): - """ - Initial migration for Castle accounting extension. - Creates tables for double-entry bookkeeping system. - """ - await db.execute( - f""" - CREATE TABLE accounts ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - account_type TEXT NOT NULL, - description TEXT, - user_id TEXT, - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now} - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_accounts_user_id ON accounts (user_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_accounts_type ON accounts (account_type); - """ - ) - - await db.execute( - f""" - CREATE TABLE journal_entries ( - id TEXT PRIMARY KEY, - description TEXT NOT NULL, - entry_date TIMESTAMP NOT NULL, - created_by TEXT NOT NULL, - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - reference TEXT - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_journal_entries_created_by ON journal_entries (created_by); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_journal_entries_date ON journal_entries (entry_date); - """ - ) - - await db.execute( - f""" - CREATE TABLE entry_lines ( - id TEXT PRIMARY KEY, - journal_entry_id TEXT NOT NULL, - account_id TEXT NOT NULL, - debit INTEGER NOT NULL DEFAULT 0, - credit INTEGER NOT NULL DEFAULT 0, - description TEXT, - metadata TEXT DEFAULT '{{}}' - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_entry_lines_journal_entry ON entry_lines (journal_entry_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_entry_lines_account ON entry_lines (account_id); - """ - ) - - # Insert default chart of accounts - default_accounts = [ - # Assets - ("cash", "Cash", "asset", "Cash on hand"), - ("bank", "Bank Account", "asset", "Bank account"), - ("lightning", "Lightning Balance", "asset", "Lightning Network balance"), - ("accounts_receivable", "Accounts Receivable", "asset", "Money owed to the Castle"), - - # Liabilities - ("accounts_payable", "Accounts Payable", "liability", "Money owed by the Castle"), - - # Equity - ("member_equity", "Member Equity", "equity", "Member contributions"), - ("retained_earnings", "Retained Earnings", "equity", "Accumulated profits"), - - # Revenue - ("accommodation_revenue", "Accommodation Revenue", "revenue", "Revenue from stays"), - ("service_revenue", "Service Revenue", "revenue", "Revenue from services"), - ("other_revenue", "Other Revenue", "revenue", "Other revenue"), - - # Expenses - ("utilities", "Utilities", "expense", "Electricity, water, internet"), - ("food", "Food & Supplies", "expense", "Food and supplies"), - ("maintenance", "Maintenance", "expense", "Repairs and maintenance"), - ("other_expense", "Other Expenses", "expense", "Miscellaneous expenses"), - ] - - for acc_id, name, acc_type, desc in default_accounts: - await db.execute( - """ - INSERT INTO accounts (id, name, account_type, description) - VALUES (:id, :name, :type, :description) - """, - {"id": acc_id, "name": name, "type": acc_type, "description": desc} - ) - - -async def m002_extension_settings(db): - """ - Create extension_settings table for Castle configuration. - """ - await db.execute( - f""" - CREATE TABLE extension_settings ( - id TEXT NOT NULL PRIMARY KEY, - castle_wallet_id TEXT, - updated_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now} - ); - """ - ) - - -async def m003_user_wallet_settings(db): - """ - Create user_wallet_settings table for per-user wallet configuration. - """ - await db.execute( - f""" - CREATE TABLE user_wallet_settings ( - id TEXT NOT NULL PRIMARY KEY, - user_wallet_id TEXT, - updated_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now} - ); - """ - ) - - -async def m004_manual_payment_requests(db): - """ - Create manual_payment_requests table for user payment requests to Castle. - """ - await db.execute( - f""" - CREATE TABLE manual_payment_requests ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - amount INTEGER NOT NULL, - description TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending', - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - reviewed_at TIMESTAMP, - reviewed_by TEXT, - journal_entry_id TEXT - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_manual_payment_requests_user_id ON manual_payment_requests (user_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_manual_payment_requests_status ON manual_payment_requests (status); - """ - ) - - -async def m005_add_flag_and_meta(db): - """ - Add flag and meta columns to journal_entries table. - - flag: Transaction status (* = cleared, ! = pending, # = flagged, x = void) - - meta: JSON metadata for audit trail (source, tags, links, notes) - """ - await db.execute( - """ - ALTER TABLE journal_entries ADD COLUMN flag TEXT DEFAULT '*'; - """ - ) - - await db.execute( - """ - ALTER TABLE journal_entries ADD COLUMN meta TEXT DEFAULT '{}'; - """ - ) - - -async def m006_hierarchical_account_names(db): - """ - Migrate account names to hierarchical Beancount-style format. - - "Cash" → "Assets:Cash" - - "Accounts Receivable" → "Assets:Receivable" - - "Food & Supplies" → "Expenses:Food:Supplies" - - "Accounts Receivable - af983632" → "Assets:Receivable:User-af983632" - """ - from .account_utils import migrate_account_name - from .models import AccountType - - # Get all existing accounts - accounts = await db.fetchall("SELECT * FROM accounts") - - # Mapping of old names to new names - name_mappings = { - # Assets - "cash": "Assets:Cash", - "bank": "Assets:Bank", - "lightning": "Assets:Bitcoin:Lightning", - "accounts_receivable": "Assets:Receivable", - - # Liabilities - "accounts_payable": "Liabilities:Payable", - - # Equity - "member_equity": "Equity:MemberEquity", - "retained_earnings": "Equity:RetainedEarnings", - - # Revenue → Income - "accommodation_revenue": "Income:Accommodation", - "service_revenue": "Income:Service", - "other_revenue": "Income:Other", - - # Expenses - "utilities": "Expenses:Utilities", - "food": "Expenses:Food:Supplies", - "maintenance": "Expenses:Maintenance", - "other_expense": "Expenses:Other", - } - - # Update default accounts using ID-based mapping - for old_id, new_name in name_mappings.items(): - await db.execute( - """ - UPDATE accounts - SET name = :new_name - WHERE id = :old_id - """, - {"new_name": new_name, "old_id": old_id} - ) - - # Update user-specific accounts (those with user_id set) - user_accounts = await db.fetchall( - "SELECT * FROM accounts WHERE user_id IS NOT NULL" - ) - - for account in user_accounts: - # Parse account type - account_type = AccountType(account["account_type"]) - - # Migrate name - new_name = migrate_account_name(account["name"], account_type) - - await db.execute( - """ - UPDATE accounts - SET name = :new_name - WHERE id = :id - """, - {"new_name": new_name, "id": account["id"]} - ) - - -async def m007_balance_assertions(db): - """ - Create balance_assertions table for reconciliation. - Allows admins to assert expected balances at specific dates. - """ - await db.execute( - f""" - CREATE TABLE balance_assertions ( - id TEXT PRIMARY KEY, - date TIMESTAMP NOT NULL, - account_id TEXT NOT NULL, - expected_balance_sats INTEGER NOT NULL, - expected_balance_fiat TEXT, - fiat_currency TEXT, - tolerance_sats INTEGER DEFAULT 0, - tolerance_fiat TEXT DEFAULT '0', - checked_balance_sats INTEGER, - checked_balance_fiat TEXT, - difference_sats INTEGER, - difference_fiat TEXT, - status TEXT NOT NULL DEFAULT 'pending', - created_by TEXT NOT NULL, - created_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - checked_at TIMESTAMP, - FOREIGN KEY (account_id) REFERENCES accounts (id) - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_balance_assertions_account_id ON balance_assertions (account_id); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_balance_assertions_status ON balance_assertions (status); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_balance_assertions_date ON balance_assertions (date); - """ - ) - - -async def m008_rename_lightning_account(db): - """ - Rename Lightning account from Assets:Lightning:Balance to Assets:Bitcoin:Lightning - for better naming consistency. - """ - await db.execute( - """ - UPDATE accounts - SET name = 'Assets:Bitcoin:Lightning' - WHERE name = 'Assets:Lightning:Balance' - """ - ) - - -async def m009_add_onchain_bitcoin_account(db): - """ - Add Assets:Bitcoin:OnChain account for on-chain Bitcoin transactions. - This allows tracking on-chain Bitcoin separately from Lightning Network payments. - """ - import uuid - - # Check if the account already exists - existing = await db.fetchone( - """ - SELECT id FROM accounts - WHERE name = 'Assets:Bitcoin:OnChain' - """ - ) - - if not existing: - # Create the on-chain Bitcoin asset account - await db.execute( - f""" - INSERT INTO accounts (id, name, account_type, description, created_at) - VALUES (:id, :name, :type, :description, {db.timestamp_now}) - """, - { - "id": str(uuid.uuid4()), - "name": "Assets:Bitcoin:OnChain", - "type": "asset", - "description": "On-chain Bitcoin wallet" - } - ) - - -async def m010_user_equity_status(db): - """ - Create user_equity_status table for managing equity contribution eligibility. - Only equity-eligible users can convert their expenses to equity contributions. - """ - await db.execute( - f""" - CREATE TABLE user_equity_status ( - user_id TEXT PRIMARY KEY, - is_equity_eligible BOOLEAN NOT NULL DEFAULT FALSE, - equity_account_name TEXT, - notes TEXT, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - revoked_at TIMESTAMP - ); - """ - ) - - await db.execute( - """ - CREATE INDEX idx_user_equity_status_eligible - ON user_equity_status (is_equity_eligible) - WHERE is_equity_eligible = TRUE; - """ - ) - - -async def m011_account_permissions(db): - """ - Create account_permissions table for granular account access control. - Allows admins to grant specific permissions (read, submit_expense, manage) to users for specific accounts. - Supports hierarchical permission inheritance (permissions on parent accounts cascade to children). - """ - await db.execute( - f""" - CREATE TABLE account_permissions ( - id TEXT PRIMARY KEY, - user_id TEXT NOT NULL, - account_id TEXT NOT NULL, - permission_type TEXT NOT NULL, - granted_by TEXT NOT NULL, - granted_at TIMESTAMP NOT NULL DEFAULT {db.timestamp_now}, - expires_at TIMESTAMP, - notes TEXT, - FOREIGN KEY (account_id) REFERENCES accounts (id) - ); - """ - ) - - # Index for looking up permissions by user - await db.execute( - """ - CREATE INDEX idx_account_permissions_user_id ON account_permissions (user_id); - """ - ) - - # Index for looking up permissions by account - await db.execute( - """ - CREATE INDEX idx_account_permissions_account_id ON account_permissions (account_id); - """ - ) - - # Composite index for checking specific user+account permissions - await db.execute( - """ - CREATE INDEX idx_account_permissions_user_account - ON account_permissions (user_id, account_id); - """ - ) - - # Index for finding permissions by type - await db.execute( - """ - CREATE INDEX idx_account_permissions_type ON account_permissions (permission_type); - """ - ) - - # Index for finding expired permissions - await db.execute( - """ - CREATE INDEX idx_account_permissions_expires - ON account_permissions (expires_at) - WHERE expires_at IS NOT NULL; - """ - ) - - -async def m012_update_default_accounts(db): - """ - Update default chart of accounts to include more detailed hierarchical structure. - Adds new accounts for fixed assets, livestock, equity contributions, and detailed expenses. - Only adds accounts that don't already exist. - """ - import uuid - from .account_utils import DEFAULT_HIERARCHICAL_ACCOUNTS - - for name, account_type, description in DEFAULT_HIERARCHICAL_ACCOUNTS: - # Check if account already exists - existing = await db.fetchone( - """ - SELECT id FROM accounts WHERE name = :name - """, - {"name": name} - ) - - if not existing: - # Create new account - await db.execute( - f""" - INSERT INTO accounts (id, name, account_type, description, created_at) - VALUES (:id, :name, :type, :description, {db.timestamp_now}) - """, - { - "id": str(uuid.uuid4()), - "name": name, - "type": account_type.value, - "description": description - } - ) - - -async def m013_remove_parent_only_accounts(db): - """ - Remove parent-only accounts from the database. - - Since Castle doesn't interface directly with Beancount (only exports to it), - we don't need parent accounts that exist only for organizational hierarchy. - The hierarchy is implicit in the colon-separated account names. - - When exporting to Beancount, the parent accounts will be inferred from the - hierarchical naming (e.g., "Assets:Bitcoin:Lightning" implies "Assets:Bitcoin" exists). - - This keeps our database clean and prevents accidentally posting to parent accounts. - - Removes: - - Assets:Bitcoin (parent of Lightning and OnChain) - - Equity (parent of user equity accounts like Equity:User-xxx) - """ - # Remove Assets:Bitcoin (parent account) - await db.execute( - "DELETE FROM accounts WHERE name = :name", - {"name": "Assets:Bitcoin"} - ) - - # Remove Equity (parent account) - await db.execute( - "DELETE FROM accounts WHERE name = :name", - {"name": "Equity"} - ) - - -async def m014_remove_legacy_equity_accounts(db): - """ - Remove legacy generic equity accounts that don't fit the user-specific equity model. - - The castle extension uses dynamic user-specific equity accounts (Equity:User-{user_id}) - created automatically when granting equity eligibility. Generic equity accounts like - MemberEquity and RetainedEarnings are not needed. - - Removes: - - Equity:MemberEquity - - Equity:RetainedEarnings - """ - # Remove Equity:MemberEquity - await db.execute( - "DELETE FROM accounts WHERE name = :name", - {"name": "Equity:MemberEquity"} - ) - - # Remove Equity:RetainedEarnings - await db.execute( - "DELETE FROM accounts WHERE name = :name", - {"name": "Equity:RetainedEarnings"} - ) - - -async def m015_convert_to_single_amount_field(db): - """ - Convert entry_lines from separate debit/credit columns to single amount field. - - This aligns Castle with Beancount's elegant design: - - Positive amount = debit (increase assets/expenses, decrease liabilities/equity/revenue) - - Negative amount = credit (decrease assets/expenses, increase liabilities/equity/revenue) - - Benefits: - - Simpler model (one field instead of two) - - Direct compatibility with Beancount import/export - - Eliminates invalid states (both debit and credit non-zero) - - More intuitive for programmers (positive/negative instead of accounting conventions) - - Migration formula: amount = debit - credit - - Examples: - - Expense transaction: - * Expenses:Food:Groceries amount=+100 (debit) - * Liabilities:Payable:User amount=-100 (credit) - - Payment transaction: - * Liabilities:Payable:User amount=+100 (debit) - * Assets:Bitcoin:Lightning amount=-100 (credit) - """ - from sqlalchemy.exc import OperationalError - - # Step 1: Add new amount column (nullable for migration) - try: - await db.execute( - "ALTER TABLE entry_lines ADD COLUMN amount INTEGER" - ) - except OperationalError: - # Column might already exist if migration was partially run - pass - - # Step 2: Populate amount from existing debit/credit - # Formula: amount = debit - credit - await db.execute( - """ - UPDATE entry_lines - SET amount = debit - credit - WHERE amount IS NULL - """ - ) - - # Step 3: Create new table with amount field as NOT NULL - # SQLite doesn't support ALTER COLUMN, so we need to recreate the table - await db.execute( - """ - CREATE TABLE entry_lines_new ( - id TEXT PRIMARY KEY, - journal_entry_id TEXT NOT NULL, - account_id TEXT NOT NULL, - amount INTEGER NOT NULL, - description TEXT, - metadata TEXT DEFAULT '{}' - ) - """ - ) - - # Step 4: Copy data from old table to new - await db.execute( - """ - INSERT INTO entry_lines_new (id, journal_entry_id, account_id, amount, description, metadata) - SELECT id, journal_entry_id, account_id, amount, description, metadata - FROM entry_lines - """ - ) - - # Step 5: Drop old table and rename new one - await db.execute("DROP TABLE entry_lines") - await db.execute("ALTER TABLE entry_lines_new RENAME TO entry_lines") - - # Step 6: Recreate indexes - await db.execute( - """ - CREATE INDEX idx_entry_lines_journal_entry ON entry_lines (journal_entry_id) - """ - ) - - await db.execute( - """ - CREATE INDEX idx_entry_lines_account ON entry_lines (account_id) - """ - ) - - -async def m016_drop_obsolete_journal_tables(db): - """ - Drop journal_entries and entry_lines tables. - - Castle now uses Fava/Beancount as the single source of truth for accounting data. - These tables are no longer written to or read from. - - All journal entry operations now: - - Write: Submit to Fava via FavaClient.add_entry() - - Read: Query Fava via FavaClient.get_entries() - - Migration completed as part of Castle extension cleanup (Nov 2025). - No backwards compatibility concerns - user explicitly approved. - """ - # Drop entry_lines first (has foreign key to journal_entries) - await db.execute("DROP TABLE IF EXISTS entry_lines") - - # Drop journal_entries - await db.execute("DROP TABLE IF EXISTS journal_entries") diff --git a/models.py b/models.py index 5199b6d..ffde1c6 100644 --- a/models.py +++ b/models.py @@ -15,18 +15,11 @@ class AccountType(str, Enum): class JournalEntryFlag(str, Enum): - """Transaction status flags (Beancount-compatible) - - Beancount only supports two user-facing flags: - - * (CLEARED): Completed transactions - - ! (PENDING): Transactions needing attention - - For voided/flagged transactions, use tags instead: - - Voided: Use "!" flag + #voided tag - - Flagged: Use "!" flag + #review tag - """ + """Transaction status flags (Beancount-style)""" CLEARED = "*" # Fully reconciled/confirmed PENDING = "!" # Not yet confirmed/awaiting approval + FLAGGED = "#" # Needs review/attention + VOID = "x" # Voided/cancelled entry class Account(BaseModel): @@ -36,8 +29,6 @@ class Account(BaseModel): description: Optional[str] = None user_id: Optional[str] = None # For user-specific accounts created_at: datetime - is_active: bool = True # Soft delete flag - is_virtual: bool = False # Virtual parent account (metadata-only, not in Beancount) class CreateAccount(BaseModel): @@ -45,21 +36,22 @@ class CreateAccount(BaseModel): account_type: AccountType description: Optional[str] = None user_id: Optional[str] = None - is_virtual: bool = False # Set to True to create virtual parent account class EntryLine(BaseModel): id: str journal_entry_id: str account_id: str - amount: int # in satoshis; positive = debit, negative = credit + debit: int = 0 # in satoshis + credit: int = 0 # in satoshis description: Optional[str] = None metadata: dict = {} # Stores currency info: fiat_currency, fiat_amount, fiat_rate, etc. class CreateEntryLine(BaseModel): account_id: str - amount: int # in satoshis; positive = debit, negative = credit + debit: int = 0 + credit: int = 0 description: Optional[str] = None metadata: dict = {} # Stores currency info @@ -131,12 +123,6 @@ class CastleSettings(BaseModel): """Settings for the Castle extension""" castle_wallet_id: Optional[str] = None # The wallet ID that represents the Castle - - # Fava/Beancount integration - ALL accounting is done via Fava - fava_url: str = "http://localhost:3333" # Base URL of Fava server - fava_ledger_slug: str = "castle-ledger" # Ledger identifier in Fava URL - fava_timeout: float = 10.0 # Request timeout in seconds - updated_at: datetime = Field(default_factory=lambda: datetime.now()) @classmethod @@ -261,172 +247,3 @@ class CreateBalanceAssertion(BaseModel): fiat_currency: Optional[str] = None tolerance_sats: int = 0 tolerance_fiat: Decimal = Decimal("0") - - -class UserEquityStatus(BaseModel): - """Tracks user's equity eligibility and status""" - user_id: str # User's wallet ID - is_equity_eligible: bool # Can user convert expenses to equity? - equity_account_name: Optional[str] = None # e.g., "Equity:Alice" - notes: Optional[str] = None # Admin notes - granted_by: str # Admin who granted eligibility - granted_at: datetime - revoked_at: Optional[datetime] = None # If eligibility was revoked - - -class CreateUserEquityStatus(BaseModel): - """Create or update user equity eligibility""" - user_id: str - is_equity_eligible: bool - equity_account_name: Optional[str] = None # Auto-generated as "Equity:User-{user_id}" if not provided - notes: Optional[str] = None - - -class UserInfo(BaseModel): - """User information including equity eligibility""" - user_id: str - is_equity_eligible: bool - equity_account_name: Optional[str] = None - - -class PermissionType(str, Enum): - """Types of permissions for account access""" - READ = "read" # Can view account and its balance - SUBMIT_EXPENSE = "submit_expense" # Can submit expenses to this account - MANAGE = "manage" # Can modify account (admin level) - - -class AccountPermission(BaseModel): - """Defines which accounts a user can access""" - id: str # Unique permission ID - user_id: str # User's wallet ID (from invoice key) - account_id: str # Account ID from accounts table - permission_type: PermissionType - granted_by: str # Admin user ID who granted permission - granted_at: datetime - expires_at: Optional[datetime] = None # Optional expiration - notes: Optional[str] = None # Admin notes about this permission - - -class CreateAccountPermission(BaseModel): - """Create account permission""" - user_id: str - account_id: str - permission_type: PermissionType - expires_at: Optional[datetime] = None - notes: Optional[str] = None - - -class BulkGrantPermission(BaseModel): - """Bulk grant same permission to multiple users""" - user_ids: list[str] # List of user IDs to grant permission to - account_id: str # Account to grant permission on - permission_type: PermissionType # Type of permission to grant - expires_at: Optional[datetime] = None # Optional expiration - notes: Optional[str] = None # Notes for all permissions - - -class BulkGrantResult(BaseModel): - """Result of bulk grant operation""" - granted: list[AccountPermission] # Successfully granted permissions - failed: list[dict] # Failed grants with errors - total: int # Total attempted - success_count: int # Number of successful grants - failure_count: int # Number of failed grants - - -class AccountWithPermissions(BaseModel): - """Account with user-specific permission metadata""" - id: str - name: str - account_type: AccountType - description: Optional[str] = None - user_id: Optional[str] = None - created_at: datetime - is_active: bool = True # Soft delete flag - is_virtual: bool = False # Virtual parent account (metadata-only) - # Only included when filter_by_user=true - user_permissions: Optional[list[PermissionType]] = None - inherited_from: Optional[str] = None # Parent account ID if inherited - # Hierarchical structure - parent_account: Optional[str] = None # Parent account name - level: Optional[int] = None # Depth in hierarchy (0 = top level) - has_children: Optional[bool] = None # Whether this account has sub-accounts - - -# ===== ROLE-BASED ACCESS CONTROL (RBAC) MODELS ===== - - -class Role(BaseModel): - """Role definition for RBAC system""" - id: str - name: str # Display name (e.g., "Employee", "Contractor") - description: Optional[str] = None - is_default: bool = False # Auto-assign this role to new users - created_by: str # User ID who created the role - created_at: datetime - - -class CreateRole(BaseModel): - """Create a new role""" - name: str - description: Optional[str] = None - is_default: bool = False - - -class UpdateRole(BaseModel): - """Update an existing role""" - name: Optional[str] = None - description: Optional[str] = None - is_default: Optional[bool] = None - - -class RolePermission(BaseModel): - """Permission granted to a role for a specific account""" - id: str - role_id: str - account_id: str - permission_type: PermissionType - notes: Optional[str] = None - created_at: datetime - - -class CreateRolePermission(BaseModel): - """Create a permission for a role""" - role_id: str - account_id: str - permission_type: PermissionType - notes: Optional[str] = None - - -class UserRole(BaseModel): - """Assignment of a user to a role""" - id: str - user_id: str # User's wallet ID - role_id: str - granted_by: str # Admin who assigned the role - granted_at: datetime - expires_at: Optional[datetime] = None - notes: Optional[str] = None - - -class AssignUserRole(BaseModel): - """Assign a user to a role""" - user_id: str - role_id: str - expires_at: Optional[datetime] = None - notes: Optional[str] = None - - -class RoleWithPermissions(BaseModel): - """Role with its associated permissions and user count""" - role: Role - permissions: list[RolePermission] - user_count: int # Number of users assigned to this role - - -class UserWithRoles(BaseModel): - """User information with their assigned roles""" - user_id: str - roles: list[Role] - direct_permissions: list[AccountPermission] # Individual permissions not from roles diff --git a/package.json b/package.json deleted file mode 100644 index f479115..0000000 --- a/package.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "name": "castle", - "version": "0.0.2", - "description": "Accounting for a collective entity", - "main": "index.js", - "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" - }, - "author": "", - "license": "ISC", - "dependencies": { - "prettier": "^3.2.5", - "pyright": "^1.1.358" - } -} diff --git a/permission_management.py b/permission_management.py deleted file mode 100644 index 7dea217..0000000 --- a/permission_management.py +++ /dev/null @@ -1,475 +0,0 @@ -""" -Bulk Permission Management Module - -Provides convenience functions for managing permissions at scale. - -Features: -- Bulk grant to multiple users -- Bulk revoke operations -- Permission templates/copying -- User offboarding -- Permission analytics - -Related: PERMISSIONS-SYSTEM.md - Improvement Opportunity #3 -""" - -from datetime import datetime -from typing import Optional -from loguru import logger - -from .crud import ( - create_account_permission, - delete_account_permission, - get_account_permissions, - get_user_permissions, - get_account, -) -from .models import ( - AccountPermission, - CreateAccountPermission, - PermissionType, -) - - -async def bulk_grant_permission( - user_ids: list[str], - account_id: str, - permission_type: PermissionType, - granted_by: str, - expires_at: Optional[datetime] = None, - notes: Optional[str] = None, -) -> dict: - """ - Grant the same permission to multiple users. - - Args: - user_ids: List of user IDs to grant permission to - account_id: Account to grant permission on - permission_type: Type of permission (READ, SUBMIT_EXPENSE, MANAGE) - granted_by: Admin user ID granting the permission - expires_at: Optional expiration date - notes: Optional notes about this bulk grant - - Returns: - dict with results: - { - "granted": 15, - "failed": 2, - "errors": ["user123: Already has permission", ...], - "permissions": [permission_obj, ...] - } - - Example: - # Grant submit_expense to all food team members - await bulk_grant_permission( - user_ids=["alice", "bob", "charlie"], - account_id="expenses_food_id", - permission_type=PermissionType.SUBMIT_EXPENSE, - granted_by="admin", - expires_at=datetime(2025, 12, 31), - notes="Q4 food team members" - ) - """ - logger.info( - f"Bulk granting {permission_type.value} permission to {len(user_ids)} users on account {account_id}" - ) - - # Verify account exists - account = await get_account(account_id) - if not account: - return { - "granted": 0, - "failed": len(user_ids), - "errors": [f"Account {account_id} not found"], - "permissions": [], - } - - granted = 0 - failed = 0 - errors = [] - permissions = [] - - for user_id in user_ids: - try: - permission = await create_account_permission( - data=CreateAccountPermission( - user_id=user_id, - account_id=account_id, - permission_type=permission_type, - expires_at=expires_at, - notes=notes, - ), - granted_by=granted_by, - ) - - permissions.append(permission) - granted += 1 - logger.debug(f"Granted {permission_type.value} to {user_id} on {account.name}") - - except Exception as e: - failed += 1 - error_msg = f"{user_id}: {str(e)}" - errors.append(error_msg) - logger.warning(f"Failed to grant permission to {user_id}: {e}") - - logger.info( - f"Bulk grant complete: {granted} granted, {failed} failed on account {account.name}" - ) - - return { - "granted": granted, - "failed": failed, - "errors": errors, - "permissions": permissions, - } - - -async def revoke_all_user_permissions(user_id: str) -> dict: - """ - Revoke ALL permissions for a user (offboarding). - - Args: - user_id: User ID to revoke all permissions from - - Returns: - dict with results: - { - "revoked": 5, - "failed": 0, - "errors": [], - "permission_types_removed": ["read", "submit_expense"] - } - - Example: - # Remove all access when user leaves - await revoke_all_user_permissions("departed_user") - """ - logger.info(f"Revoking ALL permissions for user {user_id}") - - permissions = await get_user_permissions(user_id) - - revoked = 0 - failed = 0 - errors = [] - permission_types = set() - - for perm in permissions: - try: - await delete_account_permission(perm.id) - revoked += 1 - permission_types.add(perm.permission_type.value) - logger.debug(f"Revoked {perm.permission_type.value} from {user_id}") - - except Exception as e: - failed += 1 - error_msg = f"{perm.id}: {str(e)}" - errors.append(error_msg) - logger.warning(f"Failed to revoke permission {perm.id}: {e}") - - logger.info(f"User offboarding complete: {revoked} permissions revoked for {user_id}") - - return { - "revoked": revoked, - "failed": failed, - "errors": errors, - "permission_types_removed": sorted(list(permission_types)), - } - - -async def revoke_all_permissions_on_account(account_id: str) -> dict: - """ - Revoke ALL permissions on an account (account closure). - - Args: - account_id: Account ID to revoke all permissions from - - Returns: - dict with results: - { - "revoked": 8, - "failed": 0, - "errors": [], - "users_affected": ["alice", "bob", "charlie"] - } - - Example: - # Close project and remove all access - await revoke_all_permissions_on_account("old_project_id") - """ - logger.info(f"Revoking ALL permissions on account {account_id}") - - permissions = await get_account_permissions(account_id) - - revoked = 0 - failed = 0 - errors = [] - users_affected = set() - - for perm in permissions: - try: - await delete_account_permission(perm.id) - revoked += 1 - users_affected.add(perm.user_id) - logger.debug(f"Revoked permission from {perm.user_id} on account") - - except Exception as e: - failed += 1 - error_msg = f"{perm.id}: {str(e)}" - errors.append(error_msg) - logger.warning(f"Failed to revoke permission {perm.id}: {e}") - - logger.info(f"Account closure complete: {revoked} permissions revoked") - - return { - "revoked": revoked, - "failed": failed, - "errors": errors, - "users_affected": sorted(list(users_affected)), - } - - -async def copy_permissions( - from_user_id: str, - to_user_id: str, - granted_by: str, - permission_types: Optional[list[PermissionType]] = None, - notes: Optional[str] = None, -) -> dict: - """ - Copy all permissions from one user to another (permission template). - - Args: - from_user_id: User to copy permissions from - to_user_id: User to copy permissions to - granted_by: Admin granting the new permissions - permission_types: Optional filter - only copy specific permission types - notes: Optional notes for the copied permissions - - Returns: - dict with results: - { - "copied": 5, - "failed": 0, - "errors": [], - "permissions": [permission_obj, ...] - } - - Example: - # Copy all submit_expense permissions from experienced user - await copy_permissions( - from_user_id="alice", - to_user_id="bob", - granted_by="admin", - permission_types=[PermissionType.SUBMIT_EXPENSE], - notes="Copied from Alice - new food coordinator" - ) - """ - logger.info(f"Copying permissions from {from_user_id} to {to_user_id}") - - # Get source user's permissions - source_permissions = await get_user_permissions(from_user_id) - - # Filter by permission type if specified - if permission_types: - source_permissions = [ - p for p in source_permissions if p.permission_type in permission_types - ] - - copied = 0 - failed = 0 - errors = [] - permissions = [] - - for source_perm in source_permissions: - try: - # Create new permission for target user - new_permission = await create_account_permission( - data=CreateAccountPermission( - user_id=to_user_id, - account_id=source_perm.account_id, - permission_type=source_perm.permission_type, - expires_at=source_perm.expires_at, # Copy expiration - notes=notes or f"Copied from {from_user_id}", - ), - granted_by=granted_by, - ) - - permissions.append(new_permission) - copied += 1 - logger.debug( - f"Copied {source_perm.permission_type.value} permission to {to_user_id}" - ) - - except Exception as e: - failed += 1 - error_msg = f"{source_perm.id}: {str(e)}" - errors.append(error_msg) - logger.warning(f"Failed to copy permission {source_perm.id}: {e}") - - logger.info(f"Permission copy complete: {copied} copied, {failed} failed") - - return { - "copied": copied, - "failed": failed, - "errors": errors, - "permissions": permissions, - } - - -async def get_permission_analytics() -> dict: - """ - Get analytics about permission usage (for admin dashboard). - - Returns: - dict with analytics: - { - "total_permissions": 150, - "by_type": {"read": 50, "submit_expense": 80, "manage": 20}, - "expiring_soon": [...], # Expire in next 7 days - "expired": [...], # Already expired but not cleaned up - "users_with_permissions": 45, - "users_without_permissions": ["bob", ...], - "most_permissioned_accounts": [...] - } - - Example: - stats = await get_permission_analytics() - print(f"Total permissions: {stats['total_permissions']}") - """ - from datetime import timedelta - from . import db - - logger.debug("Gathering permission analytics") - - # Total permissions - total_result = await db.fetchone("SELECT COUNT(*) as count FROM account_permissions") - total_permissions = total_result["count"] if total_result else 0 - - # By type - type_result = await db.fetchall( - """ - SELECT permission_type, COUNT(*) as count - FROM account_permissions - GROUP BY permission_type - """ - ) - by_type = {row["permission_type"]: row["count"] for row in type_result} - - # Expiring soon (next 7 days) - seven_days_from_now = datetime.now() + timedelta(days=7) - expiring_result = await db.fetchall( - """ - SELECT ap.*, a.name as account_name - FROM account_permissions ap - JOIN castle_accounts a ON ap.account_id = a.id - WHERE ap.expires_at IS NOT NULL - AND ap.expires_at > :now - AND ap.expires_at <= :seven_days - ORDER BY ap.expires_at ASC - LIMIT 20 - """, - {"now": datetime.now(), "seven_days": seven_days_from_now}, - ) - - expiring_soon = [ - { - "user_id": row["user_id"], - "account_name": row["account_name"], - "permission_type": row["permission_type"], - "expires_at": row["expires_at"], - } - for row in expiring_result - ] - - # Most permissioned accounts - top_accounts_result = await db.fetchall( - """ - SELECT a.name, COUNT(ap.id) as permission_count - FROM castle_accounts a - LEFT JOIN account_permissions ap ON a.id = ap.account_id - GROUP BY a.id, a.name - HAVING COUNT(ap.id) > 0 - ORDER BY permission_count DESC - LIMIT 10 - """ - ) - - most_permissioned_accounts = [ - {"account": row["name"], "permission_count": row["permission_count"]} - for row in top_accounts_result - ] - - # Unique users with permissions - users_result = await db.fetchone( - "SELECT COUNT(DISTINCT user_id) as count FROM account_permissions" - ) - users_with_permissions = users_result["count"] if users_result else 0 - - return { - "total_permissions": total_permissions, - "by_type": by_type, - "expiring_soon": expiring_soon, - "users_with_permissions": users_with_permissions, - "most_permissioned_accounts": most_permissioned_accounts, - } - - -async def cleanup_expired_permissions(days_old: int = 30) -> dict: - """ - Clean up permissions that expired more than N days ago. - - Args: - days_old: Delete permissions expired this many days ago - - Returns: - dict with results: - { - "deleted": 15, - "errors": [] - } - - Example: - # Delete permissions expired more than 30 days ago - await cleanup_expired_permissions(days_old=30) - """ - from datetime import timedelta - from . import db - - logger.info(f"Cleaning up permissions expired more than {days_old} days ago") - - cutoff_date = datetime.now() - timedelta(days=days_old) - - try: - result = await db.execute( - """ - DELETE FROM account_permissions - WHERE expires_at IS NOT NULL - AND expires_at < :cutoff_date - """, - {"cutoff_date": cutoff_date}, - ) - - # SQLite doesn't return rowcount reliably, so count before delete - count_result = await db.fetchone( - """ - SELECT COUNT(*) as count FROM account_permissions - WHERE expires_at IS NOT NULL - AND expires_at < :cutoff_date - """, - {"cutoff_date": cutoff_date}, - ) - deleted = count_result["count"] if count_result else 0 - - logger.info(f"Cleaned up {deleted} expired permissions") - - return { - "deleted": deleted, - "errors": [], - } - - except Exception as e: - logger.error(f"Failed to cleanup expired permissions: {e}") - return { - "deleted": 0, - "errors": [str(e)], - } diff --git a/services.py b/services.py index 1f9d826..47a3d7b 100644 --- a/services.py +++ b/services.py @@ -2,12 +2,11 @@ from .crud import ( create_castle_settings, create_user_wallet_settings, get_castle_settings, - get_or_create_user_account, get_user_wallet_settings, update_castle_settings, update_user_wallet_settings, ) -from .models import AccountType, CastleSettings, UserWalletSettings +from .models import CastleSettings, UserWalletSettings async def get_settings(user_id: str) -> CastleSettings: @@ -37,28 +36,10 @@ async def get_user_wallet(user_id: str) -> UserWalletSettings: async def update_user_wallet( user_id: str, data: UserWalletSettings ) -> UserWalletSettings: - from loguru import logger - - logger.info(f"[WALLET UPDATE] Starting update_user_wallet for user {user_id[:8]}") - settings = await get_user_wallet_settings(user_id) if not settings: - logger.info(f"[WALLET UPDATE] Creating new wallet settings for user {user_id[:8]}") settings = await create_user_wallet_settings(user_id, data) else: - logger.info(f"[WALLET UPDATE] Updating existing wallet settings for user {user_id[:8]}") settings = await update_user_wallet_settings(user_id, data) - # Proactively create core user accounts when wallet is configured - # This ensures all users have a consistent account structure from the start - logger.info(f"[WALLET UPDATE] Creating LIABILITY account for user {user_id[:8]}") - await get_or_create_user_account( - user_id, AccountType.LIABILITY, "Accounts Payable" - ) - logger.info(f"[WALLET UPDATE] Creating ASSET account for user {user_id[:8]}") - await get_or_create_user_account( - user_id, AccountType.ASSET, "Accounts Receivable" - ) - logger.info(f"[WALLET UPDATE] Completed update_user_wallet for user {user_id[:8]}") - return settings diff --git a/static/js/index.js b/static/js/index.js index 318483b..2517657 100644 --- a/static/js/index.js +++ b/static/js/index.js @@ -3,32 +3,18 @@ const mapJournalEntry = obj => { } window.app = Vue.createApp({ + el: '#vue', mixins: [windowMixin], data() { return { balance: null, allUserBalances: [], transactions: [], - transactionPagination: { - total: 0, - limit: 10, - offset: 0, - has_next: false, - has_prev: false - }, - transactionFilter: { - user_id: null, // For filtering by user - account_type: null, // For filtering by receivable/payable (asset/liability) - dateRangeType: 15, // Preset days (15, 30, 60) or 'custom' - startDate: null, // For custom date range (YYYY-MM-DD) - endDate: null // For custom date range (YYYY-MM-DD) - }, accounts: [], currencies: [], users: [], settings: null, userWalletSettings: null, - userInfo: null, // User information including equity eligibility isAdmin: false, isSuperUser: false, castleWalletConfigured: false, @@ -189,25 +175,6 @@ window.app = Vue.createApp({ } }, computed: { - transactionColumns() { - return [ - { name: 'flag', label: 'Status', field: 'flag', align: 'left', sortable: true }, - { name: 'username', label: 'User', field: 'username', align: 'left', sortable: true }, - { name: 'date', label: 'Date', field: 'entry_date', align: 'left', sortable: true }, - { name: 'description', label: 'Description', field: 'description', align: 'left', sortable: false }, - { name: 'amount', label: 'Amount (sats)', field: 'amount', align: 'right', sortable: false }, - { name: 'fiat', label: 'Fiat Amount', field: 'fiat', align: 'right', sortable: false }, - { name: 'reference', label: 'Reference', field: 'reference', align: 'left', sortable: false } - ] - }, - accountTypeOptions() { - return [ - { label: 'All Types', value: null }, - { label: 'Receivable (User owes Castle)', value: 'asset' }, - { label: 'Payable (Castle owes User)', value: 'liability' }, - { label: 'Equity (User Balance)', value: 'equity' } - ] - }, expenseAccounts() { return this.accounts.filter(a => a.account_type === 'expense') }, @@ -324,12 +291,6 @@ window.app = Vue.createApp({ } } catch (error) { LNbits.utils.notifyApiError(error) - // Set default balance to clear loading state - this.balance = { - balance: 0, - fiat_balances: {}, - accounts: [] - } } }, async loadAllUserBalances() { @@ -344,123 +305,28 @@ window.app = Vue.createApp({ console.error('Error loading all user balances:', error) } }, - async loadTransactions(offset = null) { + async loadTransactions() { try { - // Use provided offset or current pagination offset, ensure it's an integer - let currentOffset = 0 - if (offset !== null && offset !== undefined) { - currentOffset = parseInt(offset) - } else if (this.transactionPagination && this.transactionPagination.offset !== null && this.transactionPagination.offset !== undefined) { - currentOffset = parseInt(this.transactionPagination.offset) - } - - // Final safety check - ensure it's a valid number - if (isNaN(currentOffset)) { - currentOffset = 0 - } - - const limit = parseInt(this.transactionPagination.limit) || 20 - - // Build query params with filters - let queryParams = `limit=${limit}&offset=${currentOffset}` - - // Add date filter - custom range takes precedence over preset days - if (this.transactionFilter.dateRangeType === 'custom' && this.transactionFilter.startDate && this.transactionFilter.endDate) { - // Dates are already in YYYY-MM-DD format from q-date with mask - queryParams += `&start_date=${this.transactionFilter.startDate}` - queryParams += `&end_date=${this.transactionFilter.endDate}` - } else { - // Use preset days filter - const days = typeof this.transactionFilter.dateRangeType === 'number' ? this.transactionFilter.dateRangeType : 15 - queryParams += `&days=${days}` - } - - if (this.transactionFilter.user_id) { - queryParams += `&filter_user_id=${this.transactionFilter.user_id}` - } - if (this.transactionFilter.account_type) { - queryParams += `&filter_account_type=${this.transactionFilter.account_type}` - } - const response = await LNbits.api.request( 'GET', - `/castle/api/v1/entries/user?${queryParams}`, + '/castle/api/v1/entries/user', this.g.user.wallets[0].inkey ) - - // Update transactions and pagination info - this.transactions = response.data.entries - this.transactionPagination.total = response.data.total - this.transactionPagination.offset = parseInt(response.data.offset) || 0 - this.transactionPagination.has_next = response.data.has_next - this.transactionPagination.has_prev = response.data.has_prev + this.transactions = response.data } catch (error) { LNbits.utils.notifyApiError(error) - // Set empty array to clear loading state - this.transactions = [] - this.transactionPagination.total = 0 - } - }, - applyTransactionFilter() { - // Reset to first page when applying filter - this.transactionPagination.offset = 0 - this.loadTransactions(0) - }, - clearTransactionFilter() { - this.transactionFilter.user_id = null - this.transactionFilter.account_type = null - this.transactionPagination.offset = 0 - this.loadTransactions(0) - }, - onDateRangeTypeChange(value) { - // Handle date range type change (preset days or custom) - if (value !== 'custom') { - // Clear custom date range when switching to preset days - this.transactionFilter.startDate = null - this.transactionFilter.endDate = null - // Load transactions with preset days - this.transactionPagination.offset = 0 - this.loadTransactions(0) - } - // If switching to custom, don't load until user provides dates - }, - applyCustomDateRange() { - // Apply custom date range filter - if (this.transactionFilter.startDate && this.transactionFilter.endDate) { - this.transactionPagination.offset = 0 - this.loadTransactions(0) - } else { - this.$q.notify({ - type: 'warning', - message: 'Please select both start and end dates', - timeout: 3000 - }) - } - }, - nextTransactionsPage() { - if (this.transactionPagination.has_next) { - const newOffset = this.transactionPagination.offset + this.transactionPagination.limit - this.loadTransactions(newOffset) - } - }, - prevTransactionsPage() { - if (this.transactionPagination.has_prev) { - const newOffset = Math.max(0, this.transactionPagination.offset - this.transactionPagination.limit) - this.loadTransactions(newOffset) } }, async loadAccounts() { try { const response = await LNbits.api.request( 'GET', - '/castle/api/v1/accounts?filter_by_user=true&exclude_virtual=true', + '/castle/api/v1/accounts', this.g.user.wallets[0].inkey ) this.accounts = response.data } catch (error) { LNbits.utils.notifyApiError(error) - // Set empty array to clear loading state - this.accounts = [] } }, async loadCurrencies() { @@ -487,19 +353,6 @@ window.app = Vue.createApp({ console.error('Error loading users:', error) } }, - async loadUserInfo() { - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/user/info', - this.g.user.wallets[0].inkey - ) - this.userInfo = response.data - } catch (error) { - console.error('Error loading user info:', error) - this.userInfo = { is_equity_eligible: false } - } - }, async loadSettings() { try { // Try with admin key first to check settings @@ -1138,8 +991,8 @@ window.app = Vue.createApp({ this.receivableDialog.currency = null }, showSettleReceivableDialog(userBalance) { - // Only show for users who owe castle (positive balance = receivable) - if (userBalance.balance <= 0) return + // Only show for users who owe castle (negative balance) + if (userBalance.balance >= 0) return // Clear any existing polling if (this.settleReceivableDialog.pollIntervalId) { @@ -1234,21 +1087,38 @@ window.app = Vue.createApp({ clearInterval(this.settleReceivableDialog.pollIntervalId) this.settleReceivableDialog.pollIntervalId = null - // Payment detected! The webhook (on_invoice_paid in tasks.py) will automatically - // record this in Fava, so we don't need to call record-payment API here. - // Just notify the user and refresh the UI. - this.$q.notify({ - type: 'positive', - message: 'Payment received! Receivable has been settled.', - timeout: 3000 - }) + // Record payment in accounting - this creates the journal entry + // that settles the receivable + try { + const recordResponse = await LNbits.api.request( + 'POST', + '/castle/api/v1/record-payment', + this.g.user.wallets[0].adminkey, + { + payment_hash: paymentHash + } + ) + console.log('Settlement payment recorded:', recordResponse.data) - // Close dialog and refresh - this.settleReceivableDialog.show = false - await this.loadBalance() - await this.loadTransactions() - await this.loadAllUserBalances() + this.$q.notify({ + type: 'positive', + message: 'Payment received! Receivable has been settled.', + timeout: 3000 + }) + // Close dialog and refresh + this.settleReceivableDialog.show = false + await this.loadBalance() + await this.loadTransactions() + await this.loadAllUserBalances() + } catch (error) { + console.error('Error recording settlement payment:', error) + this.$q.notify({ + type: 'negative', + message: 'Payment detected but failed to record: ' + (error.response?.data?.detail || error.message), + timeout: 5000 + }) + } return true } return false @@ -1330,8 +1200,8 @@ window.app = Vue.createApp({ } }, showPayUserDialog(userBalance) { - // Only show for users castle owes (negative balance = payable) - if (userBalance.balance >= 0) return + // Only show for users castle owes (positive balance) + if (userBalance.balance <= 0) return // Extract fiat balances (e.g., EUR) const fiatBalances = userBalance.fiat_balances || {} @@ -1534,30 +1404,52 @@ window.app = Vue.createApp({ return new Date(dateString).toLocaleDateString() }, getTotalAmount(entry) { - return entry.amount + if (!entry.lines || entry.lines.length === 0) return 0 + return entry.lines.reduce((sum, line) => sum + line.debit + line.credit, 0) / 2 }, getEntryFiatAmount(entry) { - if (entry.fiat_amount && entry.fiat_currency) { - return this.formatFiat(entry.fiat_amount, entry.fiat_currency) + // Extract fiat amount from metadata if available + if (!entry.lines || entry.lines.length === 0) return null + + for (const line of entry.lines) { + if (line.metadata && line.metadata.fiat_currency && line.metadata.fiat_amount) { + return this.formatFiat(line.metadata.fiat_amount, line.metadata.fiat_currency) + } } return null }, isReceivable(entry) { // Check if this is a receivable entry (user owes castle) - if (entry.tags && entry.tags.includes('receivable-entry')) return true - if (entry.account && entry.account.includes('Receivable')) return true + // Receivables have a debit to an "Accounts Receivable" account with the user's ID + if (!entry.lines || entry.lines.length === 0) return false + + for (const line of entry.lines) { + // Look for a line with positive debit on an accounts receivable account + if (line.debit > 0) { + // Check if the account is associated with this user's receivables + const account = this.accounts.find(a => a.id === line.account_id) + if (account && account.name && account.name.includes('Assets:Receivable') && account.account_type === 'asset') { + return true + } + } + } return false }, isPayable(entry) { // Check if this is a payable entry (castle owes user) - if (entry.tags && entry.tags.includes('expense-entry')) return true - if (entry.account && entry.account.includes('Payable')) return true - return false - }, - isEquity(entry) { - // Check if this is an equity entry (user capital contribution/balance) - if (entry.tags && entry.tags.includes('equity-contribution')) return true - if (entry.account && entry.account.includes('Equity')) return true + // Payables have a credit to an "Accounts Payable" account with the user's ID + if (!entry.lines || entry.lines.length === 0) return false + + for (const line of entry.lines) { + // Look for a line with positive credit on an accounts payable account + if (line.credit > 0) { + // Check if the account is associated with this user's payables + const account = this.accounts.find(a => a.id === line.account_id) + if (account && account.name && account.name.includes('Liabilities:Payable') && account.account_type === 'liability') { + return true + } + } + } return false } }, @@ -1565,7 +1457,6 @@ window.app = Vue.createApp({ // Load settings first to determine if user is super user await this.loadSettings() await this.loadUserWallet() - await this.loadUserInfo() await this.loadExchangeRate() await this.loadBalance() await this.loadTransactions() diff --git a/static/js/permissions.js b/static/js/permissions.js deleted file mode 100644 index 0de3569..0000000 --- a/static/js/permissions.js +++ /dev/null @@ -1,1122 +0,0 @@ -window.app = Vue.createApp({ - mixins: [windowMixin], - data() { - return { - permissions: [], - accounts: [], - users: [], - filteredUsers: [], - equityEligibleUsers: [], - loading: false, - granting: false, - revoking: false, - grantingEquity: false, - revokingEquity: false, - activeTab: 'by-user', - showGrantDialog: false, - showRevokeDialog: false, - showGrantEquityDialog: false, - showRevokeEquityDialog: false, - showBulkGrantDialog: false, - showBulkGrantErrors: false, - permissionToRevoke: null, - equityToRevoke: null, - bulkGranting: false, - bulkGrantResults: null, - isSuperUser: false, - grantForm: { - user_id: '', - account_id: '', - permission_type: 'submit_expense', - notes: '', - expires_at: '' - }, - grantEquityForm: { - user_id: '', - notes: '' - }, - bulkGrantForm: { - user_ids: [], - account_id: '', - permission_type: 'submit_expense', - notes: '', - expires_at: '' - }, - permissionTypeOptions: [ - { - value: 'read', - label: 'Read', - description: 'View account and balance' - }, - { - value: 'submit_expense', - label: 'Submit Expense', - description: 'Submit expenses to this account' - }, - { - value: 'manage', - label: 'Manage', - description: 'Full account management' - } - ], - // RBAC-related data - roles: [], - selectedRole: null, - roleToDelete: null, - editingRole: false, - showCreateRoleDialog: false, - showViewRoleDialog: false, - showDeleteRoleDialog: false, - showAssignRoleDialog: false, - showRevokeUserRoleDialog: false, - savingRole: false, - deletingRole: false, - assigningRole: false, - revokingUserRole: false, - userRoleToRevoke: null, - roleForm: { - name: '', - description: '', - is_default: false - }, - assignRoleForm: { - user_id: '', - role_id: '', - expires_at: '', - notes: '' - }, - roleUsersForView: [], - rolePermissionsForView: [], - userRoles: new Map(), // Map of user_id -> array of roles - showAddRolePermissionDialog: false, - rolePermissionForm: { - account_id: '', - permission_type: '', - notes: '' - } - } - }, - - computed: { - accountOptions() { - return this.accounts.map(acc => ({ - id: acc.id, - name: acc.name, - is_virtual: acc.is_virtual || false - })) - }, - - userOptions() { - const users = this.filteredUsers.length > 0 ? this.filteredUsers : this.users - return users.map(user => ({ - id: user.id, - username: user.username || user.id, - label: user.username ? `${user.username} (${user.id.substring(0, 8)}...)` : user.id - })) - }, - - isGrantFormValid() { - return !!( - this.grantForm.user_id && - this.grantForm.account_id && - this.grantForm.permission_type - ) - }, - - isBulkGrantFormValid() { - return !!( - this.bulkGrantForm.user_ids && - this.bulkGrantForm.user_ids.length > 0 && - this.bulkGrantForm.account_id && - this.bulkGrantForm.permission_type - ) - }, - - permissionsByUser() { - const grouped = new Map() - for (const perm of this.permissions) { - if (!grouped.has(perm.user_id)) { - grouped.set(perm.user_id, []) - } - grouped.get(perm.user_id).push(perm) - } - return grouped - }, - - // Get all unique user IDs from both direct permissions and role assignments - allUserIds() { - const userIds = new Set() - - // Add users with direct permissions - for (const userId of this.permissionsByUser.keys()) { - userIds.add(userId) - } - - // Add users with role assignments - for (const userId of this.userRoles.keys()) { - userIds.add(userId) - } - - return Array.from(userIds).sort() - }, - - permissionsByAccount() { - const grouped = new Map() - for (const perm of this.permissions) { - if (!grouped.has(perm.account_id)) { - grouped.set(perm.account_id, []) - } - grouped.get(perm.account_id).push(perm) - } - return grouped - }, - - roleOptions() { - return this.roles.map(role => ({ - value: role.id, - label: role.name, - description: role.description, - is_default: role.is_default - })) - }, - - accountOptions() { - return this.accounts.map(account => ({ - value: account.id, - label: account.name, - name: account.name, - description: account.account_type, - is_virtual: account.is_virtual - })) - } - }, - - methods: { - async loadPermissions() { - if (!this.isSuperUser) { - this.$q.notify({ - type: 'warning', - message: 'Admin access required to view permissions', - timeout: 3000 - }) - return - } - - this.loading = true - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/admin/permissions', - this.g.user.wallets[0].adminkey - ) - this.permissions = response.data - } catch (error) { - console.error('Failed to load permissions:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load permissions', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.loading = false - } - }, - - async loadAccounts() { - try { - // Admin permissions UI needs to see virtual accounts to grant permissions on them - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/accounts?exclude_virtual=false', - this.g.user.wallets[0].inkey - ) - this.accounts = response.data - } catch (error) { - console.error('Failed to load accounts:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load accounts', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - async loadUsers() { - if (!this.isSuperUser) { - return - } - - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/admin/castle-users', - this.g.user.wallets[0].adminkey - ) - this.users = response.data || [] - this.filteredUsers = [] - } catch (error) { - console.error('Failed to load users:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load users', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - filterUsers(val, update) { - if (val === '') { - update(() => { - this.filteredUsers = [] - }) - return - } - - update(() => { - const needle = val.toLowerCase() - this.filteredUsers = this.users.filter(user => { - const username = user.username || '' - const userId = user.id || '' - return username.toLowerCase().includes(needle) || userId.toLowerCase().includes(needle) - }) - }) - }, - - async grantPermission() { - if (!this.isGrantFormValid) { - this.$q.notify({ - type: 'warning', - message: 'Please fill in all required fields', - timeout: 3000 - }) - return - } - - this.granting = true - try { - // Extract account_id - handle both string and object cases - const accountId = typeof this.grantForm.account_id === 'object' - ? (this.grantForm.account_id.value || this.grantForm.account_id.id) - : this.grantForm.account_id - - const payload = { - user_id: this.grantForm.user_id, - account_id: accountId, - permission_type: this.grantForm.permission_type - } - - if (this.grantForm.notes) { - payload.notes = this.grantForm.notes - } - - if (this.grantForm.expires_at) { - payload.expires_at = new Date(this.grantForm.expires_at).toISOString() - } - - await LNbits.api.request( - 'POST', - '/castle/api/v1/admin/permissions', - this.g.user.wallets[0].adminkey, - payload - ) - - this.$q.notify({ - type: 'positive', - message: 'Permission granted successfully', - timeout: 3000 - }) - - this.showGrantDialog = false - this.resetGrantForm() - await this.loadPermissions() - } catch (error) { - console.error('Failed to grant permission:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to grant permission', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.granting = false - } - }, - - confirmRevokePermission(permission) { - this.permissionToRevoke = permission - this.showRevokeDialog = true - }, - - async revokePermission() { - if (!this.permissionToRevoke) return - - this.revoking = true - try { - await LNbits.api.request( - 'DELETE', - `/castle/api/v1/admin/permissions/${this.permissionToRevoke.id}`, - this.g.user.wallets[0].adminkey - ) - - this.$q.notify({ - type: 'positive', - message: 'Permission revoked successfully', - timeout: 3000 - }) - - this.showRevokeDialog = false - this.permissionToRevoke = null - await this.loadPermissions() - } catch (error) { - console.error('Failed to revoke permission:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to revoke permission', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.revoking = false - } - }, - - resetGrantForm() { - this.grantForm = { - user_id: '', - account_id: '', - permission_type: 'submit_expense', - notes: '', - expires_at: '' - } - }, - - async bulkGrantPermissions() { - if (!this.isBulkGrantFormValid) { - this.$q.notify({ - type: 'warning', - message: 'Please fill in all required fields', - timeout: 3000 - }) - return - } - - this.bulkGranting = true - this.bulkGrantResults = null - - try { - // Extract account_id - handle both string and object cases - const accountId = typeof this.bulkGrantForm.account_id === 'object' - ? (this.bulkGrantForm.account_id.value || this.bulkGrantForm.account_id.id) - : this.bulkGrantForm.account_id - - const payload = { - user_ids: this.bulkGrantForm.user_ids, - account_id: accountId, - permission_type: this.bulkGrantForm.permission_type - } - - if (this.bulkGrantForm.notes) { - payload.notes = this.bulkGrantForm.notes - } - - if (this.bulkGrantForm.expires_at) { - payload.expires_at = new Date(this.bulkGrantForm.expires_at).toISOString() - } - - const response = await LNbits.api.request( - 'POST', - '/castle/api/v1/admin/permissions/bulk-grant', - this.g.user.wallets[0].adminkey, - payload - ) - - this.bulkGrantResults = response.data - - // Show success notification - const message = this.bulkGrantResults.failure_count > 0 - ? `Bulk grant completed: ${this.bulkGrantResults.success_count} succeeded, ${this.bulkGrantResults.failure_count} failed` - : `Successfully granted permissions to ${this.bulkGrantResults.success_count} users` - - this.$q.notify({ - type: this.bulkGrantResults.failure_count > 0 ? 'warning' : 'positive', - message: message, - timeout: 5000 - }) - - // Reload permissions to show new grants - await this.loadPermissions() - - // Don't close dialog immediately if there were failures - // (so user can review errors) - if (this.bulkGrantResults.failure_count === 0) { - setTimeout(() => { - this.closeBulkGrantDialog() - }, 2000) - } - } catch (error) { - console.error('Failed to bulk grant permissions:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to bulk grant permissions', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.bulkGranting = false - } - }, - - closeBulkGrantDialog() { - this.showBulkGrantDialog = false - this.resetBulkGrantForm() - this.bulkGrantResults = null - }, - - resetBulkGrantForm() { - this.bulkGrantForm = { - user_ids: [], - account_id: '', - permission_type: 'submit_expense', - notes: '', - expires_at: '' - } - }, - - getAccountName(accountId) { - const account = this.accounts.find(a => a.id === accountId) - return account ? account.name : accountId - }, - - getPermissionLabel(permissionType) { - const option = this.permissionTypeOptions.find(opt => opt.value === permissionType) - return option ? option.label : permissionType - }, - - getPermissionColor(permissionType) { - switch (permissionType) { - case 'read': - return 'blue' - case 'submit_expense': - return 'green' - case 'manage': - return 'red' - default: - return 'grey' - } - }, - - getPermissionIcon(permissionType) { - switch (permissionType) { - case 'read': - return 'visibility' - case 'submit_expense': - return 'add_circle' - case 'manage': - return 'admin_panel_settings' - default: - return 'security' - } - }, - - formatDate(dateString) { - if (!dateString) return '-' - const date = new Date(dateString) - return date.toLocaleString() - }, - - async loadEquityEligibleUsers() { - if (!this.isSuperUser) { - return - } - - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/admin/equity-eligibility', - this.g.user.wallets[0].adminkey - ) - this.equityEligibleUsers = response.data || [] - } catch (error) { - console.error('Failed to load equity-eligible users:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load equity-eligible users', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - async grantEquityEligibility() { - if (!this.grantEquityForm.user_id) { - this.$q.notify({ - type: 'warning', - message: 'Please select a user', - timeout: 3000 - }) - return - } - - this.grantingEquity = true - try { - const payload = { - user_id: this.grantEquityForm.user_id, - is_equity_eligible: true - } - - if (this.grantEquityForm.notes) { - payload.notes = this.grantEquityForm.notes - } - - await LNbits.api.request( - 'POST', - '/castle/api/v1/admin/equity-eligibility', - this.g.user.wallets[0].adminkey, - payload - ) - - this.$q.notify({ - type: 'positive', - message: 'Equity eligibility granted successfully', - timeout: 3000 - }) - - this.showGrantEquityDialog = false - this.resetGrantEquityForm() - await this.loadEquityEligibleUsers() - } catch (error) { - console.error('Failed to grant equity eligibility:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to grant equity eligibility', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.grantingEquity = false - } - }, - - confirmRevokeEquity(equity) { - this.equityToRevoke = equity - this.showRevokeEquityDialog = true - }, - - async revokeEquityEligibility() { - if (!this.equityToRevoke) return - - this.revokingEquity = true - try { - await LNbits.api.request( - 'DELETE', - `/castle/api/v1/admin/equity-eligibility/${this.equityToRevoke.user_id}`, - this.g.user.wallets[0].adminkey - ) - - this.$q.notify({ - type: 'positive', - message: 'Equity eligibility revoked successfully', - timeout: 3000 - }) - - this.showRevokeEquityDialog = false - this.equityToRevoke = null - await this.loadEquityEligibleUsers() - } catch (error) { - console.error('Failed to revoke equity eligibility:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to revoke equity eligibility', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.revokingEquity = false - } - }, - - resetGrantEquityForm() { - this.grantEquityForm = { - user_id: '', - notes: '' - } - }, - - // ===== RBAC ROLE MANAGEMENT METHODS ===== - - async loadRoles() { - if (!this.isSuperUser) { - return - } - - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/admin/roles', - this.g.user.wallets[0].adminkey - ) - this.roles = response.data || [] - } catch (error) { - console.error('Failed to load roles:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load roles', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - async viewRole(role) { - this.selectedRole = role - this.roleUsersForView = [] - this.rolePermissionsForView = [] - - try { - const response = await LNbits.api.request( - 'GET', - `/castle/api/v1/admin/roles/${role.id}`, - this.g.user.wallets[0].adminkey - ) - - // Create fresh arrays to ensure Vue reactivity works properly - this.rolePermissionsForView = [...(response.data.permissions || [])] - this.roleUsersForView = [...(response.data.users || [])] - - // Wait for Vue to update the DOM before showing dialog - await this.$nextTick() - this.showViewRoleDialog = true - } catch (error) { - console.error('Failed to load role details:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load role details', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - editRole(role) { - this.editingRole = true - this.selectedRole = role - this.roleForm = { - name: role.name, - description: role.description || '', - is_default: role.is_default || false - } - this.showCreateRoleDialog = true - }, - - async saveRole() { - if (!this.roleForm.name) { - this.$q.notify({ - type: 'warning', - message: 'Please enter a role name', - timeout: 3000 - }) - return - } - - this.savingRole = true - try { - const payload = { - name: this.roleForm.name, - description: this.roleForm.description || null, - is_default: this.roleForm.is_default || false - } - - if (this.editingRole) { - // Update existing role - await LNbits.api.request( - 'PUT', - `/castle/api/v1/admin/roles/${this.selectedRole.id}`, - this.g.user.wallets[0].adminkey, - payload - ) - - this.$q.notify({ - type: 'positive', - message: 'Role updated successfully', - timeout: 3000 - }) - } else { - // Create new role - await LNbits.api.request( - 'POST', - '/castle/api/v1/admin/roles', - this.g.user.wallets[0].adminkey, - payload - ) - - this.$q.notify({ - type: 'positive', - message: 'Role created successfully', - timeout: 3000 - }) - } - - this.closeRoleDialog() - await this.loadRoles() - } catch (error) { - console.error('Failed to save role:', error) - this.$q.notify({ - type: 'negative', - message: `Failed to ${this.editingRole ? 'update' : 'create'} role`, - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.savingRole = false - } - }, - - confirmDeleteRole(role) { - this.roleToDelete = role - this.showDeleteRoleDialog = true - }, - - async deleteRole() { - if (!this.roleToDelete) return - - this.deletingRole = true - try { - await LNbits.api.request( - 'DELETE', - `/castle/api/v1/admin/roles/${this.roleToDelete.id}`, - this.g.user.wallets[0].adminkey - ) - - this.$q.notify({ - type: 'positive', - message: 'Role deleted successfully', - timeout: 3000 - }) - - this.closeDeleteRoleDialog() - await this.loadRoles() - } catch (error) { - console.error('Failed to delete role:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to delete role', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.deletingRole = false - } - }, - - closeRoleDialog() { - this.showCreateRoleDialog = false - this.editingRole = false - this.selectedRole = null - this.resetRoleForm() - }, - - closeViewRoleDialog() { - this.showViewRoleDialog = false - this.selectedRole = null - this.roleUsersForView = [] - this.rolePermissionsForView = [] - }, - - closeDeleteRoleDialog() { - this.showDeleteRoleDialog = false - this.roleToDelete = null - }, - - closeAssignRoleDialog() { - this.showAssignRoleDialog = false - this.resetAssignRoleForm() - }, - - async assignRole() { - if (!this.assignRoleForm.user_id || !this.assignRoleForm.role_id) { - this.$q.notify({ - type: 'warning', - message: 'Please select both a user and a role', - timeout: 3000 - }) - return - } - - this.assigningRole = true - try { - const payload = { - user_id: this.assignRoleForm.user_id, - role_id: this.assignRoleForm.role_id - } - - if (this.assignRoleForm.notes) { - payload.notes = this.assignRoleForm.notes - } - - if (this.assignRoleForm.expires_at) { - payload.expires_at = new Date(this.assignRoleForm.expires_at).toISOString() - } - - await LNbits.api.request( - 'POST', - '/castle/api/v1/admin/user-roles', - this.g.user.wallets[0].adminkey, - payload - ) - - this.$q.notify({ - type: 'positive', - message: 'Role assigned successfully', - timeout: 3000 - }) - - this.closeAssignRoleDialog() - await this.loadRoles() - } catch (error) { - console.error('Failed to assign role:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to assign role', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.assigningRole = false - } - }, - - resetRoleForm() { - this.roleForm = { - name: '', - description: '', - is_default: false - } - }, - - resetAssignRoleForm() { - this.assignRoleForm = { - user_id: '', - role_id: '', - expires_at: '', - notes: '' - } - }, - - // Get roles for a specific user - getUserRoles(userId) { - const userRoleAssignments = this.userRoles.get(userId) || [] - // Map role assignments to role objects - return userRoleAssignments - .map(ur => this.roles.find(r => r.id === ur.role_id)) - .filter(r => r) // Filter out null/undefined - }, - - // Load all user role assignments - async loadUserRoles() { - if (!this.isSuperUser) return - try { - const response = await LNbits.api.request( - 'GET', - '/castle/api/v1/admin/users/roles', - this.g.user.wallets[0].adminkey - ) - - // Group by user_id - this.userRoles.clear() - if (response.data && Array.isArray(response.data)) { - response.data.forEach(userRole => { - if (!this.userRoles.has(userRole.user_id)) { - this.userRoles.set(userRole.user_id, []) - } - this.userRoles.get(userRole.user_id).push(userRole) - }) - } - } catch (error) { - console.error('Failed to load user roles:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to load user role assignments', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - // Get user role assignments (returns UserRole objects, not Role objects) - getUserRoleAssignments(userId) { - return this.userRoles.get(userId) || [] - }, - - // Get role name by ID - getRoleName(roleId) { - const role = this.roles.find(r => r.id === roleId) - return role ? role.name : 'Unknown Role' - }, - - // View role by ID - viewRoleById(roleId) { - const role = this.roles.find(r => r.id === roleId) - if (role) { - this.viewRole(role) - } - }, - - // Show assign role dialog with user pre-selected - showAssignRoleForUser(userId) { - this.assignRoleForm.user_id = userId - this.showAssignRoleDialog = true - }, - - // Show confirmation dialog for revoking user role - confirmRevokeUserRole(userRole) { - this.userRoleToRevoke = userRole - this.showRevokeUserRoleDialog = true - }, - - // Revoke user role - async revokeUserRole() { - if (!this.userRoleToRevoke) return - - this.revokingUserRole = true - try { - await LNbits.api.request( - 'DELETE', - `/castle/api/v1/admin/user-roles/${this.userRoleToRevoke.id}`, - this.g.user.wallets[0].adminkey - ) - - this.$q.notify({ - type: 'positive', - message: 'Role revoked successfully', - timeout: 3000 - }) - - // Reload data - await this.loadUserRoles() - await this.loadRoles() - - // Close dialog - this.showRevokeUserRoleDialog = false - this.userRoleToRevoke = null - } catch (error) { - console.error('Failed to revoke role:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to revoke role', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } finally { - this.revokingUserRole = false - } - }, - - // Add permission to role - async addRolePermission() { - if (!this.selectedRole || !this.rolePermissionForm.account_id || !this.rolePermissionForm.permission_type) { - return - } - try { - // Extract account_id - handle both string and object cases - const accountId = typeof this.rolePermissionForm.account_id === 'object' - ? (this.rolePermissionForm.account_id.value || this.rolePermissionForm.account_id.id) - : this.rolePermissionForm.account_id - - const payload = { - role_id: this.selectedRole.id, - account_id: accountId, - permission_type: this.rolePermissionForm.permission_type, - notes: this.rolePermissionForm.notes || null - } - await LNbits.api.request( - 'POST', - `/castle/api/v1/admin/roles/${this.selectedRole.id}/permissions`, - this.g.user.wallets[0].adminkey, - payload - ) - this.closeAddRolePermissionDialog() - // Reload role permissions - await this.viewRole(this.selectedRole) - this.$q.notify({ - type: 'positive', - message: 'Permission added to role successfully', - timeout: 3000 - }) - } catch (error) { - console.error('Failed to add permission to role:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to add permission to role', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }, - - // Delete role permission - async deleteRolePermission(permissionId) { - this.$q.dialog({ - title: 'Confirm', - message: 'Are you sure you want to remove this permission from the role?', - cancel: true, - persistent: true - }).onOk(async () => { - try { - await LNbits.api.request( - 'DELETE', - `/castle/api/v1/admin/roles/${this.selectedRole.id}/permissions/${permissionId}`, - this.g.user.wallets[0].adminkey - ) - // Reload role permissions - await this.viewRole(this.selectedRole) - this.$q.notify({ - type: 'positive', - message: 'Permission removed from role', - timeout: 3000 - }) - } catch (error) { - console.error('Failed to delete role permission:', error) - this.$q.notify({ - type: 'negative', - message: 'Failed to remove permission', - caption: error.message || 'Unknown error', - timeout: 5000 - }) - } - }) - }, - - // Close add role permission dialog - closeAddRolePermissionDialog() { - this.showAddRolePermissionDialog = false - this.rolePermissionForm = { - account_id: '', - permission_type: '', - notes: '' - } - } - }, - - async created() { - // Check if user is super user - this.isSuperUser = this.g.user.super_user || false - - if (this.g.user.wallets && this.g.user.wallets.length > 0) { - await this.loadAccounts() - if (this.isSuperUser) { - await Promise.all([ - this.loadPermissions(), - this.loadUsers(), - this.loadEquityEligibleUsers(), - this.loadRoles(), - this.loadUserRoles() - ]) - } - } - } -}) - -window.app.mount('#vue') diff --git a/tasks.py b/tasks.py index 1a8327d..32333e1 100644 --- a/tasks.py +++ b/tasks.py @@ -95,59 +95,6 @@ async def scheduled_daily_reconciliation(): raise -async def scheduled_account_sync(): - """ - Scheduled task that runs hourly to sync accounts from Beancount to Castle DB. - - This ensures Castle DB stays in sync with Beancount (source of truth) by - automatically adding any new accounts created in Beancount to Castle's - metadata database for permission tracking. - """ - from .account_sync import sync_accounts_from_beancount - - logger.info(f"[CASTLE] Running scheduled account sync at {datetime.now()}") - - try: - stats = await sync_accounts_from_beancount(force_full_sync=False) - - if stats["accounts_added"] > 0: - logger.info( - f"[CASTLE] Account sync: Added {stats['accounts_added']} new accounts" - ) - - if stats["errors"]: - logger.warning( - f"[CASTLE] Account sync: {len(stats['errors'])} errors encountered" - ) - for error in stats["errors"][:5]: # Log first 5 errors - logger.error(f" - {error}") - - return stats - - except Exception as e: - logger.error(f"[CASTLE] Error in scheduled account sync: {e}") - raise - - -async def wait_for_account_sync(): - """ - Background task that periodically syncs accounts from Beancount to Castle DB. - - Runs hourly to ensure Castle DB stays in sync with Beancount. - """ - logger.info("[CASTLE] Account sync background task started") - - while True: - try: - # Run sync - await scheduled_account_sync() - except Exception as e: - logger.error(f"[CASTLE] Account sync error: {e}") - - # Wait 1 hour before next sync - await asyncio.sleep(3600) # 3600 seconds = 1 hour - - def start_daily_reconciliation_task(): """ Initialize the daily reconciliation task. @@ -182,11 +129,11 @@ async def wait_for_paid_invoices(): async def on_invoice_paid(payment: Payment) -> None: """ - Handle a paid Castle invoice by automatically submitting to Fava. + Handle a paid Castle invoice by automatically creating a journal entry. This function is called automatically when any invoice on the Castle wallet is paid. It checks if the invoice is a Castle payment and records it in - Beancount via Fava. + the accounting system. """ # Only process Castle-specific payments if not payment.extra or payment.extra.get("tag") != "castle": @@ -198,119 +145,85 @@ async def on_invoice_paid(payment: Payment) -> None: return # Check if payment already recorded (idempotency) - # Query Fava for existing entry with this payment hash link - from .fava_client import get_fava_client - import httpx + from .crud import get_journal_entry_by_reference + existing = await get_journal_entry_by_reference(payment.payment_hash) + if existing: + logger.info(f"Payment {payment.payment_hash} already recorded, skipping") + return - fava = get_fava_client() + logger.info(f"Recording Castle payment {payment.payment_hash} for user {user_id[:8]}") try: - # Check if payment already recorded by fetching recent entries - # Note: We can't use BQL query with `links ~ 'pattern'` because links is a set type - # and BQL doesn't support regex matching on sets. Instead, fetch entries and filter in Python. - link_to_find = f"ln-{payment.payment_hash[:16]}" - - async with httpx.AsyncClient(timeout=5.0) as client: - # Get recent entries from Fava's journal endpoint - response = await client.get( - f"{fava.base_url}/api/journal", - params={"time": ""} # Get all entries - ) - - if response.status_code == 200: - data = response.json() - entries = data.get('entries', []) - - # Check if any entry has our payment link - for entry in entries: - entry_links = entry.get('links', []) - if link_to_find in entry_links: - logger.info(f"Payment {payment.payment_hash} already recorded in Fava, skipping") - return - - except Exception as e: - logger.warning(f"Could not check Fava for duplicate payment: {e}") - # Continue anyway - Fava/Beancount will catch duplicate if it exists - - logger.info(f"Recording Castle payment {payment.payment_hash} for user {user_id[:8]} to Fava") - - try: - from decimal import Decimal - from .crud import get_account_by_name, get_or_create_user_account - from .models import AccountType - from .beancount_format import format_net_settlement_entry + # Import here to avoid circular dependencies + from .crud import create_journal_entry, get_account_by_name, get_or_create_user_account + from .models import AccountType, CreateEntryLine, CreateJournalEntry, JournalEntryFlag # Convert amount from millisatoshis to satoshis amount_sats = payment.amount // 1000 # Extract fiat metadata from invoice (if present) - fiat_currency = None - fiat_amount = None + from decimal import Decimal + line_metadata = {} if payment.extra: fiat_currency = payment.extra.get("fiat_currency") - fiat_amount_str = payment.extra.get("fiat_amount") - if fiat_amount_str: - fiat_amount = Decimal(str(fiat_amount_str)) + fiat_amount = payment.extra.get("fiat_amount") + fiat_rate = payment.extra.get("fiat_rate") + btc_rate = payment.extra.get("btc_rate") - if not fiat_currency or not fiat_amount: - logger.error(f"Payment {payment.payment_hash} missing fiat currency/amount metadata") - return + if fiat_currency and fiat_amount: + line_metadata = { + "fiat_currency": fiat_currency, + "fiat_amount": str(fiat_amount), + "fiat_rate": fiat_rate, + "btc_rate": btc_rate, + } - # Get user's current balance to determine receivables and payables - balance = await fava.get_user_balance(user_id) - fiat_balances = balance.get("fiat_balances", {}) - total_fiat_balance = fiat_balances.get(fiat_currency, Decimal(0)) - - # Determine receivables and payables based on balance - # Positive balance = user owes castle (receivable) - # Negative balance = castle owes user (payable) - if total_fiat_balance > 0: - # User owes castle - total_receivable = total_fiat_balance - total_payable = Decimal(0) - else: - # Castle owes user - total_receivable = Decimal(0) - total_payable = abs(total_fiat_balance) - - logger.info(f"Settlement: {fiat_amount} {fiat_currency} (Receivable: {total_receivable}, Payable: {total_payable})") - - # Get account names + # Get user's receivable account (what user owes) user_receivable = await get_or_create_user_account( user_id, AccountType.ASSET, "Accounts Receivable" ) - user_payable = await get_or_create_user_account( - user_id, AccountType.LIABILITY, "Accounts Payable" - ) + + # Get lightning account lightning_account = await get_account_by_name("Assets:Bitcoin:Lightning") if not lightning_account: logger.error("Lightning account 'Assets:Bitcoin:Lightning' not found") return - # Format as net settlement transaction - entry = format_net_settlement_entry( - user_id=user_id, - payment_account=lightning_account.name, - receivable_account=user_receivable.name, - payable_account=user_payable.name, - amount_sats=amount_sats, - net_fiat_amount=fiat_amount, - total_receivable_fiat=total_receivable, - total_payable_fiat=total_payable, - fiat_currency=fiat_currency, - description=f"Lightning payment settlement from user {user_id[:8]}", - entry_date=datetime.now().date(), - payment_hash=payment.payment_hash, - reference=payment.payment_hash + # Create journal entry to record payment + # DR Assets:Bitcoin:Lightning, CR Assets:Receivable (User) + # This reduces what the user owes + entry_meta = { + "source": "lightning_payment", + "created_via": "auto_invoice_listener", + "payment_hash": payment.payment_hash, + "payer_user_id": user_id, + } + + entry_data = CreateJournalEntry( + description=f"Lightning payment from user {user_id[:8]}", + reference=payment.payment_hash, + flag=JournalEntryFlag.CLEARED, + meta=entry_meta, + lines=[ + CreateEntryLine( + account_id=lightning_account.id, + debit=amount_sats, + credit=0, + description="Lightning payment received", + metadata=line_metadata, + ), + CreateEntryLine( + account_id=user_receivable.id, + debit=0, + credit=amount_sats, + description="Payment applied to balance", + metadata=line_metadata, + ), + ], ) - # Submit to Fava - result = await fava.add_entry(entry) - - logger.info( - f"Successfully recorded payment {payment.payment_hash} to Fava: " - f"{result.get('data', 'Unknown')}" - ) + entry = await create_journal_entry(entry_data, user_id) + logger.info(f"Successfully recorded journal entry {entry.id} for payment {payment.payment_hash}") except Exception as e: logger.error(f"Error recording Castle payment {payment.payment_hash}: {e}") diff --git a/templates/castle/index.html b/templates/castle/index.html index 6648e6c..b5d6a01 100644 --- a/templates/castle/index.html +++ b/templates/castle/index.html @@ -16,13 +16,10 @@
🏰 Castle Accounting

Track expenses, receivables, and balances for the collective

-
+
Configure Your Wallet - - Manage Permissions (Admin) - Castle Settings (Super User Only) @@ -81,8 +78,8 @@ {% raw %}{{ formatDate(entry.entry_date) }}{% endraw %} - - User: {% raw %}{{ entry.username }}{% endraw %} + + User: {% raw %}{{ getUserName(entry.meta.user_id) }}{% endraw %} Ref: {% raw %}{{ entry.reference }}{% endraw %} @@ -182,7 +179,7 @@