# Beancount Patterns Analysis for Castle Extension ## Overview After analyzing the [Beancount repository](https://github.com/beancount/beancount), I've identified several excellent design patterns and architectural decisions that we should adopt or consider for the Castle Accounting extension. ## Key Patterns to Adopt ### 1. **Immutable Data Structures with NamedTuple** **Beancount Pattern:** ```python class Transaction(NamedTuple): """A transaction with immutable attributes""" meta: Meta date: datetime.date flag: Flag payee: Optional[str] narration: str tags: frozenset[str] links: frozenset[str] postings: list[Posting] class Posting(NamedTuple): """Individual posting within a transaction""" account: Account units: Optional[Amount] cost: Optional[Union[Cost, CostSpec]] price: Optional[Amount] flag: Optional[Flag] meta: Optional[Meta] ``` **Benefits:** - Immutability prevents accidental modifications - NamedTuple provides free `__repr__`, `__eq__`, `__hash__` - Type hints improve IDE support and catch errors - More memory efficient than regular classes - Thread-safe by design **Castle Application:** ```python # In models.py from typing import NamedTuple, Optional from datetime import datetime from decimal import Decimal class ImmutableJournalEntry(NamedTuple): """Immutable representation of a journal entry""" id: str description: str entry_date: datetime created_by: str created_at: datetime reference: Optional[str] lines: tuple[ImmutableEntryLine, ...] # Tuple, not list! meta: dict[str, Any] # Could store filename, lineno for audit trail class ImmutableEntryLine(NamedTuple): """Immutable representation of an entry line""" id: str journal_entry_id: str account_id: str amount: int # Beancount-style: positive = debit, negative = credit description: Optional[str] metadata: dict[str, Any] flag: Optional[str] # Like Beancount: '!', '*', etc. # Conversion functions def entry_to_immutable(entry: JournalEntry) -> ImmutableJournalEntry: """Convert mutable Pydantic model to immutable NamedTuple""" return ImmutableJournalEntry( id=entry.id, description=entry.description, entry_date=entry.entry_date, created_by=entry.created_by, created_at=entry.created_at, reference=entry.reference, lines=tuple(line_to_immutable(line) for line in entry.lines), meta={} ) ``` **Recommendation:** Use immutable NamedTuples for internal processing, Pydantic models for API I/O. --- ### 2. **Plugin Architecture** **Beancount Pattern:** ```python # In plugins/check_commodity.py __plugins__ = ('validate_commodity_directives',) def validate_commodity_directives(entries, options_map, config): """ Plugin that validates all commodities have declarations. Args: entries: List of directive entries options_map: Parser options config: Plugin-specific configuration Returns: (entries, errors) tuple """ errors = [] # ... validation logic ... return entries, errors ``` **Castle Application:** ```python # Create plugins/ directory # lnbits/extensions/castle/plugins/__init__.py from typing import Protocol, Tuple, List, Any class CastlePlugin(Protocol): """Protocol for Castle plugins""" def __call__( self, entries: List[JournalEntry], settings: dict[str, Any], config: dict[str, Any] ) -> Tuple[List[JournalEntry], List[dict]]: """ Process entries and return modified entries + errors. Args: entries: Journal entries to process settings: Castle settings config: Plugin-specific configuration Returns: (modified_entries, errors) tuple """ ... # Example plugins: # plugins/check_balanced.py __plugins__ = ('check_all_balanced',) def check_all_balanced(entries, settings, config): """Verify all journal entries balance (sum of amounts = 0)""" errors = [] for entry in entries: total_amount = sum(line.amount for line in entry.lines) if total_amount != 0: errors.append({ 'entry_id': entry.id, 'message': f'Unbalanced entry: sum of amounts={total_amount} (must equal 0)', 'severity': 'error' }) return entries, errors # plugins/auto_tags.py __plugins__ = ('auto_add_tags',) def auto_add_tags(entries, settings, config): """Automatically add tags based on rules""" tag_rules = config.get('rules', {}) for entry in entries: for pattern, tag in tag_rules.items(): if pattern in entry.description.lower(): if 'tags' not in entry.meta: entry.meta['tags'] = set() entry.meta['tags'].add(tag) return entries, [] # plugins/check_receivables.py def check_receivable_limits(entries, settings, config): """Warn if receivables exceed configured limits""" errors = [] max_per_user = config.get('max_receivable_per_user', 1_000_000) # Calculate current receivables per user receivables = {} for entry in entries: for line in entry.lines: if 'Accounts Receivable' in line.account_name: user_id = extract_user_from_account(line.account_name) receivables[user_id] = receivables.get(user_id, 0) + line.amount for user_id, amount in receivables.items(): if amount > max_per_user: errors.append({ 'user_id': user_id, 'message': f'Receivable {amount} sats exceeds limit {max_per_user}', 'severity': 'warning' }) return entries, errors ``` **Plugin Manager:** ```python # plugins/manager.py import importlib from pathlib import Path class PluginManager: def __init__(self, plugin_dir: Path): self.plugins = [] self.load_plugins(plugin_dir) def load_plugins(self, plugin_dir: Path): """Discover and load all plugins""" for plugin_file in plugin_dir.glob('*.py'): if plugin_file.name.startswith('_'): continue module_name = f"castle.plugins.{plugin_file.stem}" module = importlib.import_module(module_name) if hasattr(module, '__plugins__'): for plugin_name in module.__plugins__: plugin_func = getattr(module, plugin_name) self.plugins.append((plugin_name, plugin_func)) def run_plugins( self, entries: List[JournalEntry], settings: dict, plugin_configs: dict[str, dict] ) -> Tuple[List[JournalEntry], List[dict]]: """Run all plugins in sequence""" all_errors = [] for plugin_name, plugin_func in self.plugins: config = plugin_configs.get(plugin_name, {}) entries, errors = plugin_func(entries, settings, config) all_errors.extend(errors) return entries, all_errors ``` **Benefits:** - Extensibility without modifying core code - Easy to enable/disable validation rules - Community can contribute plugins - Each plugin is independently testable --- ### 3. **Inventory System for Position Tracking** **Beancount Pattern:** ```python class Inventory(dict[tuple[str, Optional[Cost]], Position]): """ Tracks positions keyed by (currency, cost). Handles lot matching with FIFO, LIFO, HIFO strategies. """ def add_position(self, position: Position): """Add or merge a position""" key = (position.units.currency, position.cost) if key in self: self[key] = self[key].add(position) else: self[key] = position def reduce(self, reducer_func): """Convert inventory using a reduction function""" return sum((reducer_func(pos) for pos in self), Inventory()) def get_currency_units(self, currency: str) -> Decimal: """Get total units of a specific currency""" return sum( pos.units.number for (curr, cost), pos in self.items() if curr == currency ) ``` **Castle Application:** ```python # core/inventory.py from decimal import Decimal from typing import Optional, Dict, Tuple from dataclasses import dataclass @dataclass(frozen=True) class CastlePosition: """A position in the Castle inventory""" currency: str # "SATS", "EUR", "USD" amount: Decimal cost_currency: Optional[str] = None # Original currency if converted cost_amount: Optional[Decimal] = None # Original amount date: Optional[datetime] = None metadata: Dict[str, Any] = None class CastleInventory: """ Track user balances across multiple currencies with conversion tracking. Similar to Beancount's Inventory but optimized for Castle's use case. """ def __init__(self): self.positions: Dict[Tuple[str, Optional[str]], CastlePosition] = {} def add_position(self, position: CastlePosition): """Add or merge a position""" key = (position.currency, position.cost_currency) if key in self.positions: existing = self.positions[key] self.positions[key] = CastlePosition( currency=position.currency, amount=existing.amount + position.amount, cost_currency=position.cost_currency, cost_amount=( (existing.cost_amount or Decimal(0)) + (position.cost_amount or Decimal(0)) ), date=position.date, metadata={**(existing.metadata or {}), **(position.metadata or {})} ) else: self.positions[key] = position def get_balance_sats(self) -> Decimal: """Get total balance in satoshis""" return sum( pos.amount for (curr, _), pos in self.positions.items() if curr == "SATS" ) def get_balance_fiat(self, currency: str) -> Decimal: """Get balance in specific fiat currency from cost metadata""" return sum( pos.cost_amount or Decimal(0) for (curr, cost_curr), pos in self.positions.items() if cost_curr == currency ) def to_dict(self) -> dict: """Export to dictionary""" return { "sats": float(self.get_balance_sats()), "fiat": { curr: float(self.get_balance_fiat(curr)) for curr in set( cost_curr for _, cost_curr in self.positions.keys() if cost_curr ) } } # Usage in balance calculation: async def get_user_inventory(user_id: str) -> CastleInventory: """Calculate user's inventory from journal entries""" inventory = CastleInventory() user_accounts = await get_user_accounts(user_id) for account in user_accounts: entry_lines = await get_entry_lines_for_account(account.id) for line in entry_lines: # Add as position metadata = json.loads(line.metadata) if line.metadata else {} if line.amount != 0: # Beancount-style: positive = debit, negative = credit # Adjust sign for cost amount based on amount direction cost_sign = 1 if line.amount > 0 else -1 inventory.add_position(CastlePosition( currency="SATS", amount=Decimal(line.amount), cost_currency=metadata.get("fiat_currency"), cost_amount=cost_sign * Decimal(metadata.get("fiat_amount", 0)), date=line.created_at, metadata=metadata )) return inventory ``` **Benefits:** - Cleaner abstraction for balance calculations - Separates position tracking from database queries - Easier to add lot tracking if needed later - Consistent with accounting standards --- ### 4. **Meta Attribute Pattern** **Beancount Pattern:** Every directive has a `meta: dict[str, Any]` attribute that stores: - `filename`: Source file - `lineno`: Line number - Custom metadata like tags, links, notes **Castle Application:** ```python class JournalEntryMeta(BaseModel): """Metadata for journal entries""" filename: Optional[str] = None # For imported entries lineno: Optional[int] = None source: str = "api" # "api", "import", "plugin", "migration" created_via: str = "web_ui" # "web_ui", "cli", "script" tags: set[str] = set() # e.g., {"recurring", "automated"} links: set[str] = set() # Link to other entries reviewed: bool = False reviewed_by: Optional[str] = None reviewed_at: Optional[datetime] = None notes: Optional[str] = None # Add to journal_entries table: """ ALTER TABLE journal_entries ADD COLUMN meta TEXT DEFAULT '{}'; """ # Usage: entry = await create_journal_entry( data=entry_data, created_by=user_id, meta={ "source": "api", "created_via": "web_ui", "tags": ["manual", "receivable"], "notes": "User requested invoice #1234" } ) ``` **Benefits:** - Audit trail for compliance - Better debugging ("where did this entry come from?") - Support for tags and categorization - Can track review/approval workflow --- ### 5. **Balance Assertion Pattern** **Beancount Pattern:** ```beancount 2025-10-22 balance Assets:Lightning:Balance 268548 SATS ``` This asserts that the account balance should be exactly 268,548 sats on that date. If it's not, Beancount throws an error. **Castle Application:** ```python # models.py class BalanceAssertion(BaseModel): """Assert expected balance at a specific date""" id: str date: datetime account_id: str expected_balance: int # in satoshis tolerance: int = 0 # Allow +/- this much difference checked_balance: Optional[int] = None difference: Optional[int] = None status: str = "pending" # pending, passed, failed created_by: str created_at: datetime # API endpoint @castle_api_router.post("/api/v1/assertions/balance") async def create_balance_assertion( data: CreateBalanceAssertion, wallet: WalletTypeInfo = Depends(require_admin_key), ) -> BalanceAssertion: """Create a balance assertion for reconciliation""" assertion = BalanceAssertion( id=urlsafe_short_hash(), date=data.date, account_id=data.account_id, expected_balance=data.expected_balance, tolerance=data.tolerance, created_by=wallet.wallet.user, created_at=datetime.now() ) # Check balance immediately actual_balance = await get_account_balance_at_date( data.account_id, data.date ) difference = abs(actual_balance - data.expected_balance) assertion.checked_balance = actual_balance assertion.difference = actual_balance - data.expected_balance assertion.status = "passed" if difference <= data.tolerance else "failed" await db.insert("balance_assertions", assertion) if assertion.status == "failed": raise HTTPException( status_code=HTTPStatus.CONFLICT, detail=f"Balance assertion failed: expected {data.expected_balance}, " f"got {actual_balance}, difference {assertion.difference}" ) return assertion # Periodic check job async def check_all_balance_assertions(): """Recheck all balance assertions (run daily)""" assertions = await db.fetchall( "SELECT * FROM balance_assertions WHERE status = 'pending'" ) for assertion in assertions: actual_balance = await get_account_balance_at_date( assertion.account_id, assertion.date ) difference = abs(actual_balance - assertion.expected_balance) status = "passed" if difference <= assertion.tolerance else "failed" await db.execute( """ UPDATE balance_assertions SET checked_balance = :actual, difference = :diff, status = :status WHERE id = :id """, { "actual": actual_balance, "diff": actual_balance - assertion.expected_balance, "status": status, "id": assertion.id } ) ``` **Benefits:** - Catch data entry errors early - Reconciliation checkpoints - Build confidence in accounting accuracy - Required for external audits --- ### 6. **Hierarchical Account Names** **Beancount Pattern:** ``` Assets:US:BofA:Checking Assets:US:ETrade:Cash Assets:US:ETrade:ITOT Liabilities:US:CreditCard:Amex ``` Accounts are organized hierarchically with `:` separator. **Castle Application:** ```python # Currently: "Accounts Receivable - af983632" # Better: "Assets:Receivable:User-af983632" # Update account naming: ACCOUNT_HIERARCHY = { "asset": "Assets", "liability": "Liabilities", "equity": "Equity", "revenue": "Income", # Beancount uses "Income" not "Revenue" "expense": "Expenses" } def format_account_name( account_type: AccountType, base_name: str, user_id: Optional[str] = None ) -> str: """Format account name in hierarchical style""" root = ACCOUNT_HIERARCHY[account_type.value] # Convert "Accounts Receivable" -> "Receivable" # Convert "Food & Supplies" -> "Food:Supplies" base_clean = base_name.replace("Accounts ", "").replace(" & ", ":") if user_id: return f"{root}:{base_clean}:User-{user_id[:8]}" else: return f"{root}:{base_clean}" # Examples: # "Assets:Receivable:User-af983632" # "Liabilities:Payable:User-af983632" # "Equity:MemberEquity:User-af983632" # "Income:Accommodation" # "Expenses:Food:Supplies" # "Assets:Lightning:Balance" ``` **Benefits:** - Standard accounting hierarchy - Better organization for reporting - Easier to generate balance sheet (all Assets:*, all Liabilities:*, etc.) - Compatible with Beancount import --- ### 7. **Flags for Transaction Status** **Beancount Pattern:** ```beancount 2025-10-22 * "Confirmed transaction" Assets:Checking -100 USD Expenses:Food 100 USD 2025-10-23 ! "Pending/uncleared transaction" Assets:Checking -50 USD Expenses:Gas 50 USD ``` Flags: `*` = cleared, `!` = pending, `#` = flagged for review **Castle Application:** ```python # Add flag field to journal_entries class JournalEntryFlag(str, Enum): CLEARED = "*" # Fully reconciled PENDING = "!" # Not yet confirmed FLAGGED = "#" # Needs review VOID = "x" # Voided entry class JournalEntry(BaseModel): # ... existing fields ... flag: JournalEntryFlag = JournalEntryFlag.CLEARED # Usage: # Lightning payment: flag="*" (immediately cleared) # Manual payment request: flag="!" (pending approval) # Suspicious entry: flag="#" (flagged for review) # Reversed entry: flag="x" (voided) # UI: Show icon based on flag # * = green checkmark # ! = yellow warning # # = red flag # x = gray strikethrough ``` **Benefits:** - Visual indication of entry status - Filter by flag in queries - Standard accounting practice - Supports reconciliation workflow --- ### 8. **Decimal for Money, Not Float** **Beancount Pattern:** ```python from decimal import Decimal # All amounts use Decimal, never float amount = Decimal("19.99") ``` **Castle Current Issue:** We're using `int` for satoshis (good!) but `float` for fiat amounts (bad!). **Fix:** ```python from decimal import Decimal class ExpenseEntry(BaseModel): amount: Decimal # Not float! # ... class ReceivableEntry(BaseModel): amount: Decimal # Not float! # ... # Metadata storage metadata = { "fiat_amount": str(Decimal("250.00")), # Store as string to preserve precision "fiat_rate": str(Decimal("1074.192")), } # When reading: fiat_amount = Decimal(metadata["fiat_amount"]) ``` **Benefits:** - Prevents floating point rounding errors - Required for financial applications - Exact decimal arithmetic - `Decimal("0.1") + Decimal("0.2") == Decimal("0.3")` (float doesn't!) --- ### 9. **Query Language (Future Enhancement)** **Beancount Pattern:** ```sql -- Beancount Query Language (BQL) SELECT account, sum(position) AS balance WHERE account ~ 'Assets:' GROUP BY account ORDER BY balance DESC; SELECT date, narration, position WHERE account = 'Assets:Checking' AND date >= 2025-01-01; ``` **Castle Application (Future):** ```python # Add query endpoint @castle_api_router.post("/api/v1/query") async def execute_query( query: str, wallet: WalletTypeInfo = Depends(require_invoice_key), ) -> dict: """ Execute a query against accounting data. Supports SQL-like syntax for reporting. """ # Parse query (could use SQLAlchemy for this) # Apply user permissions (filter to user's accounts) # Execute and return results pass # Example queries users could run: # - "Show all expenses in October" # - "What's my average monthly spending on food?" # - "Show all receivables older than 30 days" ``` **Benefits:** - Power users can build custom reports - Reduces need for custom endpoints - Flexible analytics --- ## Structural Improvements ### 1. **Separate Core from API** **Beancount Structure:** ``` beancount/ core/ # Pure data structures and logic data.py # Immutable data types inventory.py # Position tracking account.py # Account operations parser/ # Text file parsing plugins/ # Validation and transformation scripts/ # CLI tools tools/ # Reporting and analysis ``` **Castle Should Adopt:** ``` castle/ core/ # NEW: Pure accounting logic __init__.py inventory.py # CastleInventory for position tracking balance.py # Balance calculation logic validation.py # Entry validation (debits=credits, etc) account.py # Account hierarchy and naming models.py # Pydantic models (API I/O) crud.py # Database operations views_api.py # API endpoints plugins/ # NEW: Validation plugins __init__.py check_balanced.py check_receivables.py services.py # Business logic migrations.py templates/ static/ ``` **Benefits:** - Core logic is independent of database - Easier to test (no DB mocking needed) - Could swap DB backend - Core logic can be reused in CLI tools --- ### 2. **Error Handling with Typed Errors** **Beancount Pattern:** ```python ConfigError = collections.namedtuple('ConfigError', 'source message entry') ValidationError = collections.namedtuple('ValidationError', 'source message entry') def validate_entries(entries): errors = [] for entry in entries: if not is_valid(entry): errors.append(ValidationError( source={'filename': entry.meta['filename'], 'lineno': entry.meta['lineno']}, message="Entry is invalid", entry=entry )) return errors ``` **Castle Application:** ```python from typing import NamedTuple, Optional class CastleError(NamedTuple): """Base error type""" source: dict # {'endpoint': '...', 'user_id': '...'} message: str entry: Optional[dict] # Related entry data severity: str # 'error', 'warning', 'info' code: str # 'UNBALANCED_ENTRY', 'MISSING_ACCOUNT', etc. class UnbalancedEntryError(NamedTuple): source: dict message: str entry: dict severity: str = 'error' code: str = 'UNBALANCED_ENTRY' total_debits: int total_credits: int difference: int # Return errors from validation async def validate_journal_entry(entry: CreateJournalEntry) -> list[CastleError]: errors = [] # Beancount-style: sum of amounts must equal 0 total_amount = sum(line.amount for line in entry.lines) if total_amount != 0: errors.append(UnbalancedEntryError( source={'created_by': entry.created_by}, message=f"Entry does not balance: sum of amounts={total_amount} (must equal 0)", entry=entry.dict(), total_amount=total_amount, difference=total_amount )) return errors ``` **Benefits:** - Structured error information - Easier to log and track - Can return multiple errors at once - Better error messages for users --- ## Implementation Roadmap ### Phase 1: Foundation (High Priority) ✅ COMPLETE 1. ✅ Switch from `float` to `Decimal` for fiat amounts 2. ✅ Add `meta` field to journal entries for audit trail 3. ✅ Add `flag` field for transaction status 4. ✅ Implement hierarchical account naming ### Phase 2: Reconciliation (High Priority) ✅ COMPLETE 5. ✅ Implement balance assertions 6. ✅ Add reconciliation API endpoints 7. ✅ Build reconciliation UI 8. ✅ Add automated daily balance checks ### Phase 3: Core Logic Refactoring (Medium Priority) ✅ COMPLETE 9. ✅ Create `core/` module with pure accounting logic 10. ✅ Implement `CastleInventory` for position tracking 11. ✅ Move balance calculation to `core/balance.py` 12. ✅ Add comprehensive validation in `core/validation.py` ### Phase 4: Validation Plugins (Medium Priority) - Works better after Phase 3 13. Create plugin system architecture 14. Implement `check_balanced` plugin 15. Implement `check_receivables` plugin 16. Add plugin configuration UI ### Phase 5: Advanced Features (Low Priority) 17. Add tags and links to entries 18. Implement query language 19. Add lot tracking to inventory 20. Support multi-currency in single entry --- ## Key Takeaways **What Beancount Does Right:** 1. ✅ Immutable data structures 2. ✅ Plugin architecture for extensibility 3. ✅ Rich metadata on every directive 4. ✅ Balance assertions for reconciliation 5. ✅ Hierarchical account naming 6. ✅ Use of Decimal for precision 7. ✅ Separation of core logic from I/O 8. ✅ Comprehensive validation **What Castle Should Adopt First:** 1. **Decimal for fiat amounts** (prevent rounding errors) 2. **Meta field** (audit trail, source tracking) 3. **Flag field** (transaction status) 4. **Balance assertions** (reconciliation) 5. **Plugin system** (extensible validation) **What Can Wait:** - Query language (nice to have) - Lot tracking (not needed for current use case) - Full immutability (Pydantic models are fine for now) --- ## Conclusion Beancount's architecture is battle-tested for serious accounting work. By adopting these patterns, Castle can: - Prevent financial calculation errors (Decimal) - Support complex workflows (plugins) - Build user trust (balance assertions, audit trail) - Scale to larger deployments (clean architecture) - Export to professional tools (hierarchical accounts) The most important immediate change is switching from `float` to `Decimal` for fiat amounts. This alone will prevent serious bugs. The plugin system and balance assertions should follow shortly after as they're critical for financial correctness.