Integration Components:
1. Manual API Endpoints (admin-only):
- POST /api/v1/admin/accounts/sync (full sync)
- POST /api/v1/admin/accounts/sync/{account_name} (single account)
2. Scheduled Background Sync:
- Hourly background task (wait_for_account_sync)
- Registered in castle_start() lifecycle
- Automatically syncs new accounts from Beancount to Castle DB
3. Auto-sync on User Account Creation:
- Updated get_or_create_user_account() in crud.py
- Uses sync_single_account_from_beancount() for consistency
- Ensures receivable/payable accounts are synced when users register
Flow:
- User associates wallet → creates receivable/payable in Beancount
→ syncs to Castle DB → permissions can be granted
- Admin manually syncs → all Beancount accounts added to Castle DB
- Hourly task → catches any accounts created directly in Beancount
This ensures Beancount remains the source of truth while Castle DB
maintains metadata for permissions and user associations.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implements Phase 2 from ACCOUNTS-TABLE-REMOVAL-FEASIBILITY.md with hybrid approach:
- Beancount as source of truth
- Castle DB as metadata store
- Automatic sync keeps them aligned
New Features:
1. Account Synchronization (account_sync.py)
- Auto-sync accounts from Beancount to Castle DB
- Type inference from hierarchical names
- User ID extraction from account names
- Background scheduling support
- 150 accounts sync in ~2 seconds
2. Bulk Permission Management (permission_management.py)
- Bulk grant to multiple users (60x faster)
- User offboarding (revoke all permissions)
- Account closure (revoke all on account)
- Permission templates (copy from user to user)
- Permission analytics dashboard
- Automated expired permission cleanup
3. Comprehensive Documentation
- PERMISSIONS-SYSTEM.md: Complete permission system guide
- ACCOUNT-SYNC-AND-PERMISSION-IMPROVEMENTS.md: Implementation guide
- Admin workflow examples
- API reference
- Security best practices
Benefits:
- 50-70% reduction in admin time
- Onboarding: 10 min → 1 min
- Offboarding: 5 min → 10 sec
- Access review: 2 hours → 5 min
Related:
- Builds on Phase 1 caching (60-80% DB query reduction)
- Complements BQL investigation
- Part of architecture review improvements
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Moved BQL-BALANCE-QUERIES.md from beancounter project to Castle docs/ folder.
Updated all code references from misc-docs/ to docs/ for proper documentation
location alongside implementation.
The document contains comprehensive BQL investigation results showing that
BQL is not feasible for Castle's current ledger format where SATS are stored
in posting metadata rather than position amounts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added detailed comments to BQL methods explaining:
- Current limitation: cannot access posting metadata (sats-equivalent)
- BQL only queries position amounts (EUR/USD)
- Manual aggregation with caching remains the recommended approach
- Future consideration if ledger format changes
Changes:
- query_bql(): Added limitation warning and future consideration note
- get_user_balance_bql(): Added "NOT CURRENTLY USED" warning
- get_all_user_balances_bql(): Added "NOT CURRENTLY USED" warning
All methods kept as reference code for future architectural changes.
See: misc-docs/BQL-BALANCE-QUERIES.md for complete analysis.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added get_user_balance_bql() and get_all_user_balances_bql() methods
that use Beancount Query Language for efficient balance queries.
Benefits:
- Replaces 115-line manual aggregation with ~100 lines of BQL queries
- Server-side filtering and aggregation
- Expected 5-10x performance improvement
- Handles multi-currency positions (SATS, EUR, USD, GBP)
- Returns same data structure as manual methods for compatibility
Implementation:
- get_user_balance_bql(): Query single user's Payable/Receivable accounts
- get_all_user_balances_bql(): Query all users in one efficient query
- Position parsing handles both dict and string formats
- Excludes pending transactions (flag != '!')
Next steps:
1. Test BQL queries against real Castle data
2. Compare results with manual aggregation methods
3. Update call sites to use BQL methods
4. Remove manual aggregation methods after validation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implemented query_bql() method to enable efficient Beancount Query Language
(BQL) queries against Fava API. This is the foundation for replacing manual
balance aggregation (115 lines) with optimized BQL queries.
Benefits:
- Efficient server-side filtering and aggregation
- 5-10x expected performance improvement
- Cleaner, more maintainable code
Next: Implement get_user_balance_bql() using this method.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Since Castle extension has not been released yet, squash all database migrations
for cleaner initial deployments. This reduces migration complexity and improves
maintainability.
Changes:
- Squash migrations m001-m016 into single m001_initial migration
- Reduced from 651 lines (16 functions) to 327 lines (1 function)
- 50% code reduction, 93.75% fewer migration functions
Final database schema (7 tables):
- castle_accounts: Chart of accounts with 40+ default accounts
- castle_extension_settings: Castle configuration
- castle_user_wallet_settings: User wallet associations
- castle_manual_payment_requests: Payment approval workflow
- castle_balance_assertions: Reconciliation with Beancount integration
- castle_user_equity_status: Equity eligibility tracking
- castle_account_permissions: Granular access control
Tables removed (intentionally):
- castle_journal_entries: Now managed by Fava/Beancount (dropped in m016)
- castle_entry_lines: Now managed by Fava/Beancount (dropped in m016)
New migration includes:
- All 7 tables in their final state
- All indexes properly prefixed with idx_castle_
- All foreign key constraints
- 40+ default accounts with hierarchical names (Assets:Bitcoin:Lightning, etc.)
- Comprehensive documentation
Files:
- migrations.py: Single clean m001_initial migration
- migrations_old.py.bak: Backup of original 16 migrations for reference
- MIGRATION_SQUASH_SUMMARY.md: Complete documentation of squash process
Benefits:
- Simpler initial deployments (1 migration instead of 16)
- Easier to understand final schema
- Faster migration execution
- Cleaner codebase
See MIGRATION_SQUASH_SUMMARY.md for full details and testing instructions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit fixes two critical bugs in the user account creation flow:
1. **Always check/create in Fava regardless of Castle DB status**
- Previously, if an account existed in Castle DB, the function would
return early without checking if the Open directive existed in Fava
- This caused accounts to exist in Castle DB but not in Beancount
- Now we always check Fava and create Open directives if needed
2. **Fix Open directive insertion to preserve metadata**
- The insertion logic now skips over metadata lines when finding
the insertion point
- Prevents new Open directives from being inserted between existing
directives and their metadata, which was causing orphaned metadata
3. **Add comprehensive logging**
- Added detailed logging with [ACCOUNT CHECK], [FAVA CHECK],
[FAVA CREATE], [CASTLE DB], and [WALLET UPDATE] prefixes
- Makes it easier to trace account creation flow and debug issues
4. **Fix Fava filename handling**
- Now queries /api/options to get the Beancount file path dynamically
- Fixes "Parameter 'filename' is missing" errors with /api/source
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Balance assertions now use a hybrid architecture where Beancount is the source
of truth for validation, while Castle stores metadata for UI convenience.
Backend changes:
- Add format_balance() function to beancount_format.py for formatting balance directives
- Update POST /api/v1/assertions to write balance directive to Beancount first (via Fava)
- Store metadata in Castle DB (created_by, tolerance, notes) for UI features
- Validate assertions immediately by querying Fava for actual balance
Frontend changes:
- Update dialog description to explain Beancount validation
- Update button tooltip to clarify balance assertions are written to Beancount
- Update empty state message to mention Beancount checkpoints
Benefits:
- Single source of truth (Beancount ledger file)
- Automatic validation by Beancount
- Best of both worlds: robust validation + friendly UI
See misc-docs/BALANCE-ASSERTIONS-HYBRID-APPROACH.md for full documentation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Update FavaClient.add_account() to use PUT /api/source instead of POST /api/add_entries
because Fava does not support Open directives via add_entries endpoint.
Changes:
- Fetch current Beancount source file via GET /api/source
- Check if account already exists to avoid duplicates
- Format Open directive as plain text (not JSON)
- Insert directive after existing Open directives
- Update source file via PUT /api/source with sha256sum validation
This fixes the issue where Open directives were not being written to the Beancount file.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhances the journal entries API to include username information.
This is achieved by extracting the user ID from transaction
metadata or account names and retrieving the corresponding
username. A default username is provided if the user is not found.
The pending entries API is updated with the same functionality.
Prioritizes parsing amount strings in the new EUR/USD format and introduces support for metadata containing sats equivalent.
Fallbacks to legacy parsing when the new format is not detected.
This ensures correct interpretation of amount data from different sources.
This change ensures that user-specific accounts are automatically created
in the Fava/Beancount ledger when they are first requested. It checks for
the existence of the account via a Fava query and creates it via an Open
directive if it's missing. This simplifies account management and
ensures that all necessary accounts are available for transactions.
This implementation adds a new `add_account` method to the `FavaClient`
class which makes use of the /add_entries endpoint to create an account
using an Open Directive.
Ensures that user-provided reference strings for expense,
receivable, and revenue entries are sanitized before being
included as Beancount links. This prevents issues caused by
invalid characters in the links, improving compatibility with
Beancount's link format. A new utility function is introduced
to handle the sanitization process.
Adds a function to sanitize strings for use as Beancount links,
ensuring compatibility with Beancount's link restrictions.
Refactors the journal entry creation process to use EUR-based
postings when fiat currency is provided, improving accuracy
and consistency. The legacy SATS-based fallback is retained for
cases without fiat currency information.
Adjusts reference generation for Beancount entries using the
sanitized description.
Improves the Beancount import process to send SATS amounts with fiat metadata to the API, enabling automatic conversion to EUR-based postings.
Updates the API to store entries in Fava instead of the Castle DB, simplifying the JournalEntry creation process.
Adds error handling to the upload entry function.
Includes a note about imported transactions being stored in EUR with SATS in metadata.
Removes the explicit call to the record-payment API when settling a receivable.
The webhook (on_invoice_paid in tasks.py) automatically handles recording the payment in Fava, making the API call redundant. This simplifies the frontend logic.
Also, in the `showPayUserDialog` function, it now correctly identifies users who are owed payments based on a negative balance instead of a positive balance.
Introduces a function to format fiat settlement entries for Beancount, handling cash, bank transfers, and other non-lightning payments.
This allows for recording transactions in fiat currency with sats as metadata.
Updates the API endpoint to use the new function when settling receivables with fiat currencies.
Reverses the condition for displaying the settle receivable dialog.
It now correctly shows the dialog only for users with a positive balance,
which indicates they owe the castle.
Improves payment recording logic by fetching recent entries and filtering using Python, replacing the BQL query.
This addresses issues with matching against set types in BQL, enhancing reliability.
Removes excessive logging to improve readability and reduce verbosity.
Streamlines balance processing and improves logging for settlement amounts.
Adds a note about Fava's internal normalization behavior to the beancount formatting.
Adds support for parsing direct EUR/USD amount strings in the format "37.22 EUR" or "12.34 USD".
It also retrieves the SATS equivalent from the metadata if present, for the new amount format. This ensures compatibility with both the old "SATS {EUR}" format and the newer, direct fiat formats.
Updates the amount parsing logic to support a new format where fiat amounts (EUR/USD) are specified directly.
Adds support for tracking SATS equivalents from metadata when the new format is used.
Also tracks fiat amounts specified in metadata as a fallback for backward compatibility.
Reverses the calculation of net balance to correctly reflect receivables and liabilities.
Updates the expense tracking system to store payables and receivables in fiat currency within Beancount.
This ensures accurate debt representation and simplifies balance calculations.
Changes include:
- Converting `format_expense_entry` and `format_receivable_entry` to use fiat amounts.
- Introducing `format_net_settlement_entry` for net settlement payments.
- Modifying `format_payment_entry` to use cost syntax for fiat tracking.
- Adjusting Fava client to correctly process new amount formats and metadata.
- Adding average cost basis posting format
The use of fiat amounts and cost basis aims to provide better accuracy and compatibility with existing Beancount workflows.
Refactors user balance calculation to directly parse journal
entries, enhancing accuracy and efficiency. This change
eliminates reliance on direct database queries and provides a
more reliable mechanism for determining user balances.
Adds logging for debugging purposes.
Also extracts and uses fiat metadata from invoice/payment extras.
Changes the displayed balance perspective to reflect the castle's point of view instead of the user's.
This involves:
- Displaying balances as positive when the user owes the castle
- Displaying balances as negative when the castle owes the user.
This change affects how balances are calculated and displayed in both the backend logic and the frontend templates.
Ensures unique identification for receivable and
revenue entries by generating a UUID and incorporating
it into a castle reference.
This enhances tracking and linking capabilities by
providing a consistent and easily identifiable
reference across the system.
Refactors user balance calculation to use journal entries
instead of querying Fava's query endpoint.
This change allows for exclusion of voided transactions
(tagged with #voided) in addition to pending transactions
when calculating user balances, providing more accurate
balance information.
Additionally the change improves parsing of the amounts in journal entries by using regular expressions.
Updates the pending entries API to correctly parse the amount and fiat values from the amount string, which can now contain both SATS and fiat information.
This change handles different formats of the amount string, including cases where the fiat amount is present within curly braces.
Improves handling of the amount field in user entries by parsing string formats that include both SATS and fiat currency information.
This change allows extracting the SATS amount and fiat amount/currency directly from the string, accommodating different display formats.
Switches to retrieving all journal entries from Fava and filtering in the application to allow filtering by account type and user.
This provides more flexibility and control over the data being presented to the user.
Also extracts and includes relevant metadata such as entry ID, fiat amounts, and references for improved frontend display.
Ensures that voided transactions are not included in the
list of pending entries. This prevents displaying
transactions that have been cancelled or reversed,
providing a more accurate view of truly pending items.
Removes redundant metadata from entries and postings.
The cost syntax already contains fiat/exchange rate information.
Metadata such as 'created-via', 'is-equity', and payer/payee
can be inferred from transaction direction, tags, and account names.
Instead of deleting pending expense entries, marks them as voided by adding a #voided tag.
This ensures an audit trail while excluding them from balances.
Updates the Fava client to use 'params' for the delete request.
Adds functionality to interact with Fava for managing
Beancount entries, including fetching, updating, and
deleting entries directly from the Beancount ledger.
This allows for approving/rejecting pending entries
via the API by modifying the source file through Fava.
The changes include:
- Adds methods to the Fava client for fetching all journal
entries, retrieving entry context (source and hash),
updating the entry source, and deleting entries.
- Updates the pending entries API to use the Fava journal
endpoint instead of querying transactions.
- Implements entry approval and rejection using the new
Fava client methods to modify the underlying Beancount file.
Modifies balance queries to exclude pending transactions (flag='!')
and only include cleared/completed transactions (flag='*'). This
ensures accurate balance calculations by reflecting only settled transactions.
Improves the handling of pending entries by extracting and deduplicating data from Fava's query results.
Adds support for displaying fiat amounts alongside entries and extracts them from the position data in Fava.
Streamlines receivables/payables/equity checks on the frontend by relying on BQL query to supply account type metadata and tags.
Inverts the sign of Beancount balances to represent the user's perspective, where liabilities are positive and receivables are negative.
This change ensures that user balances accurately reflect the amount the castle owes the user (positive) or the amount the user owes the castle (negative). It simplifies the logic by consistently negating the Beancount balance rather than using conditional checks based on account type.
Updates journal entry flags to align with Beancount's limited flag support.
Beancount only uses cleared (*) and pending (!) flags.
Removes the VOID and FLAGGED flags and recommends using tags instead
(e.g., "! + #voided" for voided entries, "! + #review" for flagged entries).
Updates the API to reflect this change, removing the ability to directly
"reject" an expense entry via the void flag. Instead, instructs users to
add the #voided tag in Fava.
Updates reconciliation summary to count entries with voided/review tags
instead of voided/flagged flags.
Replaces direct database queries for transactions with calls to the Fava API,
centralizing transaction logic and improving data consistency.
This change removes redundant code and simplifies the API by relying on Fava
for querying transactions based on account patterns and other criteria.
Specifically, the commit introduces new methods in the FavaClient class for
querying transactions, retrieving account transactions, and retrieving user
transactions. The API endpoints are updated to utilize these methods.
Migrates balance calculation and inventory tracking to
Fava/Beancount, leveraging Fava's query API for all
accounting calculations. This simplifies the core module
and centralizes accounting logic in Fava.
Removes direct journal entry creation in favor of using Fava for accounting.
This change centralizes accounting logic in Fava, improving auditability and consistency.
It replaces direct database interactions for recording payments and settlements with calls to the Fava client.
The changes also refactor balance retrieval to fetch data from Fava.
