lnbits-exts/castle
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

add dev docs

Browse source
This commit is contained in:
padreug 2025-11-04 01:19:30 +01:00
parent 3add13075c
commit 3248d3dad6
7 changed files with 0 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

936
docs/BEANCOUNT_PATTERNS.md Normal file
View file

@ -0,0 +1,936 @@
# Beancount Patterns Analysis for Castle Extension
## Overview
After analyzing the [Beancount repository](https://github.com/beancount/beancount), I've identified several excellent design patterns and architectural decisions that we should adopt or consider for the Castle Accounting extension.
## Key Patterns to Adopt
### 1. **Immutable Data Structures with NamedTuple**
**Beancount Pattern:**
```python
class Transaction(NamedTuple):
"""A transaction with immutable attributes"""
meta: Meta
date: datetime.date
flag: Flag
payee: Optional[str]
narration: str
tags: frozenset[str]
links: frozenset[str]
postings: list[Posting]
class Posting(NamedTuple):
"""Individual posting within a transaction"""
account: Account
units: Optional[Amount]
cost: Optional[Union[Cost, CostSpec]]
price: Optional[Amount]
flag: Optional[Flag]
meta: Optional[Meta]
```
**Benefits:**
- Immutability prevents accidental modifications
- NamedTuple provides free `__repr__`, `__eq__`, `__hash__`
- Type hints improve IDE support and catch errors
- More memory efficient than regular classes
- Thread-safe by design
**Castle Application:**
```python
# In models.py
from typing import NamedTuple, Optional
from datetime import datetime
from decimal import Decimal
class ImmutableJournalEntry(NamedTuple):
"""Immutable representation of a journal entry"""
id: str
description: str
entry_date: datetime
created_by: str
created_at: datetime
reference: Optional[str]
lines: tuple[ImmutableEntryLine, ...] # Tuple, not list!
meta: dict[str, Any] # Could store filename, lineno for audit trail
class ImmutableEntryLine(NamedTuple):
"""Immutable representation of an entry line"""
id: str
journal_entry_id: str
account_id: str
debit: int
credit: int
description: Optional[str]
metadata: dict[str, Any]
flag: Optional[str] # Like Beancount: '!', '*', etc.
# Conversion functions
def entry_to_immutable(entry: JournalEntry) -> ImmutableJournalEntry:
"""Convert mutable Pydantic model to immutable NamedTuple"""
return ImmutableJournalEntry(
id=entry.id,
description=entry.description,
entry_date=entry.entry_date,
created_by=entry.created_by,
created_at=entry.created_at,
reference=entry.reference,
lines=tuple(line_to_immutable(line) for line in entry.lines),
meta={}
)
```
**Recommendation:** Use immutable NamedTuples for internal processing, Pydantic models for API I/O.
---
### 2. **Plugin Architecture**
**Beancount Pattern:**
```python
# In plugins/check_commodity.py
__plugins__ = ('validate_commodity_directives',)
def validate_commodity_directives(entries, options_map, config):
"""
Plugin that validates all commodities have declarations.
Args:
entries: List of directive entries
options_map: Parser options
config: Plugin-specific configuration
Returns:
(entries, errors) tuple
"""
errors = []
# ... validation logic ...
return entries, errors
```
**Castle Application:**
```python
# Create plugins/ directory
# lnbits/extensions/castle/plugins/__init__.py
from typing import Protocol, Tuple, List, Any
class CastlePlugin(Protocol):
"""Protocol for Castle plugins"""
def __call__(
self,
entries: List[JournalEntry],
settings: dict[str, Any],
config: dict[str, Any]
) -> Tuple[List[JournalEntry], List[dict]]:
"""
Process entries and return modified entries + errors.
Args:
entries: Journal entries to process
settings: Castle settings
config: Plugin-specific configuration
Returns:
(modified_entries, errors) tuple
"""
...
# Example plugins:
# plugins/check_balanced.py
__plugins__ = ('check_all_balanced',)
def check_all_balanced(entries, settings, config):
"""Verify all journal entries have debits = credits"""
errors = []
for entry in entries:
total_debits = sum(line.debit for line in entry.lines)
total_credits = sum(line.credit for line in entry.lines)
if total_debits != total_credits:
errors.append({
'entry_id': entry.id,
'message': f'Unbalanced entry: debits={total_debits}, credits={total_credits}',
'severity': 'error'
})
return entries, errors
# plugins/auto_tags.py
__plugins__ = ('auto_add_tags',)
def auto_add_tags(entries, settings, config):
"""Automatically add tags based on rules"""
tag_rules = config.get('rules', {})
for entry in entries:
for pattern, tag in tag_rules.items():
if pattern in entry.description.lower():
if 'tags' not in entry.meta:
entry.meta['tags'] = set()
entry.meta['tags'].add(tag)
return entries, []
# plugins/check_receivables.py
def check_receivable_limits(entries, settings, config):
"""Warn if receivables exceed configured limits"""
errors = []
max_per_user = config.get('max_receivable_per_user', 1_000_000)
# Calculate current receivables per user
receivables = {}
for entry in entries:
for line in entry.lines:
if 'Accounts Receivable' in line.account_name:
user_id = extract_user_from_account(line.account_name)
receivables[user_id] = receivables.get(user_id, 0) + line.debit - line.credit
for user_id, amount in receivables.items():
if amount > max_per_user:
errors.append({
'user_id': user_id,
'message': f'Receivable {amount} sats exceeds limit {max_per_user}',
'severity': 'warning'
})
return entries, errors
```
**Plugin Manager:**
```python
# plugins/manager.py
import importlib
from pathlib import Path
class PluginManager:
def __init__(self, plugin_dir: Path):
self.plugins = []
self.load_plugins(plugin_dir)
def load_plugins(self, plugin_dir: Path):
"""Discover and load all plugins"""
for plugin_file in plugin_dir.glob('*.py'):
if plugin_file.name.startswith('_'):
continue
module_name = f"castle.plugins.{plugin_file.stem}"
module = importlib.import_module(module_name)
if hasattr(module, '__plugins__'):
for plugin_name in module.__plugins__:
plugin_func = getattr(module, plugin_name)
self.plugins.append((plugin_name, plugin_func))
def run_plugins(
self,
entries: List[JournalEntry],
settings: dict,
plugin_configs: dict[str, dict]
) -> Tuple[List[JournalEntry], List[dict]]:
"""Run all plugins in sequence"""
all_errors = []
for plugin_name, plugin_func in self.plugins:
config = plugin_configs.get(plugin_name, {})
entries, errors = plugin_func(entries, settings, config)
all_errors.extend(errors)
return entries, all_errors
```
**Benefits:**
- Extensibility without modifying core code
- Easy to enable/disable validation rules
- Community can contribute plugins
- Each plugin is independently testable
---
### 3. **Inventory System for Position Tracking**
**Beancount Pattern:**
```python
class Inventory(dict[tuple[str, Optional[Cost]], Position]):
"""
Tracks positions keyed by (currency, cost).
Handles lot matching with FIFO, LIFO, HIFO strategies.
"""
def add_position(self, position: Position):
"""Add or merge a position"""
key = (position.units.currency, position.cost)
if key in self:
self[key] = self[key].add(position)
else:
self[key] = position
def reduce(self, reducer_func):
"""Convert inventory using a reduction function"""
return sum((reducer_func(pos) for pos in self), Inventory())
def get_currency_units(self, currency: str) -> Decimal:
"""Get total units of a specific currency"""
return sum(
pos.units.number
for (curr, cost), pos in self.items()
if curr == currency
)
```
**Castle Application:**
```python
# core/inventory.py
from decimal import Decimal
from typing import Optional, Dict, Tuple
from dataclasses import dataclass
@dataclass(frozen=True)
class CastlePosition:
"""A position in the Castle inventory"""
currency: str # "SATS", "EUR", "USD"
amount: Decimal
cost_currency: Optional[str] = None # Original currency if converted
cost_amount: Optional[Decimal] = None # Original amount
date: Optional[datetime] = None
metadata: Dict[str, Any] = None
class CastleInventory:
"""
Track user balances across multiple currencies with conversion tracking.
Similar to Beancount's Inventory but optimized for Castle's use case.
"""
def __init__(self):
self.positions: Dict[Tuple[str, Optional[str]], CastlePosition] = {}
def add_position(self, position: CastlePosition):
"""Add or merge a position"""
key = (position.currency, position.cost_currency)
if key in self.positions:
existing = self.positions[key]
self.positions[key] = CastlePosition(
currency=position.currency,
amount=existing.amount + position.amount,
cost_currency=position.cost_currency,
cost_amount=(
(existing.cost_amount or Decimal(0)) +
(position.cost_amount or Decimal(0))
),
date=position.date,
metadata={**(existing.metadata or {}), **(position.metadata or {})}
)
else:
self.positions[key] = position
def get_balance_sats(self) -> Decimal:
"""Get total balance in satoshis"""
return sum(
pos.amount
for (curr, _), pos in self.positions.items()
if curr == "SATS"
)
def get_balance_fiat(self, currency: str) -> Decimal:
"""Get balance in specific fiat currency from cost metadata"""
return sum(
pos.cost_amount or Decimal(0)
for (curr, cost_curr), pos in self.positions.items()
if cost_curr == currency
)
def to_dict(self) -> dict:
"""Export to dictionary"""
return {
"sats": float(self.get_balance_sats()),
"fiat": {
curr: float(self.get_balance_fiat(curr))
for curr in set(
cost_curr
for _, cost_curr in self.positions.keys()
if cost_curr
)
}
}
# Usage in balance calculation:
async def get_user_inventory(user_id: str) -> CastleInventory:
"""Calculate user's inventory from journal entries"""
inventory = CastleInventory()
user_accounts = await get_user_accounts(user_id)
for account in user_accounts:
entry_lines = await get_entry_lines_for_account(account.id)
for line in entry_lines:
# Add as position
metadata = json.loads(line.metadata) if line.metadata else {}
if line.debit > 0:
inventory.add_position(CastlePosition(
currency="SATS",
amount=Decimal(line.debit),
cost_currency=metadata.get("fiat_currency"),
cost_amount=Decimal(metadata.get("fiat_amount", 0)),
date=line.created_at,
metadata=metadata
))
if line.credit > 0:
inventory.add_position(CastlePosition(
currency="SATS",
amount=-Decimal(line.credit),
cost_currency=metadata.get("fiat_currency"),
cost_amount=-Decimal(metadata.get("fiat_amount", 0)),
date=line.created_at,
metadata=metadata
))
return inventory
```
**Benefits:**
- Cleaner abstraction for balance calculations
- Separates position tracking from database queries
- Easier to add lot tracking if needed later
- Consistent with accounting standards
---
### 4. **Meta Attribute Pattern**
**Beancount Pattern:**
Every directive has a `meta: dict[str, Any]` attribute that stores:
- `filename`: Source file
- `lineno`: Line number
- Custom metadata like tags, links, notes
**Castle Application:**
```python
class JournalEntryMeta(BaseModel):
"""Metadata for journal entries"""
filename: Optional[str] = None # For imported entries
lineno: Optional[int] = None
source: str = "api" # "api", "import", "plugin", "migration"
created_via: str = "web_ui" # "web_ui", "cli", "script"
tags: set[str] = set() # e.g., {"recurring", "automated"}
links: set[str] = set() # Link to other entries
reviewed: bool = False
reviewed_by: Optional[str] = None
reviewed_at: Optional[datetime] = None
notes: Optional[str] = None
# Add to journal_entries table:
"""
ALTER TABLE journal_entries ADD COLUMN meta TEXT DEFAULT '{}';
"""
# Usage:
entry = await create_journal_entry(
data=entry_data,
created_by=user_id,
meta={
"source": "api",
"created_via": "web_ui",
"tags": ["manual", "receivable"],
"notes": "User requested invoice #1234"
}
)
```
**Benefits:**
- Audit trail for compliance
- Better debugging ("where did this entry come from?")
- Support for tags and categorization
- Can track review/approval workflow
---
### 5. **Balance Assertion Pattern**
**Beancount Pattern:**
```beancount
2025-10-22 balance Assets:Lightning:Balance 268548 SATS
```
This asserts that the account balance should be exactly 268,548 sats on that date. If it's not, Beancount throws an error.
**Castle Application:**
```python
# models.py
class BalanceAssertion(BaseModel):
"""Assert expected balance at a specific date"""
id: str
date: datetime
account_id: str
expected_balance: int # in satoshis
tolerance: int = 0 # Allow +/- this much difference
checked_balance: Optional[int] = None
difference: Optional[int] = None
status: str = "pending" # pending, passed, failed
created_by: str
created_at: datetime
# API endpoint
@castle_api_router.post("/api/v1/assertions/balance")
async def create_balance_assertion(
data: CreateBalanceAssertion,
wallet: WalletTypeInfo = Depends(require_admin_key),
) -> BalanceAssertion:
"""Create a balance assertion for reconciliation"""
assertion = BalanceAssertion(
id=urlsafe_short_hash(),
date=data.date,
account_id=data.account_id,
expected_balance=data.expected_balance,
tolerance=data.tolerance,
created_by=wallet.wallet.user,
created_at=datetime.now()
)
# Check balance immediately
actual_balance = await get_account_balance_at_date(
data.account_id,
data.date
)
difference = abs(actual_balance - data.expected_balance)
assertion.checked_balance = actual_balance
assertion.difference = actual_balance - data.expected_balance
assertion.status = "passed" if difference <= data.tolerance else "failed"
await db.insert("balance_assertions", assertion)
if assertion.status == "failed":
raise HTTPException(
status_code=HTTPStatus.CONFLICT,
detail=f"Balance assertion failed: expected {data.expected_balance}, "
f"got {actual_balance}, difference {assertion.difference}"
)
return assertion
# Periodic check job
async def check_all_balance_assertions():
"""Recheck all balance assertions (run daily)"""
assertions = await db.fetchall(
"SELECT * FROM balance_assertions WHERE status = 'pending'"
)
for assertion in assertions:
actual_balance = await get_account_balance_at_date(
assertion.account_id,
assertion.date
)
difference = abs(actual_balance - assertion.expected_balance)
status = "passed" if difference <= assertion.tolerance else "failed"
await db.execute(
"""
UPDATE balance_assertions
SET checked_balance = :actual, difference = :diff, status = :status
WHERE id = :id
""",
{
"actual": actual_balance,
"diff": actual_balance - assertion.expected_balance,
"status": status,
"id": assertion.id
}
)
```
**Benefits:**
- Catch data entry errors early
- Reconciliation checkpoints
- Build confidence in accounting accuracy
- Required for external audits
---
### 6. **Hierarchical Account Names**
**Beancount Pattern:**
```
Assets:US:BofA:Checking
Assets:US:ETrade:Cash
Assets:US:ETrade:ITOT
Liabilities:US:CreditCard:Amex
```
Accounts are organized hierarchically with `:` separator.
**Castle Application:**
```python
# Currently: "Accounts Receivable - af983632"
# Better: "Assets:Receivable:User-af983632"
# Update account naming:
ACCOUNT_HIERARCHY = {
"asset": "Assets",
"liability": "Liabilities",
"equity": "Equity",
"revenue": "Income", # Beancount uses "Income" not "Revenue"
"expense": "Expenses"
}
def format_account_name(
account_type: AccountType,
base_name: str,
user_id: Optional[str] = None
) -> str:
"""Format account name in hierarchical style"""
root = ACCOUNT_HIERARCHY[account_type.value]
# Convert "Accounts Receivable" -> "Receivable"
# Convert "Food & Supplies" -> "Food:Supplies"
base_clean = base_name.replace("Accounts ", "").replace(" & ", ":")
if user_id:
return f"{root}:{base_clean}:User-{user_id[:8]}"
else:
return f"{root}:{base_clean}"
# Examples:
# "Assets:Receivable:User-af983632"
# "Liabilities:Payable:User-af983632"
# "Equity:MemberEquity:User-af983632"
# "Income:Accommodation"
# "Expenses:Food:Supplies"
# "Assets:Lightning:Balance"
```
**Benefits:**
- Standard accounting hierarchy
- Better organization for reporting
- Easier to generate balance sheet (all Assets:*, all Liabilities:*, etc.)
- Compatible with Beancount import
---
### 7. **Flags for Transaction Status**
**Beancount Pattern:**
```beancount
2025-10-22 * "Confirmed transaction"
Assets:Checking -100 USD
Expenses:Food 100 USD
2025-10-23 ! "Pending/uncleared transaction"
Assets:Checking -50 USD
Expenses:Gas 50 USD
```
Flags: `*` = cleared, `!` = pending, `#` = flagged for review
**Castle Application:**
```python
# Add flag field to journal_entries
class JournalEntryFlag(str, Enum):
CLEARED = "*" # Fully reconciled
PENDING = "!" # Not yet confirmed
FLAGGED = "#" # Needs review
VOID = "x" # Voided entry
class JournalEntry(BaseModel):
# ... existing fields ...
flag: JournalEntryFlag = JournalEntryFlag.CLEARED
# Usage:
# Lightning payment: flag="*" (immediately cleared)
# Manual payment request: flag="!" (pending approval)
# Suspicious entry: flag="#" (flagged for review)
# Reversed entry: flag="x" (voided)
# UI: Show icon based on flag
# * = green checkmark
# ! = yellow warning
# # = red flag
# x = gray strikethrough
```
**Benefits:**
- Visual indication of entry status
- Filter by flag in queries
- Standard accounting practice
- Supports reconciliation workflow
---
### 8. **Decimal for Money, Not Float**
**Beancount Pattern:**
```python
from decimal import Decimal
# All amounts use Decimal, never float
amount = Decimal("19.99")
```
**Castle Current Issue:**
We're using `int` for satoshis (good!) but `float` for fiat amounts (bad!).
**Fix:**
```python
from decimal import Decimal
class ExpenseEntry(BaseModel):
amount: Decimal # Not float!
# ...
class ReceivableEntry(BaseModel):
amount: Decimal # Not float!
# ...
# Metadata storage
metadata = {
"fiat_amount": str(Decimal("250.00")), # Store as string to preserve precision
"fiat_rate": str(Decimal("1074.192")),
}
# When reading:
fiat_amount = Decimal(metadata["fiat_amount"])
```
**Benefits:**
- Prevents floating point rounding errors
- Required for financial applications
- Exact decimal arithmetic
- `Decimal("0.1") + Decimal("0.2") == Decimal("0.3")` (float doesn't!)
---
### 9. **Query Language (Future Enhancement)**
**Beancount Pattern:**
```sql
-- Beancount Query Language (BQL)
SELECT account, sum(position) AS balance
WHERE account ~ 'Assets:'
GROUP BY account
ORDER BY balance DESC;
SELECT date, narration, position
WHERE account = 'Assets:Checking'
AND date >= 2025-01-01;
```
**Castle Application (Future):**
```python
# Add query endpoint
@castle_api_router.post("/api/v1/query")
async def execute_query(
query: str,
wallet: WalletTypeInfo = Depends(require_invoice_key),
) -> dict:
"""
Execute a query against accounting data.
Supports SQL-like syntax for reporting.
"""
# Parse query (could use SQLAlchemy for this)
# Apply user permissions (filter to user's accounts)
# Execute and return results
pass
# Example queries users could run:
# - "Show all expenses in October"
# - "What's my average monthly spending on food?"
# - "Show all receivables older than 30 days"
```
**Benefits:**
- Power users can build custom reports
- Reduces need for custom endpoints
- Flexible analytics
---
## Structural Improvements
### 1. **Separate Core from API**
**Beancount Structure:**
```
beancount/
core/ # Pure data structures and logic
data.py # Immutable data types
inventory.py # Position tracking
account.py # Account operations
parser/ # Text file parsing
plugins/ # Validation and transformation
scripts/ # CLI tools
tools/ # Reporting and analysis
```
**Castle Should Adopt:**
```
castle/
core/ # NEW: Pure accounting logic
__init__.py
inventory.py # CastleInventory for position tracking
balance.py # Balance calculation logic
validation.py # Entry validation (debits=credits, etc)
account.py # Account hierarchy and naming
models.py # Pydantic models (API I/O)
crud.py # Database operations
views_api.py # API endpoints
plugins/ # NEW: Validation plugins
__init__.py
check_balanced.py
check_receivables.py
services.py # Business logic
migrations.py
templates/
static/
```
**Benefits:**
- Core logic is independent of database
- Easier to test (no DB mocking needed)
- Could swap DB backend
- Core logic can be reused in CLI tools
---
### 2. **Error Handling with Typed Errors**
**Beancount Pattern:**
```python
ConfigError = collections.namedtuple('ConfigError', 'source message entry')
ValidationError = collections.namedtuple('ValidationError', 'source message entry')
def validate_entries(entries):
errors = []
for entry in entries:
if not is_valid(entry):
errors.append(ValidationError(
source={'filename': entry.meta['filename'], 'lineno': entry.meta['lineno']},
message="Entry is invalid",
entry=entry
))
return errors
```
**Castle Application:**
```python
from typing import NamedTuple, Optional
class CastleError(NamedTuple):
"""Base error type"""
source: dict # {'endpoint': '...', 'user_id': '...'}
message: str
entry: Optional[dict] # Related entry data
severity: str # 'error', 'warning', 'info'
code: str # 'UNBALANCED_ENTRY', 'MISSING_ACCOUNT', etc.
class UnbalancedEntryError(NamedTuple):
source: dict
message: str
entry: dict
severity: str = 'error'
code: str = 'UNBALANCED_ENTRY'
total_debits: int
total_credits: int
difference: int
# Return errors from validation
async def validate_journal_entry(entry: CreateJournalEntry) -> list[CastleError]:
errors = []
total_debits = sum(line.debit for line in entry.lines)
total_credits = sum(line.credit for line in entry.lines)
if total_debits != total_credits:
errors.append(UnbalancedEntryError(
source={'created_by': entry.created_by},
message=f"Entry does not balance: debits={total_debits}, credits={total_credits}",
entry=entry.dict(),
total_debits=total_debits,
total_credits=total_credits,
difference=total_debits - total_credits
))
return errors
```
**Benefits:**
- Structured error information
- Easier to log and track
- Can return multiple errors at once
- Better error messages for users
---
## Implementation Roadmap
### Phase 1: Foundation (High Priority) ✅ COMPLETE
1. ✅ Switch from `float` to `Decimal` for fiat amounts
2. ✅ Add `meta` field to journal entries for audit trail
3. ✅ Add `flag` field for transaction status
4. ✅ Implement hierarchical account naming
### Phase 2: Reconciliation (High Priority) ✅ COMPLETE
5. ✅ Implement balance assertions
6. ✅ Add reconciliation API endpoints
7. ✅ Build reconciliation UI
8. ✅ Add automated daily balance checks
### Phase 3: Core Logic Refactoring (Medium Priority) ✅ COMPLETE
9. ✅ Create `core/` module with pure accounting logic
10. ✅ Implement `CastleInventory` for position tracking
11. ✅ Move balance calculation to `core/balance.py`
12. ✅ Add comprehensive validation in `core/validation.py`
### Phase 4: Validation Plugins (Medium Priority) - Works better after Phase 3
13. Create plugin system architecture
14. Implement `check_balanced` plugin
15. Implement `check_receivables` plugin
16. Add plugin configuration UI
### Phase 5: Advanced Features (Low Priority)
17. Add tags and links to entries
18. Implement query language
19. Add lot tracking to inventory
20. Support multi-currency in single entry
---
## Key Takeaways
**What Beancount Does Right:**
1. ✅ Immutable data structures
2. ✅ Plugin architecture for extensibility
3. ✅ Rich metadata on every directive
4. ✅ Balance assertions for reconciliation
5. ✅ Hierarchical account naming
6. ✅ Use of Decimal for precision
7. ✅ Separation of core logic from I/O
8. ✅ Comprehensive validation
**What Castle Should Adopt First:**
1. **Decimal for fiat amounts** (prevent rounding errors)
2. **Meta field** (audit trail, source tracking)
3. **Flag field** (transaction status)
4. **Balance assertions** (reconciliation)
5. **Plugin system** (extensible validation)
**What Can Wait:**
- Query language (nice to have)
- Lot tracking (not needed for current use case)
- Full immutability (Pydantic models are fine for now)
---
## Conclusion
Beancount's architecture is battle-tested for serious accounting work. By adopting these patterns, Castle can:
- Prevent financial calculation errors (Decimal)
- Support complex workflows (plugins)
- Build user trust (balance assertions, audit trail)
- Scale to larger deployments (clean architecture)
- Export to professional tools (hierarchical accounts)
The most important immediate change is switching from `float` to `Decimal` for fiat amounts. This alone will prevent serious bugs. The plugin system and balance assertions should follow shortly after as they're critical for financial correctness.

232
docs/DAILY_RECONCILIATION.md Normal file
View file

@ -0,0 +1,232 @@
# Automated Daily Reconciliation
The Castle extension includes automated daily balance checking to ensure accounting accuracy.
## Overview
The daily reconciliation task:
- Checks all balance assertions
- Identifies discrepancies
- Logs results
- Can send alerts (future enhancement)
## Manual Trigger
You can manually trigger the reconciliation check from the UI or via API:
### Via API
```bash
curl -X POST https://your-lnbits-instance.com/castle/api/v1/tasks/daily-reconciliation \
-H "X-Api-Key: YOUR_ADMIN_KEY"
```
## Automated Scheduling
### Option 1: Cron Job (Recommended)
Add to your crontab:
```bash
# Run daily at 2 AM
0 2 * * * curl -X POST http://localhost:5000/castle/api/v1/tasks/daily-reconciliation -H "X-Api-Key: YOUR_ADMIN_KEY" >> /var/log/castle-reconciliation.log 2>&1
```
To edit crontab:
```bash
crontab -e
```
### Option 2: Systemd Timer
Create `/etc/systemd/system/castle-reconciliation.service`:
```ini
[Unit]
Description=Castle Daily Reconciliation Check
After=network.target
[Service]
Type=oneshot
User=lnbits
ExecStart=/usr/bin/curl -X POST http://localhost:5000/castle/api/v1/tasks/daily-reconciliation -H "X-Api-Key: YOUR_ADMIN_KEY"
```
Create `/etc/systemd/system/castle-reconciliation.timer`:
```ini
[Unit]
Description=Run Castle reconciliation daily
[Timer]
OnCalendar=daily
OnCalendar=02:00
Persistent=true
[Install]
WantedBy=timers.target
```
Enable and start:
```bash
sudo systemctl enable castle-reconciliation.timer
sudo systemctl start castle-reconciliation.timer
```
### Option 3: Docker/Kubernetes CronJob
For containerized deployments:
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: castle-reconciliation
spec:
schedule: "0 2 * * *" # Daily at 2 AM
jobTemplate:
spec:
template:
spec:
containers:
- name: reconciliation
image: curlimages/curl:latest
args:
- /bin/sh
- -c
- curl -X POST http://lnbits:5000/castle/api/v1/tasks/daily-reconciliation -H "X-Api-Key: ${ADMIN_KEY}"
restartPolicy: OnFailure
```
## Response Format
The endpoint returns:
```json
{
"task_id": "abc123",
"timestamp": "2025-10-23T02:00:00",
"total": 15,
"checked": 15,
"passed": 14,
"failed": 1,
"errors": 0,
"failed_assertions": [
{
"id": "assertion_id",
"account_id": "account_id",
"expected_sats": 100000,
"actual_sats": 99500,
"difference_sats": -500
}
]
}
```
## Monitoring
### Check Logs
```bash
# View cron logs
grep CRON /var/log/syslog
# View custom log (if using cron with redirect)
tail -f /var/log/castle-reconciliation.log
```
### Success Criteria
- `failed: 0` - All assertions passed
- `errors: 0` - No errors during checks
- `checked === total` - All assertions were checked
### Failure Scenarios
If `failed > 0`:
1. Check the `failed_assertions` array for details
2. Investigate discrepancies in the Castle UI
3. Review recent transactions
4. Check for data entry errors
5. Verify exchange rate conversions (for fiat)
## Future Enhancements
Planned features:
- [ ] Email notifications on failures
- [ ] Webhook notifications
- [ ] Slack/Discord integration
- [ ] Configurable schedule from UI
- [ ] Historical reconciliation reports
- [ ] Automatic retry on transient errors
## Troubleshooting
### Task Not Running
1. **Check cron service**:
```bash
sudo systemctl status cron
```
2. **Verify API key**:
- Ensure you're using the admin wallet API key
- Key must belong to the super user
3. **Check network connectivity**:
```bash
curl http://localhost:5000/castle/api/v1/reconciliation/summary -H "X-Api-Key: YOUR_KEY"
```
### Permission Denied
- Ensure the user running cron has execute permissions
- Check file permissions on any scripts
- Verify API key is valid and belongs to super user
### High Failure Rate
- Review your balance assertions
- Some assertions may need tolerance adjustments
- Check for recent changes in exchange rates
- Verify all transactions are properly cleared
## Best Practices
1. **Set Reasonable Tolerances**: Use tolerance levels to account for rounding
2. **Regular Review**: Check reconciliation dashboard weekly
3. **Assertion Coverage**: Create assertions for critical accounts
4. **Maintenance Window**: Run reconciliation during low-activity periods
5. **Backup First**: Run manual check before configuring automation
6. **Monitor Logs**: Set up log rotation and monitoring
7. **Alert Integration**: Plan for notification system integration
## Example Setup Script
```bash
#!/bin/bash
# setup-castle-reconciliation.sh
# Configuration
LNBITS_URL="http://localhost:5000"
ADMIN_KEY="your_admin_key_here"
LOG_FILE="/var/log/castle-reconciliation.log"
# Create log file
touch "$LOG_FILE"
chmod 644 "$LOG_FILE"
# Add cron job
(crontab -l 2>/dev/null; echo "0 2 * * * curl -X POST $LNBITS_URL/castle/api/v1/tasks/daily-reconciliation -H 'X-Api-Key: $ADMIN_KEY' >> $LOG_FILE 2>&1") | crontab -
echo "Daily reconciliation scheduled for 2 AM"
echo "Logs will be written to: $LOG_FILE"
# Test the endpoint
echo "Running test reconciliation..."
curl -X POST "$LNBITS_URL/castle/api/v1/tasks/daily-reconciliation" \
-H "X-Api-Key: $ADMIN_KEY"
```
Make executable and run:
```bash
chmod +x setup-castle-reconciliation.sh
./setup-castle-reconciliation.sh
```

1213
docs/DOCUMENTATION.md Normal file
View file

File diff suppressed because it is too large Load diff

167
docs/EXPENSE_APPROVAL.md Normal file
View file

@ -0,0 +1,167 @@
# Expense Approval Workflow
## Overview
The Castle extension now requires admin approval for all user-submitted expenses. This prevents invalid or incorrect expenses from affecting balances until they are verified by the Castle admin.
## How It Works
### 1. User Submits Expense
- User fills out the expense form with description, amount, category, etc.
- Expense is created with `flag='!'` (PENDING status)
- Entry is saved to the database but **does not affect balances**
### 2. Admin Reviews Pending Expenses
- Admin sees "Pending Expense Approvals" card on the main page
- Card shows all pending expenses with:
- Description and amount
- User who submitted it
- Date submitted
- Fiat amount (if applicable)
- Reference number
### 3. Admin Takes Action
#### Option A: Approve
- Admin clicks "Approve" button
- Entry flag changes from `!` to `*` (CLEARED)
- Entry **now affects balances** (user's balance updates)
- User sees the expense in their transaction history
- Entry appears with green checkmark icon
#### Option B: Reject
- Admin clicks "Reject" button
- Entry flag changes from `!` to `x` (VOID)
- Entry **never affects balances**
- Entry appears with grey cancel icon (voided)
## Balance Calculation
Only entries with `flag='*'` (CLEARED) are included in balance calculations:
```sql
-- Balance query excludes pending/flagged/voided entries
SELECT SUM(debit), SUM(credit)
FROM entry_lines el
JOIN journal_entries je ON el.journal_entry_id = je.id
WHERE el.account_id = :account_id
AND je.flag = '*' -- Only cleared entries
```
## Transaction Flags
| Flag | Symbol | Status | Affects Balance | Description |
|------|--------|--------|----------------|-------------|
| `*` | ✅ | CLEARED | Yes | Confirmed and reconciled |
| `!` | ⏱️ | PENDING | No | Awaiting approval |
| `#` | 🚩 | FLAGGED | No | Needs review |
| `x` | ❌ | VOID | No | Cancelled/rejected |
## API Endpoints
### Get Pending Entries (Admin Only)
```
GET /castle/api/v1/entries/pending
Authorization: Admin Key
Returns: list[JournalEntry]
```
### Approve Expense (Admin Only)
```
POST /castle/api/v1/entries/{entry_id}/approve
Authorization: Admin Key
Returns: JournalEntry (with flag='*')
```
### Reject Expense (Admin Only)
```
POST /castle/api/v1/entries/{entry_id}/reject
Authorization: Admin Key
Returns: JournalEntry (with flag='x')
```
## Implementation Details
### Files Modified
1. **views_api.py**
- Line 284: Set expenses to `JournalEntryFlag.PENDING` on creation
- Lines 181-197: Added `/api/v1/entries/pending` endpoint
- Lines 972-1011: Added approve endpoint
- Lines 1013-1053: Added reject endpoint
2. **crud.py**
- Lines 315-329: Updated `get_account_balance()` to filter by flag
- Lines 367-376: Updated fiat balance calculation to filter by flag
- Lines 238-269: Fixed `get_all_journal_entries()` to parse flag/meta
3. **index.html**
- Lines 157-209: Added "Pending Expense Approvals" card
4. **index.js**
- Line 68: Added `pendingExpenses` data property
- Lines 497-511: Added `loadPendingExpenses()` method
- Lines 545-563: Added `approveExpense()` method
- Lines 564-580: Added `rejectExpense()` method
- Line 731: Load pending expenses on page load for admins
## User Experience
### For Regular Users
1. Submit expense via "Add Expense" button
2. See expense with orange pending icon (⏱️) in transaction list
3. Balance does NOT change yet
4. Wait for admin approval
### For Admin (Super User)
1. See "Pending Expense Approvals" card at top of page
2. Review expense details
3. Click "Approve" → User's balance updates
4. Click "Reject" → Expense is voided, no balance change
## Security
- All approval endpoints require admin key
- Super user check prevents regular users from approving their own expenses
- Voided entries are never included in balance calculations
- Full audit trail in `meta` field tracks who created and reviewed each entry
## Testing
1. **Submit test expense as regular user**
```
POST /castle/api/v1/entries/expense
{
"description": "Test groceries",
"amount": 50.00,
"currency": "EUR",
"expense_account": "utilities",
"is_equity": false
}
```
2. **Verify it's pending**
- Check user balance → should NOT include this expense
- Check transaction list → should show orange pending icon
3. **Login as admin and approve**
- See expense in "Pending Expense Approvals"
- Click "Approve"
- Verify user balance updates
4. **Submit another expense and reject it**
- Submit expense
- Admin clicks "Reject"
- Verify balance never changed
- Entry shows grey cancel icon
## Future Enhancements
- [ ] Email notifications when expense is approved/rejected
- [ ] Bulk approve/reject multiple expenses
- [ ] Admin notes when rejecting (reason for rejection)
- [ ] Expense revision system (user can edit and resubmit rejected expenses)
- [ ] Approval workflow with multiple approvers

200
docs/PHASE1_COMPLETE.md Normal file
View file

@ -0,0 +1,200 @@
# Phase 1 Implementation - Complete ✅
## Summary
We've successfully implemented the core improvements from Phase 1 of the Beancount patterns adoption:
## ✅ Completed
### 1. **Decimal Instead of Float for Fiat Amounts**
- **Files Changed:**
- `models.py`: Changed all fiat amount fields from `float` to `Decimal`
- `ExpenseEntry.amount`
- `ReceivableEntry.amount`
- `RevenueEntry.amount`
- `UserBalance.fiat_balances` dictionary values
- `crud.py`: Updated fiat balance calculations to use `Decimal`
- `views_api.py`: Store fiat amounts as strings with `str(amount.quantize(Decimal("0.001")))`
- **Benefits:**
- Prevents floating point rounding errors
- Exact decimal arithmetic
- Financial-grade precision
### 2. **Meta Field for Journal Entries**
- **Database Migration:** `m005_add_flag_and_meta`
- Added `meta TEXT DEFAULT '{}'` column to `journal_entries` table
- **Model Changes:**
- Added `meta: dict = {}` to `JournalEntry` and `CreateJournalEntry`
- Meta stores: source, created_via, user_id, payment_hash, etc.
- **CRUD Updates:**
- `create_journal_entry()` now stores meta as JSON
- `get_journal_entries_by_user()` parses meta from JSON
- **API Integration:**
- Expense entries: `{"source": "api", "created_via": "expense_entry", "user_id": "...", "is_equity": false}`
- Receivable entries: `{"source": "api", "created_via": "receivable_entry", "debtor_user_id": "..."}`
- Payment entries: `{"source": "lightning_payment", "created_via": "record_payment", "payment_hash": "...", "payer_user_id": "..."}`
- **Benefits:**
- Full audit trail for every transaction
- Source tracking (where did this entry come from?)
- Can add tags, links, notes in future
- Essential for compliance and debugging
### 3. **Flag Field for Transaction Status**
- **Database Migration:** `m005_add_flag_and_meta`
- Added `flag TEXT DEFAULT '*'` column to `journal_entries` table
- **Model Changes:**
- Created `JournalEntryFlag` enum:
- `*` = CLEARED (confirmed/reconciled)
- `!` = PENDING (awaiting confirmation)
- `#` = FLAGGED (needs review)
- `x` = VOID (cancelled)
- Added `flag: JournalEntryFlag` to `JournalEntry` and `CreateJournalEntry`
- **CRUD Updates:**
- `create_journal_entry()` stores flag as string value
- `get_journal_entries_by_user()` converts string to enum
- **API Logic:**
- Expense entries: Default to CLEARED (immediately confirmed)
- Receivable entries: Start as PENDING (unpaid debt)
- Payment entries: Mark as CLEARED (payment received)
- **Benefits:**
- Visual indication of transaction status in UI
- Filter transactions by status
- Supports reconciliation workflows
- Standard accounting practice (Beancount-style)
## 📊 Migration Details
**Migration `m005_add_flag_and_meta`:**
```sql
ALTER TABLE journal_entries ADD COLUMN flag TEXT DEFAULT '*';
ALTER TABLE journal_entries ADD COLUMN meta TEXT DEFAULT '{}';
```
**To Apply:**
1. Stop LNbits server (if running)
2. Restart LNbits - migration runs automatically
3. Check logs for "m005_add_flag_and_meta" success message
## 🔧 Technical Implementation Details
### Decimal Handling
```python
# Store as string for precision
metadata = {
"fiat_amount": str(data.amount.quantize(Decimal("0.001"))),
}
# Parse back to Decimal
fiat_decimal = Decimal(str(fiat_amount))
```
### Flag Handling
```python
# Set flag on creation
entry_data = CreateJournalEntry(
flag=JournalEntryFlag.PENDING, # or CLEARED
# ...
)
# Parse from database
flag = JournalEntryFlag(entry_data.get("flag", "*"))
```
### Meta Handling
```python
# Create with meta
entry_meta = {
"source": "api",
"created_via": "expense_entry",
"user_id": wallet.wallet.user,
}
entry_data = CreateJournalEntry(
meta=entry_meta,
# ...
)
# Parse from database
meta = json.loads(entry_data.get("meta", "{}")) if entry_data.get("meta") else {}
```
## 🎯 What's Next (Remaining Phase 1 Items)
### Hierarchical Account Naming (In Progress)
Implement Beancount-style account hierarchy:
- Current: `"Accounts Receivable - af983632"`
- Better: `"Assets:Receivable:User-af983632"`
### UI Updates for Flags
Display flag icons in transaction list:
- ✅ `*` = Green checkmark (cleared)
- ⚠️ `!` = Yellow/Orange badge (pending)
- 🚩 `#` = Red flag (needs review)
- ❌ `x` = Strikethrough (voided)
## 🧪 Testing Recommendations
1. **Test Decimal Precision:**
```python
# Create expense with fiat amount
POST /api/v1/entries/expense
{"amount": "36.93", "currency": "EUR", ...}
# Verify stored as exact string
SELECT metadata FROM entry_lines WHERE ...
# Should see: {"fiat_amount": "36.930", ...}
```
2. **Test Flag Workflow:**
```python
# Create receivable (should be PENDING)
POST /api/v1/entries/receivable
# Check: flag = '!'
# Pay receivable (creates CLEARED entry)
POST /api/v1/record-payment
# Check: payment entry flag = '*'
```
3. **Test Meta Audit Trail:**
```python
# Create any entry
# Check database:
SELECT meta FROM journal_entries WHERE ...
# Should see: {"source": "api", "created_via": "...", ...}
```
## 🎉 Success Metrics
- ✅ No more floating point errors in fiat calculations
- ✅ Every transaction has source tracking
- ✅ Transaction status is visible (pending vs cleared)
- ✅ Database migration successful
- ✅ All API endpoints updated
- ✅ CRUD operations handle new fields
## 📝 Notes
- **Backward Compatibility:** Old entries will have default values (`flag='*'`, `meta='{}'`)
- **Performance:** No impact - added columns have defaults and indexes not needed yet
- **Storage:** Minimal increase (meta typically < 200 bytes per entry)
## ✅ Phase 1 Complete!
All Phase 1 tasks have been completed:
1. ✅ Decimal instead of float for fiat amounts
2. ✅ Meta field for journal entries (audit trail)
3. ✅ Flag field for transaction status
4. ✅ Hierarchical account naming (Beancount-style)
5. ✅ UI updated to display flags and metadata
**Next:** Move to Phase 2 (Core logic refactoring) when ready.

273
docs/PHASE2_COMPLETE.md Normal file
View file

@ -0,0 +1,273 @@
# Phase 2: Reconciliation - COMPLETE ✅
## Summary
Phase 2 of the Beancount-inspired refactor focused on **reconciliation and automated balance checking**. This phase builds on Phase 1's foundation to provide robust reconciliation tools that ensure accounting accuracy and catch discrepancies early.
## Completed Features
### 1. Balance Assertions ✅
**Purpose**: Verify account balances match expected values at specific points in time (like Beancount's `balance` directive)
**Implementation**:
- **Models** (`models.py:184-219`):
- `AssertionStatus` enum (pending, passed, failed)
- `BalanceAssertion` model with sats and optional fiat checks
- `CreateBalanceAssertion` request model
- **Database** (`migrations.py:275-320`):
- `balance_assertions` table with expected/actual balance tracking
- Tolerance levels for flexible matching
- Status tracking and timestamps
- Indexes for performance
- **CRUD** (`crud.py:773-981`):
- `create_balance_assertion()` - Create and store assertion
- `get_balance_assertion()` - Fetch single assertion
- `get_balance_assertions()` - List with filters
- `check_balance_assertion()` - Compare expected vs actual
- `delete_balance_assertion()` - Remove assertion
- **API Endpoints** (`views_api.py:1067-1230`):
- `POST /api/v1/assertions` - Create and check assertion
- `GET /api/v1/assertions` - List assertions with filters
- `GET /api/v1/assertions/{id}` - Get specific assertion
- `POST /api/v1/assertions/{id}/check` - Re-check assertion
- `DELETE /api/v1/assertions/{id}` - Delete assertion
- **UI** (`templates/castle/index.html:254-378`):
- Balance Assertions card (super user only)
- Failed assertions prominently displayed with red banner
- Passed assertions in collapsible panel
- Create assertion dialog with validation
- Re-check and delete buttons
- **Frontend** (`static/js/index.js:70-79, 602-726`):
- Data properties and computed values
- CRUD methods for assertions
- Automatic loading on page load
### 2. Reconciliation API Endpoints ✅
**Purpose**: Provide comprehensive reconciliation tools and reporting
**Implementation**:
- **Summary Endpoint** (`views_api.py:1236-1287`):
- `GET /api/v1/reconciliation/summary`
- Returns counts of assertions by status
- Returns counts of journal entries by flag
- Total accounts count
- Last checked timestamp
- **Check All Endpoint** (`views_api.py:1290-1325`):
- `POST /api/v1/reconciliation/check-all`
- Re-checks all balance assertions
- Returns summary of results (passed/failed/errors)
- Useful for manual reconciliation runs
- **Discrepancies Endpoint** (`views_api.py:1328-1357`):
- `GET /api/v1/reconciliation/discrepancies`
- Returns all failed assertions
- Returns all flagged journal entries
- Returns all pending entries
- Total discrepancy count
### 3. Reconciliation UI Dashboard ✅
**Purpose**: Visual dashboard for reconciliation status and quick access to reconciliation tools
**Implementation** (`templates/castle/index.html:380-499`):
- **Summary Cards**:
- Balance Assertions stats (total, passed, failed, pending)
- Journal Entries stats (total, cleared, pending, flagged)
- Total Accounts count with last checked timestamp
- **Discrepancies Alert**:
- Warning banner when discrepancies found
- Shows count of failed assertions and flagged entries
- "View Details" button to expand discrepancy list
- **Discrepancy Details**:
- Failed assertions list with expected vs actual balances
- Flagged entries list
- Quick access to problematic transactions
- **Actions**:
- "Check All" button to run full reconciliation
- Loading states during checks
- Success message when all accounts reconciled
**Frontend** (`static/js/index.js:80-85, 727-779, 933-934`):
- Reconciliation data properties
- Methods to load summary and discrepancies
- `runFullReconciliation()` method with notifications
- Automatic loading on page load for super users
### 4. Automated Daily Balance Checks ✅
**Purpose**: Run balance checks automatically on a schedule to catch discrepancies early
**Implementation**:
- **Tasks Module** (`tasks.py`):
- `check_all_balance_assertions()` - Core checking logic
- `scheduled_daily_reconciliation()` - Scheduled wrapper
- Results logging and reporting
- Error handling
- **API Endpoint** (`views_api.py:1363-1390`):
- `POST /api/v1/tasks/daily-reconciliation`
- Can be triggered manually or via cron
- Returns detailed results
- Super user only
- **Documentation** (`DAILY_RECONCILIATION.md`):
- Comprehensive setup guide
- Multiple scheduling options (cron, systemd, k8s)
- Monitoring and troubleshooting
- Best practices
- Example scripts
## Benefits
### Accounting Accuracy
- ✅ Catch data entry errors early
- ✅ Verify balances at critical checkpoints
- ✅ Build confidence in accounting accuracy
- ✅ Required for external audits
### Operational Excellence
- ✅ Automated daily checks reduce manual work
- ✅ Dashboard provides at-a-glance reconciliation status
- ✅ Discrepancies are immediately visible
- ✅ Historical tracking of assertions
### Developer Experience
- ✅ Clean API for programmatic reconciliation
- ✅ Well-documented scheduling options
- ✅ Flexible tolerance levels
- ✅ Comprehensive error reporting
## File Changes
### New Files Created
1. `tasks.py` - Background tasks for automated reconciliation
2. `DAILY_RECONCILIATION.md` - Setup and scheduling documentation
3. `PHASE2_COMPLETE.md` - This file
### Modified Files
1. `models.py` - Added `BalanceAssertion`, `CreateBalanceAssertion`, `AssertionStatus`
2. `migrations.py` - Added `m007_balance_assertions` migration
3. `crud.py` - Added balance assertion CRUD operations
4. `views_api.py` - Added assertion, reconciliation, and task endpoints
5. `templates/castle/index.html` - Added assertions and reconciliation UI
6. `static/js/index.js` - Added assertion and reconciliation functionality
7. `BEANCOUNT_PATTERNS.md` - Updated roadmap to mark Phase 2 complete
## API Endpoints Summary
### Balance Assertions
- `POST /api/v1/assertions` - Create assertion
- `GET /api/v1/assertions` - List assertions
- `GET /api/v1/assertions/{id}` - Get assertion
- `POST /api/v1/assertions/{id}/check` - Re-check assertion
- `DELETE /api/v1/assertions/{id}` - Delete assertion
### Reconciliation
- `GET /api/v1/reconciliation/summary` - Get reconciliation summary
- `POST /api/v1/reconciliation/check-all` - Check all assertions
- `GET /api/v1/reconciliation/discrepancies` - Get discrepancies
### Automated Tasks
- `POST /api/v1/tasks/daily-reconciliation` - Run daily reconciliation check
## Usage Examples
### Create a Balance Assertion
```bash
curl -X POST http://localhost:5000/castle/api/v1/assertions \
-H "X-Api-Key: ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"account_id": "lightning",
"expected_balance_sats": 268548,
"tolerance_sats": 100
}'
```
### Get Reconciliation Summary
```bash
curl http://localhost:5000/castle/api/v1/reconciliation/summary \
-H "X-Api-Key: ADMIN_KEY"
```
### Run Full Reconciliation
```bash
curl -X POST http://localhost:5000/castle/api/v1/reconciliation/check-all \
-H "X-Api-Key: ADMIN_KEY"
```
### Schedule Daily Reconciliation (Cron)
```bash
# Add to crontab
0 2 * * * curl -X POST http://localhost:5000/castle/api/v1/tasks/daily-reconciliation -H "X-Api-Key: ADMIN_KEY"
```
## Testing Checklist
- [x] Create balance assertion (UI)
- [x] Create balance assertion (API)
- [x] Assertion passes when balance matches
- [x] Assertion fails when balance doesn't match
- [x] Tolerance levels work correctly
- [x] Fiat balance assertions work
- [x] Re-check assertion updates status
- [x] Delete assertion removes it
- [x] Reconciliation summary shows correct stats
- [x] Check all assertions endpoint works
- [x] Discrepancies endpoint returns correct data
- [x] Dashboard displays summary correctly
- [x] Discrepancy alert shows when issues exist
- [x] "Check All" button triggers reconciliation
- [x] Daily reconciliation task executes successfully
- [x] Failed assertions are logged
- [x] All endpoints require super user access
## Next Steps
**Phase 3: Core Logic Refactoring (Medium Priority)**
- Create `core/` module with pure accounting logic
- Implement `CastleInventory` for position tracking
- Move balance calculation to `core/balance.py`
- Add comprehensive validation in `core/validation.py`
**Phase 4: Validation Plugins (Medium Priority)**
- Create plugin system architecture
- Implement `check_balanced` plugin
- Implement `check_receivables` plugin
- Add plugin configuration UI
**Phase 5: Advanced Features (Low Priority)**
- Add tags and links to entries
- Implement query language
- Add lot tracking to inventory
- Support multi-currency in single entry
## Conclusion
Phase 2 successfully implements Beancount's reconciliation philosophy in the Castle extension. With balance assertions, comprehensive reconciliation APIs, a visual dashboard, and automated daily checks, users can:
- **Trust their data** with automated verification
- **Catch errors early** through regular reconciliation
- **Save time** with automated daily checks
- **Gain confidence** in their accounting accuracy
The implementation follows Beancount's best practices while adapting to LNbits' architecture and use case. All reconciliation features are admin-only, ensuring proper access control for sensitive accounting operations.
**Phase 2 Status**: ✅ COMPLETE
---
*Generated: 2025-10-23*
*Next: Phase 3 - Core Logic Refactoring*

365
docs/PHASE3_COMPLETE.md Normal file
View file

@ -0,0 +1,365 @@
# Phase 3: Core Logic Refactoring - COMPLETE ✅
## Summary
Phase 3 of the Beancount-inspired refactor focused on **separating business logic from database operations** and creating a clean, testable core module. This phase improves code quality, maintainability, and follows best practices from Beancount's architecture.
## Completed Features
### 1. Core Module Structure ✅
**Purpose**: Separate pure accounting logic from database and API concerns
**Implementation** (`core/__init__.py`):
- Created `core/` module package
- Exports main classes and functions
- Clean separation of concerns
**Benefits**:
- Testable without database
- Reusable across different storage backends
- Easier to audit and verify
- Clear architecture
### 2. CastleInventory for Position Tracking ✅
**Purpose**: Track balances across multiple currencies with cost basis information (following Beancount's Inventory pattern)
**Implementation** (`core/inventory.py`):
**CastlePosition** (Lines 11-84):
- Immutable dataclass representing a single position
- Tracks currency, amount, cost basis, and metadata
- Supports addition and negation operations
- Automatic Decimal conversion in `__post_init__`
```python
@dataclass(frozen=True)
class CastlePosition:
currency: str # "SATS", "EUR", "USD"
amount: Decimal
cost_currency: Optional[str] = None
cost_amount: Optional[Decimal] = None
date: Optional[datetime] = None
metadata: Dict[str, Any] = field(default_factory=dict)
```
**CastleInventory** (Lines 87-201):
- Container for multiple positions
- Positions keyed by `(currency, cost_currency)` tuple
- Methods for querying balances:
- `get_balance_sats()` - Total satoshis
- `get_balance_fiat(currency)` - Fiat balance for specific currency
- `get_all_fiat_balances()` - All fiat balances
- Utility methods:
- `is_empty()` - Check if no positions
- `is_zero()` - Check if all positions sum to zero
- `to_dict()` - Export to dictionary
### 3. BalanceCalculator ✅
**Purpose**: Pure logic for calculating balances from journal entries
**Implementation** (`core/balance.py`):
**AccountType Enum** (Lines 13-19):
```python
class AccountType(str, Enum):
ASSET = "asset"
LIABILITY = "liability"
EQUITY = "equity"
REVENUE = "revenue"
EXPENSE = "expense"
```
**BalanceCalculator Class** (Lines 22-217):
**Static Methods**:
1. **`calculate_account_balance()`** (Lines 29-54):
- Calculate balance based on account type
- Normal balances:
- Assets/Expenses: Debit balance (debit - credit)
- Liabilities/Equity/Revenue: Credit balance (credit - debit)
2. **`build_inventory_from_entry_lines()`** (Lines 56-117):
- Build CastleInventory from journal entry lines
- Handles both sats and fiat currency tracking
- Accounts for account type when determining sign
3. **`calculate_user_balance()`** (Lines 119-168):
- Calculate user's total balance across all accounts
- Returns both sats balance and fiat balances by currency
- Properly handles asset (receivable) vs liability (payable) accounts
4. **`check_balance_matches()`** (Lines 170-187):
- Verify balance assertion for sats
5. **`check_fiat_balance_matches()`** (Lines 189-202):
- Verify balance assertion for fiat currency
### 4. Comprehensive Validation ✅
**Purpose**: Validation rules for accounting operations
**Implementation** (`core/validation.py`):
**ValidationError Exception** (Lines 10-18):
- Custom exception for validation failures
- Includes detailed error information
**Validation Functions**:
1. **`validate_journal_entry()`** (Lines 21-124):
- Checks:
- At least 2 lines (double-entry requirement)
- Entry is balanced (debits = credits)
- Valid amounts (non-negative)
- No line has both debit and credit
- All lines have account_id
2. **`validate_balance()`** (Lines 127-177):
- Validates balance assertions
- Checks both sats and fiat within tolerance
3. **`validate_receivable_entry()`** (Lines 180-199):
- Validates receivable (user owes castle) entries
- Ensures positive amount
- Ensures revenue account type
4. **`validate_expense_entry()`** (Lines 202-227):
- Validates expense entries
- Ensures positive amount
- Checks account type (expense or equity)
5. **`validate_payment_entry()`** (Lines 230-245):
- Validates payment entries
- Ensures positive amount
6. **`validate_metadata()`** (Lines 248-284):
- Validates entry line metadata
- Checks for required keys
- Validates fiat currency/amount consistency
- Validates Decimal conversion
### 5. Refactored CRUD Operations ✅
**Purpose**: Use core logic in database operations
**Modified Files**: `crud.py`
**Changes**:
1. **Imports** (Lines 26-36):
- Import core accounting logic
- Import validation functions
2. **`get_account_balance()`** (Lines 347-377):
- Refactored to use `BalanceCalculator.calculate_account_balance()`
- Removed duplicate logic
3. **`get_user_balance()`** (Lines 380-435):
- Completely refactored to use:
- `BalanceCalculator.build_inventory_from_entry_lines()`
- `BalanceCalculator.calculate_user_balance()`
- Cleaner separation of database queries vs business logic
4. **`get_all_user_balances()`** (Lines 438-459):
- Simplified to call `get_user_balance()` for each user
- Eliminates code duplication
## Architecture
### Before Phase 3
```
views_api.py → crud.py (mixed DB + logic)
↓
database
```
All accounting logic was embedded in crud.py alongside database operations.
### After Phase 3
```
views_api.py → crud.py → core/
↓ ↓
database Pure Logic
(testable)
```
**Separation of Concerns**:
- `core/` - Pure accounting logic (no DB dependencies)
- `crud.py` - Database operations + orchestration
- `views_api.py` - HTTP API layer
## Benefits
### Code Quality
- ✅ **Testability**: Core logic can be tested without database
- ✅ **Maintainability**: Clear separation makes code easier to understand
- ✅ **Reusability**: Core logic can be used in different contexts
- ✅ **Consistency**: Centralized accounting rules
### Developer Experience
- ✅ **Type Safety**: Immutable dataclasses with proper types
- ✅ **Documentation**: Well-documented core functions
- ✅ **Debugging**: Easier to trace accounting logic
- ✅ **Refactoring**: Safer to make changes
### Reliability
- ✅ **Validation**: Comprehensive validation rules
- ✅ **Correctness**: Pure functions easier to verify
- ✅ **Auditability**: Clear accounting rules
## File Structure
```
lnbits/extensions/castle/
├── core/
│ ├── __init__.py # Module exports
│ ├── inventory.py # CastleInventory, CastlePosition
│ ├── balance.py # BalanceCalculator
│ └── validation.py # Validation functions
├── crud.py # DB operations (refactored to use core/)
├── models.py # Pydantic models
├── views_api.py # API endpoints
└── PHASE3_COMPLETE.md # This file
```
## Usage Examples
### Using CastleInventory
```python
from decimal import Decimal
from castle.core.inventory import CastleInventory, CastlePosition
# Create inventory
inv = CastleInventory()
# Add positions
inv.add_position(CastlePosition(
currency="SATS",
amount=Decimal("100000")
))
inv.add_position(CastlePosition(
currency="SATS",
amount=Decimal("50000"),
cost_currency="EUR",
cost_amount=Decimal("25.00")
))
# Query balances
total_sats = inv.get_balance_sats() # Decimal("150000")
eur_balance = inv.get_balance_fiat("EUR") # Decimal("25.00")
# Export
data = inv.to_dict()
# {"sats": 150000, "fiat": {"EUR": 25.00}}
```
### Using BalanceCalculator
```python
from castle.core.balance import BalanceCalculator, AccountType
# Calculate account balance
balance = BalanceCalculator.calculate_account_balance(
total_debit=100000,
total_credit=50000,
account_type=AccountType.ASSET
)
# Returns: 50000 (debit balance for asset)
# Build inventory from entry lines
entry_lines = [
{"debit": 100000, "credit": 0, "metadata": '{"fiat_currency": "EUR", "fiat_amount": "50.00"}'},
{"debit": 0, "credit": 50000, "metadata": "{}"}
]
inventory = BalanceCalculator.build_inventory_from_entry_lines(
entry_lines,
AccountType.ASSET
)
# Check balance matches
is_valid = BalanceCalculator.check_balance_matches(
actual_balance_sats=100000,
expected_balance_sats=99900,
tolerance_sats=100
)
# Returns: True (within tolerance)
```
### Using Validation
```python
from castle.core.validation import validate_journal_entry, ValidationError
entry = {
"id": "abc123",
"description": "Test entry",
"entry_date": datetime.now()
}
entry_lines = [
{"account_id": "acc1", "debit": 100000, "credit": 0},
{"account_id": "acc2", "debit": 0, "credit": 100000}
]
try:
validate_journal_entry(entry, entry_lines)
print("Valid!")
except ValidationError as e:
print(f"Invalid: {e.message}")
print(f"Details: {e.details}")
```
## Testing Checklist
- [x] CastleInventory created and tested
- [x] CastlePosition addition works
- [x] Inventory balance calculations work
- [x] BalanceCalculator account balance calculation works
- [x] BalanceCalculator inventory building works
- [x] BalanceCalculator user balance calculation works
- [x] Validation functions work
- [x] crud.py refactored to use core logic
- [x] Existing balance calculations still work
- [ ] Unit tests for core module (future work)
## Next Steps
**Phase 4: Validation Plugins** (Medium Priority)
- Create plugin system architecture
- Implement `check_balanced` plugin
- Implement `check_receivables` plugin
- Add plugin configuration UI
**Future Enhancements**:
- Add unit tests for core/ module
- Add integration tests
- Add lot tracking to inventory
- Support multi-currency in single entry
- Add more validation plugins
## Conclusion
Phase 3 successfully refactors Castle's accounting logic into a clean, testable core module. By following Beancount's architecture patterns, we've created:
- **Pure accounting logic** separated from database concerns
- **CastleInventory** for position tracking across currencies
- **BalanceCalculator** for consistent balance calculations
- **Comprehensive validation** for data integrity
The refactoring improves code quality, maintainability, and sets the foundation for Phase 4's plugin system.
**Phase 3 Status**: ✅ COMPLETE
---
*Generated: 2025-10-23*
*Next: Phase 4 - Validation Plugins*