diff --git a/misc-docs/ORDER-DISCOVERY-ANALYSIS.md b/misc-docs/ORDER-DISCOVERY-ANALYSIS.md new file mode 100644 index 0000000..de393e2 --- /dev/null +++ b/misc-docs/ORDER-DISCOVERY-ANALYSIS.md @@ -0,0 +1,320 @@ +# Nostrmarket Order Discovery Analysis + +## Executive Summary + +This document analyzes the order discovery mechanism in the Nostrmarket extension and identifies why merchants must manually refresh to see new orders instead of receiving them automatically through persistent subscriptions. + +--- + +## Current Architecture + +### Two Subscription Systems + +The Nostrmarket extension implements two distinct subscription mechanisms for receiving Nostr events: + +#### 1. **Persistent Subscriptions (Background Task)** + +**Purpose**: Continuous monitoring for new orders, products, and merchant events + +**Implementation**: + +- Runs via `wait_for_nostr_events()` background task +- Initiated on extension startup (15-second delay) +- Creates subscription ID: `nostrmarket-{hash}` +- Monitors all merchant public keys continuously + +**Code Location**: `/nostrmarket/tasks.py:37-49` + +```python +async def wait_for_nostr_events(nostr_client: NostrClient): + while True: + try: + await subscribe_to_all_merchants() + while True: + message = await nostr_client.get_event() + await process_nostr_message(message) +``` + +**Subscription Filters**: + +- Direct messages (kind 4) - for orders +- Stall events (kind 30017) +- Product events (kind 30018) +- Profile updates (kind 0) + +#### 2. **Temporary Subscriptions (Manual Refresh)** + +**Purpose**: Catch up on missed events when merchant clicks "Refresh from Nostr" + +**Implementation**: + +- Duration: 10 seconds only +- Triggered by user action +- Creates subscription ID: `merchant-{hash}` +- Fetches ALL events from time=0 + +**Code Location**: `/nostrmarket/nostr/nostr_client.py:100-120` + +```python +async def merchant_temp_subscription(self, pk, duration=10): + dm_filters = self._filters_for_direct_messages([pk], 0) + # ... creates filters with time=0 (all history) + await self.send_req_queue.put(["REQ", subscription_id] + merchant_filters) + asyncio.create_task(unsubscribe_with_delay(subscription_id, duration)) +``` + +--- + +## Problem Identification + +### Why Manual Refresh is Required + +#### **Issue 1: Timing Window Problem** + +The persistent subscription uses timestamps from the last database update: + +```python +async def subscribe_to_all_merchants(): + last_dm_time = await get_last_direct_messages_created_at() + last_stall_time = await get_last_stall_update_time() + last_prod_time = await get_last_product_update_time() + + await nostr_client.subscribe_merchants( + public_keys, last_dm_time, last_stall_time, last_prod_time, 0 + ) +``` + +**Problem**: Events that occur between: + +- The last database update time +- When the subscription becomes active + ...are potentially missed + +#### **Issue 2: Connection Stability** + +The WebSocket connection between components may be unstable: + +``` +[Nostrmarket] <--WebSocket--> [Nostrclient] <--WebSocket--> [Nostr Relays] + Extension Extension (Global) +``` + +**Potential failure points**: + +1. Connection drops between nostrmarket → nostrclient +2. Connection drops between nostrclient → relays +3. Reconnection doesn't re-establish subscriptions + +#### **Issue 3: Subscription State Management** + +**Current behavior**: + +- Single persistent subscription per merchant +- No automatic resubscription on failure +- No heartbeat/keepalive mechanism +- No verification that subscription is active + +#### **Issue 4: Event Processing Delays** + +The startup sequence has intentional delays: + +```python +async def _subscribe_to_nostr_client(): + await asyncio.sleep(10) # Wait for nostrclient + await nostr_client.run_forever() + +async def _wait_for_nostr_events(): + await asyncio.sleep(15) # Wait for extension init + await wait_for_nostr_events(nostr_client) +``` + +**Problem**: Orders arriving during initialization are missed + +--- + +## Why Manual Refresh Works + +The temporary subscription succeeds because: + +1. **Fetches from time=0**: Gets ALL historical events +2. **Fresh connection**: Creates new subscription request +3. **Immediate processing**: No startup delays +4. **Direct feedback**: User sees results immediately + +```python +# Temporary subscription uses time=0 (all events) +dm_filters = self._filters_for_direct_messages([pk], 0) # ← 0 means all time + +# Persistent subscription uses last update time +dm_filters = self._filters_for_direct_messages(public_keys, dm_time) # ← can miss events +``` + +--- + +## Impact Analysis + +### User Experience Issues + +1. **Merchants miss orders** without manual refresh +2. **No real-time notifications** for new orders +3. **Uncertainty** about order status +4. **Extra manual steps** required +5. **Delayed order fulfillment** + +### Technical Implications + +1. **Not truly decentralized** - requires active monitoring +2. **Scalability concerns** - manual refresh doesn't scale +3. **Reliability issues** - depends on user action +4. **Performance overhead** - fetching all events repeatedly + +--- + +## Recommended Solutions + +### Solution A: Enhanced Persistent Subscriptions + +**Implement redundant subscription mechanisms:** + +```python +class EnhancedSubscriptionManager: + def __init__(self): + self.last_heartbeat = time.time() + self.subscription_active = False + + async def maintain_subscription(self): + while True: + if not self.subscription_active or \ + time.time() - self.last_heartbeat > 30: + await self.resubscribe_with_overlap() + await asyncio.sleep(10) + + async def resubscribe_with_overlap(self): + # Use timestamp with 5-minute overlap + overlap_time = int(time.time()) - 300 + await subscribe_to_all_merchants(since=overlap_time) +``` + +### Solution B: Periodic Auto-Refresh + +**Add automatic temporary subscriptions:** + +```python +async def auto_refresh_loop(): + while True: + await asyncio.sleep(60) # Every minute + merchants = await get_all_active_merchants() + for merchant in merchants: + await merchant_temp_subscription(merchant.pubkey, duration=5) +``` + +### Solution C: WebSocket Health Monitoring + +**Implement connection health checks:** + +```python +class WebSocketHealthMonitor: + async def check_connection_health(self): + try: + # Send ping to nostrclient + response = await nostr_client.ping(timeout=5) + if not response: + await self.reconnect_and_resubscribe() + except Exception: + await self.reconnect_and_resubscribe() +``` + +### Solution D: Event Gap Detection + +**Detect and fill gaps in event sequence:** + +```python +async def detect_event_gaps(): + # Check for gaps in event timestamps + last_known = await get_last_event_time() + current_time = int(time.time()) + + if current_time - last_known > 60: # 1 minute gap + # Perform temporary subscription to fill gap + await fetch_missing_events(since=last_known) +``` + +--- + +## Implementation Priority + +### Phase 1: Quick Fixes (1-2 days) + +1. [DONE] Increase temp subscription duration (10s → 30s) +2. [DONE] Add connection health logging +3. [DONE] Reduce startup delays + +### Phase 2: Reliability (3-5 days) + +1. [TODO] Implement subscription heartbeat +2. [TODO] Add automatic resubscription on failure +3. [TODO] Create event gap detection + +### Phase 3: Full Solution (1-2 weeks) + +1. [TODO] WebSocket connection monitoring +2. [TODO] Redundant subscription system +3. [TODO] Real-time order notifications +4. [TODO] Event deduplication logic + +--- + +## Testing Recommendations + +### Test Scenarios + +1. **Order during startup**: Send order within 15 seconds of server start +2. **Long-running test**: Keep server running for 24 hours, send periodic orders +3. **Connection interruption**: Disconnect nostrclient, send order, reconnect +4. **High volume**: Send 100 orders rapidly +5. **Network latency**: Add artificial delay between components + +### Monitoring Metrics + +- Time between order sent → order discovered +- Percentage of orders requiring manual refresh +- WebSocket connection uptime +- Subscription success rate +- Event processing latency + +--- + +## Conclusion + +The current order discovery system relies on manual refresh due to: + +1. **Timing gaps** in persistent subscriptions +2. **Connection stability** issues +3. **Lack of redundancy** in subscription management +4. **No automatic recovery** mechanisms + +While the temporary subscription (manual refresh) provides a workaround, a proper solution requires implementing connection monitoring, subscription health checks, and automatic gap-filling mechanisms to ensure reliable real-time order discovery. + +--- + +## Appendix: Code References + +### Key Files + +- `/nostrmarket/tasks.py` - Background task management +- `/nostrmarket/nostr/nostr_client.py` - Nostr client implementation +- `/nostrmarket/services.py` - Order processing logic +- `/nostrmarket/views_api.py` - API endpoints for refresh + +### Relevant Functions + +- `wait_for_nostr_events()` - Main event loop +- `subscribe_to_all_merchants()` - Persistent subscription +- `merchant_temp_subscription()` - Manual refresh +- `process_nostr_message()` - Event processing + +--- + +_Document prepared: January 2025_ +_Analysis based on: Nostrmarket v1.0_ +_Status: Active Investigation_