nostrmarket/misc-docs/ORDER-DISCOVERY-ANALYSIS.md

Nostrmarket Order Discovery Analysis

Executive Summary

This document analyzes the order discovery mechanism in the Nostrmarket extension and identifies why merchants must manually refresh to see new orders instead of receiving them automatically through persistent subscriptions.

---

Current Architecture

Two Subscription Systems

The Nostrmarket extension implements two distinct subscription mechanisms for receiving Nostr events:

1. Persistent Subscriptions (Background Task)

Purpose: Continuous monitoring for new orders, products, and merchant events

Implementation:

• Runs via wait_for_nostr_events() background task
• Initiated on extension startup (15-second delay)
• Creates subscription ID: nostrmarket-{hash}
• Monitors all merchant public keys continuously

Code Location: /nostrmarket/tasks.py:37-49

async def wait_for_nostr_events(nostr_client: NostrClient):
    while True:
        try:
            await subscribe_to_all_merchants()
            while True:
                message = await nostr_client.get_event()
                await process_nostr_message(message)

Subscription Filters:

• Direct messages (kind 4) - for orders
• Stall events (kind 30017)
• Product events (kind 30018)
• Profile updates (kind 0)

2. Temporary Subscriptions (Manual Refresh)

Purpose: Catch up on missed events when merchant clicks "Refresh from Nostr"

Implementation:

• Duration: 10 seconds only
• Triggered by user action
• Creates subscription ID: merchant-{hash}
• Fetches ALL events from time=0

Code Location: /nostrmarket/nostr/nostr_client.py:100-120

async def merchant_temp_subscription(self, pk, duration=10):
    dm_filters = self._filters_for_direct_messages([pk], 0)
    # ... creates filters with time=0 (all history)
    await self.send_req_queue.put(["REQ", subscription_id] + merchant_filters)
    asyncio.create_task(unsubscribe_with_delay(subscription_id, duration))

---

Problem Identification

Why Manual Refresh is Required

Issue 1: Timing Window Problem

The persistent subscription uses timestamps from the last database update:

async def subscribe_to_all_merchants():
    last_dm_time = await get_last_direct_messages_created_at()
    last_stall_time = await get_last_stall_update_time()
    last_prod_time = await get_last_product_update_time()

    await nostr_client.subscribe_merchants(
        public_keys, last_dm_time, last_stall_time, last_prod_time, 0
    )

Problem: Events that occur between:

• The last database update time
• When the subscription becomes active ...are potentially missed

Issue 2: Connection Stability

The WebSocket connection between components may be unstable:

[Nostrmarket] <--WebSocket--> [Nostrclient] <--WebSocket--> [Nostr Relays]
 Extension                      Extension                      (Global)

Potential failure points:

1. Connection drops between nostrmarket → nostrclient
2. Connection drops between nostrclient → relays
3. Reconnection doesn't re-establish subscriptions

Issue 3: Subscription State Management

Current behavior:

• Single persistent subscription per merchant
• No automatic resubscription on failure
• No heartbeat/keepalive mechanism
• No verification that subscription is active

Issue 4: Event Processing Delays

The startup sequence has intentional delays:

async def _subscribe_to_nostr_client():
    await asyncio.sleep(10)  # Wait for nostrclient
    await nostr_client.run_forever()

async def _wait_for_nostr_events():
    await asyncio.sleep(15)  # Wait for extension init
    await wait_for_nostr_events(nostr_client)

Problem: Orders arriving during initialization are missed

---

Why Manual Refresh Works

The temporary subscription succeeds because:

1. Fetches from time=0: Gets ALL historical events
2. Fresh connection: Creates new subscription request
3. Immediate processing: No startup delays
4. Direct feedback: User sees results immediately

# Temporary subscription uses time=0 (all events)
dm_filters = self._filters_for_direct_messages([pk], 0)  # ← 0 means all time

# Persistent subscription uses last update time
dm_filters = self._filters_for_direct_messages(public_keys, dm_time)  # ← can miss events

---

Impact Analysis

User Experience Issues

1. Merchants miss orders without manual refresh
2. No real-time notifications for new orders
3. Uncertainty about order status
4. Extra manual steps required
5. Delayed order fulfillment

Technical Implications

1. Not truly decentralized - requires active monitoring
2. Scalability concerns - manual refresh doesn't scale
3. Reliability issues - depends on user action
4. Performance overhead - fetching all events repeatedly

---

Recommended Solutions

Solution A: Enhanced Persistent Subscriptions

Implement redundant subscription mechanisms:

class EnhancedSubscriptionManager:
    def __init__(self):
        self.last_heartbeat = time.time()
        self.subscription_active = False

    async def maintain_subscription(self):
        while True:
            if not self.subscription_active or \
               time.time() - self.last_heartbeat > 30:
                await self.resubscribe_with_overlap()
            await asyncio.sleep(10)

    async def resubscribe_with_overlap(self):
        # Use timestamp with 5-minute overlap
        overlap_time = int(time.time()) - 300
        await subscribe_to_all_merchants(since=overlap_time)

Solution B: Periodic Auto-Refresh

Add automatic temporary subscriptions:

async def auto_refresh_loop():
    while True:
        await asyncio.sleep(60)  # Every minute
        merchants = await get_all_active_merchants()
        for merchant in merchants:
            await merchant_temp_subscription(merchant.pubkey, duration=5)

Solution C: WebSocket Health Monitoring

Implement connection health checks:

class WebSocketHealthMonitor:
    async def check_connection_health(self):
        try:
            # Send ping to nostrclient
            response = await nostr_client.ping(timeout=5)
            if not response:
                await self.reconnect_and_resubscribe()
        except Exception:
            await self.reconnect_and_resubscribe()

Solution D: Event Gap Detection

Detect and fill gaps in event sequence:

async def detect_event_gaps():
    # Check for gaps in event timestamps
    last_known = await get_last_event_time()
    current_time = int(time.time())

    if current_time - last_known > 60:  # 1 minute gap
        # Perform temporary subscription to fill gap
        await fetch_missing_events(since=last_known)

---

Implementation Priority

Phase 1: Quick Fixes (1-2 days)

1. [DONE] Increase temp subscription duration (10s → 30s)
2. [DONE] Add connection health logging
3. [DONE] Reduce startup delays

Phase 2: Reliability (3-5 days)

1. [TODO] Implement subscription heartbeat
2. [TODO] Add automatic resubscription on failure
3. [TODO] Create event gap detection

Phase 3: Full Solution (1-2 weeks)

1. [TODO] WebSocket connection monitoring
2. [TODO] Redundant subscription system
3. [TODO] Real-time order notifications
4. [TODO] Event deduplication logic

---

Testing Recommendations

Test Scenarios

1. Order during startup: Send order within 15 seconds of server start
2. Long-running test: Keep server running for 24 hours, send periodic orders
3. Connection interruption: Disconnect nostrclient, send order, reconnect
4. High volume: Send 100 orders rapidly
5. Network latency: Add artificial delay between components

Monitoring Metrics

• Time between order sent → order discovered
• Percentage of orders requiring manual refresh
• WebSocket connection uptime
• Subscription success rate
• Event processing latency

---

Conclusion

The current order discovery system relies on manual refresh due to:

1. Timing gaps in persistent subscriptions
2. Connection stability issues
3. Lack of redundancy in subscription management
4. No automatic recovery mechanisms

While the temporary subscription (manual refresh) provides a workaround, a proper solution requires implementing connection monitoring, subscription health checks, and automatic gap-filling mechanisms to ensure reliable real-time order discovery.

---

Appendix: Code References

Key Files

• /nostrmarket/tasks.py - Background task management
• /nostrmarket/nostr/nostr_client.py - Nostr client implementation
• /nostrmarket/services.py - Order processing logic
• /nostrmarket/views_api.py - API endpoints for refresh

Relevant Functions

• wait_for_nostr_events() - Main event loop
• subscribe_to_all_merchants() - Persistent subscription
• merchant_temp_subscription() - Manual refresh
• process_nostr_message() - Event processing

---

Document prepared: January 2025
Analysis based on: Nostrmarket v1.0
Status: Active Investigation