lnbits-exts/nostrmarket
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

Adds order discovery analysis document

Browse source 
Adds a document analyzing the order discovery mechanism in Nostrmarket.

The document identifies the reasons merchants need to manually refresh to see new orders, instead of receiving them automatically. It analyzes timing window issues, connection stability, subscription state management, and event processing delays. It proposes solutions such as enhanced persistent subscriptions, periodic auto-refresh, WebSocket health monitoring, and event gap detection.
This commit is contained in:
padreug 2025-11-04 00:38:29 +01:00
parent 725037e669
commit 4856bf2f89
1 changed files with 320 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

320
misc-docs/ORDER-DISCOVERY-ANALYSIS.md Normal file
View file

@ -0,0 +1,320 @@
# Nostrmarket Order Discovery Analysis
## Executive Summary
This document analyzes the order discovery mechanism in the Nostrmarket extension and identifies why merchants must manually refresh to see new orders instead of receiving them automatically through persistent subscriptions.
---
## Current Architecture
### Two Subscription Systems
The Nostrmarket extension implements two distinct subscription mechanisms for receiving Nostr events:
#### 1. **Persistent Subscriptions (Background Task)**
**Purpose**: Continuous monitoring for new orders, products, and merchant events
**Implementation**:
- Runs via `wait_for_nostr_events()` background task
- Initiated on extension startup (15-second delay)
- Creates subscription ID: `nostrmarket-{hash}`
- Monitors all merchant public keys continuously
**Code Location**: `/nostrmarket/tasks.py:37-49`
```python
async def wait_for_nostr_events(nostr_client: NostrClient):
while True:
try:
await subscribe_to_all_merchants()
while True:
message = await nostr_client.get_event()
await process_nostr_message(message)
```
**Subscription Filters**:
- Direct messages (kind 4) - for orders
- Stall events (kind 30017)
- Product events (kind 30018)
- Profile updates (kind 0)
#### 2. **Temporary Subscriptions (Manual Refresh)**
**Purpose**: Catch up on missed events when merchant clicks "Refresh from Nostr"
**Implementation**:
- Duration: 10 seconds only
- Triggered by user action
- Creates subscription ID: `merchant-{hash}`
- Fetches ALL events from time=0
**Code Location**: `/nostrmarket/nostr/nostr_client.py:100-120`
```python
async def merchant_temp_subscription(self, pk, duration=10):
dm_filters = self._filters_for_direct_messages([pk], 0)
# ... creates filters with time=0 (all history)
await self.send_req_queue.put(["REQ", subscription_id] + merchant_filters)
asyncio.create_task(unsubscribe_with_delay(subscription_id, duration))
```
---
## Problem Identification
### Why Manual Refresh is Required
#### **Issue 1: Timing Window Problem**
The persistent subscription uses timestamps from the last database update:
```python
async def subscribe_to_all_merchants():
last_dm_time = await get_last_direct_messages_created_at()
last_stall_time = await get_last_stall_update_time()
last_prod_time = await get_last_product_update_time()
await nostr_client.subscribe_merchants(
public_keys, last_dm_time, last_stall_time, last_prod_time, 0
)
```
**Problem**: Events that occur between:
- The last database update time
- When the subscription becomes active
...are potentially missed
#### **Issue 2: Connection Stability**
The WebSocket connection between components may be unstable:
```
[Nostrmarket] <--WebSocket--> [Nostrclient] <--WebSocket--> [Nostr Relays]
Extension Extension (Global)
```
**Potential failure points**:
1. Connection drops between nostrmarket → nostrclient
2. Connection drops between nostrclient → relays
3. Reconnection doesn't re-establish subscriptions
#### **Issue 3: Subscription State Management**
**Current behavior**:
- Single persistent subscription per merchant
- No automatic resubscription on failure
- No heartbeat/keepalive mechanism
- No verification that subscription is active
#### **Issue 4: Event Processing Delays**
The startup sequence has intentional delays:
```python
async def _subscribe_to_nostr_client():
await asyncio.sleep(10) # Wait for nostrclient
await nostr_client.run_forever()
async def _wait_for_nostr_events():
await asyncio.sleep(15) # Wait for extension init
await wait_for_nostr_events(nostr_client)
```
**Problem**: Orders arriving during initialization are missed
---
## Why Manual Refresh Works
The temporary subscription succeeds because:
1. **Fetches from time=0**: Gets ALL historical events
2. **Fresh connection**: Creates new subscription request
3. **Immediate processing**: No startup delays
4. **Direct feedback**: User sees results immediately
```python
# Temporary subscription uses time=0 (all events)
dm_filters = self._filters_for_direct_messages([pk], 0) # ← 0 means all time
# Persistent subscription uses last update time
dm_filters = self._filters_for_direct_messages(public_keys, dm_time) # ← can miss events
```
---
## Impact Analysis
### User Experience Issues
1. **Merchants miss orders** without manual refresh
2. **No real-time notifications** for new orders
3. **Uncertainty** about order status
4. **Extra manual steps** required
5. **Delayed order fulfillment**
### Technical Implications
1. **Not truly decentralized** - requires active monitoring
2. **Scalability concerns** - manual refresh doesn't scale
3. **Reliability issues** - depends on user action
4. **Performance overhead** - fetching all events repeatedly
---
## Recommended Solutions
### Solution A: Enhanced Persistent Subscriptions
**Implement redundant subscription mechanisms:**
```python
class EnhancedSubscriptionManager:
def __init__(self):
self.last_heartbeat = time.time()
self.subscription_active = False
async def maintain_subscription(self):
while True:
if not self.subscription_active or \
time.time() - self.last_heartbeat > 30:
await self.resubscribe_with_overlap()
await asyncio.sleep(10)
async def resubscribe_with_overlap(self):
# Use timestamp with 5-minute overlap
overlap_time = int(time.time()) - 300
await subscribe_to_all_merchants(since=overlap_time)
```
### Solution B: Periodic Auto-Refresh
**Add automatic temporary subscriptions:**
```python
async def auto_refresh_loop():
while True:
await asyncio.sleep(60) # Every minute
merchants = await get_all_active_merchants()
for merchant in merchants:
await merchant_temp_subscription(merchant.pubkey, duration=5)
```
### Solution C: WebSocket Health Monitoring
**Implement connection health checks:**
```python
class WebSocketHealthMonitor:
async def check_connection_health(self):
try:
# Send ping to nostrclient
response = await nostr_client.ping(timeout=5)
if not response:
await self.reconnect_and_resubscribe()
except Exception:
await self.reconnect_and_resubscribe()
```
### Solution D: Event Gap Detection
**Detect and fill gaps in event sequence:**
```python
async def detect_event_gaps():
# Check for gaps in event timestamps
last_known = await get_last_event_time()
current_time = int(time.time())
if current_time - last_known > 60: # 1 minute gap
# Perform temporary subscription to fill gap
await fetch_missing_events(since=last_known)
```
---
## Implementation Priority
### Phase 1: Quick Fixes (1-2 days)
1. [DONE] Increase temp subscription duration (10s → 30s)
2. [DONE] Add connection health logging
3. [DONE] Reduce startup delays
### Phase 2: Reliability (3-5 days)
1. [TODO] Implement subscription heartbeat
2. [TODO] Add automatic resubscription on failure
3. [TODO] Create event gap detection
### Phase 3: Full Solution (1-2 weeks)
1. [TODO] WebSocket connection monitoring
2. [TODO] Redundant subscription system
3. [TODO] Real-time order notifications
4. [TODO] Event deduplication logic
---
## Testing Recommendations
### Test Scenarios
1. **Order during startup**: Send order within 15 seconds of server start
2. **Long-running test**: Keep server running for 24 hours, send periodic orders
3. **Connection interruption**: Disconnect nostrclient, send order, reconnect
4. **High volume**: Send 100 orders rapidly
5. **Network latency**: Add artificial delay between components
### Monitoring Metrics
- Time between order sent → order discovered
- Percentage of orders requiring manual refresh
- WebSocket connection uptime
- Subscription success rate
- Event processing latency
---
## Conclusion
The current order discovery system relies on manual refresh due to:
1. **Timing gaps** in persistent subscriptions
2. **Connection stability** issues
3. **Lack of redundancy** in subscription management
4. **No automatic recovery** mechanisms
While the temporary subscription (manual refresh) provides a workaround, a proper solution requires implementing connection monitoring, subscription health checks, and automatic gap-filling mechanisms to ensure reliable real-time order discovery.
---
## Appendix: Code References
### Key Files
- `/nostrmarket/tasks.py` - Background task management
- `/nostrmarket/nostr/nostr_client.py` - Nostr client implementation
- `/nostrmarket/services.py` - Order processing logic
- `/nostrmarket/views_api.py` - API endpoints for refresh
### Relevant Functions
- `wait_for_nostr_events()` - Main event loop
- `subscribe_to_all_merchants()` - Persistent subscription
- `merchant_temp_subscription()` - Manual refresh
- `process_nostr_message()` - Event processing
---
_Document prepared: January 2025_
_Analysis based on: Nostrmarket v1.0_
_Status: Active Investigation_