Stream Recovery E2E Testbook

End-to-end test for stream interruption, recovery, and cleanup scenarios.

Prerequisites

• Active browser stream (complete steps 1-7 from browser-streaming-e2e.md)
• Access to hydraneckwebrtc API
• Access to hydraguard for network interruption tests

---

Step 1: Start browser stream

Follow steps 1-7 from browser-streaming-e2e.md to establish a working video stream.

Expected: Video stream is active in the browser. Note the session ID from:

curl -s https://hydraneckwebrtc.experiencenet.com/api/v1/sessions | jq '.[0].id'

Step 2: Kill session server-side

# Replace SESSION_ID with the actual session ID
curl -X DELETE https://hydraneckwebrtc.experiencenet.com/api/v1/sessions/SESSION_ID

Expected within 5 seconds:

• Browser shows "stream ended" overlay
• Video freezes, then overlay appears
• moonlight-web-stream process terminates on the server

Step 3: Verify "Start new stream" button

1. Observe the stream-ended overlay in the browser
2. Click "Start new stream" / reconnect button

Expected: A new session is created, new moonlight-web-stream process spawns, video resumes.

Step 4: Verify new session

curl -s https://hydraneckwebrtc.experiencenet.com/api/v1/sessions | jq '.'

Expected: A new session with a different ID. The old session is gone.

---

Step 5: Test network interruption (WireGuard)

With an active stream:

1. Briefly disrupt WireGuard connectivity:

# On the hydraguard node — temporarily bring down the WG interface
# WARNING: this affects all traffic through the tunnel
wg-quick down wg0
sleep 10
wg-quick up wg0

2. Observe the browser during the interruption

Expected:

• Video freezes during the outage
• Within 30 seconds of WireGuard restoration, the stream recovers
• If the 30-second grace period expires, the session is terminated and cleanup runs

Step 6: Verify grace period behavior

1. Start a new stream
2. Note the session ID
3. Kill the moonlight-web-stream process directly:

# On the hydraneckwebrtc server
# Find the moonlight process for this session
ps aux | grep moonlight-web-stream
kill PID

4. Wait and observe:

Expected within 30 seconds:

• hydraneckwebrtc detects the dead process
• Session is marked as ended
• Browser shows stream-ended overlay
• No orphaned processes remain

Step 7: Verify orphan cleanup

After all tests, verify no orphaned resources:

# No stale moonlight processes
ps aux | grep moonlight-web-stream | grep -v grep

# No stale sessions
curl -s https://hydraneckwebrtc.experiencenet.com/api/v1/sessions | jq 'length'

# Body shows idle
curl -s -H "Authorization: Bearer TOKEN" \
  https://hydrabodystatus.experiencenet.com/api/v1/bodies | \
  jq '.[] | select(.stream_status == "streaming")'

Expected: No orphaned moonlight processes, no stale sessions, no bodies stuck in "streaming" state.

---

Recovery Scenarios Summary

| Scenario | Expected behavior | Timeout | |----------|------------------|---------| | Server kills session | Overlay in browser within 5s | 5s | | Network brief outage (<30s) | Stream resumes automatically | 30s grace | | Network long outage (>30s) | Session terminated, cleanup | 30s | | moonlight process crash | hydraneckwebrtc detects, cleans up | 30s | | Browser tab closed | 30s grace, then full cleanup | 30s | | Body becomes unreachable | Session times out, cleanup | Configurable |

Troubleshooting

| Symptom | Check | |---------|-------| | No overlay after kill | WebSocket/SSE connection to hydraneckwebrtc | | Stream doesn't recover | WireGuard tunnel status, ICE reconnection | | Orphaned processes | hydraneckwebrtc cleanup goroutine, process monitoring | | Stale sessions | Session TTL, heartbeat timeout config | | Body stuck streaming | hydrabody heartbeat, hydrabodystatus data freshness |