← Back to docs

Event Stream Validation Runbook

Pre-event validation of mercator-talks streams across all event districts using a flatscreen kiosk head and the streaming monitor dashboard.

1. Overview

Purpose: validate that mercator-talks streams correctly from each render body to cheeky-cactus-86 (hydraheadflatscreen) before bringing iPad heads online for an event.

Components involved:

2. Prerequisites

Before starting, verify all of the following:

Verify via hydracluster API:

curl -s -H "Authorization: Bearer <admin_token>" https://hydracluster.experiencenet.com/api/v1/nodes \
  | jq '.[] | select(.roles[] == "hydrabody") | {name, district, venue, status, stream_status}'

3. Screenshot Validation Protocol

Apply this protocol for each district. Capture all screenshots for the record.

| Step | Action | Expected result | |------|--------|----------------| | 1 | Screenshot before launch | Idle home screen on cheeky — baseline | | 2 | Move cheeky to target district (see §4) | API returns 200 | | 3 | Launch mercator-talks from cheeky | Only mercator-talks visible in the grid | | 4 | Screenshot after ~5s | Loading indicator visible | | 5 | Screenshot during active stream (×3-5) | mercator-talks running in portrait orientation | | 6 | Stop stream | — | | 7 | Screenshot after stop (×2-3) | Clean exit, back to home screen |

Screenshot via exec (requires hydraheadflatscreen v2.0.38+):

curl -s -H "Authorization: Bearer <admin_token>" \
  https://hydracluster.experiencenet.com/api/v1/nodes/<cheeky-node-id>/screenshot

4. Streaming Monitor Checks

Open hydrastreamingmonitor.experiencenet.com and monitor throughout each validation.

| Phase | Expected state | |-------|---------------| | Before launch | body=online, neck=healthy, no active sessions | | Stream loading | body=connecting, neck=relay-establishing | | Active stream | body=streaming, neck=relaying, head=connected, session=active | | After stop | All components idle, session=ended |

Any deviation from expected state is a blocker — do not proceed to the next district until resolved.

5. Session Registry Checks (hydracluster v2.0.84+)

The session registry infers sessions from body status transitions and tracks the active head via live_body_id heartbeats. Check it alongside the streaming monitor.

Before launch — no active sessions:

curl -s -H "Authorization: Bearer <admin_token>" \
  https://hydracluster.experiencenet.com/api/v1/sessions | jq .
# Expect: empty array []

During active stream — one session per district:

curl -s -H "Authorization: Bearer <admin_token>" \
  https://hydracluster.experiencenet.com/api/v1/sessions | jq .
# Expect: session with body_id=<target-body-node-id>, head_id=<cheeky-node-id>,
#         app="mercator-talks", started_at set, ended_at null

After stop — session closed, history updated:

curl -s -H "Authorization: Bearer <admin_token>" \
  https://hydracluster.experiencenet.com/api/v1/sessions | jq .
# Expect: empty array [] (no active sessions)

curl -s -H "Authorization: Bearer <admin_token>" \
  https://hydracluster.experiencenet.com/api/v1/sessions/history | jq .
# Expect: closed session appears with ended_at set

Any orphan active session after a clean stop is a blocker — it indicates the session registry is out of sync with body state.

6. Per-District Sequence

Run districts in this order (start with the district least likely to affect others if something goes wrong).

6.1 — Move cheeky to target district

Only the district changes. Cheeky's venue stays fixed at its physical location (e.g. nerdland) throughout — cross-venue streaming (nerdland head → cloud-seven body) is supported.

curl -s -X POST https://hydracluster.experiencenet.com/api/v1/nodes/<cheeky-node-id>/district \
  -H "Authorization: Bearer <admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"district":"<target-district>","venue":"nerdland"}'

6.2 — Validate

Follow the screenshot protocol (§3), streaming monitor checks (§4), and session registry checks (§5).

6.3 — Pass criteria

6.4 — Move to next district

Repeat §6.1-6.3 for each remaining district.

7. iPad Go-Live (after all cheeky districts pass)

Once all cheeky validations pass:

  1. Power on each event iPad
  2. Verify each iPad comes online in hydracluster (status=online)
  3. Each iPad should see only mercator-talks in its district
  4. Each iPad streams from the body in its district
  5. Confirm portrait orientation

iPad districts must match body districts 1:1 (same district, same owner).

8. Go / No-Go Criteria

| Check | Pass condition | |-------|---------------| | All cheeky districts validated | Screenshot protocol passed for each | | No unexpected experiences visible | Only mercator-talks in event districts | | Streaming monitor clean | Start + stop clean for every district | | Session registry clean | Active session during stream, no orphans after stop | | iPads online | Each iPad connects and streams |

Do not open the event if any criterion is unresolved.

9. Troubleshooting

| Symptom | Where to look | |---------|---------------| | Body not streaming | hydrabody/docs/runbooks/ | | Neck relay not establishing | hydraneck/docs/runbooks/ | | Streaming monitor shows wrong state | hydrastreamingmonitor/docs/runbooks/session-monitoring.md | | Flatscreen kiosk not responding | hydraheadflatscreen/docs/runbooks/runbook.md | | iPad not connecting | hydraheadipad/docs/runbooks/ | | Experience grid shows wrong experiences | hydraexperiencelibrary API — verify district venue lists | | Body online but no experiences visible | Check head district matches body district and owner | | Session stays active after stream stop | Body status stuck at streaming — force stop via POST /api/v1/nodes/{id}/exec with stream/stop, then verify session/history | | Session registry returns 404 | hydracluster not yet on v2.0.84+ — skip session checks and rely on streaming monitor |

10. Post-Event