Log Sources Reference

Where to find operational logs for every component in the streaming path, and what the streaming monitor surfaces vs. what requires direct access.

How sources feed into the streaming monitor

The /logs page merges three live feeds, each pulled from a different backend:

| /logs source | Backed by | What flows through | |---------------|-----------|-------------------| | Body Logs | hydracluster → hydrabody.log on each body | All [vdd], [portrait], [stream], [gpu-mismatch], [sunshine] lines written by hydrabody; also ffplay/ffmpeg running-status from the heartbeat | | Sessions | hydracluster session records | Session open/close events scoped to a body+head pair, with duration and reason | | Anomalies | hydrabodystatus anomaly detector | orphan_stream detections (GPU high, stream idle) and auto-relaunch events |

Everything else — FFmpeg session output, VDD config files, Sunshine log, hydraneckwebrtc, raw hydracluster server logs — is not surfaced on /logs and requires direct access as described below.

hydrabody.log  ──── hydracluster ────► Body Logs  ┐
hydracluster sessions ───────────────► Sessions   ├─► /logs (unified feed)
hydrabodystatus anomalies ───────────► Anomalies  ┘

C:\ProgramData\hydravoice\session-debug.log  ─► exec only
C:\Sunshine\config\sunshine.log              ─► exec only
C:\VirtualDisplayDriver\vdd_settings.xml     ─► exec only
C:\Sunshine\config\display_device.state      ─► exec only
hydraneckwebrtc journalctl                   ─► SSH only
hydracluster server journalctl               ─► SSH only

What the streaming monitor surfaces

Unified view

/logs — start here during an incident. Merges body logs, session start/end events, and anomalies from the last 2 hours into a single timestamp-sorted feed. Filter by source (Body Logs / Sessions / Anomalies) and by individual body using the checkboxes at the top. Auto-refreshes every 30 s.

Individual sources

| Source | What you get | Where in the UI | |--------|-------------|-----------------| | hydrabodystatus | GPU temp/utilization, stream status, provider status, recent cleanups | /stream, /bodies | | hydrabodystatus anomalies | Relaunch events (24h) and GPU mismatch (orphan_stream) detections | /api/v1/body-anomalies and /logs (source: anomaly) | | hydracluster body logs | Full hydrabody log for a node | /logs (source: body), /bodies/{id}/logs, Recent Logs panel on /stream | | hydracluster session logs | Paired body + head log lines scoped to a session time window | /sessions/{id}/logs (the "logs" link in the sessions table) | | hydracluster events | Live SSE stream: session opened/closed, stream started/ended, watchdog events | /events | | hydracluster nodes | Node ID, venue, owner, stream_status, WireGuard IP | /bodies, /stream | | hydracluster heads | LiveBodyID (which body the head is paired to), kiosk name, diagnostics | /heads | | hydracluster sessions | Active and history sessions with body/head/experience/duration | /sessions and /logs (source: session) |

What requires direct access

These sources are not proxied through the streaming monitor. Access them via journalctl on the server or hydracluster exec on the body.

hydraneckwebrtc

WebRTC session lifecycle, moonlight-web-stream spawns, mic relay events.

ssh root@46.225.220.240 \
  "journalctl -u hydraneckwebrtc.service -n 100 --no-pager"

Key log patterns:

session created id=... — new WebRTC session spawned
session ended id=... reason=... — session teardown
mic: ... — mic relay events (RTP forwarding, drops)
moonlight-web-stream PID changes — process crash/restart

FFmpeg / hydravoice (audio pipeline, on the body machine)

FFmpeg is part of the audio pipeline: iPad mic → RTP/UDP:47995 → hydravoice → ffmpeg → hydravoice-player → VB-Cable → Unreal Engine. It runs as a subprocess of hydravoice.exe on the body machine (C:\hydravoice\ffmpeg.exe).

C:\ProgramData\hydravoice\session-debug.log — captures every ffmpeg session: the full command line, stdout/stderr from ffmpeg, and session start/end events. Append-only; grows across stream sessions. This is the primary source for audio pipeline failures (Opus decode errors, device not found, pipe breaks).

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\ProgramData\\hydravoice\\session-debug.log' -Tail 60\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" --timeout 10s

FFmpeg does not appear on /logs — hydrabody.log surfaces the audio status (ffplay_running, ffplay_pid) as part of the heartbeat, but ffmpeg's own session output is only in session-debug.log.

Key patterns:

[ffmpeg] started in session N (PID XXXXX) — ffmpeg launched
ffmpeg stderr lines showing codec/device errors
Silence after a stream start with microphone enabled — ffmpeg failed to start; check session-debug.log for the error

hydracluster (server-side)

Session watchdog decisions, head/body heartbeat timeouts, stream open/close audit trail.

ssh root@hydracluster.experiencenet.com \
  "journalctl -u hydracluster.service -n 200 --no-pager"

Or via API (same data, structured):

# Live SSE
curl -sf -N -H "Authorization: Bearer $HYDRACLUSTER_TOKEN" \
  "https://hydracluster.experiencenet.com/api/v1/events"

Key log patterns:

[session-watchdog] body/head heartbeat silent >60s — marking terminate pending
[session] opened/closed body=... reason=... duration=...
[body] stream started/ended body=...
[session] terminate signal drained by body=...

hydrabody (on the body machine)

The authoritative record for what hydrabody did: ForceStop attempts, process kills, VDD restarts, GPU mismatch tracking.

Access via hydracluster exec (do not SSH directly to bodies):

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\Windows\\System32\\config\\systemprofile\\.hydrabody\\hydrabody.log' -Tail 50\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" \
  --timeout 15s 2>&1 | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g'

The same log is available in the UI at /bodies/{id}/logs (replace {id} with the node ID, e.g. node-b961f1c8).

Key log patterns:

[sunshine] stream started/ended: <appName> — Sunshine hook fired
[stream] stream started/ended: <appName> orientation=... — hydrabody recorded it
[stream] hydracluster requested stream termination — calling ForceStop
[stream] ForceStop failed (close running app: HTTP 400: ...) — Content-Type bug (issue #367), falls back to process kill
[stream] killed N process(es) for <appName> by name — fallback process kill succeeded
[gpu-mismatch] detected: GPU at N% with stream status idle — mismatch alarm started
[gpu-mismatch] cleared — GPU dropped, alarm resolved
[portrait] deleted display_device.state for VDD GUID reset — portrait cleanup, Sunshine restart imminent
[vdd] restarting Sunshine to apply config changes... — body unavailable for ~3 minutes

Sunshine (on the body machine)

Low-level stream hook events. Rarely needed; hydrabody.log usually captures what matters.

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\Program Files\\Sunshine\\config\\sunshine.log' -Tail 40\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" \
  --timeout 15s 2>&1 | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g'

Note: Sunshine log path is C:\Sunshine\config\sunshine.log (not C:\Program Files\Sunshine) on Hydra bodies — the provider is extracted to C:\Sunshine.

Virtual Display Driver (VDD)

The VDD (MttVDD) has no dedicated log file. Relevant state is spread across three files, all on the body machine:

C:\VirtualDisplayDriver\vdd_settings.xml — driver configuration: monitor count, GPU name, available resolutions. Written by hydrabody during provisioning. Changes require a driver reinstall or reboot to take effect. Check this when a portrait resolution (1080×1920) is missing or a custom resolution is not offered by Sunshine.

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\VirtualDisplayDriver\\vdd_settings.xml'\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" --timeout 10s

C:\Sunshine\config\display_device.state — Sunshine's display topology snapshot (JSON). Contains the GUID of every display path Sunshine has seen. hydrabody deletes this file before a portrait stream to force Sunshine to rediscover the VDD GUID on the next stream start. If this file is stale after a VDD disable/enable cycle, Sunshine will reference the wrong GUID and portrait switching will fail silently. The hydrabody log pattern [portrait] deleted display_device.state for VDD GUID reset confirms the delete fired.

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\Sunshine\\config\\display_device.state'\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" --timeout 10s

C:\Windows\System32\config\systemprofile\.hydrabody\vdd_version.txt and vdd_sunshine_configured.txt — hydrabody marker files. vdd_version.txt stores the installed VDD version; vdd_sunshine_configured.txt marks that Sunshine has already been configured for VDD. If either is missing, hydrabody will re-run the provisioning sequence on next start.

$HYDRACLUSTER_BIN exec "$NODE_ID" \
  "powershell -NoProfile -Command \"Get-Content 'C:\\Windows\\System32\\config\\systemprofile\\.hydrabody\\vdd_version.txt'; Get-Content 'C:\\Windows\\System32\\config\\systemprofile\\.hydrabody\\vdd_sunshine_configured.txt'\"" \
  --server "$HYDRACLUSTER_SERVER" --admin-token "$HYDRACLUSTER_TOKEN" --timeout 10s

VDD driver events do not have a dedicated Windows Event Log channel. The hydrabody.log [vdd] and [portrait] lines are the primary diagnostic source — and these do appear on /logs (Body Logs source) in real time. The config files above require direct exec access.

hydrabodystatus (server-side)

Aggregated heartbeat store. The streaming monitor reads its API; direct log access is rarely needed.

ssh root@78.47.174.83 \
  "journalctl -u hydrabodystatus.service -n 100 --no-pager"

hydrastreamingmonitor (server-side)

ssh root@78.47.174.83 \
  "journalctl -u hydrastreamingmonitor.service -n 100 --no-pager"

Quick decision tree

"A head can't start a stream"

Check /bodies Active Problems panel — is the body orphaned?
Check /stream — is the body online, idle, and Sunshine running?
Check /bodies/{id}/logs — look for [vdd] restarting Sunshine (3-minute blackout window after portrait stream)
If body is idle and Sunshine is running but head still fails: check hydraneckwebrtc logs for session creation errors

"A stream ended unexpectedly"

Check /sessions — find the session, click "logs"
In the session logs, look for [stream] hydracluster requested stream termination (watchdog) vs. [sunshine] stream ended (Sunshine-initiated)
Check /events for the session close event and reason

"GPU is high but no stream is active"

GET /api/v1/body-anomalies — orphan_stream entry confirms the mismatch
/bodies/{id}/logs — look for [gpu-mismatch] lines and whether [stream] killed N process(es) fired
If mismatch persists >3 minutes: hydrabody's 180s watchdog will fire automatically; use the Active Problems panel Stop button to force it sooner

"ForceStop HTTP 400 errors in body logs" Known bug (issue #367): hydrabody does not send Content-Type: application/json to Sunshine's CloseRunningApp endpoint. Sunshine rejects the request. The fallback process kill by name succeeds. No action needed — streams do terminate cleanly via the fallback.

Environment variables for log access

These are set in the session-monitoring runbook setup section:

HYDRACLUSTER_TOKEN="c21ff820b95c59c5301d797b58fa262240a127c81b45262f314022647623b76d"
HYDRACLUSTER_BIN="/home/claude-user/hydracluster/bin/hydracluster"
HYDRACLUSTER_SERVER="https://hydracluster.experiencenet.com"
BODYSTATUS_TOKEN="b191130cb189b0b74337532e766220d9cda223a38bc7282a67f976167b9dfaeb"