Analysis-run performance telemetry

Streamlit UI load profiling (core/observability/perf.py, TRANSCRIPTX_STREAMLIT_PERF) is unrelated. This document covers analysis-run wall time, module duration_ms, and .transcriptx/run_performance.json.

Phase 0 findings

  • Optional module_outcomes[].duration_ms / used_cache survive load_run_results without a schema bump (RUN_RESULTS_SCHEMA_VERSION remains 2).

  • Prefer a narrow validator before a full typed List[Dict] → model migration if churn appears.

Wall-clock scope (timing_scope_version: 1)

Transcript

  1. Start at RunOrchestrator.run entry (perf_counter).

  2. Include preparation, execution, and all required persistence.

  3. Stop after required persistence; write optional run_performance.json outside the measured interval, still under the same per-run lease.

Group

  1. Start at entry to the group branch of run_analysis_pipeline (perf_counter).

  2. Include sequential member execution and required group finalisation persistence (aggregation/synthesis, required group artifacts, final group run_results.json commit and validation).

  3. Stop immediately before the optional performance-sidecar write; write .transcriptx/run_performance.json while the group writer lease remains held.

Group runs use a separate recorder; members keep their own. The group recorder is not bound as the active ContextVar during member execution (each member RunOrchestrator binds and restores its own recorder).

Group wall time is an independent end-to-end measurement. Do not calculate it by summing member wall times. Group wall may exceed the sum of member walls because it includes preparation, aggregation, and persistence overhead.

Authority

  • Module status + duration_ms: run_results.json only (for members: each member’s file; group rollup does not invent group module timings). Member module timings remain authoritative in member run_results.json files.

  • Sidecar: non-authoritative run-level telemetry + stable interpretative context (mode/profile/runtime/workload snapshots allowed). Group sidecars include wall time, status, analysis context, workload when available, and GroupPerformanceMeta — not member durations or LLM aggregates.

  • Group LLM metrics are not collected in schema v1 (omit llm; do not report zeros). This is an intentional instrumentation gap, not “zero calls”.

  • run_report.json: labelled legacy display fallback only.

  • Optional group sidecar write failure must not invalidate an otherwise committed group run; coded warnings may appear under group_phase_metadata.performance_sidecar_warning.

Loader statuses

missing | malformed | unsupported_schema | oversized | io_error — the loader cannot know “legacy”; the UI uses surrounding run metadata to distinguish old runs from telemetry loss.

Phase 2 — Retained-run snapshot exporter

Optional Prometheus textfile exporter over currently retained committed runs (valid run_results.json on disk). Independent of the Streamlit Performance page; GUI never depends on it.

Design

  • Each export cycle rescans transcript + group output trees and regenerates the entire metric snapshot.

  • Metrics are gauges (including histogram-bucket cumulative counts). When a run directory is deleted, the next scan simply omits it — gauges shrink.

  • No mtime-based ingest state file and no Prometheus counters that cannot delete observations.

  • Committed-run inventory is path-safe, deterministically ordered, capped (max_runs), and isolates per-candidate faults. It does not use RunIndex.list_runs (user-visible artifact filter).

How to run

python scripts/export_run_performance_snapshot.py
python scripts/export_run_performance_snapshot.py -o /var/lib/node_exporter/textfile/transcriptx_run_perf.prom
python scripts/export_run_performance_snapshot.py --outputs-dir /path/to/outputs --max-runs 5000

Programmatic:

from transcriptx.core.observability.run_performance import (
    SnapshotExportConfig,
    export_retained_run_snapshot,
)

export_retained_run_snapshot(
    SnapshotExportConfig(
        outputs_dir=...,
        group_outputs_dir=...,
        textfile_path=...,
    )
)

Config knobs

Env / flag

Meaning

TRANSCRIPTX_RUN_PERF_EXPORT_PATH / --output

Textfile destination (default: $TRANSCRIPTX_DATA_DIR/state/run_performance_snapshot.prom)

TRANSCRIPTX_RUN_PERF_EXPORT_MAX_RUNS / --max-runs

Scan cap (default 10000)

TRANSCRIPTX_OUTPUT_DIR / --outputs-dir

Outputs root

Compose / node_exporter profile wiring is deferred (plan: later).

Metric universe (low cardinality)

All # TYPE … gauge. Labels never include run_id, transcript_key, paths, fingerprints, or exception text. Model identity is capped/normalised; mode is quick | full | other | unknown.

Metric

Meaning

transcriptx_retained_runs{target_type,execution_status,mode,cache_provenance}

Count of retained committed runs

transcriptx_retained_run_wall_seconds_bucket{target_type,le}

Cumulative wall-duration histogram buckets (seconds)

transcriptx_retained_run_wall_seconds_sum / _count

Wall duration sum / count

transcriptx_retained_module_outcomes{module_id,status}

Module outcome counts from run_results

transcriptx_retained_module_duration_seconds_bucket{module_id,le}

Started-module duration buckets

transcriptx_retained_llm_calls{model,result}

Logical LLM success/failure from sidecars (when present)

transcriptx_retained_scan_*

Last-scan candidates / errors / truncated / sidecar presence

Package layout: inventory.py (scan), exporter.py (aggregate + textfile), scripts/export_run_performance_snapshot.py.