Lexical diversity analysis

Deterministic module (lexical_diversity, tier T0, category light) that measures vocabulary variety per speaker and globally. It does not call an LLM and has no pydantic config — thresholds and bucket size are module constants.

Distinct from understandability.lexical_density (NLTK tokenizer / readability context). This module uses a pinned Unicode tokenizer and reports ttr, mtld, and hapax_rate.

Requirements

Registry requirements: SEGMENTS + SPEAKER_LABELS only (gate_on_turn_taking_speakers: true).

Segment timestamps are optional. When valid finite non-negative start/end values are present on all eligible segments, global time buckets are emitted; otherwise time_buckets is [] and analysis still succeeds.

Metrics

Metric

Definition

Empty / short text

token_count / type_count / hapax_count

Counts after tokenization

0 when empty

ttr

type_count / token_count

null when empty

hapax_rate

hapax_count / type_count (types denominator)

null when empty

mtld

Forward + reverse MTLD (factor threshold 0.72), averaged

null when tokens below MIN_MTLD_TOKENS (50)

JSON never contains NaN or Infinity. Canonical JSON keeps full finite precision; CSV rounds display floats to 6 decimal places and writes empty cells for null MTLD/rates.

Tokenizer (v1)

  • Case-fold, then match [^\W\d_]+(?:['\u2019-][^\W\d_]+)*

  • Tokens shorter than 2 characters are dropped

  • Contractions/possessives/hyphenated forms stay one token (don't, speaker's, state-of-the-art)

  • Digits alone and underscores as token characters are excluded (hello_world → hello, world)

Metadata in every payload includes schema_id, algorithm_version, tokenizer_version, mtld_factor_threshold, min_mtld_tokens, and bucket_seconds (60).

Interpretation limits

  • TTR is highly length-sensitive — do not treat higher TTR as unconditionally “better”.

  • MTLD reduces length sensitivity but is unstable on short inputs (hence the 50-token floor).

  • Hapax rate can reflect names, ASR errors, or topic-specific vocabulary.

Output layout

lexical_diversity/data/global/{base}_lexical_diversity.json
lexical_diversity/data/global/{base}_lexical_diversity.csv
lexical_diversity/charts/.../lexical-ttr.png   # and mtld / hapax-rate when data allows

JSON envelope (schema_id: transcriptx.lexical_diversity.v1):

  • metadata — algorithm/tokenizer constants

  • global_stats — metrics over eligible concatenated text (same population as speaker analysis)

  • speaker_stats — per eligible turn-taking speaker

  • time_buckets — global-only buckets ([t0 + k*60, t0 + (k+1)*60) by segment start); empty when timestamps unavailable

  • exclusions — skipped segment counts / reasons

CSV is one tidy table with columns: scope, speaker, bucket_start, bucket_end, metric fields. Scopes: global | speaker | time_bucket.

UI and export

  • Insights (default layout): block lexical_diversity_block shows global metrics, per-speaker table, and optional time buckets.

  • Overview module metrics: summary extractor shows global TTR/MTLD/hapax with a length-sensitivity caption.

  • Charts gallery: viz IDs lexical_diversity.ttr.speaker, .mtld.speaker, .hapax_rate.speaker (MTLD omits null bars).

  • Zip export: JSON, CSV, and chart PNGs are included via the manifest-driven export path.

Group aggregation

Session/speaker rows are collected for group charts. Allowlisted numeric fields: ttr, mtld, hapax_rate, token_count.

  • token_count is additive across transcripts.

  • type_count is not summed (vocabulary overlap would invalidate derived TTR).

  • Mean/median of ratio metrics across sessions are descriptive approximations, not exact pooled lexical diversity. No full token vocabularies are persisted for exact cross-session recomputation.