Evals Hub
This page organizes Dramatica evaluations as benchmark cards so researchers and studio technical teams can inspect signal quality quickly.
Framing
Each benchmark card is reported with:
- Task
- Data
- Metric
- Baselines
- Verifier loop result
- Qualitative notes
Constraints Layer
Benchmark Card: Storyform Constraint Compliance
- Task: detect hard Storyform violations in candidate drafts
- Data: prompt-to-draft pairs conditioned on fixed Storyform specs
- Metric: pass rate (constraint-valid outputs)
- Baselines: vanilla generation, prompting-only guardrails
- Verifier loop result: Placeholder
- Qualitative notes: exposes early structural drift before full rewrite cycles
Benchmark Card: Throughline Role Integrity
- Task: validate whether Objective Story, Main Character, Influence Character, and Relationship Story roles remain distinct and consistent
- Data: controlled scenarios with targeted Throughline swaps
- Metric: violation detection accuracy
- Baselines: prompting-only checks
- Verifier loop result: Placeholder
- Qualitative notes: catches hidden role collapse that often appears late in drafts
Alignment Layer
Finding: StoryBeat Alignment Across Domains
Dramatica evaluates StoryBeat generation for structural alignment, not just style or plausibility. A generated StoryBeat should illustrate the requested Throughline, Domain, Signpost, and scope while still reading as usable storytelling.
In internal evaluation runs, Objective Story Signposts passed StoryBeat alignment checks across all four Domains. Each run was tested both without extra notes and with ordinary writer-provided ideas, such as a concrete event, discovery, split, or reversal that needed to be preserved inside the Storyform.
| Objective Story Domain | Signpost Sequence | Result |
|---|---|---|
| Physics | Understanding, Learning, Doing, Obtaining | Passed internal alignment evaluation |
| Psychology | Conceptualizing, Being, Becoming, Conceiving | Passed internal alignment evaluation |
| Universe | Past, Progress, Future, Present | Passed internal alignment evaluation |
| Mind | Memory, Preconscious, Subconscious, Conscious | Passed internal alignment evaluation |
This matters because a StoryBeat can sound dramatic while still doing the wrong structural job. Alignment checks help separate "this reads like a scene" from "this beat is carrying the intended Storyform pressure."
Writer notes remain creative input. A note like "they escape" or "the family splits in two" is treated as story material, not as a replacement for the Storyform. The alignment pass preserves the idea while checking whether the resulting beat still performs the correct narrative function.
Writing use: When reviewing generated Storybeats, ask whether the beat changes the story through the intended structural pressure, not merely whether it mentions the right Dramatica term.
Benchmark Card: Throughline Voice Classification
- Task: classify whether narrative pressure matches intended Throughline perspective
- Data: candidate passages labeled by intended Throughline pressure
- Metric: agreement and classification accuracy
- Baselines: vanilla model, prompting-only
- Verifier loop result: Placeholder
- Qualitative notes: strongest gains appear on Influence Character vs Main Character disambiguation
Benchmark Card: Dynamic Consistency
- Task: evaluate whether Dynamics remain coherent through revisions
- Data: multi-draft sequences under fixed Dynamic settings
- Metric: consistency score, pairwise agreement
- Baselines: best-of-N without verifier
- Verifier loop result: Placeholder
- Qualitative notes: reduces “dead-on-arrival” rewrites that read fine but violate intended argument
Benchmark Card: Storybeat Advancement
- Task: verify that Storybeats advance intended narrative pressure rather than repeat surface events or drift into the wrong Throughline
- Data: beat-level generation traces across Objective Story Signpost sequences in Physics, Psychology, Universe, and Mind
- Metric: alignment pass rate by Domain, Signpost, and note condition
- Baselines: vanilla generation
- Verifier loop result: Objective Story Signpost fixtures passed internal StoryBeat alignment evaluation across all four Domains, both with and without writer-provided notes
- Qualitative notes: strongest signal appears when the writer note is concrete but structurally ambiguous; the verifier checks whether the note remains story material while the beat keeps the correct Storyform job
Quality and Diversity Layer
Benchmark Card: Diverse Valid Candidates
- Task: measure whether best-of-N preserves distinct options after verification
- Data: same prompt and Storyform across N candidates
- Metric: count of distinct valid candidates per run
- Baselines: unconstrained best-of-N, single-pass generation
- Verifier loop result: Placeholder
- Qualitative notes: confirms the system narrows to meaning-correct without forcing one style
Benchmark Card: Human Selection Lift
- Task: compare human preference between verified candidate sets and baseline sets when intent must remain fixed
- Data: blind human selection rounds
- Metric: selection rate lift
- Baselines: prompting-only and vanilla sets
- Verifier loop result: Placeholder
- Qualitative notes: separates intent correctness from taste choice in review workflows
Pipeline summary
Storyform Spec -> Generator -> Dramatica Verifier (constraints + scores) -> Best-of-N selection -> Human choice
The verifier narrows outputs to meaning-correct candidates. Human reviewers choose voice and taste.