Story Quality

Verified 2026-09-06 by real runs against production: a multi-voice audio file imported through the new file picker, transcribed by Gemini with speaker turns, converted into a chapter, and read back. Timings below are measured, not estimated. Throwaway data deleted.

Scope: the pipeline that turns a recording, an uploaded file, or an Elena conversation into written prose — how faithful it is, and how long it takes. Paths are relative to apps/web-app.

1. How a recording becomes a story

Every stage is announced to the browser as it starts, over SSE (Accept: text/event-stream), and every stage is timed into story_conversions.metadata.stage_timings.

2. Why the gate was slow

The gate ran a full JSON assessment — episodes, gaps, suggested questions, guidance — on gemini-3.6-flash before a single word was written.

Two things made it worse than the sum suggests. Accepting “Create a shorter story anyway” re-ran the whole request, so a thin clip paid for the readiness call twice. And every model call goes through executeFeature, which makes five Supabase round trips before it reaches the model.

The deciding observation: the arithmetic was already doing the real work. The word floors capped how optimistic the model was allowed to be, so a verdict of “ready” on 100 words was overridden to “partial” regardless. What the model uniquely added was episode segmentation — which only matters when several sources are combined.

3. What changed

• The gate is arithmetic. assessStoryReadiness is synchronous and never leaves the process. Measured readiness_ms: 0.
• Richness rides with generation. One call returns the story and a self-assessment (richness, coherence, storyteller, voice_markers_kept), so the judgement is paid for once instead of before and again after.
• Segmentation only for multi-source. A single recording, upload or conversation never pays for it.
• Pre-checks run together. Sources, book access and target chapter are one Promise.all.
• Output budgets are bounded, with room for thinking. Gemini bills reasoning against maxOutputTokens and nothing in lib/ai sets a thinkingConfig, so a prose-sized budget truncates: gemini-3.6-flash spent 1,215 tokens thinking and returned a 27-word stub with finishReason: MAX_TOKENS.
• Revision runs only when something is flagged.

After

Single voice is 6x faster than the old best case and 14x faster than the worst. Upload and transcription sit outside this: B2 upload 1.5 – 2.7s, Gemini transcription 7.9 – 11.2s for a 69-second clip.

4. The authenticity rules

The generation prompt (lib/ai/story-grounding-prompts.ts, seeded into ai_prompt_templates) is built around one test: if the storyteller read this aloud, would they recognise it as the way they talk?

1. Their words, first person. Vocabulary, sentence rhythm, dialect, idiom, humour and asides are kept exactly — including constructions a copy-editor would “correct”. The old prompt asked for a “memoir ghostwriter” writing “polished, engaging and scene-driven” prose, which is an instruction to replace the person with a writer.
2. Light editing only. Filler and false starts out, an obvious transcription slip fixed, paragraphs added. No reordering, condensing, explaining or improving.
3. No invention. No fact, name, date, feeling or scene that was not said. Uncertainty stays as spoken (“I think it was ‘62”). Vagueness stays vague. The order of telling is kept. No moral, no summing-up final line.
4. Other voices stay other voices. Interviewer questions are removed and their context folded into the answer. A friend’s or relative’s contribution is kept as a short attributed line, never absorbed into the narrator’s “I”. A name is used only when the transcript ties that name to that voice — otherwise “one of the others said”. Transcript labels (Speaker 2:) must never reach the page.
5. Style continuity. Up to two excerpts from chapters this storyteller already finished are passed as a voice reference, explicitly not as content, so chapter seven sounds like chapter one.

Output is HTML paragraphs, not markdown. stories.content is rendered with dangerouslySetInnerHTML and 19 of the 20 real stories in production hold HTML; the old prompt asked for markdown, so every converted chapter rendered as one unbroken blob with a literal # in front of it. parseGeneratedStory wraps plain text as a floor, so the contract holds even when a model ignores it.

Gemini’s transcription prompt labels speakers only when more than one voice is clearly present, keeps the numbering stable, and gives Speaker 1 to whoever is telling the story.

5. The eval

scripts/story-quality-eval.ts runs five fixtures (scripts/fixtures/story-quality/) through the old and new prompts and scores each with a Gemini judge plus a deterministic named-entity/date check.

cd /Users/abhay/Documents/workspace/next-js/my-story-flow
NODE_PATH=./node_modules node_modules/.bin/tsx \
  apps/web-app/scripts/story-quality-eval.ts --variant=both --runs=2

Flags: --variant=before|after|both, --model=<gemini id>, --fixture=<substr>, --runs=<n>, --json=<path>.

Fixtures: a 3-minute dialect monologue with a joke; a 6-question Elena interview; “dad with two friends” where the friends add two facts; a 40-second clip; a false start with a corrected date.

⚠️ gemini-3.6-flash hit MAX_TOKENS on 10/10 cells — its scores measure the token budget, not the model. It rejects thinkingConfig, so thinking cannot be disabled. It is disqualified for generation.

Production runs generation on gemini-3.5-flash-lite — best on every axis and the fastest, so there was no trade to make. It is also the only model that renders another speaker as direct quotation, which preserves “his mother” as spoken; gemini-2.5-flash’s indirect phrasing drifted the referent.

The judge is a ceiling, not a verdict. It scored the old prompt 4.50 on the “friends” fixture while that output absorbed a second speaker’s correction into the narrator’s first person and silently picked a winner in a disagreement — exactly the defect the new prompt was written to prevent.

The fact-check runs at two speeds

Measured on six drafts, one clean and five each carrying a single planted violation:

An absorbed line is invisible to a reader — every fact really was said in the room, just not by the narrator — and it is impossible when there is only one voice. So single-voice sources get the sub-second check and multi-voice sources pay ~10s more, spending the time exactly where the invisible mistake can happen.

6. Importing an existing recording

“Upload a recording” now appears on app/books/[id]/record/page.tsx and on the recording history list (components/voice/AudioFileUpload.tsx). It takes audio/*, video/mp4 and video/quicktime, reads the duration client-side, shows a real XMLHttpRequest progress bar, forwards ?storyId= so the file fills the chapter it was chosen for, and then follows exactly the same path as a live recording.

The size limit is 9 MB, not the 500 MB originally intended. Measured against the running server: 9.1 MB succeeds, 10.1 MB is refused by the platform inside request.formData() before any handler code runs. The constraint is the request body — 10 MiB in dev and smaller again on Vercel’s serverless functions — not storage. A 20 MB file therefore cannot be verified through this route today. The picker refuses oversized files with an explanation rather than letting somebody wait through an upload that was never going to land.

7. Still open

• Direct-to-Backblaze upload. Until the browser uploads straight to storage with a pre-signed URL and posts only metadata, imports are capped at 9 MB (~half an hour of a phone voice memo) and the “3 hours” goal is unreachable.
• An absorbed line still slips through occasionally on multi-voice material — observed once in three runs before the fact-check split, and the reason the thorough check exists. It is caught after the fact, not prevented.
• The multi-voice path is 15.9s, over the 15s target, and it is deliberate: the alternative is a 4s conversion that ships a story quietly reassigning somebody’s memory.
• maxTokens never reaches the primary model call. executeFeature uses ai_models.max_tokens, so the per-stage budgets in story-conversion-service.ts apply only to the fallback leg. Budgets are currently controlled through the ai_models rows instead.
• __tests__/api/recordings/tagging.test.ts has had 8 failing tests since before this work (verified identical on HEAD); it mocks generateFileName while the route calls generateOrganizedFileName.
• Attribution reads mechanically when the same neutral frame (“Somebody else put in:”) repeats several times in one story.