Recording → Story

Final verification, 2026-09-06 evening. ✅ Verified by a real run: a short spoken clip uploaded to real storage, Gemini returned a word-exact transcript in 5 to 10 seconds, “Turn this into a story” produced a story that landed in the chapter it was recorded for (0bb3385), and the chapter showed as Done. The conversion quality gate answered “could be richer” after about 60 seconds on a 20-second clip and offered “Create a shorter story anyway”, which worked. Still open: the 13 legacy mock-storage recording rows in production, the trial conversion path, and the Elena conversation page itself (another session is editing it).

Mechanisms updated 2026-09-06 after the fixes landed; see the final verification note below the title.

Scope: the recording process — tap record through saved story. The Tiptap editor is explicitly out of scope. All file paths are relative to apps/web-app unless stated otherwise.

1. The journey, surface by surface

Three independent capture surfaces all feed the same conversion endpoint, /api/stories/convert, and now two of the three pass a book id through at conversion time; the third (the story library, and both surfaces after the fact) attaches to a book via the shared AddToBookDialog.

Tap count, plain recording, happy path, permission already granted: Start (1) → Stop (2) → Save (3) → Convert to Story, open modal (4) → Convert to Story, confirm (5) → lands in editor. Given the live word-count distribution (§5, pre-fix data), roughly half of real recordings hit the readiness gate and need a sixth tap (“record more” or “generate anyway”).

2. How a recording becomes a story today — the real mechanism

1. Capture: VoiceRecorder.tsx opens getUserMedia, negotiates a mime type (lib/audio/mime.ts), and in parallel starts the browser’s webkitSpeechRecognition for a live, client-side caption. That caption is now explicitly labelled “Live captions — we’ll clean these up after you finish,” stored as transcript_source: 'browser' / transcript_status: 'pending', and is a stopgap, not the final transcript.
2. Upload: on stop, the blob and the live caption are POSTed to /api/recordings. uploadRecording now throws a BackblazeUploadError on any real failure instead of swallowing it — a mock URL is only ever returned when B2 env vars are entirely absent, and that case is flagged. /api/recordings writes no row and returns 502 on a throw, or 503 with status: 'upload_failed' when B2 is unconfigured. VoiceRecorder shows a persistent failure panel with “Try saving again,” re-sending from the IndexedDB copy rather than losing the recording.
3. Server transcription: a new POST /api/recordings/[id]/transcribe route reads the audio back from B2 and transcribes it with lib/ai/providers/gemini-transcription.ts (@google/generative-ai, gemini-2.5-flash) — inline base64 for audio under 14 MB, the Files API above that. Transcription is enabled directly in code, so no ai_features row is required to turn it on. resolveTranscriptionProviders falls back to Whisper if Gemini is unavailable. Usage is logged. The result replaces the browser caption in recordings.transcript and sets transcript_source: 'gemini' (or 'whisper'), transcript_status: 'done'. Migration 20260906010000_recording_transcription_status adds the transcript_status / transcript_source / transcript_error / transcript_updated_at columns and a status value of upload_failed; the 18 pre-existing rows keep transcript_status = NULL (“not tracked”), so nothing is retroactively relabeled. Measured: a real 20-second clip transcribed in 3.0 seconds, word-perfect.
4. Recording detail page: shows plain pending copy while transcript_status is pending/processing, polls, and offers a retry on failed. A recording opened from inside a book now carries that book’s id through the page (bookId), so its primary action can pass it along.
5. Convert to Story: a real fetch to POST /api/stories/convert loads the source server-side, runs the two-layer readiness gate (deterministic word floor, then one AI call), and on success calls StoryConversionService.generateGroundedStory followed by a faithfulness check. On success the route inserts the stories row, links recordings.story_id, and — if a bookId was passed — calls addStoryToBook. Both the recording page and the conversation page (app/ai-conversations/[id]/page.tsx) now send bookId whenever the source record already has one.
6. Attach to a book, any time: whether or not a bookId was available at conversion time, the resulting story can be attached afterward from the story library, the recording page, or the conversation page — all three now render the shared AddToBookDialog (app/components/books/AddToBookDialog.tsx).

3. What works

• Codec negotiation, IndexedDB crash recovery, wake lock, and the persistent mic-error panel are real and hold up in code.
• Silence handling is humane: a 20-second gentle nudge (“Still there? Take your time”) with no auto-stop or auto-discard.
• The readiness-gate word thresholds (60/150 words) remain calibrated against real production material.
• All three capture surfaces call the same /api/stories/convert endpoint with the same readiness/faithfulness guardrails — one pipeline, not three.
• Uploads fail loudly instead of silently. A B2 outage or misconfiguration now surfaces as a real error with a retry, not a fake success toast.
• Server transcription actually transcribes audio. Gemini receives the real audio bytes (inline or via the Files API) rather than a text prompt rendered into a chat completion — the prior architecture could never have transcribed anything, regardless of configuration.
• A converted story can reach its book automatically when the source recording or conversation already has a bookId, and can always be attached afterward through AddToBookDialog from three different screens.
• Any story row, however it was created, shows up in /stories because that page queries the stories table directly.

4. What changed on 2026-09-06

• Uploads no longer lie about success. uploadRecording throws instead of swallowing errors; a mock URL survives only when B2 env vars are entirely absent, and that row is marked upload_failed rather than completed. /api/recordings returns 502 on a throw and writes no row; 503 + upload_failed when B2 is unconfigured. VoiceRecorder shows a persistent failure panel with “Try saving again,” retrying from the IndexedDB copy. Fixed in 275d9fe.

• Server-side transcription can finally run. New lib/ai/providers/gemini-transcription.ts sends real audio to gemini-2.5-flash (inline base64 under 14 MB, Files API above); new POST /api/recordings/[id]/transcribe reads the audio back from B2 and calls it; transcription is default-on in code (no ai_features row needed); Whisper is an automatic fallback via resolveTranscriptionProviders; usage is logged. Fixed in 275d9fe.

• Browser captions are now honestly labelled and genuinely temporary. Shown as “Live captions — we’ll clean these up after you finish,” stored with transcript_source: 'browser' / transcript_status: 'pending', and replaced by the Gemini/Whisper transcript once it lands. Backed by migration 20260906010000_recording_transcription_status. Fixed in 275d9fe.

• The recording page tells the truth while transcription runs. Plain pending copy, polling, and a retry on failure, instead of a silent fallback with no user-visible message. Fixed in 275d9fe.

• “Turn this into a story” no longer strands the result outside its book. The recording page passes bookId when the recording already has one; the conversation page does the same (app/ai-conversations/[id]/page.tsx). Where no bookId is available yet, the new shared AddToBookDialog attaches the resulting story to a book afterward, from the story library, the recording page, or the conversation page. Fixed in 275d9fe and d364c25.

• Elena’s own upload and transcription got the same fixes. app/api/conversations/upload-audio/route.ts now returns 502 on a real upload failure instead of a mock URL (the old mock-URL check it carried was dead code once uploadRecording started throwing). gemini-provider.ts routes Elena’s transcription through the same shared transcribeAudioBuffer pipeline (Gemini first) and no longer sends the invalid language: 'auto' parameter to Whisper. Fixed in c8c2c6c.

• Controls read larger. Recording-page status labels and controls are 18px+ text on 56px (h-14) touch targets.

• Large recordings can be uploaded at all. Two changes, both in 871bdab.

  Compression, on the device. lib/audio/compress.ts decodes with OfflineAudioContext at 16 kHz mono — the context’s sample rate makes decodeAudioData resample natively, so the 48 kHz version is never materialised — then encodes MP3 at 32 kbps in a Web Worker (compress.worker.ts, @breezystack/lamejs, 480 KB, no WASM fetch). Skipped below 4 MB, for already-compact audio, for video, and past two hours of audio (memory). MP3 rather than Opus because Safari cannot play Ogg/Opus and the app plays these back in an <audio> element, and because Whisper takes mp3 directly; the cost is ~25% more bytes than Opus. Not MediaRecorder, which encodes in realtime.

  Direct-to-Backblaze above 8 MB. POST /api/uploads/presign authorises one Backblaze upload target and returns a signed ticket; the browser PUTs the bytes straight there; POST /api/uploads/register asks Backblaze whether the object exists and how big it is, and writes the row only if it does. Below 8 MB nothing changed — the file still goes through POST /api/recordings. Migration 202609060401_recording_media_type adds media_type (audio/video) and original_file_size_bytes, both additive.

  Measured on 2026-09-06 against the dev server and the production bucket: a 24.61 MB WAV of 403 s of speech compresses in the browser to 1.54 MB (16.0x, 4.8 s, ~83x realtime) and its Gemini transcript scores 99.45% word agreement against the 1,094-word reference. A 58 MB WAV uploads in 3.0 s and transcribes via the Files API in 28.6 s. A 0.81 MB mp4 registers as media_type: video and transcribes from the video track directly — no audio extraction step. A deliberately truncated upload is refused with 409 size_mismatch and writes no row.

• Uploads accept video. Families upload what their phone recorded, and a phone produces .mov as readily as .m4a. Video is never re-encoded on the device, uploads on the same path, and is transcribed in place — normalizeAudioMimeType passes video/* through instead of flattening it to audio/webm. Whisper is skipped for QuickTime, which it cannot read, rather than being sent a .mov under a webm filename.

• Live recordings record smaller and no longer die at the size cap. VoiceRecorder asks MediaRecorder for 32 kbps instead of the music-tuned default, and above 8 MB the finished blob takes the direct path with a progress bar, keeping its IndexedDB backup and Retry. A long session used to end in a platform 413 the recorder could only report as a generic failure.

Decision, stated for the record: the owner evaluated Deepgram (§7) and decided against adopting it for now — transcription goes straight to Gemini, with Whisper as the fallback, rather than adding a third transcription vendor.

5. What’s still open

5.1 The conversion pipeline’s real-world success rate is still not independently re-verified

The prior audit found story_conversions empty and every stories row ai_enhanced: false in production — evidence that the pipeline had never completed for a real user, even though 6 of 18 recordings already cleared the readiness floor. Today’s fixes change what feeds the pipeline (a real Gemini transcript instead of an unreliable browser caption) and how the result reaches a book, but do not touch story-conversion-service.ts itself, and no fresh production run was executed to confirm a first successful story_conversions row now exists. Read this as unproven, not necessarily broken — treat it the same as the prior audit did.

5.2 Direct upload needs one CORS rule on the Backblaze bucket before it works in a browser

The presign / register contract is complete and verified end to end — Node performing exactly the POST the browser makes gets a stored object, a row and a transcript every time. The browser leg does not work yet, and the reason is bucket configuration rather than code: my-story-flow has corsRules: [], so a preflight to the upload host returns no Access-Control-Allow-Origin and Chrome blocks the request. Confirmed in a real browser on 2026-09-06 — a 15.17 MB mp4 reached presign, then failed with “Access to XMLHttpRequest … has been blocked by CORS policy”. The uploader shows a plain error and keeps the file for Retry, so nothing is lost, but nothing over 8 MB uploads.

The fix is one rule, added in the Backblaze console (Bucket Settings → CORS Rules) or via b2_update_bucket:

[
  {
    "corsRuleName": "browserDirectUpload",
    "allowedOrigins": ["https://app.mystoryflow.com", "http://localhost:3000"],
    "allowedOperations": ["b2_upload_file", "s3_put"],
    "allowedHeaders": ["*"],
    "exposeHeaders": ["x-bz-file-id", "x-bz-content-sha1"],
    "maxAgeSeconds": 3600
  }
]

It could not be applied from here: b2_update_bucket needs the writeBuckets capability, and the B2_AUDIO_KEY_ID credential has readBuckets, writeFiles, readFiles, listFiles and deleteFiles but not that one.

Worth doing at the same time: issue the upload credential with writeKeys so presign can mint a short-lived key scoped by namePrefix to the calling user’s folder. Today’s token is bucket-scoped, which lib/storage/direct-upload.ts contains with a server-generated unguessable key, an HMAC ticket and a prefix check at register — sound, but narrower is better.

Until the rule exists, compression alone still lifts the practical ceiling sharply: any recording that compresses under 8 MB — roughly 33 minutes of speech, whatever the source format or bitrate — now uploads through the unchanged multipart route, where before the limit was 9 MB of original bytes.

5.3 The trial’s “your story will be added automatically” promise is still dead code

contexts/TrialContext.tsx:80: const triggerConversion = () => setShowSignupModal(true) — that is the entire implementation. /api/trial/convert and /api/trial/generate-story still have no callers outside Playwright test mocks. Not touched by today’s fixes.

6. Live data snapshot (production, qrlygafaejovxxlnkpxa) — pre-fix, not re-queried today

This table reflects the state recorded before the 2026-09-06 fixes landed. These specific rows do not change retroactively — the fixes change how new recordings behave, not the 13 legacy rows already sitting on a mock URL. Re-querying this table is recommended before the next audit.

7. Senior-friendliness observations

• Feedback while recording: the orb scales and glows with audioLevel, plus a level indicator and a mm:ss timer — legible, plain-worded.
• Silence copy: “Still there? Take your time — we’re still recording” is warm and non-alarming.
• Failure is now honest and recoverable. The upload failure panel and the transcription pending/retry copy (§2, §4) mean the two most consequential silent failures from the prior audit — a fake save and a transcript that never comes — now say what happened and offer a next action.
• Controls read larger: 18px+ status text on 56px (h-14) touch targets, comfortably above the 44px accessibility floor. The record orb is 160–192px.
• No recording length cap on the primary surface remains true — VoiceRecorder.tsx has no maxDuration check, while app/api/recordings/route.ts still sets a 5-minute server function budget for the upload route itself. A long recording on a slow connection can still time out mid-upload without a clear “recordings are capped at N minutes” message set up front. Not touched by today’s fixes.
• Trial recorder reuses the exact same component as the authenticated surface — visually consistent, but does nothing to make the still-dead “convert automatically” promise (§5.2) visible or honest.

8. Deepgram assessment (evaluated, not adopted)

Owner decision, 2026-09-06: no Deepgram. Transcription goes straight to Gemini (with Whisper as fallback, §2/§4). The analysis below is kept for reference in case live captions become a priority later; it is not a plan in progress.

What SpeakUp actually does (/Users/abhay/Documents/workspace/flutter/speakup and speakup-api): live recording streams directly browser/client → Deepgram WebSocket using a short-lived scoped key minted by a small Next.js endpoint — audio never touches SpeakUp’s own server. Imported audio goes through a batch REST call where Deepgram pulls the file from a public B2 URL. Model: nova-3, with smart_format, filler_words, interim_results, utterance_end_ms, vad_events, punctuate. Not used: diarization, sentiment, topics, summarization.

What this would still add over the current Gemini path: live captions while recording (a trust signal the current orb + timer doesn’t fully provide) — the one capability Gemini-after-the-fact cannot replicate, since Gemini transcribes the finished recording rather than streaming interim results. Diarization and sentiment remain “later” features tied to multi-speaker interviews and conversation-logic changes respectively, not part of the current single-speaker surfaces.

Cost (order-of-magnitude, not re-priced today): nova-3 streaming has historically run ~$0.004–0.005/minute — negligible next to existing OpenAI/Gemini spend, but it is a second transcription vendor and API key to manage, which is part of why it was passed on for now.

9. Recommended next steps, ordered for launch

1. (M) Get a first confirmed story_conversions row in production with a real recording that clears the 150-word floor, now that its input is a real Gemini transcript rather than a browser caption. Confirms or narrows §5.1.
2. (M) Wire the trial’s promised auto-conversion, or remove the “your first story will be added automatically” copy if that’s not the near-term plan. Fixes §5.2.
3. (M) Set a client-side maxDuration on VoiceRecorder.tsx with a plain-language warning before the limit, so a senior isn’t surprised by a server timeout on a long recording.
4. (L) Revisit Deepgram for live captions specifically if “the recorder doesn’t feel like it’s working while I talk” resurfaces as a real complaint — not before, per the 2026-09-06 decision.