Neural Summary generates 30 types of professional documents from conversation transcripts. Each document is produced in the user's language. Each one is a typed JSON structure, not markdown. And each one is generated by a single LLM call with a carefully designed prompt.
This post covers the prompt engineering patterns we developed over eight months and 30 template iterations. Some of these patterns are obvious in retrospect. Most were not obvious when we started. One note before we dive in: the prompt excerpts below are illustrative sketches that show the shape of each pattern, not our production prompts.
The decision: JSON over markdown
Our V1 system generated markdown. The LLM received a transcript and returned formatted text with headings, bullet points, and bold emphasis. This worked for rendering a single document in English. It failed at everything else.
Markdown is a rendering format. It encodes both content and presentation in one string. You cannot search it semantically. You cannot translate it without also translating the formatting markers. You cannot validate whether it contains the sections you expect. You cannot render it differently on web versus mobile.
In V2, every template generates a typed JSON structure. An executive summary returns an object with a recommendation, a list of findings (each carrying its evidence and implication), decisions (each with an owner and rationale), and risks (each with a severity and mitigation).
This separation of content from presentation enables five things that matter:
- 1The same output renders differently on desktop and mobile.
- 2The same output displays in five languages without re-generation.
- 3Schema validation catches incomplete or malformed output before it reaches the user.
- 4Semantic search indexes specific fields (decisions, action items, risks) rather than full text.
- 5Export to markdown, PDF, or clipboard produces format-appropriate output from the same data.
The upfront cost is higher. You need a JSON schema for every template. You need validation logic. You need a rendering component for each schema. But the payback is immediate and compounds.
Why generic prompts produced generic output
Early prompts said things like "Summarize this transcript and identify key themes." The output was generic. Correct, but unremarkable.
The fix was positioning. Every prompt now opens with a specific professional identity. The flavor of it, in illustrative terms: a summary lens written from the desk of a management consultant who presents to steering committees, a backlog lens written by a product practitioner who has shipped real backlogs to real teams, a coaching lens grounded in how people actually develop, a competitive lens framed as a seasoned market analyst rather than a note-taker.
This is not anthropomorphism. It is context setting. The positioning activates the domain knowledge embedded in the model's training data. A prompt that says "as a seasoned market analyst, produce a win strategy" generates output with the specificity and structure that role would actually expect.
The more specific the positioning, the better the output. "You are a helpful assistant" produces assistant-quality output. "You are a management consultant preparing a brief a steering committee will act on" produces consultant-quality output.
Showing the model what good looks like (and what bad looks like)
Telling the LLM what you want is less effective than showing it. Every template prompt includes examples of both good and bad output for critical fields.
For action items:
GOOD: "Send updated contract to legal by Tuesday → owner: Priya, priority: high"
BAD: "We should probably get the contract reviewed at some point"
For user stories in the agile backlog template:
GOOD: "As a warehouse planner, I can flag low-stock items
so that reorders happen before a stockout"
BAD: "As a user, I want to manage inventory"
The bad examples are as important as the good ones. Without them, the model gravitates toward the median quality of its training data, which for meeting notes is low.
Forcing language compliance, not suggesting it
Supporting five languages means the LLM must generate all content, including field labels and structural text, in the user's language. This is harder than it sounds.
Left to its own devices, the model generates JSON keys in English and values in mixed languages. German output had English section headers. French output fell back to English for technical terms. We tested polite formulations and they produced English leakage in roughly 15% of non-English outputs.
Our solution is a mandatory language block injected into every prompt with emphatic language. In sketch form:
LANGUAGE RULE (non-negotiable): every string value in the
output is written in {language}. Headings, labels, body
content, all of it. An English fallback in any field is
a defect.
The emphatic version reduced English leakage to under 1%. Without strong instruction, the model treats language as a suggestion rather than a constraint.
Constraining verbosity before it reaches the user
LLMs are verbose by default. Without constraints, an action item that should say "Ship revised pricing page by Friday" becomes "It was discussed and agreed upon that the team should prioritize the revision of the pricing page, with a target completion date of Friday."
We enforce conciseness in two ways.
Word limits per field. Each template specifies maximum word counts for specific fields. Action item descriptions get a hard cap of roughly a dozen words. Key finding summaries: one sentence. Section headings: a few words, no more.
Action-verb-first patterns. For tasks, action items, and next steps, the prompt requires that every item begin with a verb in imperative form. "Ship revised pricing page" not "The pricing page should be revised." This single constraint eliminates most filler language.
Detecting conversation type to improve output relevance
Not all conversations are equal. A sales discovery call has different content than a sprint retrospective. A coaching session has different dynamics than a board meeting.
Our system auto-detects the conversation type during the initial transcription pass. We classify into 14 types: sales call, interview, brainstorm, sprint planning, coaching session, one-on-one, board meeting, and so on. This classification is stored alongside the transcript and used in two ways.
First, it informs template recommendations. A detected sales call surfaces CRM notes, deal qualification, and follow-up email as suggested templates. A detected retrospective surfaces the retrospective template and action items.
Second, the conversation type is passed to the prompt as context. The same executive summary template produces subtly different output for a sales call versus a strategy session, because the prompt knows what kind of conversation it is analyzing.
Validating output before the user sees it
LLMs do not always produce valid JSON. Even when they do, the JSON may not conform to the expected schema. A coaching notes output might be missing the key moments array. An executive summary might have a recommendation field that is null.
We validate every output against the template's expected schema before storing it. Required fields must exist and have the correct type. Arrays must not be empty when they are expected to contain items. String fields must not be empty.
When validation fails, we have two options: reject and retry, or patch the output with defaults. For critical fields (the executive summary's recommendation, the action items list), we retry once with a slightly modified prompt. For optional enhancement fields (risk severity, priority badges), we patch with sensible defaults.
This validation layer catches roughly 3-5% of outputs. Without it, users would occasionally see empty sections or missing data.
The hardest language: German
German is our most challenging language. Not because of character encoding or font rendering, but because of capitalization.
German capitalizes all nouns. English capitalizes proper nouns and sentence beginnings. Spanish, French, and Dutch follow English-like rules with minor variations.
When the LLM generates a heading like "Key Findings" in English, the equivalent in German is "Wichtigste Erkenntnisse," where "Erkenntnisse" (findings) is capitalized because it is a noun. But the LLM, trained primarily on English text, sometimes produces "wichtigste erkenntnisse" (all lowercase after the first word) or "Wichtigste erkenntnisse" (missing the noun capitalization).
We solved this with explicit capitalization instructions per language in the prompt, plus a post-processing step that checks German output for common capitalization errors. The post-processing is imperfect, but it catches the most visible mistakes.
What the prompts look like
A single template prompt averages 400-600 words. It contains:
- 1Domain expert positioning (2-3 sentences)
- 2Output format specification with JSON schema
- 3Language requirement block
- 4Good and bad examples for critical fields
- 5Conciseness constraints and word limits
- 6Template-specific instructions, leaning on an established framework where the domain has one (a standard qualification framework for sales lenses, a recognized briefing structure for executive output)
- 7Edge case handling, so that when a transcript contains no decisions, the output says so instead of inventing them
The prompts are the most carefully maintained code in the system. A small change in wording can meaningfully shift output quality. We version them, review changes, and test against a consistent set of sample transcripts before deploying updates.
What this changed about how we build AI features
Structured output is strictly better than markdown for any application that needs to do more than render text. The upfront cost is higher, but the optionality it creates is worth more than the cost within weeks.
Prompt quality matters more than model size. A well-designed prompt on a mid-range model consistently outperforms a vague prompt on a larger model. Invest in the prompt.
Language support is not a feature, it is an architecture decision. If you build an English-first system and add languages later, you will rewrite the prompt layer. If you build language-aware from the start, additional languages are incremental.
Specificity is proof. A vague summary feels generated. A specific finding with evidence, an implication, and a cited quote feels like it was written by someone who listened. The same principle from good writing applies to good prompt engineering: concrete details are what make output credible.
Thirty templates. Five languages. Structured JSON. Built one prompt iteration at a time.
