How to integrate an LLM into your product

Short answer: start with a well-written prompt and a strong hosted model. Add retrieval (RAG) when the model needs your data. Consider fine-tuning last, and only for behaviour, not knowledge. Before any of that, collect real examples with known good results, because without an evaluation set you cannot tell whether any change is an improvement. Then engineer the surroundings: privacy, guardrails, streaming, fallbacks and human review.

Start with the task, not the model

Write down four things before choosing any technology:

• Input and output. What goes in, what comes out, and in what format. “Support email in; category, urgency and a draft reply out.”
• Who consumes the output. A person who can spot errors, or code that will act on it without review.
• The cost of a wrong answer. Mild annoyance, a lost customer, or a legal problem.
• Twenty real examples with the result you would consider correct.

If you cannot produce the examples, the task is not yet defined well enough to build. If a fixed rule would produce the right output, write the rule. The broader question of whether AI suits the problem is covered in how to build an AI product.

Prompting, RAG or fine-tuning

Decision rules

1. Try prompting first, with a capable model, clear instructions and a few examples. Measure it.
2. If failures are caused by missing information, add retrieval.
3. If the information is small and stable, such as a policy document of a few pages, skip retrieval and place it in the prompt, using the provider’s prompt caching to keep cost down.
4. If failures are about consistent format or tone despite good prompts, or you need a small, cheap model to perform a narrow task at volume, consider fine-tuning.
5. These combine. A fine-tuned model can still use retrieval, and most agents use all of prompting, retrieval and tools.

Getting retrieval right

In a RAG system, answer quality is capped by retrieval quality. If the right passage is not in the prompt, no model will rescue the answer. The parts that matter:

• Parsing. Clean text out of PDFs, slides, spreadsheets and web pages, keeping headings and tables. Scanned documents need OCR. This step is dull and decides a great deal.
• Chunking. Split along the document’s own structure (sections, headings, question-and-answer pairs), not at arbitrary character counts. Store the title and section path with each chunk.
• Hybrid search. Combine vector similarity with keyword search. Embeddings are good at meaning and weak at exact product codes, names and numbers; keyword search is the reverse.
• Reranking. Retrieve generously, then reorder with a reranking model and keep the top few. Fewer, better passages improve both quality and cost.
• Permissions. Filter by the asking user’s access rights inside the search query, before anything reaches the model. Filtering afterwards risks leaks through summaries.
• Freshness. Re-index when sources change, and remove deleted content from the index promptly.
• Citations and refusal. Have the model cite the passages it used, and tell it to say when the sources do not contain the answer.

Evaluate retrieval on its own: for each test question, was the passage containing the answer among the top results? That single number locates most RAG problems faster than reading generated answers.

Data privacy and security

An LLM call sends data to a third party. Treat it like any other sub-processor, with some points particular to the technology.

• Read the provider’s business API terms on three points: whether inputs are used for training, how long they are retained, and where they are processed. Consumer chat products often have different terms from the API. Terms change, so record the date you checked.
• Sign a data processing agreement and add the provider to your own sub-processor list where your customer contracts require it.
• Send the minimum. Remove fields the task does not need; consider masking names, emails and identifiers before the call and restoring them afterwards.
• Check regional processing options if you have residency commitments.
• Keep tenants apart in multi-tenant products: retrieval scoped by tenant, no shared caches across tenants, no customer data in prompts shared between customers.
• Decide what you log. Prompts and outputs are valuable for debugging and may contain personal data. Set retention and access rules for those logs.
• Keep provider API keys on the server, never in a browser or mobile app.

Prompt injection

Any text the model reads can contain instructions: a web page, an uploaded PDF, an incoming email. A model cannot reliably tell your instructions from an attacker’s. Design on the assumption that injection will sometimes succeed:

• Treat all model output as untrusted input to the rest of your system
• Give tools the narrowest permissions possible, acting as the current user, never as an administrator
• Require human confirmation for actions that send data out, spend money or delete things
• Do not place secrets in prompts
• Be especially careful when one feature combines untrusted content, access to private data and an ability to communicate externally; removing any one of the three greatly reduces the risk

Evaluation: the part teams skip

Without evaluation, prompt work is guesswork: a change that fixes one example quietly breaks three others. A workable process for a small team:

1. Collect 50–200 real inputs covering typical cases, edge cases and inputs that should be refused. Add every production failure you find later.
2. Define “correct” for each. An exact label, facts that must appear, facts that must not, or a reference answer.
3. Automate the scoring. Code checks where possible: valid schema, correct category, required citation present, number matches. Use a model as a grader for open-ended text, with a written rubric, and spot-check the grader against human judgement.
4. Keep a baseline score and run the set on every change to prompts, models, retrieval settings or tools.
5. Track cost and latency in the same run. A change that gains two points of quality and doubles the cost is a business decision.
6. Continue in production. Thumbs up/down, edits users make to drafts, escalation rates, and a weekly review of a sample of real conversations.

Agree the pass mark with the business before building. “92% of categories correct and no wrong refund amounts” is a target. “It should be accurate” is not.

Guardrails

The guiding idea: the model proposes and your code decides. Anything that must always be true belongs in code, not in a prompt.

Latency and streaming

Responses from language models take seconds, not milliseconds, and output is generated token by token. What users perceive is mostly the time until something appears.

• Stream conversational and long-form output to the interface as it is generated.
• Do not stream into code. When the output is structured data for your system, wait for the full response, validate it, then act.
• Move long work to background jobs with progress updates. Agents and document processing should not hold a web request open.
• Cut tokens. Shorter prompts and capped output lengths reduce latency as well as cost.
• Use a smaller model for steps that do not need a large one, such as classification and routing.
• Run independent calls in parallel, and cache stable prompt prefixes where the provider supports it.
• Show progress honestly in the interface: “searching documents”, “drafting”. A named step feels shorter than a spinner.

Fallbacks and failure handling

Provider APIs have outages, rate limits and slow periods. Model versions are retired on a schedule. Plan for each.

• Timeouts and retries with backoff for transient errors, with a ceiling so that retries do not multiply cost.
• A provider abstraction. One internal interface for “complete this prompt”, so that a second model or provider can be configured as a fallback. Test the fallback against your evaluation set, because prompts do not transfer perfectly between models.
• Graceful degradation. If the AI step fails, the product should still work: show the search results without the generated summary, queue the task, or pass it to a person.
• Invalid output. When schema validation fails, retry once with the error included, then fall back. Never pass unvalidated output downstream.
• Pinned model versions. Use explicit version identifiers, not “latest” aliases, so that behaviour changes only when you decide. Put provider deprecation dates in your calendar.
• Budget circuit-breakers. Per-tenant and global spending limits that degrade the feature before they surprise finance.

Human in the loop

The right level of autonomy depends on how reversible the action is and how well the error rate has been measured.

Build the review interface with care: show the sources, make editing quick, and record every correction. Those corrections are your best evaluation data, and the measured approval rate is the evidence you need before moving a task one row down the table.

A sensible build order

1. Task definition and twenty examples
2. Prompt-only prototype on a strong model; measure
3. Expand the evaluation set; add retrieval if failures are about missing knowledge
4. Structured outputs and validation
5. Privacy review: provider terms, data minimisation, logging policy
6. Interface with streaming, error states and feedback capture
7. Guardrails, limits, metering and spend alerts
8. Fallbacks and the off switch
9. Try smaller models against the evaluation set to reduce cost
10. Limited rollout, weekly review of real usage, then wider release

Steps 1 to 3 form the prototype stage and answer whether the feature is worth building. What each stage costs, including the token arithmetic, is in AI integration cost. If the feature lives in a multi-tenant product, the tenancy and metering groundwork is described in our SaaS architecture guide. And if you would like a team to carry this out, BBR’s AI development and LLM integration service follows this sequence.