Short answer: choose a task where AI has a real advantage and errors are tolerable or reviewable. Define success as a number before writing code. Prototype on real data in two to four weeks and be prepared to stop. Prefer fixed workflows to autonomous agents wherever the steps are known. Build the product around the model with evaluation, guardrails and cost limits. Roll out gradually and keep measuring. And check first whether ordinary software would solve the problem, because it frequently would.
How AI products differ from ordinary software
| Conventional feature | AI feature | |
|---|---|---|
| Behaviour | Specified; same input gives same output | Probabilistic; varies between runs and between model versions |
| Feasibility | Known in advance with reasonable confidence | Unknown until tried on real data |
| Testing | Pass or fail | A score on an evaluation set, with a threshold |
| Marginal cost per use | Near zero | Real, and proportional to usage |
| Failure | Errors and crashes you can see | Fluent, plausible output that is wrong |
| Dependencies | Libraries you pin and control | A third-party model that will be updated and eventually retired |
None of this makes AI products unusually hard. It means the process needs a feasibility stage that ordinary projects can skip, and a measurement habit that lasts as long as the product does.
When AI is the wrong tool
The cheapest AI project is the one you correctly decide not to start. Be sceptical when:
- The logic can be written as rules. Pricing, eligibility, validation, routing on fixed criteria. Code is cheaper, instant, auditable and gives the same answer every time.
- The answer has to be exact. Totals, balances, stock levels, dates. Query the database. A model may help a user phrase the question, but the figure should come from the system of record.
- A wrong answer is costly and nobody checks. Unsupervised decisions in medicine, law, finance or safety are a poor match for a component with a non-zero error rate.
- The decision must be explained and reproduced. Regulators and auditors want a rule they can inspect.
- There is no data to ground it and no examples to test it. You will be unable to make it good, or to know whether it is.
- The real problem is findability or interface design. Many requests for a chatbot are requests for better search, clearer navigation or a shorter form.
- The arithmetic fails. If a request costs more than the value it creates, the feature loses money faster as it succeeds.
- The volume is tiny. A task done five times a month does not justify a build; a person with a general-purpose assistant will do.
Where language models do have a real advantage: reading and writing unstructured text, handling inputs too varied for rules, summarising and extracting, classifying with nuance, translating between formats, answering questions across a large body of documents, and serving as a flexible interface to structured systems. Good AI products sit squarely inside that list.
The process
1. Choose the task (about a week)
State it in one sentence with a user, an input and an output: “A claims handler uploads a repair invoice and receives the line items, totals and any policy exclusions that apply.” Then check three things: how often it happens, how long it takes now, and what happens when the result is wrong. High frequency, real time savings and recoverable errors make a good candidate.
2. Define success as a number (days)
Covered in the next section. Do this before the prototype, so that the result cannot be argued into a success after the fact.
3. Audit the data (about a week)
What will the model need to read? Where is it, in what format, who may see it, and how often does it change? What personal or confidential data is involved? Collect 30 to 50 real examples with the correct result. Missing or inaccessible data ends more AI projects than model limitations do.
4. Prototype for feasibility (2–4 weeks)
Build the narrowest version that can run the examples: a prompt, retrieval if needed, a script and a spreadsheet of scores. No polished interface. Measure quality, latency and cost per request. This is a proof of concept in the strict sense described in our comparison of MVPs, prototypes and proofs of concept: its only job is to answer “can this work well enough?”
There are three honest outcomes. Proceed. Narrow the task (perhaps it works for invoices but not handwritten estimates). Or stop. A project stopped in week three for a few thousand dollars is a success of the process.
5. Design for imperfection (1–3 weeks)
Assume some outputs will be wrong and design the experience around that: show sources, make results easy to edit, mark low-confidence cases, give a clear route to a person, and collect feedback with one click. Decide which actions need approval. An interface that presents AI output as a draft earns more trust than one that presents it as fact.
6. Build the production system (4–12 weeks for a feature; longer for a new product)
Retrieval pipelines, tool integrations, permissions, structured outputs, validation, streaming, fallbacks, metering and limits, logging and the evaluation tooling. The engineering detail is in our guide to integrating an LLM into a product. For a new AI-first product, this runs alongside the usual product work: accounts, billing and administration, as described on our SaaS development page.
7. Roll out gradually (2–4 weeks)
Internal users first, then a small customer group, then everyone. Review real conversations or outputs every week. Add every failure to the evaluation set. Watch spend daily for the first month.
8. Operate
Models are updated and retired, your content changes, and users find new ways to ask things. Re-run the evaluation on any change, review a sample of production output regularly, and revisit cheaper models every few months, since a task that needed a large model last year may run on a small one now.
Defining success metrics
You need metrics at three levels, plus limits.
| Level | Question | Examples |
|---|---|---|
| Task quality | Is the output right? | Share of extractions with all fields correct; share of answers supported by the cited source; rate of harmful errors |
| User outcome | Does it help the person? | Drafts accepted without major edits; time per case; share of queries resolved without escalation |
| Business outcome | Does it matter to the company? | Cost per ticket; conversion to a paid plan; retention of accounts that use the feature |
| Limits | What must stay within bounds? | Cost per task; response time; rate of unsafe or out-of-scope output |
Three practical points:
- Measure the human baseline. If people doing the task today agree with each other 90% of the time, demanding 99% from the system is demanding more than the task’s definition supports.
- Separate error types. A missed category is an inconvenience; a wrong refund amount is an incident. Set a different threshold for each.
- Write the threshold down first. “At least 90% of fields correct, zero wrong totals on the test set, under five seconds, under three cents a document.” That sentence is the acceptance criterion for the prototype.
Where agents fit, and where they fail
An agent is a model in a loop: it is given a goal and a set of tools, chooses an action, reads the result, and continues until it decides the task is done. It helps to see agents as one end of a range:
- Single call. One prompt, one output. Summarise, classify, extract.
- Fixed workflow. Several model calls in an order your code controls: classify, then retrieve, then draft, then check.
- Router. The model picks which of a few fixed workflows to run.
- Agent. The model decides the steps, their order and when to stop.
Every step down the list adds flexibility, cost, latency and unpredictability. Use the simplest level that does the job. Many products described as “agents” are level 2 or 3, and are better for it.
| Agents tend to work when | Agents tend to fail when |
|---|---|
| The steps cannot be known in advance (research, investigation, debugging) | The steps are known; a fixed workflow would be cheaper and more reliable |
| Success can be checked: a test passes, a record validates, a person approves | Nothing verifies the result, so the agent declares success on its own authority |
| There are a few well-described tools with clean inputs and outputs | There are dozens of overlapping tools, or APIs that return ambiguous errors |
| Actions are read-only or reversible | Actions send messages, move money or delete data without approval |
| Tasks are short: a handful of steps | Tasks are long: small per-step error rates compound |
| It reads trusted content | It reads untrusted content while holding sensitive permissions, which invites prompt injection |
| A slower, costlier answer is acceptable | Users expect an instant reply, or the margin per task is thin |
The compounding point deserves a number. If each step goes right 95% of the time, a 10-step task succeeds about 60% of the time and a 20-step task about 36%. Reliability comes from shorter chains, checks between steps, and the ability to recover, not from hoping each step is perfect.
Rules for agents in production
- Narrow scope: one job, a few tools, a clear definition of done
- Tools validate their own inputs and act with the current user’s permissions
- Hard limits on steps, time and spend per task
- Human approval for irreversible or external actions
- Every step logged, so that a failure can be replayed and understood
- An evaluation set of whole tasks, scored on the outcome and not only on individual replies
- A fallback when the agent gives up: hand the case to a person with the work done so far
Build vs buy
There are several layers, and the answer differs for each.
| Layer | Default | Reasoning |
|---|---|---|
| Foundation model | Buy (hosted API) | Training your own is a research programme. Self-host an open-weight model only for data-control reasons or at very high steady volume |
| Generic capability your staff need (meeting notes, writing help, coding assistance) | Buy an existing product | Mature products exist, priced per seat. Building your own is rarely justified |
| Standard function in a standard setting (support chatbot over a help centre) | Buy first, or use the AI features of the platform you already have | Build only if the off-the-shelf tools cannot reach your data, workflow or quality bar |
| AI inside your own product’s workflow | Build, on bought models | It depends on your data model, permissions and interface, and it is part of what customers pay you for |
| Supporting infrastructure (vector search, logging, evaluation tooling) | Start with what you already run, add tools as needed | PostgreSQL and plain logs go a long way; specialist tools help at scale |
| Evaluation sets, prompts, domain rules | Always yours | This is where your knowledge of the domain lives, and it carries over between models |
A useful test before building: give a general-purpose assistant the same input your product would receive. If it does the job about as well, your product needs to add something it cannot: access to private data, integration into the workflow, guarantees, or scale. If you cannot name that addition, reconsider.
Budget figures for the build route, along with the running-cost arithmetic, are in what it costs to build and run AI features.
Pitfalls that repeat
- Starting with “we should use AI”. Start with a task someone is tired of doing.
- Judging by the demo. Five chosen examples prove nothing about five hundred real ones.
- No evaluation set. Every later change becomes a matter of opinion.
- Reaching for an agent first. Most of the value is usually available from a fixed workflow at a fraction of the cost.
- Leaving out the person who knows the domain. Engineers can build the system; only a practitioner can say whether the output is right.
- Presenting output as fact. Without sources and an edit path, one visible mistake destroys trust in the whole feature.
- Unlimited usage on a flat price. Meter from the first day.
- Building the platform before the feature. One narrow feature in production teaches more than a general “AI layer” that serves nothing yet.
- Treating launch as the end. Models change underneath you. Quality is something you maintain.
A realistic timeline
| Stage | AI feature in an existing product | New AI-first product |
|---|---|---|
| Task, metrics, data audit | 1–2 weeks | 2–3 weeks, including product scoping |
| Feasibility prototype | 2–4 weeks | 2–4 weeks |
| Design | 1–2 weeks | 2–4 weeks |
| Production build | 4–12 weeks | 10–20 weeks |
| Gradual rollout | 2–4 weeks | 4–8 weeks of private beta |
These are planning ranges that assume an experienced team and reasonably accessible data. The stage most likely to overrun is the one that is not about AI at all: getting access to the data, and agreeing what a correct answer is.
If you would like a team to run this process with you, BBR’s AI development service starts with the task review and prototype described here, and says so plainly when the prototype shows that AI is not the answer.