The pattern is familiar by now. Someone builds an AI feature, it demos beautifully, everybody is impressed, and then it meets real data and real users and quietly falls apart. Nothing was faked. The demo was real. It just was not a product.
The distance between the two is engineering, and it is mostly the same engineering any serious feature needs. What is the success measure? How do we tell, after a change, whether the thing got better or worse? What happens when the provider is slow, or down, or returns something malformed? What does it cost per thousand uses, and what happens to that number when usage triples? Who reviews the output when being wrong actually matters?
We treat an AI feature as a product feature. It gets a written specification with a defined success measure. It gets an evaluation set, a fixed collection of real inputs with known-good outputs, so a change can be proven better rather than assumed better. Without one, every prompt tweak is a guess, and teams end up rewriting prompts in circles because nobody can measure whether last week's version was worse.
It gets fallbacks, because you are now depending on somebody else's infrastructure, and that infrastructure has bad days. It gets cost controls, because spend grows with success and a feature that becomes unaffordable when it becomes popular is a trap. It gets logging, so that when someone asks what the system did and why, there is an answer.
And it gets a human in the loop wherever being confidently wrong would cause damage. That is not pessimism about models, it is the same rule we would apply to any automated decision with consequences.
The other half of the job is saying no. Plenty of problems that arrive described as AI problems are better solved with ordinary code: a lookup, a rule, a well-designed form. Ordinary code is cheaper, faster, testable, and does not drift when a provider updates a model. We would rather tell you that than take the more interesting brief.
We are also explicit about data. Before anything is built, it is on record what gets sent to which provider, what is retained, and what stays inside your own systems. That is a conversation to have at the start, when it can still shape the design, not a question to answer after something has already been sent.
None of this is exotic. It is the ordinary discipline of shipping software, applied to a component that happens to be probabilistic. Skip it and you have a demo. Do it and you have a feature.
