Where an LLM earns its place — and where it doesn't
Most AI features fail before a single token is generated, because the wrong problem was chosen. Here's the filter I use with clients.
Every AI consulting engagement I've taken on starts the same way: someone has decided the product needs AI, and the question I'm handed is "how", when the question that matters is "where".
The filter
An LLM earns its place when three things are true at once.
- The task has many valid answers rather than one correct one. Summarizing, rewriting, classifying loosely, drafting. If there is exactly one right answer and a deterministic way to compute it, write the function.
- Being wrong is cheap and visible. The user sees the output and can reject it. A wrong suggestion costs a click; a wrong bank transfer costs a lawsuit.
- The input is messy in a way that resists parsing. Free text, mixed formats, human phrasing. If your input is already structured, you probably want a query, not a model.
Miss any one of the three and you are building something that will look impressive in the demo and quietly erode trust in production.
The failure I see most
Teams reach for an LLM to avoid writing rules, then discover the rules were the easy part. What they actually needed was the data model underneath, and the model is now papering over its absence — expensively, non-deterministically, and without a way to test it.
The tell is when someone asks "how do we make it more accurate" before anyone can say what accurate means for that feature. That's not a prompting problem.
What to do first
Write down what a correct output looks like, concretely enough that two people would grade the same output the same way. If you can't, you don't have an evaluation, and without an evaluation you don't have a way to know whether any change you make is an improvement.
That document is worth more than the prompt. The prompt is the easy part — it's the thing you'll rewrite twenty times, and each rewrite is only worth doing because the evaluation tells you whether it helped.