An employee asked our chatbot about a company policy. The answer was clear and well written. It was also from an old version of the policy.
That chatbot is the best answer I have to "what is AI engineering?". So I'll start there, and come to the definitions after.
How I got here
I was CTO of a logistics startup. After that I became a product manager at a software services company. Our work there was adding AI features to products that already existed and already had users.
One of them was an employee engagement platform. It had the usual social features: posts, comments, announcements. Senior people in each client organisation published policies on it. The feature we were asked for sounded simple. When an employee asks a question about a policy, the answer should come from that policy.
The chatbot that "just does not work"
Our first version was a simple RAG setup that we inherited from the previous team. Find the policy text that is most similar to the question, give it to the model, return the answer.
The review from the client was one line: "It just does not work as expected."
So we started reading the responses, one by one. The pattern was easy to see once we looked. The answer depended on the top few semantic similarity matches, and nothing in the system cared about when a document was written. A company publishes a leave policy. Later it publishes a new version. The two documents are almost the same text, so both match the question well. Whichever one matched first decided the answer. Sometimes that was the old one.
The model did nothing wrong. It answered faithfully from the text it was given. The mistake was in what we gave it. Chip Huyen's book AI Engineering mentions this exact need: for time-sensitive applications, retrieved documents can be reranked by time so newer data counts for more. We had no such step. The most similar text and the policy that applies today can be two different documents.
"Isn't it just calling an API?"
It is a fair question from a backend developer. My answer: that's where it starts, but the probabilistic nature and the long-running processes make it much more than that.
The probabilistic part first. Most developers I worked with were used to deterministic systems. You send a query, you get the same rows back every time. An LLM call is plain text in and plain text out, and the same input can give a different output. Huyen calls this the probabilistic nature of these models, and it changes how you test everything around them.
The long-running part. One call is easy. A real feature often needs several calls, retrieval, checks and retries, and that takes long enough that a normal request and response stops being enough. I wrote about both sides separately: what you build around an LLM call and why chat needs SSE.
Where the day goes
An AI engineer spends most of the day figuring out where the feature fails.
A backend engineer has a luxury here. They can focus on the things that work and add constraints for everything else. The API body has a structure. Anything unexpected is rejected, with a proper reason, before it touches the logic.
An AI engineer always receives a block of text that we call a prompt. The question is whether that text has enough context for the model to answer correctly. If it doesn't, the answer still comes back. It just seems a little less adequate, and nobody gets an error. The policy chatbot is that failure in one picture: the prompt was well formed, but the context in it was wrong.
So the work is reading outputs, finding the inputs that break the feature, and tracing each bad answer back to what the model was given.
The requirements changed too
Before AI features, a requirement could become a strict test case. Given this input, return exactly this. With AI features the requests started sounding like "something like this, for something like that". You can't write an assertion for "something like".
Our features demoed well and then fell flat in production on real user data. The demo used inputs we had picked. Production used everything else.
The team was reluctant to change how we worked. That changed when the client kept rejecting what we shipped. Then even the team had to rethink its methods. We started defining each request in much more detail: real example inputs, and what a good output looks like for each.
How others define it
Huyen defines AI engineering as "building applications on top of foundation models". That is the cleanest definition I know.
In 2023, Latent Space's essay The Rise of the AI Engineer described a new software engineering role, focused on applying models rather than training them. It argued that shipping AI products needs engineers more than researchers, and that product-specific data and evals belong to this role.
By 2026 the title is everywhere, and it is getting fuzzy. A Hacker News thread asks whether "AI Engineer" now just means someone who uses AI tools to write ordinary software. I don't think it does. Using Claude to write a backend is a good habit. Making a model behave inside a product is a different job.
My own version, from the chatbot: AI engineering is not getting a model to answer. It is making sure the answer comes from the right context, and knowing where it doesn't.
What I would do differently
In the first month, I would have started by defining the requirements better: detailed input and output expectations, written down, before any code. The line between technical requirements and domain requirements gets blurrier every day. The person who knows that a policy has versions is the person who can tell you what "correct" means. Get them in the room early.
If you are building AI features for an existing product and want a second pair of eyes on the step from demo to production, that is the work we do in prototype to production.
One thing to do this week
Take one AI feature you work on. Write down ten real inputs, and next to each one, the output a domain expert would accept. Include one input where an old document or a near-duplicate could fool it. Run all ten.
You'll learn more from the one it gets wrong than from the nine it gets right.