Question 1

How do you know your LLM feature is working?

Accepted Answer

Whether you have an evaluation practice or you shipped on impressions. This is the fastest way to separate people who have run something in production from people who have built demos. Describe your eval set: how many cases, where they came from (production failures are the best answer), and what you measure. Say explicitly what your pass bar was and what you did when a change regressed it. If you only had manual spot checks, say so and explain what you would build first.

Question 2

Your RAG system returns confident but wrong answers. Walk me through diagnosing it.

Accepted Answer

Whether you can separate retrieval failure from generation failure. Most candidates immediately blame the model, which is usually wrong. Check retrieval first and in isolation: for a failing question, is the correct chunk in the retrieved set at all? If it is not, the problem is chunking, embeddings, or reranking — not the model. Only if the right context was retrieved and ignored is it a generation or prompt problem. Mention measuring recall@k rather than eyeballing.

Question 3

When would you not use an LLM?

Accepted Answer

Engineering judgment. Enthusiasm is cheap in this field; knowing the limits is what makes someone safe to hire. Name a concrete case: deterministic extraction from structured documents, anything with a hard correctness requirement and no review step, latency budgets under roughly 100ms, or a task a regex and a lookup table already solve. The strongest version is a time you argued against using one and were right.

Question 4

How would you cut the inference cost of a feature in half without users noticing?

Accepted Answer

Whether you treat cost as an engineering constraint. This is the question most likely to expose someone who has never owned a bill. Work down the obvious levers: route easy requests to a smaller model with a quality gate, cache aggressively at the prompt-prefix and response level, trim retrieved context (most RAG prompts carry far more than they need), and shorten outputs. Say how you would verify quality held — this loops back to evals.

Question 5

How do you handle prompt injection in a system with tool access?

Accepted Answer

Security instinct for nondeterministic systems. Increasingly asked, and frequently answered with hand-waving. Treat model output as untrusted input. Constrain what tools can do rather than trying to instruct the model out of misbehaving — allowlists, scoped credentials, no destructive actions without confirmation. Separate retrieved content from instructions in the prompt. Say plainly that prompt-level defences alone are not sufficient.

Question 6

Tell me about a model change you shipped that made things worse.

Accepted Answer

Whether you can detect regression at all, which requires having had a baseline. Name what regressed and how you found out — ideally from an eval suite or a monitored metric rather than a user complaint. Explain the rollback and what you added so the same class of regression would be caught before release next time.

AI Engineer interview questions

“How do you know your LLM feature is working?”

“Your RAG system returns confident but wrong answers. Walk me through diagnosing it.”

“When would you not use an LLM?”

“How would you cut the inference cost of a feature in half without users noticing?”

“How do you handle prompt injection in a system with tool access?”

“Tell me about a model change you shipped that made things worse.”

Before the interview, get past the screen

Related roles