AI System Testing: How to Test LLMs, RAG and AI Agents Before Production

Why is traditional QA not enough for AI systems?

In traditional software, a large part of testing is based on predictable system behaviour. Given a particular input and set of conditions, the application is expected to produce a specific result. With generative AI components, responses may be non-deterministic – two different outputs may both be acceptable.

This changes how quality needs to be defined. Instead of checking only whether the result exactly matches an expected output, teams may need to evaluate factual correctness, completeness, adherence to context, security and behaviour across different scenarios.

Worth knowing: NIST uses the concept of Testing, Evaluation, Validation and Verification (TEVV) when discussing AI systems, while the AI Risk Management Framework addresses risk management across the AI lifecycle. See NIST AI RMF 1.0.

7 layers of AI system testing

Testing the model endpoint and declaring the entire solution validated is not enough. In a production environment, the model is only one component of a larger architecture.

1. Data and input: are we testing what the AI actually receives?

Even a capable model can produce poor results when input data is incomplete, outdated or inconsistent. AI testing should therefore begin before the model itself.

The evaluation set should reflect not only ideal scenarios but also the type of data the system is likely to encounter in real production use.

If problems already exist at the level of source systems, ownership, freshness or pipelines, first assess whether your data is ready for AI.

2. How do you test RAG and retrieval?

In a RAG architecture, answer quality depends on more than the capabilities of the LLM. The system first needs to retrieve the right knowledge. If retrieval returns irrelevant or incorrect documents, the model may generate a highly convincing answer based on the wrong context.

Evaluation should therefore be separated into at least two levels:

RAG testing should verify, among other things:

• whether the correct document appears among the retrieved results,
• whether retrieval returns too much irrelevant context,
• how the system behaves when the answer does not exist in the knowledge base,
• whether updated information becomes available after a source document changes,
• whether users receive answers based only on information they are authorised to access,
• whether chunking and indexing cause important context to be lost.

Expert source: Microsoft Foundry separates RAG evaluation into answer-quality evaluation and retrieval-process evaluation, including dimensions such as groundedness, relevance and document retrieval. See Microsoft Foundry RAG Evaluators.

3. How do you evaluate LLM outputs?

Generative AI often does not have a single perfect answer that can be stored as a traditional expected result. Teams therefore need explicitly defined evaluation criteria.

Build a representative evaluation dataset

A useful evaluation baseline is a stable set of representative scenarios containing user inputs, required context, acceptance criteria and cases where the system should refuse to answer or request additional information.

Such a dataset makes it possible to compare new prompt versions, model versions, retrieval configurations and other changes against the same quality criteria. This helps determine not only whether a change improves one scenario, but also whether it introduces regressions elsewhere.

4. AI security testing: what happens when the user or input data does not follow the intended scenario?

AI systems should not be tested only against valid user requests. A user may attempt to bypass restrictions, access sensitive information or alter the intended behaviour of the system. Threats may also originate from external content processed by the AI system.

Expert source: OWASP describes both direct and remote/indirect prompt injection, as well as risks such as RAG poisoning and attack scenarios relevant to agentic systems. See the OWASP LLM Prompt Injection Prevention Cheat Sheet.

5. How do you test AI agents and the actions they perform?

A wrong answer from a chatbot may primarily create an information problem. An error in an agentic system can have a broader impact because the AI may modify a record, send a message, create a ticket, call an API or trigger another business process.

Testing AI agents should therefore cover not only the quality of the final outcome, but also the observable execution path and boundaries of autonomy: which tools the agent selects, which parameters it passes, in what order it performs actions and when it should stop and request human approval.

Expert source: Microsoft Foundry distinguishes between evaluating the overall result of an agent workflow and evaluating the individual steps in the process. Process-level evaluation can include tool selection, tool input accuracy, tool output utilisation and tool-call success. See Microsoft Foundry Agent Evaluators.

6. Latency, availability and cost: AI also requires non-functional testing

An AI system can produce excellent answers and still be unsuitable for production. Response time, API limitations, scalability or the cost of each interaction may become the real blocker.

This is often where problems hidden during demonstrations become visible. We discuss these challenges in more detail in our guide to why AI projects stall between PoC and production.

7. AI QA does not end at deployment: regression testing and production monitoring

In traditional software, a new code release is an obvious trigger for regression testing. AI systems have more potential sources of behavioural change.

What should be monitored after deployment?

Expert source: NIST highlights the difference between pre-deployment evaluation in controlled environments and post-deployment monitoring in real operating conditions, including the need to identify unexpected outputs and consequences that emerge in use. See NIST AI 800-4 – Challenges to the Monitoring of Deployed AI Systems.

How do you define acceptance criteria for an AI system?

Running tests alone does not tell the team whether the system is ready for production. The project needs acceptance criteria that define what level of quality, risk, performance and cost is acceptable for the specific use case.

There is no universal threshold that works for every AI system. An internal assistant used to search documentation may have different requirements from an agent performing operations in a financial system. Acceptance criteria should reflect the use case, the consequences of potential errors and the organisation’s risk tolerance.

AI testing checklist: is the system ready for production validation?

Before release, check whether the team can confidently answer “yes” to the following questions.

How can Edge One Solutions support AI system testing?

At Edge One Solutions, we look at AI system quality beyond the model itself. A production solution also includes data, applications, integrations, access control, infrastructure and the operating model used after deployment.

Depending on the architecture and use case, we can combine QA, Artificial Intelligence, Data Engineering, development and maintenance capabilities to design testing across the entire system flow.

If your project is still at an earlier stage, start by assessing whether your data is ready for AI. If the PoC already works but scaling, integrations and operations remain unresolved, see our guide to moving AI from PoC to production.

For AI solutions affected by regulatory requirements, see our guide to the technical implications of the EU AI Act for businesses.