Most teams know how to test whether an AI feature works. The harder question is whether it works well enough for people to trust, understand and use in the context it was designed for.
For product teams, that distinction becomes especially important as AI moves beyond isolated experiments and into everyday digital experiences. A recommendation engine may return technically valid results but misunderstand what a user actually needs. A generative AI assistant may produce fluent answers while giving users no clear indication of when those answers are uncertain. A workflow may behave correctly in the happy path but leave users confused when the model fails, refuses a request or produces an unexpected output.
These are not problems that can always be solved by improving the model. They are often problems of validation.
AI validation and verification provide a broader way to evaluate whether an AI system was built correctly, whether it performs as intended and whether its behaviour remains appropriate in the environment where people actually use it. This is becoming increasingly relevant as AI governance develops through regulations such as the EU AI Act and frameworks including the NIST AI Risk Management Framework and ISO/IEC 42001. NIST, for example, treats testing, evaluation, verification and validation (TEVV) as part of the broader process of building trustworthy AI systems.
For designers, this raises an important question: when an AI product is being validated, are we validating the model alone, or the experience people have with it?
AI validation and verification are closely related, but they address different parts of the problem.
Verification asks whether the system was built according to its requirements. It examines whether the implementation matches the technical specifications, architecture and assumptions defined during development. If a team specifies how an AI feature should behave, verification helps determine whether the resulting system actually follows those specifications.
Validation asks whether the resulting system is appropriate for its intended purpose. It looks beyond technical implementation to evaluate whether the system performs reliably in the context where it will be used, whether its outputs meet the relevant requirements and whether its behaviour creates risks that were not apparent during development.
This distinction is particularly useful in product design. A team may successfully verify that an AI assistant returns an answer within the expected interface state, but that does not tell us whether users understand the answer, know when to trust it or have a reasonable way to correct it when it is wrong. Similarly, an AI recommendation component can satisfy its technical requirements while still producing recommendations that are irrelevant or difficult for users to interpret.
The difference becomes more important because AI systems are not operating in static environments. Data changes, user behaviour changes, models are updated and surrounding product systems evolve. NIST’s AI Risk Management Framework therefore recommends that AI systems be tested before deployment and regularly while in operation, with documented metrics, performance assessments and monitoring of system behaviour in production.
For product teams, validation should therefore extend beyond the moment when an AI feature is approved for launch. It should continue as the experience, model and context evolve.
The growing focus on AI validation is not simply a regulatory exercise. It reflects a more fundamental shift in how businesses build products with AI.
Traditional software tends to behave according to rules explicitly defined by the product team. AI introduces a different relationship between the system, its data and its users. Outputs can vary, models can behave differently across contexts and new failure modes can emerge when the system encounters situations that were not represented during development.
That makes the usual product development question — does this feature work? — less sufficient.
Teams also need to understand what happens when the system is uncertain, when it makes an incorrect inference, when its data changes or when users behave differently from what the original design assumed.
Regulation is adding another layer of urgency. The EU AI Act takes a risk-based approach and introduces requirements around areas such as risk management, data governance, technical documentation, human oversight, accuracy, robustness and cybersecurity for AI systems within its scope. Article 10 specifically addresses data governance for high-risk AI systems that use model training, including the management of training, validation and testing datasets.
The timeline also needs to be understood correctly in 2026. The AI Act entered into force in 2024, and many provisions become applicable from 2 August 2026. However, the EU’s 2026 AI Omnibus extended the application timeline for high-risk AI rules: certain high-risk systems covered by Annex III are scheduled for 2 December 2027, while high-risk AI embedded in regulated products under Annex I has an extended timeline to 2 August 2028.
That does not make validation a future problem. It makes the current period an opportunity for organisations to establish the processes, documentation and product practices they will need before those requirements become applicable.
NIST takes a voluntary but useful framework-based approach. Its AI RMF is designed to help organisations incorporate trustworthiness considerations into the design, development, use and evaluation of AI products and systems. The framework explicitly recognises characteristics such as validity and reliability, safety, security, transparency, explainability, privacy and fairness.
ISO/IEC 42001 approaches the issue from an organisational perspective. It provides requirements for establishing, implementing, maintaining and continually improving an AI Management System, giving businesses a structured way to manage AI-related risks and opportunities across their operations.
For design and product teams, the implication is practical: AI quality can no longer be separated completely from the experience built around it.
AI validation is not a single test. It involves several connected layers, and each one can affect the quality of the final product.
Data validation asks whether the data supporting an AI system is appropriate for its intended use.
For AI products, this matters long before a user interacts with the interface. A model trained or evaluated on data that does not adequately represent its real operating environment can produce unreliable results even when the underlying implementation is technically sound.
The questions are therefore broader than whether a dataset contains errors. Teams need to consider where the data came from, what it represents, how it was prepared, whether there are meaningful gaps, and whether potential biases could affect the system’s behaviour.
The EU AI Act explicitly addresses these considerations for relevant high-risk AI systems. Article 10 requires appropriate data governance and management practices for training, validation and testing datasets, including considerations around data collection, preparation, assumptions, suitability and potential bias.
For designers, this matters because data quality can eventually become experience quality. If an AI assistant consistently misunderstands a particular group of users or an AI recommendation system repeatedly produces irrelevant results, the problem may appear in the interface even though its origin sits much deeper in the data layer.
Model validation evaluates whether the model performs appropriately against the requirements of its intended use.
Depending on the product, this may involve accuracy, precision, recall, robustness, reliability, fairness or other measures. The right metric depends on the consequences of getting an output wrong. A model recommending music does not need the same validation criteria as a system supporting a high-impact decision.
This is also why a single benchmark score rarely tells the complete story. Product teams need to understand the conditions under which the model performs well and where its limitations begin.
NIST emphasises that AI evaluations should consider conditions similar to those in deployment and that the behaviour and functionality of AI systems should be monitored once they are in production.
That creates a direct connection to product design. If a model has known limitations, the interface should not hide them. The experience may need to communicate uncertainty, request clarification, offer alternative actions or involve a human reviewer depending on the use case.
A model is rarely the entire product.
An AI experience can depend on retrieval systems, APIs, databases, business rules, third-party models, content sources and human intervention. A model can perform well in isolation while the overall experience fails because one of these surrounding components behaves differently in production.
System validation therefore looks at the complete experience in its intended environment.
For designers, this is where AI validation becomes especially tangible. The system needs to be evaluated not only when everything goes according to plan, but also when the AI is uncertain, produces an incorrect answer, receives an unexpected input or cannot complete a request.
These states are often treated as edge cases during product development. In AI products, they are part of the experience itself.
A well-designed AI system therefore needs more than a polished happy path. It needs clear states for uncertainty, correction, escalation and recovery. Users need to understand what the system has done, what it has not done and what they can do next when the output is not useful.
That is a design responsibility as much as a technical one.
The final layer concerns the evidence behind the system.
An organisation may conduct extensive testing and still struggle to demonstrate what it has learned if the methods, results, limitations and decisions were never properly documented. Compliance validation connects technical evaluation with the evidence needed for governance, review and audit.
NIST’s Measure function specifically calls for test sets, metrics and details about the tools and methods used during TEVV to be documented. It also recommends regular assessment of AI systems and involvement from appropriate experts and independent assessors where necessary.
For product teams, documentation does not necessarily mean creating another layer of bureaucracy. It can mean recording the assumptions behind an AI experience, the scenarios tested, known failure modes, user feedback, escalation rules and the decisions made in response.
The more consequential the AI experience, the more valuable that record becomes.
AI changes the role of validation in product design because the interface is no longer simply presenting deterministic functionality.
With conventional software, a designer can often define a relatively clear relationship between an action and a resulting state. AI introduces uncertainty into that relationship. The same input can produce different outputs, and an output that looks convincing may still be incorrect.
This means designers need to consider a wider range of states when designing an AI experience.
Instead of validating only whether a user can complete a task, teams may need to ask whether users can recognise when the AI is uncertain, understand why a result was produced, correct an incorrect output and recover when the system cannot complete the task. These questions become particularly important when AI outputs influence decisions rather than simply assisting with low-risk tasks.
The validation process can therefore extend the familiar design sequence. Research still helps teams understand user needs, usability testing still helps evaluate interactions and prototyping still helps explore possible solutions. But AI products require teams to test how users interpret the system’s behaviour as well as whether they can navigate the interface.
For example, an AI financial assistant might generate a clear and useful explanation for most queries. A conventional usability test could show that users understand the interface and complete their tasks efficiently. A stronger AI validation process would also examine what happens when the assistant lacks enough information, gives an uncertain answer or encounters a request outside its intended scope.
The design question becomes less about making AI appear seamless and more about making its behaviour understandable.
That is an important shift for designers. The goal is not to design around the assumption that AI will always be right. It is to design an experience that remains useful when AI is uncertain, imperfect or wrong.
The term AI audit can refer to two different activities, and separating them helps clarify what businesses actually need to evaluate.
The first is an audit of an AI system. This involves assessing the system’s performance, risks, reliability, security, governance or compliance.
The second is using AI to conduct an audit. In this case, AI becomes part of the auditing process, helping review documents, analyse transactions, identify anomalies or compare large volumes of information against defined controls.
AI can make certain auditing processes more comprehensive by analysing datasets at a scale that would be difficult to handle manually. But the use of AI does not eliminate the need for human judgement or validation. If an AI system is responsible for identifying anomalies or producing evidence for an audit, the organisation still needs to understand the reliability and limitations of those outputs.
For product teams designing AI-powered audit tools, this distinction is particularly important. The interface should make the relationship between AI output and human decision-making visible. Users should be able to understand what the system has identified, inspect supporting information and decide when further investigation is necessary.
In other words, the design of an AI audit product should not simply optimise for automation. It should help users exercise appropriate judgement around AI-generated information.
The right AI audit tool depends on what the organisation is trying to validate, but several capabilities are broadly useful.
Traceability is one of them. AI-generated insights should be connected to the underlying information that supports them. Users need a way to understand where an output came from and, where appropriate, investigate the evidence behind it.
Continuous monitoring is equally important for production AI. A point-in-time assessment can show how a system behaved during one evaluation, but it cannot establish that the same behaviour will continue as the operating environment changes. NIST recommends monitoring AI functionality and behaviour in production as part of ongoing measurement.
Shadow AI visibility is another consideration. An organisation’s official AI inventory may not capture the tools employees are actually using. External AI applications can enter workflows through individual teams, creating questions around data privacy, security and governance.
Finally, privacy and security controls need to be evaluated before an AI audit platform receives access to sensitive business information. Data access, retention, encryption, authentication, and third-party subprocessors all become part of the tool-selection decision.
For designers, there is one additional consideration: how much of the evidence is actually understandable to the people using the tool? A technically complete audit trail is less useful if the interface makes it difficult for users to interpret what happened or decide what action to take next.
There is a different kind of validation happening outside the organisation that product and design teams should pay attention to.
AI systems are increasingly being used to discover and explain businesses. When someone asks an AI search system about a company, the system has to form an understanding from the information it can access. That may include the company’s website, structured data, external publications, directories and other sources.
The quality of that information affects what the AI system can understand and ultimately what it can say about the business.
This creates an external-facing version of the validation problem.
A company may have carefully validated the AI systems inside its products while the information available about the company itself remains difficult for AI systems to interpret. Important facts may be buried in unstructured pages. Business entities may be inconsistent across sources. Structured data may be incomplete. Product and service information may lack enough context to establish exactly what the business provides.
For a design studio, this is particularly relevant because discoverability is closely connected to how information is structured and communicated. Information architecture, content hierarchy, entity clarity and structured content are not only concerns for human visitors. They also influence how machine systems can interpret a digital presence.
An AI discoverability audit addresses this external layer. It examines whether a business’s public-facing digital signals give AI systems enough accessible and consistent information to discover, understand and represent the company accurately.
This should not be presented as a substitute for AI system validation or regulatory compliance. The two address different problems. Internal AI validation asks whether the AI a company builds or uses is fit for purpose. AI discoverability validation asks whether AI systems can form an accurate understanding of the company from the information available to them.
For businesses increasingly discovered through AI-powered search, both questions are worth asking.
Explore the AI Readiness Audit with Lollypop Spot
A practical AI validation process begins with visibility. Before an organisation can evaluate its AI systems, it needs to understand which systems exist, where they are being used and what role they play in the business.
Start by creating an AI inventory that includes both formally deployed systems and tools adopted by individual teams. Classify each system according to its intended purpose, risk and applicable governance requirements.
Next, document the data and evaluation criteria behind each system. Teams should understand what data is being used, what assumptions were made, which metrics define acceptable performance and what limitations were identified during testing.
For production systems, establish ongoing monitoring rather than relying solely on the original validation result. Define the signals that indicate a meaningful behaviour change and establish what happens when those signals are triggered. NIST recommends regular testing and monitoring because AI risks and system behaviour can change after deployment.
Product teams should also document AI-specific experience states. This includes what happens when the system is uncertain, produces an incorrect result, cannot fulfil a request or needs human intervention. These states should be considered part of the product experience rather than exceptions to it.
Finally, look at the organisation from the outside. Conduct an AI discoverability audit to understand whether AI systems can find and interpret the information that represents the business online.
Together, these checks create a more complete picture of AI readiness: how the system performs, how it behaves for users, how its risks are governed and how effectively the business itself can be understood by AI systems.
AI validation is often approached as something that happens behind the scenes: engineers test a model, compliance teams review the documentation and the product eventually goes live.
For AI products, that separation is becoming harder to maintain.
The way an AI system behaves is experienced through the product. Users encounter its recommendations, explanations, refusals and errors through an interface. They decide whether to trust an output based partly on how the experience communicates uncertainty and gives them control. When an AI system fails, the quality of the recovery experience can determine whether the failure becomes a minor inconvenience or a serious usability problem.
That makes validation relevant to the entire product team.
Designers do not need to become model engineers or compliance specialists to participate meaningfully in AI validation. Their role is to understand how the system behaves, identify where that behaviour affects users and design an experience that remains clear when the AI is uncertain or imperfect. This aligns with the broader view of trustworthy AI in NIST’s framework, which considers AI systems in relation to human behaviour and the context in which they are deployed.
There is also a second layer to consider. As AI systems increasingly mediate discovery, businesses themselves are becoming objects of machine interpretation. The information architecture and content that designers help shape can influence whether AI systems can understand what a company does, what it offers and why its claims should be trusted.
The result is a broader design challenge. Teams need to think about how AI behaves inside the product and how AI systems understand the business outside it.
That is why AI validation in 2026 is no longer only a question for model developers or compliance teams. It is becoming part of how responsible AI products are designed, tested and maintained — with evidence, user context and clarity built into the process from the start.
AI validation is the process of evaluating whether an AI system is appropriate and reliable for its intended purpose. It involves assessing system behaviour and outputs against relevant requirements and conditions, both before deployment and as the system operates in the real world.
Verification focuses on whether an AI system was built according to its technical requirements and specifications. Validation focuses on whether the resulting system performs appropriately for its intended purpose and context. Both sit within the broader testing, evaluation, verification and validation process used to assess trustworthy AI.
The EU AI Act does not define one universal process called “AI validation”. Instead, it establishes requirements for AI systems within its scope across areas including risk management, data governance, documentation, human oversight, accuracy, robustness and cybersecurity. For relevant high-risk AI systems, Article 10 specifically addresses the governance and quality of training, validation and testing datasets.
The implementation timeline also depends on the type of high-risk system. Following the 2026 AI Omnibus, certain Annex III high-risk AI rules are scheduled to apply from 2 December 2027, while high-risk AI embedded in regulated products has a transition to 2 August 2028.
An AI audit can refer to evaluating an AI system’s performance, risks, governance or compliance. It can also refer to using AI technology to support an audit of business processes, transactions or documents. The scope needs to be defined before selecting an AI audit tool.
Designers should validate more than the interface and happy-path interaction. They should consider whether users understand AI outputs, recognise uncertainty, know when the system may be wrong, can correct or challenge an output and have a clear path forward when the AI cannot complete a task. The exact requirements depend on the product’s risk and intended use.
AI systems increasingly use websites and external sources to understand and describe businesses. If important information is difficult to access, inconsistent across sources or poorly structured, AI systems may struggle to represent the business accurately.
An AI discoverability audit can help identify those gaps. It complements internal AI validation and governance rather than replacing them.
