​
Overview

The Relevance Check guardrail uses advanced AI to verify that AI-generated responses are relevant, accurate, and aligned with the user’s query. It analyses both the user’s input and the AI’s response to ensure the content is appropriate, factual, and directly addresses the user’s needs. Key Points:
  • Verifies response relevance and accuracy against user queries
  • Prevents AI from generating off-topic or misleading information
  • Uses context-aware analysis to maintain conversation quality
  • Ensures responses are factual and well-supported
This ensures all AI interactions remain relevant, accurate, and trustworthy while maintaining high-quality user experiences.

​
What the Guardrail Does

​
Purpose

The primary goal of the Relevance Check guardrail is to maintain high-quality AI interactions by ensuring responses are relevant, accurate, and directly address user queries while preventing the spread of misinformation or off-topic content. By enabling this guardrail, organisations can improve user satisfaction, maintain content quality, ensure factual accuracy, and uphold responsible AI usage across all interactions.

​
Scope

​
Comprehensive Relevance Analysis

The Relevance Check guardrail applies advanced content analysis to:
  • Input – Applies the selected behaviour to what users send to the model.
  • Output – Applies the selected behaviour to what the model returns as a response.
  • Both – Full bidirectional coverage

​
Operational Modes

  • Monitor – Lets you review input or output content without taking any action—used for observation and diagnostics.
  • Block – Automatically stops content from being processed if it violates the selected guardrail rules.

​
Set Detection Threshold

Under Set Guardrail Threshold select the required detection sensitivity:
  • Low: Filters only the most clearly irrelevant responses. Content with low or uncertain relevance issues is allowed.
  • Medium: Filters responses that are likely or certainly irrelevant. Lower-confidence relevance issues are still allowed.
  • High: Applies strict filtering—blocks any response that may be irrelevant, even with low confidence.

​
Detection Categories

The guardrail monitors multiple aspects of response quality:
  • Relevance: Ensures responses directly address the user’s query and stay on topic
  • Accuracy: Verifies that information provided is factual and well-supported
  • Completeness: Checks that responses provide adequate information to answer the query
  • Context: Ensures responses maintain appropriate context and don’t introduce irrelevant information
  • Misinformation: Detects potential false or misleading information that could spread misinformation

​
Contextual Analysis

The Relevance Check guardrail performs comprehensive contextual analysis by evaluating:
  • Query Alignment: Whether the response directly answers the user’s specific question
  • Source Grounding: Whether the response is based on available information sources
  • Factual Consistency: Whether the response contains accurate information without introducing unverified claims
  • Contextual Relevance: Whether the response maintains appropriate context throughout the conversation

​
Key Features

Relevance Verification

Ensures AI responses directly address user queries and maintain conversation focus.

Accuracy Assessment

Verifies factual accuracy and prevents the spread of misinformation in responses.

Context-Aware Analysis

Advanced understanding of conversation context and query intent for accurate relevance checking.

Configurable Sensitivity

Adjustable detection thresholds for different use cases with Low, Medium, and High options.

Low Latency

High-performance detection that doesn’t impact response times or user experience.

Quality Assurance

Maintains high standards of response quality and user satisfaction across all interactions.

​
Why Use This Guardrail?

​
Benefits

  • Quality Assurance: Ensures AI responses are relevant and directly address user queries
  • Misinformation Prevention: Prevents the spread of false or misleading information
  • User Satisfaction: Improves user experience by maintaining high-quality, relevant responses
  • Brand Protection: Protects organisational reputation by ensuring accurate and helpful AI interactions
  • Trust Building: Maintains user trust through reliable and relevant AI responses

​
Use Case: Customer Support AI Assistant

​
Scenario

A technology company deploys an AI assistant to handle customer support inquiries across their product range. The assistant must provide accurate, relevant responses that directly address customer questions while preventing the spread of incorrect information about their products or services.

​
Challenge

The organisation must ensure that:
  • AI responses are always relevant to the customer’s specific query
  • Information provided is accurate and up-to-date
  • Responses don’t contain misleading or incorrect product information
  • All interactions maintain high quality and customer satisfaction

​
Solution: Implementing Relevance Check

  1. Comprehensive Quality Control
    • Enabled for both user inputs and AI responses
    • Configured to verify relevance and accuracy across all interactions
  2. Appropriate Enforcement
    • Set to Block to prevent irrelevant or inaccurate responses
    • Provides clear, helpful fallback responses when quality issues are detected
  3. Optimised Configuration
    • Used Medium sensitivity threshold for balanced quality control
    • Maintains detection effectiveness across diverse query types and topics

​
How to Use the Guardrail

Note: The steps below guide you through configuring the Relevance Check using the Guardrail Setup.

​
Step 1: Navigate to the Guardrail Setup

  1. From the Home Page, open the AI System Dashboard by selecting View to open your AI system from the AI System Table.
  2. In the guardrails section of the AI System Overview, click Edit Guardrails to launch the guardrail configuration workflow.

​
Step 2: Select and Enable the Relevance Check

  1. In the Configure Guardrails page, a list of available guardrails will be displayed.
  2. Click on Relevance Check to open its configuration options on the right-hand side of the screen.
  3. Toggle the Enable Policy switch to ON to begin configuration.

​
Step 3: Set Application Scope

  1. Under Apply Guardrail To, choose where the policy will be enforced:
    • Input – Applies the selected behaviour to what users send to the model.
    • Output – Applies the selected behaviour to what the model returns as a response.
    • Both – Full bidirectional coverage

​
Step 4: Configure Enforcement Behaviour

  1. Under Select Guardrail Behaviour, choose how the system should respond to relevance issues:
    • Monitor – Lets you review input or output content without taking any action—used for observation and diagnostics.
    • Block – Automatically stops content from being processed if it violates the selected guardrail rules.

​
Step 5: Adjust Detection Threshold

  1. Under Set Guardrail Threshold, select the required detection sensitivity:
    • 0.00 - 0.33 (Strict): Applies strict filtering—blocks any response that may be irrelevant, even with low confidence.
    • 0.34 - 0.66 (Balanced): Filters responses that are likely or certainly irrelevant. Lower-confidence relevance issues are still allowed.
    • 0.67 - 0.99 (Permissive): Filters only the most clearly irrelevant responses. Content with low or uncertain relevance issues is allowed.

​
Step 6: Save, Test, and Apply the Guardrail

  1. Click Save & Continue to store your configuration settings.
  2. Go to the Test Guardrails step to evaluate how the guardrail behaves in real time with a chatbot.
  3. After saving, you can proceed to the Summary section to review your configuration, save all changes, and view your AI System overview.
The Relevance Check guardrail provides enterprise-grade quality control for AI responses, ensuring your AI interactions remain relevant, accurate, and trustworthy while maintaining the highest standards of user satisfaction.

​
Relevance Check Categories

The Relevance Check guardrail is designed to identify and manage various aspects of response quality. Below is an overview of the primary categories our system monitors:
CategoryDescriptionExample
RelevanceEnsures responses directly address the user’s query and maintain conversation focusUser asks about product pricing, AI responds with unrelated product features
AccuracyVerifies that information provided is factual and well-supportedAI provides outdated pricing information or incorrect technical specifications
CompletenessChecks that responses provide adequate information to answer the queryUser asks for troubleshooting steps, AI provides incomplete or partial instructions
ContextEnsures responses maintain appropriate context and don’t introduce irrelevant informationAI includes unrelated product information when user asks about a specific feature
MisinformationDetects potential false or misleading information that could spread misinformationAI provides incorrect product capabilities or false claims about features

​
Understanding Response Quality Issues

The Relevance Check guardrail evaluates responses to ensure they directly address user queries. Here are the key scenarios it detects:

​
Relevant Responses

What it is: Responses that directly address the user’s specific question and maintain conversation focus. Example: 
- **User Query:** "What are the monthly fees for a checking account?"
- **Good Response:** "The monthly fee for maintaining a checking account is $10, and there are no fees associated with opening one."
- **Why it's good:** Directly answers the question about checking account fees.

​
Irrelevant Responses

What it is: Responses that don’t address the user’s specific question or introduce unrelated information. Example: 
- **User Query:** "What are the charges for using a checking account?"
- **Poor Response:** "The late payment fee for credit cards is 23.99%"
- **Why it's flagged:** Provides information about credit cards when the user asked about checking accounts.
Another Example: 
- **User Query:** "How do I reset my password?"
- **Poor Response:** "Our customer service hours are Monday to Friday, 9 AM to 5 PM."
- **Why it's flagged:** Provides customer service hours instead of password reset instructions.
Each category is monitored with configurable sensitivity, allowing organisations to maintain appropriate quality standards while ensuring smooth user interactions. The guardrail helps create a reliable AI environment while preserving the intended functionality and trustworthiness of your automated workflows.

​
Best Practices for Relevance Checking

​
Optimising Response Quality

When configuring the Relevance Check guardrail, consider these best practices for optimal performance: Threshold Selection:
  • The guardrail generates confidence scores for both relevance and accuracy
  • Higher thresholds increase the likelihood of blocking poor-quality content
  • Lower thresholds allow more content through but may include lower-quality responses
  • Configure thresholds between 0 and 0.99 for optimal filtering
Application Scope:
  • Apply to Output for most use cases to ensure response quality
  • Use Both for comprehensive quality control across the entire conversation
  • Consider Input monitoring for query validation in critical applications
Regular Review:
  • Monitor guardrail performance through the dashboard
  • Review blocked responses to identify false positives
  • Update configuration based on new use cases or quality requirements
Content Evaluation:
  • The guardrail evaluates three key components: available information sources, user queries, and AI responses
  • Ensure your information sources are comprehensive and up-to-date for accurate evaluation
  • Consider the relationship between user questions and available information when configuring thresholds
  • Monitor for patterns in blocked responses to identify potential improvements in information sources or AI training