AI QA Framework & Testing
Operationalize and execute client-approved QA frameworks, evaluation rubrics, and test scripts tailored for AI chatbots, virtual agents, and copilot tools.
Perform routine pre-release User Acceptance Testing (UAT) and regression testing across prompt deployments, Knowledge Base (KB) updates, and automated workflow releases within the operational environment.
Operational Auditing & Performance Monitoring
Conduct daily audits of live AI-driven conversations, automated resolutions, and self-service interactions to detect intent misclassifications, system hallucinations, incorrect policy applications, or journey drops.
Audit the handover bridge between AI and Human-Assist agents to evaluate context retention, CRM data pass-through accuracy, and agent handle times post-transfer.
Closed-Loop Vendor Feedback & Defect Analysis
Perform root-cause defect analysis on resolution failures, escalation spikes, and recurring customer friction points.
Synthesize audit data into actionable weekly reports; feed continuous improvement insights to Client AI Automation Leads, KB Managers, and Product Engineering for system retuning and sprint updates.
Identify out-of-scope traveler queries and KB content gaps, recommending new intents and articles to safely broaden containment coverage.
Operational Metrics & SLA Compliance
Measure, track, and report on primary operational and AI quality KPIs:
Intent Recognition Accuracy
Resolution Precision & Answer Accuracy
Containment Rate & Defect Rate
Handover Context Retention Accuracy
Traveler Sentiment Score (TSS) / CSAT Impact
Hold operational teams and frontline agents accountable for adhering to updated AI copilot workflows and knowledge standards.
Experience
Contact Center QA Background: 2+ years of experience in quality management, operational auditing, or process improvement within a high-volume BPO/contact center environment.
AI & Automation Auditing: Demonstrated experience testing, auditing, or evaluating AI chatbots, conversational flows, copilot integrations, or workflow automation tools.
Travel Industry (Preferred): Prior experience in travel, hospitality, or tech-enabled customer support operations is strongly preferred.
Methodology: Familiarity with operational process improvement tools (Lean, Six Sigma, Root Cause Analysis / 5 Whys).
Core Competencies & Skills
Conversational AI Fluency: Deep understanding of natural language processing (NLP/NLU) behavior, intent mapping structures, and failure patterns (e.g., hallucination, drift, misclassification).
Analytical & Data-Driven: Ability to turn transcript audits, CSS feedback, and performance dashboard data into actionable insights, using AI tools to support analysis and validating outputs for accuracy and relevance.
Cross-Functional Communication: Clear written and verbal reporting skills to present quality defects and technical feedback to both BPO leadership and client technical teams.
Quality Standard-Setting: Meticulous attention to detail with an unyielding commitment to brand trust and effortless traveler resolution.
All your information will be kept confidential according to EEO guidelines.