AI QA Senior Educational qualification: BTech/Masters/PhD The opportunity We are seeking an experienced AI QA Engineer with 5?9+ years of professional experience and deep expertise in AI testing, quality engineering, validation frameworks, and enterprise AI solution assurance. The ideal candidate should possess strong hands-on experience validating AI systems, Agentic AI workflows, RAG applications, conversational AI solutions, and enterprise-scale AI platforms. The individual will play a key role in ensuring the quality, reliability, security, safety, and performance of AI systems while collaborating with AI engineers, architects, developers, and business stakeholders to establish robust quality assurance practices across AI and Generative AI implementations. Important note: This role requires professional implementation experience in relevant AI technologies. Personal portfolio projects, cloned demos, certifications, or tutorial-based builds are helpful, but are not sufficient on their own. Application guidance: Candidates should be able to clearly explain at least two relevant AI implementations, including the business use case, their direct contribution, the technical approach, and the outcome delivered. Your
key responsibilities: Design, develop, and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems.
Validate Large Language Model outputs for accuracy, consistency, relevance, safety, and compliance requirements. Develop test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications. Design and execute evaluation frameworks for RAG systems, retrieval pipelines, vector search solutions, and knowledge-based AI applications. Perform functional, integration, regression, performance, reliability, and scalability testing of AI-powered applications. Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines. Validate prompt engineering approaches, agent workflows, tool usage patterns, and reasoning capabilities in Agentic AI systems. Execute AI model testing across conversational AI, multimodal AI, and intelligent automation solutions. Perform AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and security validation activities. Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance requirements. Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms. Utilize RAG evaluation frameworks such as Ragas to measure retrieval quality, answer faithfulness, relevance, and response effectiveness. Develop testing dashboards, quality metrics, defect reporting processes, and continuous quality improvement practices. Collaborate closely with AI engineers and architects to identify quality risks and improve operational reliability. Contribute to quality standards, governance controls, testing accelerators, reusable frameworks, and best practices. Stay current with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging quality assurance technologies. Proven experience building production, pre-production, or enterprise pilot AI applications. Strong hands-on depth in at least 3 of the following areas, with meaningful ownership in at least 2: LLM application development. Retrieval pipelines. Agent or tool orchestration. Evaluation, guardrails, or quality systems. Model adaptation or fine-tuning. Cloud deployment or optimization of AI applications. Conversational AI. Multimodal AI. Backend engineering for AI systems. Strong software engineering foundation with ability to build reliable, maintainable AI services. Able to clearly explain direct contributions to delivered AI implementations. Educational Background Bachelor?s/master?s degree in computer science, Data Science, Engineering, or related field. Technical Skills Strong understanding of Large Language Models (LLMs), Generative AI, Agentic AI, and Agentic RAG systems. Hands-on experience validating and testing AI-powered applications and enterprise AI platforms.
Experience: developing AI testing strategies, validation frameworks, and automated test suites.
Strong expertise in AI Model Testing, functional testing, regression testing, integration testing, and end-to-end quality validation.
Experience: performing AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and vulnerability assessments.
Strong understanding of prompt engineering validation, reasoning evaluation, and response quality assessment techniques.
Experience: implementing evaluation frameworks and quality measurements for Agentic AI workflows and AI agents.
Hands-on experience with Ragas and similar evaluation frameworks for assessing retrieval quality, answer correctness, faithfulness, relevance, and grounding.
Experience: validating Agentic RAG systems, retrieval pipelines, and knowledge-based AI applications.
Experience: testing AI integrations across APIs, enterprise applications, databases, and cloud services.
Strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking.
Experience: working with Microsoft Azure AI Platform and Azure AI services.
Hands-on experience developing automated test frameworks and validation utilities using Python. Familiarity with FastAPI-based services and API testing frameworks.
Experience: designing and executing automated API testing strategies.
Knowledge of Responsible AI principles, AI safety controls, governance requirements, and regulatory compliance considerations.
Experience: evaluating hallucinations, bias, toxicity, safety violations, and model robustness.
Strong understanding of software quality engineering, secure development practices, test automation frameworks, and CI/CD testing integration.
Experience: working with performance testing, scalability testing, reliability testing, and production-readiness validation.
Familiarity with retrieval augmentation architectures, vector databases, embeddings, and enterprise search solutions. Strong analytical, troubleshooting, and root-cause analysis skills.
Soft Skills: Excellent problem-solving skills and ability to connect AI capabilities to business value.
Strong communication and presentation skills. Other Responsibilities: Contribute to end-to-end quality assurance activities across AI and GenAI engagements. Collaborate with AI engineers, architects, data scientists, and product teams to establish robust testing and validation practices. Participate in AI design reviews, risk assessments, model evaluations, and quality governance initiatives. Ensure AI solutions meet functional, security, compliance, governance, and business requirements before production deployment. Support development of reusable testing assets, automation frameworks, quality accelerators, and evaluation methodologies. Contribute to AI governance, Responsible AI, risk management, and compliance initiatives. Support presales, innovation initiatives, technical demonstrations, and client workshops where AI testing expertise is required. Provide mentorship to junior QA engineers and help establish AI quality engineering best practices.