We’re Hiring: Principal QA Engineer (AI Systems & Platform) – Remote
Job Title: Principal QA Engineer
Work Mode: Remote
Shift: 7:30 PM IST – 4:30 AM IST
Experience: 7+ Years
About the Role
We are looking for a highly skilled Principal QA Engineer to build and lead the QA function from scratch for cutting-edge AI systems.
This is a founding-level role where you will define quality standards, design testing frameworks, and ensure AI-generated insights are accurate, reliable, and production-ready. You will act as the final quality gate before outputs reach business leaders and clients.
Key Responsibilities
A. QA Foundation & Strategy
- Build QA processes from the ground up (unit, integration, end-to-end testing)
- Define quality benchmarks, coverage standards, and documentation practices
- Audit platform risks and prepare for client deployments
B. AI Systems & Evaluation
- Design evaluation frameworks for non-deterministic AI/LLM outputs
- Implement prompt regression testing and model drift detection
- Build automated testing for AI agent orchestration and pipelines
C. Platform & Integration Testing
- Test end-to-end data ingestion pipelines across multiple systems
- Validate multi-model routing, streaming responses, and system performance
- Ensure reliability of file ingestion pipelines and enterprise data workflows
D. Data & Knowledge Systems Validation
- Validate knowledge graphs, data accuracy, and retrieval quality
- Ensure reliability of AI-generated insights before client delivery
E. Monitoring & Reliability
- Track system performance, failures, and latency thresholds
- Ensure graceful handling of model failures and degraded scenarios
F. Team Building & Leadership
- Hire, mentor, and lead QA engineers
- Establish a strong quality-first engineering culture
- Collaborate with engineering and product teams in Agile environments
Required Skills
- 7+ years in QA Engineering (3+ years in leadership roles)
- Strong experience testing AI/LLM-based applications
- Expertise in Python and automated testing frameworks
- Hands-on experience with Playwright and Vitest
- Experience with API testing, async systems, and distributed architectures
- Strong understanding of CI/CD pipelines and test automation
- Excellent problem-solving and leadership skills
Tech Stack Exposure
- AI/LLM: Anthropic Claude, OpenAI GPT, multi-model routing
- Backend: Python (FastAPI), async systems
- Data: Memgraph, Neo4j, Qdrant, PostgreSQL, Redis
- Frontend: React, Next.js, TypeScript
- Cloud: Google Cloud Platform, Azure
- DevOps: GitHub Actions, Docker
- Testing: Playwright, Vitest
Nice to Have
- Experience with LLM evaluation frameworks (LangSmith, PromptFlow, etc.)
- Exposure to agent-based systems or orchestration frameworks
- Experience in regulated industries (finance, healthcare, etc.)
- Experience working with client-facing or deployment teams
Salary
As per industry standards