Software QA Lead – AI & GenAI Platforms

Software QA Lead – AI & GenAI Platforms Chicago, IL J10150857 Information Technology Corporate Corporate Job Description Build the quality engine behind enterprise AI. We’re looking for a hands-on QA leader to drive the strategy and execution of quality assurance for enterprise-grade Generative AI assistant applications. You’ll own how AI systems are tested, validated, and scaled, spanning full-stack applications powered by LLMs and RAG workflows. This role is critical to ensuring reliable AI outputs, strong user experiences, and responsible, ethical AI behavior in production. This position will offer flexibility for hybrid work schedules to include both in-office presence and telecommute/virtual work, to be based in Chicago, IL. Key Responsibilities Define and execute QA strategy for end-to-end AI applications (frontend, backend, and AI workflows) Lead validation of LLM outputs for accuracy, coherence, and reliability across diverse scenarios Test and optimize RAG pipelines to ensure high-quality retrieval and grounded responses Design user-centric test scenarios (multi-turn conversations, edge cases, failure handling) Conduct performance, load, and latency testing across AI and application layers Build and scale automated testing frameworks (API, UI, integration) within CI/CD pipelines Establish and track quality metrics (accuracy, relevance, latency, user satisfaction) Ensure ethical AI standards, including bias detection and responsible output validation Partner cross-functionally to embed QA practices across the development lifecycle Qualifications Minimum Qualifications Bachelor's Degree plus at least 10 years of software development experience, or demonstrated equivalency of experience and/or education Proven experience leading QA for complex, full-stack applications Strong expertise in testing Generative AI systems (LLMs, RAG workflows) Hands-on experience with test automation tools (e.g., Selenium, Appium) Experience with API testing, distributed systems, and CI/CD pipelines Familiarity with performance testing tools (e.g., JMeter, LoadRunner) Strong analytical, problem-solving, and communication skills Preferred Qualifications Experience with AI evaluation techniques (prompt testing, output scoring, bias testing) Background in conversational AI, NLP, or chatbot testing Familiarity with enterprise data/retrieval systems and knowledge bases Experience working in cloud environments (AWS, Azure, or GCP) Additional Information Additional Information Relocation assistance is not available for this position Sponsorship for US work authorization is not available for this position, now or in the future. Offered rate of compensation will be based on individual education, qualifications, experience, and work location. The range for this position is $150000 to $190000. About AECOM AECOM is proud to offer comprehensive benefits to meet the diverse needs of our employees. Depending on your employment status, AECOM benefits may include medical, dental, vision, life, AD&D, disability benefits, paid time off, leaves of absences, voluntary benefits, perks, flexible work options, well-being resources, employee assistance program, business travel insurance, service recognition awards, retirement savings plan, and employee stock purchase plan. AECOM is the global infrastructure leader, committed to delivering a better world. As a trusted professional services firm powered by deep technical abilities, we solve our clients’ complex challenges in water, environment, energy, transportation and buildings. Our teams partner with public- and private-sector clients to create innovative, sustainable and resilient solutions throughout the project lifecycle – from advisory, planning, design and engineering to program and construction management. AECOM is a Fortune 500 firm that had revenue of $16.1 billion in fiscal year 2025. Learn more at aecom.com. What makes AECOM a great place to work You will be part of a global team that champions your growth and career ambitions. Work on groundbreaking projects - both in your local community and on a global scale - that are transforming our industry and shaping the future. With cutting-edge technology and a network of experts, you’ll have the resources to make a real impact. Our award-winning training and development programs are designed to expand your technical expertise and leadership skills, helping you build the career you’ve always envisioned. Here, you’ll find a welcoming workplace built on respect, collaboration and community—where you have the freedom to grow in a world of opportunity. As an Equal Opportunity Employer, we believe in your potential and are here to help you achieve it. All your information will be kept confidential according to EEO guidelines. ReqID: J10150857 Business Line: Corporate Business Group: Corporate Strategic Business Unit: Information Technology Career Area: Information Technology Work Location Model: Hybrid Compensation: USD 150000 - USD 190000 - yearly