Software Engineer in Test (AI)

Position Overview

An organization is seeking a Software Developer in Test to support quality engineering for a core technology platform. This role will focus on backend, API, automation, AI/ML-enabled application testing, and CI/CD-integrated quality practices. The position requires a hands-on test engineer with experience improving automation frameworks, partnering with engineering teams, and supporting feature-based testing across distributed systems.

Mandatory Requirements:

• 10 years of QA engineering delivery experience, with a strong focus on backend and API testing.
• 6 years of experience delivering within Agile SDLC teams, ideally in CI/CD environments.
• Experience testing AI/ML or LLM-powered systems.
• Experience with model validation, prompt testing, AI behavior regression testing, and evaluation of non-deterministic outputs.
• Familiarity with AI testing methods and metrics, including precision, recall, drift detection, bias assessment, and offline versus online evaluation.
• Working knowledge of AI-enabled development tools or platforms, such as AI coding assistants, test generation tools, or internal AI platforms.
• Strong experience testing REST APIs, backend microservices, and distributed systems.
• Solid understanding of CI/CD tooling and pipelines, especially GitLab CI/CD.
• Experience with Python, Behave, PyTest, Postman or REST testing tools, JMeter, or similar performance testing tools.
• Experience with GitLab, Git, Jenkins, and Docker.

Key Responsibilities

• Design, build, and maintain automated testing suites for backend services and API layers.
• Develop testing approaches for AI/ML models and AI-enabled internal applications, including validation of model responses, prompt performance, repeatability, and release-over-release regression.
• Create automated evaluation frameworks for AI systems, covering accuracy, consistency, hallucination detection, bias, and model drift.
• Incorporate AI evaluation and automated testing processes into CI/CD pipelines.
• Work with AI/ML engineers, platform teams, and governance stakeholders to support responsible AI practices, including traceability, test coverage, and production readiness.
• Integrate automated test suites into GitLab CI/CD pipelines and deployment workflows.
• Enhance automation frameworks to support changing application architecture, including blockchain-related components where applicable.
• Collaborate with software engineers, product managers, and DevOps teams to promote quality and testability across distributed systems.
• Manage defect tracking workflows using tools such as JIRA, ensuring issues are clearly documented, communicated, and traceable.
• Define and report QA metrics related to test automation development, test execution, defect trends, and overall release quality.
• Build and support performance and load testing for critical backend services and smart contract interactions when relevant.
• Perform functional, API, automation, performance, and integration testing as needed.
• Contribute to ongoing improvements in QA processes, tools, standards, and Agile testing practices.

Required Qualifications

• 10 years of experience in QA engineering delivery, with emphasis on backend systems or API testing.
• 6 years of experience working within Agile SDLC teams, preferably with CI/CD delivery models.
• Experience testing AI/ML or LLM-based systems, including model output validation, prompt testing, regression testing, and non-deterministic response evaluation.
• Knowledge of AI testing metrics and evaluation techniques, including precision and recall, drift detection, bias assessment, and offline versus online evaluation methods.
• Understanding of AI-enabled software development tools and their impact on SDLC processes, testing practices, and quality governance.
• Strong hands-on experience with REST API testing, backend microservices, and distributed systems.
• Practical knowledge of CI/CD tools and pipeline integration, particularly GitLab CI/CD.
• Experience using Python, Behave, PyTest, Postman or comparable REST testing tools, JMeter, or similar performance testing platforms.
• Experience with GitLab, Git, Jenkins, and Docker.
• Ability to work closely with developers, product managers, DevOps engineers, AI/ML teams, and governance stakeholders.
• Strong understanding of test automation frameworks, defect management, and Agile QA delivery.

Preferred Qualifications

• Experience working with blockchain technologies, including smart contracts, distributed ledgers, or blockchain node interactions.
• AI integration experience.
• Exposure to banking, financial services, or other regulated environments.
• Prior experience enhancing existing automation frameworks.
• Experience supporting quality governance for AI-enabled applications.

Why Join This Team / Organization Summary

This role offers the opportunity to support quality engineering for complex backend, API, AI-enabled, and distributed technology systems. The position is suited for a senior QA automation professional who can combine hands-on test development, AI testing practices, CI/CD integration, and cross-functional collaboration.