AI Automation Tester

Overview

We are looking for an AI Automation Tester to support IBM on a major programme of work

The role will focus on designing and executing automated testing across an AI-enabled solution, covering traditional functional and integration testing alongside the additional challenges involved in validating AI-generated responses and behaviours.

You’ll work as part of the wider delivery team to ensure the solution is reliable, performs as expected and continues to meet requirements as it develops.

Key Responsibilities

• Design, build and maintain automated tests across the AI solution
• Execute functional, integration and regression testing
• Develop and maintain automated test suites
• Perform API testing across integrated services
• Validate AI-generated responses and system behaviours
• Design appropriate test scenarios for AI functionality
• Manage and maintain test data required across different testing activities
• Test integrations between the AI solution and wider systems/services
• Support performance testing and identify potential issues or bottlenecks
• Analyse test results, identify defects and work with technical teams through resolution
• Support continuous testing throughout the delivery lifecycle
• Ensure appropriate regression coverage as the solution evolves
• Work closely with developers, engineers and wider delivery stakeholders

Key Skills & Experience

We’re looking for candidates with strong experience across:

• Test Automation
• API Testing
• Functional Testing
• Integration Testing
• Regression Testing
• Performance Testing
• Automated test frameworks and scripting
• Test data management
• Continuous testing / CI/CD environments
• Defect identification, investigation and management

Experience testing AI, Generative AI or LLM-based solutions would be particularly valuable, including experience evaluating the quality, consistency and accuracy of AI responses.

AI Testing

Alongside traditional automation testing, an important part of the role will be understanding how to effectively test an AI-enabled solution.

Relevant experience could include:

• AI response evaluation and validation
• Testing non-deterministic AI outputs
• Developing test scenarios and evaluation criteria for AI responses
• Testing AI APIs and integrations
• Automated evaluation of AI functionality
• Identifying unexpected, inconsistent or inappropriate model responses
• Building AI-specific testing into wider automated test suites

Candidates don’t necessarily need to come exclusively from an “AI Tester” background, but should have strong automation testing fundamentals and ideally some practical exposure to testing AI/LLM-enabled applications.

Reasonable Adjustments:

Respect and equality are core values to us. We are proud of the diverse and inclusive community we have built, and we welcome applications from people of all backgrounds and perspectives. Our success is driven by our people, united by the spirit of partnership to deliver the best resourcing solutions for our clients.

If you need any help or adjustments during the recruitment process for any reason, please let us know when you apply or talk to the recruiters directly so we can support you.