Software Quality Assurance Engineer – GenAI / LLM

Primary Focus: Software Quality Assurance / Software Testing / Test Automation – GenAI, LLM & Agentic AI

Secondary Exposure: Solution Analysis / Technology Solution Design / Enterprise Integration

Domain / Project: Global Markets, Capital Markets Banking Technology & Market Risk Technology

Role Overview

We are looking for a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications within a complex enterprise environment.

This is not a traditional manual QA or software testing role.

The role combines:

Software Quality Engineering

GenAI / LLM Testing & Evaluation

Agentic AI / AI Agent Testing

UI & API Testing

Test Automation

Solution Analysis

Enterprise Integration Testing

Observability & Troubleshooting

You will work across discovery, solution design, development, testing and release, translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations.

Key Responsibilities

GenAI / LLM Quality Engineering

Define and execute end-to-end quality engineering and test strategies covering:

Web / UI workflows

REST APIs

Backend services

Enterprise integrations

GenAI applications

LLM workflows

RAG pipelines

Agentic AI / AI Agent interfaces

Perform GenAI / LLM testing and evaluation covering:

Response quality

Task completion

Grounding

Faithfulness

Relevance

Consistency

Citation accuracy

Hallucination risk

Safe failure behaviour

Test non-deterministic / probabilistic AI systems using:

Evaluation datasets

Repeat testing

Quality thresholds

Acceptance criteria

Regression evaluation

Validate RAG / Retrieval-Augmented Generation solutions, including retrieval quality, grounding and response accuracy.

Agentic AI / AI Agent Testing

Test end-to-end Agentic AI and AI Agent workflows, including:

Multi-turn conversations

Context handling

Agent planning

Tool selection

Tool calling / function calling

Tool inputs and outputs

State transitions

Memory and state

Human-in-the-loop approvals

Handoffs

Retries

Timeouts

Fallback behaviour

Error recovery

Termination conditions

Partial failures

Validate that AI agents behave correctly across both successful and failure scenarios.

Software & API Quality Engineering

Perform:

Functional Testing

Integration Testing

API Testing

Regression Testing

Exploratory Testing

Negative Testing

Resilience Testing

Basic Performance Testing

End-to-End Testing

Design comprehensive REST API tests covering:

API contracts

Authentication

Authorisation

Input validation

Error handling

Idempotency

Rate limits

Downstream system failures

Test web application behaviour across browsers and realistic end-user journeys, including:

Loading states

Interrupted sessions

Error messages

Feedback capture

Accessibility fundamentals

Test Automation

Develop and maintain risk-based test automation that reduces:

Regression testing time

Manual testing effort

Release cycle time

Production risk

Use automation frameworks and tools such as:

Playwright

Cypress

Selenium

pytest

REST Assured

Postman

Equivalent UI / API automation frameworks

Apply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios.

GenAI Evaluation & AI Safety Testing

Validate LLM and GenAI applications for:

Grounded responses

Hallucinations

Retrieval quality

Citation accuracy

Prompt behaviour

Prompt injection

Unsupported requests

Restricted content handling

Safe failure behaviour

Adversarial scenarios

Support AI evaluation / LLM evaluation using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches.

Exposure to AI Red Teaming / Adversarial Testing would be advantageous.

Observability & Troubleshooting

Use application and GenAI observability to identify the source of defects across:

Application

LLM / Model

RAG / Retrieval

Data

API / Integration

Platform

Analyse:

Logs

Distributed traces

API requests / responses

Payloads

Network calls

Database records

Agent execution traces

Exposure to observability and LLM evaluation tools such as:

Langfuse

LangSmith

OpenTelemetry

Elastic / Elasticsearch

Splunk

is advantageous.

Solution Analysis & Design

The role also acts as a hands-on Solution Analyst for GenAI applications.

Responsibilities include:

Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design.

Analyse proposed GenAI use cases and determine whether the requirement should use:

Conventional application logic

Deterministic business rules

Search / retrieval

RAG

Workflow automation

Agentic AI

Human approval

Translate business requirements into:

Functional requirements

End-to-end solution flows

User journeys

Acceptance criteria

Interface behaviour

Decision rules

Non-functional requirements

Map interactions across:

User Interfaces

APIs

LLMs / Models

Prompts

RAG / Retrieval components

Enterprise data sources

AI Agent tools

Downstream enterprise systems

Analyse solution design trade-offs involving:

Quality

Complexity

Cost

Latency

Security

Data access

Maintainability

Operational risk

Identify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins.

Support the design of:

Human-in-the-loop approval

Fallback flows

Escalation

Exception handling

Solution Documentation

Produce practical technical and functional artefacts including:

Process Flows

Sequence Diagrams

Context Diagrams

Interface Specifications

Decision Tables

User Stories

Acceptance Criteria

Test Scenarios

Traceability Documentation

Maintain traceability across:

Business Requirement → Solution Design → Implementation → Test / Evaluation Scenario → Release Evidence

Release Quality & Governance

Create and maintain:

Test scenarios

Test datasets

Reusable regression scenarios

Test evidence

Defect reports

Quality metrics

Release