AI Quality and Evaluation Manager

Lorien • London, United Kingdom • 6h ago

AI Quality & Evaluation Manager (Contract)

Location: Hybrid working - Blackfriars 3 days per weekContract: 6 months, Outside IR35

Are you passionate about building the future of AI quality? Do you thrive in hands-on roles where you can shape frameworks from the ground up and make a real impact? We're looking for an experienced AI Quality & Evaluation Manager to join our team on a contract basis and lay the foundations for robust, reliable, and user-focused AI services across our business.

What You'll Do

Design and implement a comprehensive AI testing and evaluation framework for all AI solutions, including LLM-based tools, RAG systems, and third-party platforms.
Define and document quality standards for semantic accuracy, factual consistency, bias, tone, and relevance.
Develop reusable testing templates, data sets, and evaluation methods that can be scaled and maintained by internal teams.
Run hands-on testing of AI prototypes and production tools to assess technical performance and business value.
Collaborate with business users to guide practical testing and feedback processes.
Deliver training and upskilling materials to empower internal staff to sustain the framework after your contract ends.
Support vendor evaluations and POC assessments with robust test protocols.
Establish baseline metrics and dashboards to measure ongoing AI quality and relevance.
Work closely with engineering and product leads to embed testing into delivery workflows.
Champion responsible AI practices to ensure fairness, transparency, and user trust.

What You'll Bring

Strong hands-on experience in testing and evaluation of AI or software systems, ideally with NLP or LLM-based applications.
Understanding of prompt evaluation, semantic search, and LLM behaviour (accuracy, hallucination, bias, tone, etc.).
Familiarity with tools like Trulens, HumanLoop, PromptLayer, or similar; experience designing QA approaches for GenAI environments.
Knowledge of modern AI architectures (RAG pipelines, embeddings, API integrations such as OpenAI, Azure OpenAI, Anthropic).
Experience designing and implementing structured test regimes in fast-evolving contexts.
Excellent communication and facilitation skills, engaging both technical and business audiences.
Proven ability to create sustainable frameworks, documentation, and training materials.

Who You Are

A builder who loves creating practical, scalable solutions.
Hands-on and analytical, balancing experimentation with process.
Collaborative and empathetic, bridging technical and non-technical teams.
User-focused, driven by delivering real value.
Committed to responsible AI, fairness, and transparency.

Ready to shape the future of AI quality with us?Apply now and help us ensure our AI-enabled services are accurate, consistent, and trusted by all.

Carbon60, Lorien & SRG - The Impellam Group STEM Portfolio are acting as an Employment Business in relation to this vacancy.

Related Jobs

Digital Product Manager - Insurance Broker Model

Lorien • London, United Kingdom • 6h ago

Outside IR35

Digital Product Manager - Insurance Broker Model Contract: 6 months (Outside IR35) Location: Hybrid - London Head Office We're looking for an experienced Product Manager to support a major transformation project within o...

6h ago

Apply

Part Time M365 IT & Security Support Engineer £300 - £350 Outside IR35 Onsite

Opus Recruitment Solutions Ltd • London, United Kingdom • 6h ago

Engineer Outside IR35

Part Time M365 IT & Security Support Engineer £300 - £350 Outside IR35 Onsite We're recruiting an experienced M365 IT & Security Support Engineer to assist our client through a busy transitional period. This 2-mo...

6h ago

Apply

Part Time | M365 IT & Security Support Engineer | £300 - £350 | Outside IR35 | Onsite

Opus Recruitment Solutions Ltd • ATS • London, Not Specified, United Kingdom • £300 - £350 / * • 10h ago

Engineer Outside IR35

Part Time | M365 IT & Security Support Engineer | £300 - £350 | Outside IR35 | Onsite We're recruiting an experienced M365 IT & Security Support Engineer to assist our client through a busy transitional period. T...

10h ago

Apply

Contract Data Engineer - eDV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £450 - £650 / * • 10h ago

Engineer Outside IR35

CONTRACT DATA ENGINEER - eDV CLEARED NEW OUTSIDE IR35 CONTRACT OPPORTUNITY AVAILABLE WITHIN A LEADING NATIONAL SECURITY SME FOR A DATA ENGINEER WITH eDV CLEARANCE Contract job opportunity for a Data Engineer National Sec...

10h ago

Apply

Contract Senior Software Engineer - eDV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £500 - £750 / * • 10h ago

Engineer Outside IR35

CONTRACT SENIOR SOFTWARE ENGINEER - eDV CLEARED NEW CONTRACT JOB OPPORTUNITY AVAILABLE WITHIN A LEADING NATIONAL SECURITY COMPANY FOR A SENIOR SOFTWARE ENGINEER WITH eDV CLEARANCE Contract job opportunity for a Senior So...

10h ago

Apply

Contract Solution Architect - eDV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £600 - £850 / * • 10h ago

Architect Outside IR35

Contract Solution Architect - Enhanced DV Cleared New contract opportunity in London for a Solution Architect with eDV Clearance to work across Operational, National Security projects New contract opportunity for a Solut...

10h ago

Apply

Contract Palantir Foundry Engineer - DV Cleared

Searchability NS&D • TRAFFIC • London, Not Specified, United Kingdom • £500 - £650 / * • 10h ago

Engineer Outside IR35

CONTRACT PALANTIR DATA ENGINEER - DV CLEARED NEW OUTSIDE IR35 CONTRACT OPPORTUNITY AVAILABLE WITHIN A HIGHLY SECURE DELIVERY PROGRAMME FOR A DV CLEARED PALANTIR FOUNDRY DATA ENGINEER. Contract opportunity for a Palantir...

10h ago

Apply

Palantir Engineer / Developer - Perm & outside IR35 Contract roles - DV Cleared - London

FryerMiles Recruitment • TRAFFIC • London, Not Specified, United Kingdom • £500 - £750 / * • 12h ago

Developer Outside IR35

Palantir Engineer / Developer - Perm & outside IR35 Contract roles - DV Cleared - London FryerMiles are delighted to be partnering with a renowned consultant to assist with their recruitment of multiple Palantir Engi...

12h ago

Apply

IT Controls Analyst - Audits

ARM (Advanced Resource Managers) • APPLICATION • London, Not Specified, United Kingdom • 15h ago

Outside IR35

IT Controls Analyst - IT Audits 6 months Remote £Negotiable - INSIDE IR35 Good practical experience of controls testing, including knowledge of key control areas, such as security and IT resilience IT Audits and strategi...

15h ago

Apply

Senior Complaints Handler - Contract | London, UK | Hybrid

Hunter Bond • London, England, GB • 17h ago

Outside IR35

My leading FinTech client are looking for someone to take responsibility for dealing with high level / FOS complaints and managing them end to end, maintaining their internal FOS and ADR logs, collating reporting for man...

17h ago

Apply