New cohorts enrolling monthly — onsite, online & hybrid formats. Modern SQA Formula Bootcamp 2026 Registration ongoing Corporate teams: ask about custom bootcamp pricing.
AI quality team evaluating LLM responses retrieval evidence safety latency and cost
SQA, Test Automation & SDET · Course #057

LLM Application Testing

The purpose of LLM Application Testing is to help participants test applications built on large language models across task quality, retrieval, safety, latency, cost and changing model behaviour with methods they can defend and adapt. Core ideas such as llm application architecture and probabilistic behaviour, intended use, users, harms and testable quality criteria and evaluation datasets are explored through focused demonstrations and applied tasks. Participants define task-specific criteria, create evaluation datasets, test prompts and retrieval context, compare deterministic and human judging, probe injection and harmful behaviour and monitor regressions. Learners also consider quality risks, useful documentation and the hand-offs required across a modern software team. They build an evaluation harness concept that acknowledges probabilistic outputs and separates model, orchestration, data and user-experience failures.

Enroll Interest Back to Courses

Overview

The purpose of LLM Application Testing is to help participants test applications built on large language models across task quality, retrieval, safety, latency, cost and changing model behaviour with methods they can defend and adapt. Core ideas such as llm application architecture and probabilistic behaviour, intended use, users, harms and testable quality criteria and evaluation datasets are explored through focused demonstrations and applied tasks. Participants define task-specific criteria, create evaluation datasets, test prompts and retrieval context, compare deterministic and human judging, probe injection and harmful behaviour and monitor regressions. Learners also consider quality risks, useful documentation and the hand-offs required across a modern software team. They build an evaluation harness concept that acknowledges probabilistic outputs and separates model, orchestration, data and user-experience failures.

Course Highlights

Task-specific LLM evaluation

Golden and challenge datasets

RAG and grounding checks

Safety and injection testing

Human and automated judging

Regression, latency and cost monitoring

Modules & Curriculum

LLM application architecture and probabilistic behaviour

Intended use, users, harms and testable quality criteria

Evaluation datasets: representative, edge and adversarial cases

Response quality: correctness, relevance and instruction following

Retrieval-augmented generation: context quality and groundedness; safety, refusal behaviour and prompt-injection resistance

Tool use, function calls and workflow state

Automated evaluators, human review and judge reliability

Regression, latency, availability and cost measures; evaluation harness and release-evidence design

Learning Outcomes

  • Decompose an LLM application into testable system components.
  • Define task-specific quality criteria and evaluation datasets.
  • Evaluate groundedness, relevance and retrieval contribution.
  • Probe prompt injection, unsafe output and misuse scenarios.
  • Combine deterministic checks, model grading and human review.

Ready to Build Your Future-Ready Skills?

Build repeatable LLM evaluations that expose quality, safety and regression risks.

Enroll Interest

Related Courses

Beginner Junior QA learners mapping assurance activities across a software delivery lifecycle

Software Quality Assurance (SQA) Fundamentals

View Course
Beginner Software team tracing requirements through development testing release and maintenance stages

Software Development Life Cycle (SDLC)

View Course
Beginner QA trainees organising test planning execution defect tracking and closure artefacts

Software Testing Life Cycle (STLC)

View Course