-
🛑 Class Recording Will be Here
7 Lessons-
StartAI Automation Agents Building with MCP
-
StartPlaywright MCP Server
-
StartPlaywright MCP CLI Full Class
-
StartSelenium to Playwright MCP Migration 🚀 Complete Step-by-Step Guide
-
StartSelenium to Playwright MCP Migration 🚀 Complete Step-by-Step Guide Part 2
-
StartDeepEval for QA: Introduction, Install, Objectives & Metric Architecture
-
PreviewDeepEval Project: Grade a Chatbot, RAG Pipeline & Live Bot Step by Step
-
E2E Testing ChatBot, AI Agent, RAG, MCP Server with DeepEval Framework
Test and evaluate LLM apps, chatbots, AI agents, RAG pipelines, and MCP servers with DeepEval.
3
hands-on projects
29
metric rows you will run
4
systems: chatbot · RAG · agent · MCP
25
pytest files
2
surfaces: CI gate + dashboard
✓ DeepEval 3.9.9✓ LLM-as-judge✓ pytest + CI✓ FastAPI dashboard✓ OpenAI · Groq · Ollama✓ Full GitHub project
Course Summary
A complete guide to testing AI systems with the DeepEval framework. Build evaluation suites for chatbots, AI agents, RAG pipelines, and MCP servers: measure hallucination, answer relevancy, faithfulness, and tool-use correctness, then wire evals into CI. Hands-on projects for QA engineers and AI developers who need reliable, measurable AI quality.
the problem
Your test suite cannot see an AI bug
Ship a chatbot and the first bug report is never a stack trace. It is "the bot told a customer refunds take seven days." Status code 200. Schema valid. Response time fine. Every existing test green. The defect lives entirely inside the meaning of a sentence — and assertEquals cannot reach it.
| What you used to assert | Why it breaks on an LLM app | What replaces it |
|---|---|---|
assertEquals(expected, actual) | The same prompt returns different wording every call | Answer Relevancy — on topic, however phrased |
| 200 OK + schema valid | A perfectly-shaped JSON can contain an invented policy | Faithfulness / Hallucination |
| Regex / substring match | A paraphrase kills the match — the answer was still right | G-Eval Correctness — your rubric, graded by an LLM |
| Nobody tested tone or privacy | Bias, toxicity and PII leaks ship silently | Bias · Toxicity · PII Leakage |
| Manual spot-checks in a spreadsheet | Not repeatable, cannot gate a build | A pytest suite with thresholds in CI |
the outcome
What you will actually be able to do
Put a number on quality
Turn "the bot feels worse this week" into a score, a threshold and a red build.
Grade a chatbot
10 metrics on a real React + FastAPI + Groq support bot: relevancy, faithfulness, hallucination, bias, toxicity, PII, multi-turn memory.
Grade a RAG pipeline
11 metrics including the three retrieval metrics that tell you whether the retriever or the generator caused the wrong answer.
Grade an AI agent
Task success, tool choice and argument correctness as G-Eval rubrics on top of the standard quality rows.
Grade an MCP server
Wrap a tool call so the result becomes the output under test — then every faithfulness and correctness metric applies unchanged.
Grade a bot you do not control
The realistic enterprise case: a live vendor bot, plain-text responses, no visibility into its prompt.
Run it in CI
Marker-sliced fast gates on every pull request, a full 29-row sweep nightly, HTML reports as build artefacts.
Ship a dashboard
A FastAPI app on
:8203 — click a metric, get a live score, pass/fail and the judge's written reason.how it works
The one diagram behind the whole course
end to end
The complete curriculum
the full coverage map
The 29 metrics you will run
⬆⬇ The thing most people get wrong
Quality metrics are a floor — the score must climb above the threshold. Safety metrics are a ceiling — hallucination, bias, toxicity and PII leakage must stay below it. Get that backwards and your report says the opposite of the truth. We cover the exact direction for all 29 rows.
audience
Who this is for
Manual and automation QAs
Your team just shipped an AI feature and asked you to sign it off. This is the missing skill.
SDETs and automation leads
You already own the CI pipeline. Now you own the eval gate that sits inside it.
QA leads and managers
You need a defensible answer to "is it good enough to ship?" — a table of numbers, not a vibe.
Developers building AI features
Stop shipping prompt changes blind. Score them before the customer does.
Anyone testing chatbots, RAG, agents or MCP
If it answers over HTTP, everything in this course applies to it.
| You should already have | Depth needed |
|---|---|
| Python | Comfortable reading and writing functions and classes — no advanced work required |
pytest | Basics: test functions, fixtures, parametrize, markers. We revise markers in class. |
| HTTP and JSON | You can read a curl command and a JSON response |
| LLM experience | None assumed. Prompts, embeddings, RAG and judging are all explained from zero. |
| An API key | Helpful, not required — local Ollama and mock mode let you follow along free |
deliverables
What you take home
- A working evaluation harness with 29 metric rows across three real systems under test
- Three target clients that absorb three different HTTP contracts, including plain text
- 41 goldens you authored and grounded yourself, plus safety and PII probe sets
- 25
pytestfiles, marker-sliced so you can run exactly what you need - A FastAPI dashboard with live scores, judge reasons and local run history
- A GitHub Actions workflow you can paste into your own repository tomorrow
- The reusable skill playbook that rebuilds this architecture for your product
- Two written lessons, permanent access, exhaustively diagrammed
what class looks like
A taste of the code
this is a complete, real LLM test
import pytest from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import FaithfulnessMetric from datasets.rag_goldens import RAG_GOLDENS @pytest.mark.rag @pytest.mark.quality @pytest.mark.needs_rag @pytest.mark.parametrize("golden", RAG_GOLDENS, ids=lambda g: g["input"][:40]) def test_rag_faithfulness(golden, rag, judge): res = rag.chat(golden["input"], top_k=4) case = LLMTestCase( input=golden["input"], actual_output=res.reply, retrieval_context=res.retrieval_context, # the chunks, verbatim ) # fails if ANY claim in the answer is not supported by the chunks assert_test(case, [FaithfulnessMetric(threshold=0.7, model=judge)])and this is how you run it
pytest tests/rag/ -v # the whole target pytest -m "rag and retrieval" -v # retriever bugs only pytest -m safety -v # bias · toxicity · PII pytest tests/ -n 4 --html=report.html # the nightly sweep uvicorn dashboard.app:app --port 8203 # the live dashboard
FAQ
Common questions
Do I need an OpenAI API key?
No. Set
JUDGE_PROVIDER=ollama and the judge runs locally with no key and no bill. The chatbot and RAG apps also fall back to mock mode without a key, so the framework always has a target to hit. A key just makes the scores steadier and the runs faster.Is this only about DeepEval, or transferable?
The metrics come from DeepEval, but the architecture — registry of metric-by-target rows, target clients that absorb HTTP contracts, versioned goldens, thresholds as the gate — is framework-agnostic. Swap the metric library and the design survives.
I have never worked with LLMs. Will I keep up?
Yes. Prompts, embeddings, chunking, retrieval and LLM-as-judge are all built from zero in Lesson 1. The Python and
pytest is the part we assume, and that is the part you already have.Does this cover AI agents and MCP servers, or just chatbots?
Both are covered explicitly. Agents get task-success and tool-choice rubrics on top of the standard quality rows. MCP servers slot in by treating a tool call as a chat turn — the tool result becomes the output under test and the tool's data becomes the ground-truth context.
Will the judge scores be stable enough to gate a build?
Stable enough if you use them correctly: a small low-temperature judge, thresholds set with headroom, and trends read across runs rather than single numbers. We cover exactly where LLM judging is reliable and where it is not.
How much will the judge calls cost me?
Almost nothing if you follow the cost discipline in the course: develop against two or three goldens, use
gpt-4o-mini, run the full 29-row sweep nightly rather than per commit, or run entirely on local Ollama for free.Do I get the full source project?
Yes — the complete
Chapter_19_DeepEval_Framework project on GitHub: all three subsystems, the registry, the goldens, the dashboard, all 25 test files and the reusable skill playbook.What if I get stuck?
Every lesson ships with a troubleshooting table built from the errors students actually hit — the broken CLI version, the empty-context crash, the 429s, the mock-mode confusion and the inverted thresholds.
Course Curriculum
Pramod Dutta
Founder of The Testing Academy, a YouTube Channel with 95K subscribers where Pramod teaches about Software testing & Test Automation. With overall 10+ years of experience in Software Testing & Test Automation, he has mentored 10,000+ students in Software Testing, API Testing, Test Automation.
Pramod Dutta is working as SDET Manager at Tekion | Ex BrowserStack Employee & BrowserStack Champion & Certified Scrum Master. Pramod has a vast range of experience handling from Manual Testing, Mobile & Web Automation, Desktop & cloud services like AWS, GCP extra.
By joining this Automation Testing Course , you’ll have the opportunity to take control of your life, work in an exciting industry with infinite possibilities and live the life you want.
