-
🛑 LLM Fundamentals for QA — Class Recordings
7 Lessons-
StartHow LLMs Actually Work: Tokens, Embeddings, Transformers and Inference in Plain English
-
StartContext Windows, Temperature, Top-p and System Prompts: The Knobs That Change Your Output
-
StartPrompt Engineering Patterns for QA: Test Cases, Test Data and Edge Case Generation
-
StartRAG for Testers: Grounding an LLM in Your Requirements, Specs and Bug History
-
StartHallucinations, Grounding and Guardrails: Making LLM Output Safe to Ship
-
StartEvaluating LLMs: Golden Datasets, LLM-as-a-Judge and Your Own Eval Harness
-
StartLLMs in Real QA Workflows: Bug Triage, Log Analysis, Flaky Test Diagnosis and Reporting
-
This course is not open for enrollment
LLM Fundamentals for QA
Understand how large language models actually work and apply them to real software testing work.
7
deep-dive class recordings
12+
prompt patterns you reuse
1
RAG pipeline you build end to end
0
maths or ML background needed
100%
QA-focused examples
How LLMs really workContext windows and tokensPrompt patterns for testersRAG on your own specsHallucination guardrailsEvaluating LLM output
Course Summary
A ground-up LLM course written for QA engineers. Learn tokens, embeddings, context windows, temperature and prompting patterns, then apply them to test case generation, bug triage, log analysis and test data creation. No maths degree required, and every concept is taught through testing examples you can run the same day.
Why this course exists
You are using LLMs every day without knowing why they fail
Most testers learned prompting by trial and error. When the model invents a field that does not exist, truncates your spec, or answers differently on Tuesday than it did on Monday, there is no mental model to fall back on. This course gives you that model, built entirely out of testing examples.
No maths, no training theory, no research papers. Just the concepts that change what you do at your desk: what a token costs you, why the middle of a long document disappears, what temperature actually changes, and how to make an answer traceable back to your own specification.
The model invents requirements
It predicts plausible text rather than reading your document. Without grounding, confident fabrication is the default behaviour, not a bug.
The same prompt gives different answers
Temperature and top-p are left at chat defaults and there is no system prompt, so nothing is repeatable enough to review.
Long specs get silently truncated
The context window filled up and the middle of the document was dropped. Nobody was told, and the answer looked fine.
Generated test cases are generic
Broad prompting produces broad output. Structured patterns that force boundary, negative and state-transition thinking produce something usable.
Outcomes
What you will be able to do by the last lesson
Every outcome is something you can demonstrate at your desk the same week.
1
Explain how an LLM produces an answer
Tokens, embeddings, attention and inference, in language you could use in a design review without hand-waving.
2
Predict and control cost
Count tokens, budget context, choose chunk sizes and know what a given workload will cost before you run it.
3
Set the decoding knobs deliberately
Temperature, top-p, penalties, seeds and system prompts — chosen for the task rather than left at defaults.
4
Apply prompt patterns for QA
Twelve reusable patterns for test cases, test data, edge cases, boundary analysis and negative paths.
5
Ground answers in your own documents
A working retrieval pipeline over your specs, requirements and bug history, with citations on every claim.
6
Reduce hallucination measurably
Grounding, constrained output, refusal behaviour and verification steps — with a before-and-after number.
7
Evaluate output objectively
Golden datasets, rubric scoring, model-as-judge and a small evaluation harness you own.
8
Wire LLMs into daily QA work
Bug triage, log analysis, flaky test diagnosis and report summarisation, as repeatable workflows.
The method
Seven sessions, one working LLM toolkit for your team
Concepts first, then control, then application, then measurement. Each lesson depends on the one before it.
1
Build the mental model
How the model actually turns your text into an answer. Everything later depends on this being clear.
2
Take control of the output
The parameters and system prompts that decide repeatability, length, tone and cost.
3
Learn the patterns
Structured prompt patterns aimed specifically at test design and test data.
4
Ground it in your world
Retrieval over your specs and bug history, so answers cite your documents instead of inventing them.
5
Make it safe
Hallucination control, constrained output formats, refusal behaviour and verification.
6
Measure it
Golden datasets and an evaluation harness so you can prove a change was an improvement.
7
Ship it
Real QA workflows: triage, log analysis, flake diagnosis and reporting.
Curriculum in detail
Every lesson, topic by topic
Seven class recordings. Here is exactly what each one covers and what you finish it with.
1
How LLMs Actually Work: Tokens, Embeddings, Transformers and Inference in Plain English
Lesson 1 · FoundationsThe mental model, taught without a single equation. By the end you can explain to a colleague why the model behaves the way it does — and predict several of its failure modes before they happen.
- ▸Tokens: what they are, how text is split, why it matters for cost
- ▸Tokenisation quirks: numbers, code, non-English text and whitespace
- ▸Embeddings and what semantic similarity really means
- ▸Attention and why position in the prompt affects the answer
- ▸Inference: next-token prediction and why fabrication is the default
- ▸Base models, instruction tuning and chat models
- ▸Model size, capability and cost trade-offs
- ▸Why the same model behaves differently through different interfaces
- ▸Failure modes you can predict from the architecture alone
You build: A token-counting and cost-estimation script you run against your own specs and prompts.
2
Context Windows, Temperature, Top-p and System Prompts: The Knobs That Change Your Output
Lesson 2 · ControlEvery parameter that changes the answer, explained and demonstrated side by side. This is the lesson that makes output repeatable enough to review.
- ▸Context window size and what happens when you exceed it
- ▸The lost-in-the-middle effect and how to lay out a long prompt
- ▸Temperature: what it changes and sensible values per task
- ▸Top-p, top-k and how they interact with temperature
- ▸Frequency and presence penalties
- ▸Seeds, determinism and why identical output is not guaranteed
- ▸System prompts: role, constraints, format and refusal rules
- ▸Message structure, few-shot examples and ordering effects
- ▸Max tokens, stop sequences and truncation handling
- ▸Measuring the effect of each knob on a fixed task
You build: A parameter comparison harness that runs one task across settings so you can see the effect rather than guess at it.
3
Prompt Engineering Patterns for QA: Test Cases, Test Data and Edge Case Generation
Lesson 3 · ApplicationTwelve reusable patterns aimed squarely at testing work. Each one is demonstrated on a real requirement and compared against naive prompting.
- ▸Role, task, context and format as a prompt skeleton
- ▸Boundary value analysis as a structured prompt pattern
- ▸Equivalence partitioning and decision table prompting
- ▸Negative and error path generation
- ▸State transition and workflow test prompting
- ▸Test data generation with realistic constraints
- ▸Synthetic data with referential integrity across tables
- ▸Chain of thought and when it genuinely helps
- ▸Self-critique and revision loops
- ▸Output formats: tables, Gherkin, JSON and CSV
- ▸Reusable prompt templates and a team prompt library
- ▸Comparing pattern output against naive prompting
You build: A personal prompt library of twelve QA patterns, each with a worked example and the output it produces.
4
RAG for Testers: Grounding an LLM in Your Requirements, Specs and Bug History
Lesson 4 · GroundingThe single biggest reliability upgrade available to you. We build a retrieval pipeline over real documents so the model answers from your material rather than its training data.
- ▸Why retrieval beats a longer prompt
- ▸Document ingestion: specs, requirements, tickets and bug history
- ▸Chunking strategy: size, overlap and respecting structure
- ▸Embeddings and vector storage
- ▸Semantic search versus keyword search, and hybrid retrieval
- ▸Re-ranking retrieved chunks
- ▸Assembling context within a token budget
- ▸Citations: making every claim traceable to a source line
- ▸Handling documents that contradict each other
- ▸Keeping the index fresh as documents change
- ▸Measuring retrieval quality separately from answer quality
You build: A working RAG pipeline over your own specs and bug history that answers questions with cited source lines.
5
Hallucinations, Grounding and Guardrails: Making LLM Output Safe to Ship
Lesson 5 · ReliabilityWhy fabrication happens, and the layered controls that reduce it. We measure the improvement rather than assuming it.
- ▸Why hallucination is inherent to next-token prediction
- ▸Grounding as the primary control
- ▸Constrained output: JSON schemas and structured generation
- ▸Teaching the model to say it does not know
- ▸Confidence signals and abstention thresholds
- ▸Verification passes and cross-checking claims
- ▸Input validation and prompt injection from untrusted content
- ▸Handling sensitive data and what never goes into a prompt
- ▸Human review gates and where to place them
- ▸Measuring hallucination rate before and after each control
You build: A guardrail layer with schema-constrained output and a measured reduction in fabricated claims on a fixed test set.
6
Evaluating LLMs: Golden Datasets, LLM-as-a-Judge and Your Own Eval Harness
Lesson 6 · MeasurementTesters are unusually well suited to this lesson. Building an evaluation harness is, after all, just test design applied to a non-deterministic system.
- ▸Building a golden dataset from real examples
- ▸Choosing evaluation criteria that matter for your task
- ▸Deterministic checks: schema, format, required content
- ▸Rubric-based scoring and inter-rater consistency
- ▸LLM-as-a-judge: how to do it and where it misleads
- ▸Handling non-determinism with multiple trials
- ▸Regression testing prompts across model versions
- ▸Cost and latency as first-class evaluation metrics
- ▸Comparing models honestly on your own task
- ▸Reporting evaluation results to stakeholders
You build: An evaluation harness with a golden dataset that scores any prompt or model change against a fixed benchmark.
7
LLMs in Real QA Workflows: Bug Triage, Log Analysis, Flaky Test Diagnosis and Reporting
Lesson 7 · IntegrationEverything assembled into four workflows you can run on Monday, each with the grounding, guardrails and evaluation from earlier lessons applied.
- ▸Bug triage: duplicate detection, severity suggestion and routing
- ▸Log analysis: summarising, clustering and finding the anomaly
- ▸Flaky test diagnosis from failure history and traces
- ▸Release and test report summarisation for non-technical readers
- ▸Requirement review: finding ambiguity and missing acceptance criteria
- ▸Choosing the right model per workflow to control cost
- ▸Automating a workflow end to end with review gates
- ▸Measuring time saved honestly
- ▸A thirty-day plan to introduce one workflow at a time
You build: Four working QA workflows, each grounded, guarded, evaluated and ready to introduce to your team.
Tooling
What you use along the way
Deliberately lightweight. You can complete this course with a laptop and one API key.
| Tool or concept | How it is used in this course |
|---|---|
| An LLM API | The main working surface for every lesson. Provider-agnostic; a local model is shown as a cheaper alternative. |
| Tokeniser library | Counting tokens, estimating cost and diagnosing truncation. |
| Embeddings model | Semantic search over your specs, requirements and bug history. |
| Vector store | Storing and retrieving document chunks for the RAG pipeline. |
| Structured output / JSON schema | Constraining generation so downstream code can rely on the shape. |
| Golden dataset | A fixed set of inputs and expected outcomes for evaluation. |
| Evaluation harness | Scoring prompt and model changes against that dataset. |
| Python or Node | Either is fine; examples are kept simple and language-light. |
| Your own documents | Specs, tickets and bug history — the material that makes the RAG lesson real. |
Before you start
What you need, and what you honestly do not
This is the most beginner-friendly course in the series on the AI side, and it stays firmly in QA territory throughout.
What you need before lesson one
- ✓Working knowledge of software testing
- ✓Basic scripting ability in Python or JavaScript
- ✓An API key for one LLM provider
- ✓Comfort reading JSON
- ✓Some of your own specs or tickets to practise on
Helpful, but not required
- +Previous experience prompting a chat assistant seriously
- +Familiarity with vector databases
- +Some exposure to API-based automation
- +An existing bug backlog you would like to triage faster
There is no maths and no machine learning background required. Concepts like embeddings and attention are taught through testing examples rather than equations, and lesson one is explicitly designed for people who have never read an AI paper.
Fit check
Is this the right course for you?
A short, honest answer about who benefits.
This is for you if
- ✓You use AI assistants for testing work but do not know why they fail
- ✓You want to ground AI answers in your own specs and bug history
- ✓You need to evaluate LLM output objectively rather than by vibe
- ✓You are being asked to bring AI into your QA process
- ✓You want the fundamentals before taking an agents or MCP course
- ✓You want to reduce hallucination and prove the reduction with a number
This is probably not for you if
- ✗You want to train or fine-tune models — this is applied usage only
- ✗You are looking for machine learning theory or research depth
- ✗You want a Playwright automation course — start with the CLI masterclass
- ✗You are already building production RAG systems and evaluation pipelines
What you get
Everything you keep after the last lesson
Small, practical assets that fit into your existing work.
Seven class recordings
Full sessions with every concept demonstrated on real testing material. Lifetime access.
A QA prompt library
Twelve reusable patterns for test cases, test data, edge cases and negative paths.
A working RAG pipeline
Ingestion, chunking, embedding, retrieval, re-ranking and citation over your own documents.
A guardrail configuration
Schema-constrained output, refusal behaviour and verification steps.
An evaluation harness
Golden dataset, rubric scoring and regression detection across prompt and model changes.
Four QA workflows
Bug triage, log analysis, flaky test diagnosis and report summarisation, ready to adapt.
The difference
What actually changes in your week
Before and after, for a tester who works through the whole course.
| Today | After this course |
|---|---|
| The model invents requirements and you spot it late | Answers cite the exact source line in your own specification |
| The same prompt gives a different answer each run | Parameters and system prompt chosen deliberately, output repeatable enough to review |
| Long specs get silently truncated | Token budgets measured, documents chunked, nothing dropped without you knowing |
| Generated test cases are generic and shallow | Structured patterns forcing boundary, negative and state-transition coverage |
| You judge output by whether it feels right | A golden dataset and rubric score for every prompt change |
| Bug triage is entirely manual | Duplicate detection and severity suggestion running as a reviewed workflow |
| AI cost is a mystery | Token counts and cost per task estimated before you run anything |
Your instructor
Pramod Dutta
Pramod Dutta is the founder of The Testing Academy and has spent over a decade in software testing and automation. This course exists because of a pattern he kept seeing: testers using LLMs daily with no mental model of what was happening underneath.
The teaching approach here is deliberately concrete. Every abstract idea — tokens, embeddings, attention, retrieval — is introduced through a testing task, and the recordings show the failures as well as the successes.
No AI background is assumed at any point. If you can read a specification and write a simple script, lesson one starts exactly where you are.
10+ yrs
in testing and automation
1M+
learners reached through The Testing Academy
0
maths prerequisites
FAQ
Everything people ask before enrolling
If your question is not answered here, ask support before enrolling.
Do I need a machine learning background?
No, and that is the point of the course. Every concept is taught through testing examples. There are no equations and no training theory.
Which LLM provider should I use?
Any mainstream one. The course is provider-agnostic and also shows running a smaller local model for the cheaper tasks.
Roughly what will this cost me in API usage?
Small. The exercises are deliberately compact, token counting is taught in lesson one, and cost estimation is part of every workflow.
Is this Python or JavaScript?
Examples are kept simple enough to follow in either. The concepts, prompts and evaluation designs are language-independent.
Do I need my own documents for the RAG lesson?
It helps a great deal, and sample documents are provided if you cannot use internal material. Using your own specs makes the lesson far more valuable.
How does this relate to the AI Agents and MCP courses?
This is the foundation. Agents and MCP both assume you understand context, cost, grounding and evaluation. Take this first if any of those terms are fuzzy.
Will this teach me to fine-tune a model?
No. It is applied usage: prompting, grounding, guardrails and evaluation. Fine-tuning is discussed only to explain when it is and is not the right answer.
Is the material likely to go out of date?
Model versions change constantly, but tokens, context, retrieval, grounding and evaluation are stable fundamentals. The course teaches those rather than one vendor interface.
Is access limited in time?
No. Lifetime access to all recordings and any updates.
Ready when you are
Stop guessing why the model got it wrong.
Seven sessions from tokens to a grounded, evaluated, working LLM toolkit for QA — with no maths and no machine learning background required.
- ✓Seven full class recordings with lifetime access
- ✓A twelve-pattern QA prompt library
- ✓A working RAG pipeline over your own documents
- ✓A guardrail layer with measured hallucination reduction
- ✓An evaluation harness and golden dataset
- ✓Four ready-to-adapt QA workflows
Course Curriculum
Pramod Dutta
Founder of The Testing Academy, a YouTube Channel with 95K subscribers where Pramod teaches about Software testing & Test Automation. With overall 10+ years of experience in Software Testing & Test Automation, he has mentored 10,000+ students in Software Testing, API Testing, Test Automation.
Pramod Dutta is working as SDET Manager at Tekion | Ex BrowserStack Employee & BrowserStack Champion & Certified Scrum Master. Pramod has a vast range of experience handling from Manual Testing, Mobile & Web Automation, Desktop & cloud services like AWS, GCP extra.
By joining this Automation Testing Course , you’ll have the opportunity to take control of your life, work in an exciting industry with infinite possibilities and live the life you want.
