Workshops: Prompt Engineering
Measuring Prompt Quality: Evaluation & Testing Workshop
This workshop shows how to measure the quality of prompts and AI assistants objectively instead of relying on a few examples and gut feeling. You will build test sets, define grading criteria and set up a process that checks every prompt or model change in a traceable way.
Who is this workshop for?
- Teams providing Custom GPTs, Claude Projects or Copilot agents to many users
- Developers of LLM applications
- Product owners for AI features
- Quality management and subject experts signing off AI output
- AI champions who need to assess model changes
What you will take away
- You define what a good result concretely means for your use case.
- You compile a representative test set with typical, difficult and edge cases.
- You develop rubrics for human and automated grading.
- You apply LLM-as-a-judge and understand its weaknesses.
- You compare prompt variants and models systematically and document decisions.
- You embed evals as regression tests in your prompt maintenance process.
Agenda
Module 1
Why gut feeling is not enough
- Understanding variability in LLM output
- Typical fallacies of testing with three examples
Module 2
Defining quality
- Criteria: correctness, completeness, format, tone, safety
- Rubrics and reference answers
- Exercise: criteria for one of your own use cases
Module 3
Building test sets
- Collecting and anonymising real requests
- Adding edge cases, noisy input and misuse attempts
Module 4
Grading: human and machine
- Organising human review efficiently
- LLM-as-a-judge: setup, calibration, bias
- Automated checks for format and required content
Module 5
Evals in daily work
- A/B comparison of prompt variants and models
- Regression tests before every change
- Tools from spreadsheets to eval platforms
No measurement, no improvement
A prompt that shines on three test questions may fail on the fortieth. A new model may be better in one area and worse in another. Anyone rolling out AI assistants widely needs a method to make quality measurable and assess changes safely.
Evals sound like a developer topic, but they matter just as much to business teams: the best grading criteria come from where the domain knowledge sits.
Two tracks: business teams and developers
We adapt the workshop to your participants. Business teams work with spreadsheets and manual grading in ChatGPT, Claude, Microsoft 365 Copilot, Gemini or Mistral Le Chat. Developer teams automate evals via API and integrate them into CI pipelines, supported by coding tools such as Claude Code or Cursor.
Quality as part of governance
Documented tests also help with compliance: they show that an AI deployment is tested and monitored. For high-risk systems, the EU AI Act sets requirements on accuracy, robustness and monitoring. Evals are a practical building block but do not replace a legal assessment.
Frequently asked questions
How many test cases do we need?
To start, 30 to 50 well-chosen cases are often enough if they cover typical and difficult requests. Representativeness matters more than volume.
Is LLM-as-a-judge reliable?
With clear criteria and calibration against human ratings it can work well. It has its own biases, which we expose and test for in the workshop.
Do we need special software?
No. Many teams start with a spreadsheet and a fixed grading routine. Dedicated eval tools pay off as volume and change frequency grow.
Workshop
Get in touch
Shall we build a test set for your most important AI assistant together?
Tell us a little about your team – we will get back to you within 24 hours with a proposal for content, format and dates.
Prefer to talk?
More workshops
- Workshops: Prompt EngineeringUnderstanding and Defending Against Prompt Injection
- Workshops: Prompt EngineeringPrompt Engineering Basics Training for ChatGPT, Claude & Copilot
- Workshops: Prompt EngineeringAdvanced Prompt Engineering Workshop: Beyond the Basics
- Deploy, Convert & ScaleMigrating Off an AI App Builder (Base44, Lovable) to Your Stack
