tecnovy AI Suite

AI Evaluation & Observability

Swap in a new model or reword a prompt, and an AI application's answers can get worse without anyone noticing: no test fails, there is no baseline to compare against, and nobody watches production. In AI Evaluation & Observability you set quality criteria, build evaluation datasets and automated scoring, and set up tracing and alerts. Afterwards you show with measurements whether a change made quality better or worse.

Attendance certificate4.8/5 on ProvenExpert

What this module delivers.

Anyone who develops, tests or signs off LLM applications or agents builds quality criteria, evaluation datasets and production monitoring in tecnovy's AI Evaluation & Observability. Releases are then decided against criteria the team agreed on.

How the training runs

Method: Hands-on with an example application, from test data to monitoring
Basis: tecnovy curriculum for evaluating and observing AI systems
Outcome: tecnovy certificate of participation, no examination

Dates & booking

Choose a date that fits

No public dates are scheduled yet.

New dates are published regularly. Join the waitlist or plan an in-house course for your team.

Request in-house training

Fit

Who this module is designed for

Typical roles

You develop LLM applications or agents and want evidence of whether a change makes them better.
You test AI systems in quality assurance whose outputs differ from one run to the next.
You decide whether an AI application may go into production.

Prerequisites

No prerequisites

You can start right away. What you need: Python skills, experience with LLM applications.

Consider instead Advanced RAG For anyone whose quality problem is the retrieval and answers of a RAG system: you find the weak points of an existing system and improve it one technique at a time, measuring each.

Curriculum

AI Evaluation & Observability in Detail

Five parts on one continuous example application: you work out how AI evaluation differs from conventional testing, build datasets and assessments, evaluate RAG systems and agents, and finally set up monitoring for production.

01Why AI evaluation is different

This part explains why a fixed expected value rarely works for AI applications.

Non-deterministic outputs: the same input, different answers
Open tasks with no single right answer
Quality dimensions: correctness, relevance, safety, tone

02Evaluation datasets

This part builds the data every later change is measured against.

Deriving test cases from real usage data
Adding edge cases that are rare but matter
Including attack scenarios in the dataset

03Assessment methods

You learn to assess answers automatically and by people, and to combine the two.

Rule-based checks and metrics
LLM-as-a-judge: a language model scores the answers
Human assessment and comparing it with the model's verdicts

04Evaluating agents and RAG

Systems made of several steps need measuring at more than one point.

RAG: measuring retrieval quality and answer quality separately
Assessing agents by task success
Assessing agents by their individual steps

05Observability in production

Finally you follow the quality of an application that is already running.

Tracing: which steps a request passes through
Metrics, feedback and alerts
Turning production data into new test cases

Outcome

What you will be able to do afterwards

01
You define quality criteria for an AI application, such as correctness, relevance, safety and tone.
02
You derive evaluation datasets from real usage data and add edge cases and attack scenarios.
03
You check outputs automatically with rule-based checks and metrics.
04
You combine LLM-as-a-judge with human assessment and check where the model judges differently from people.
05
You measure retrieval quality and answer quality in RAG systems separately.
06
You assess agents by whether they complete the task and by their individual steps.
07
You set up tracing, metrics, feedback and alerts for production.
08
You turn production data into new test cases for your evaluation datasets.

tecnovy Teilnahmezertifikat

Confirms your attendance and the curriculum covered in the training. Issued by tecnovy, shareable on LinkedIn.

Open the Certificate Showroom tecnovy →

Offer scope

What is included

Included

Templates for evaluation plans and test datasetsYou receive the templates and can use them for your own AI applications after the training.

Why tecnovy

What you get on top with us

No Slideshow, Hands-On!

Promised: no PowerPoint marathon. We work in groups, tie theory to practice, and you get real project examples from our experienced trainers plus the exchange with like-minded people.

Certificate Showroom

Get your certificate of participation and, if you have one, add your exam certificate from E-Learning. Fully automated, beautifully designed. Just for you, only at tecnovy.

Flexible Date Change

If you are not able to attend the course, you can rebook your training free of charge up to one week before the start of the training.

Attend Twice, Pay Once

You are welcome to attend the training online again within a year as a refresher.

Learn from Experts

We always guarantee you the use of didactically and methodically first-class qualified trainers who draw their knowledge from training experience as well as professional practical and project experience.

FAQs

Frequently asked questions

01Why can't an AI application be tested like conventional software?

Because a language model can answer the same input differently, and open tasks often have no single right answer. In tecnovy's AI Evaluation & Observability training you therefore test against quality criteria such as correctness, relevance, safety and tone rather than one fixed expected value.

02How reliable is LLM-as-a-judge?

Only as reliable as your checks on it: in LLM-as-a-judge a language model scores answers, and Zheng et al. (2023) describe biases from answer order, answer length and a preference for the model's own answers. In tecnovy's AI Evaluation & Observability training you therefore compare the model's verdicts with human assessment.

03Is AI Evaluation & Observability suitable for testers?

Yes, the tecnovy training is also meant for people who test AI systems in quality assurance. You should bring Python skills and experience with LLM applications.

04How do you assess an AI agent?

On two levels: whether the agent completed its task overall, and how good its individual steps along the way were. You practise both in tecnovy's AI Evaluation & Observability training, along with measuring retrieval and answer quality separately in RAG systems.

05What does tracing mean for AI applications?

A trace records the steps a single request passes through, such as model calls, searches and tool calls. In tecnovy's AI Evaluation & Observability training you use traces together with metrics, feedback and alerts to follow quality in production. Shared names for such data are defined in OpenTelemetry's semantic conventions for generative AI, status Development (as of November 2026).

06How does AI Evaluation & Observability differ from Advanced RAG?

AI Evaluation & Observability covers evaluating LLM applications and agents in general, plus ongoing operation; RAG systems are one part of it. tecnovy's Advanced RAG starts from an existing RAG system and improves retrieval and document preparation one technique at a time, measuring each.

07What do the AI Evaluation & Observability exercises need technically?

The exercises in tecnovy's AI Evaluation & Observability training run on an example application that calls language models through an API, so they need API access with a key. Every automated LLM-as-a-judge assessment is one more model call.

08Does an evaluation find every error in an AI application?

No, an evaluation only shows how the application handles the cases in your dataset. In tecnovy's AI Evaluation & Observability training you therefore extend datasets with edge cases, monitor production and turn production data into new test cases.

09Is there an examination or a certification at the end of AI Evaluation & Observability?

No, AI Evaluation & Observability is a tecnovy training with no examination and no certification. If you attend at least 80% of it, you receive the tecnovy certificate of participation.

What does your training at tecnovy look like?

AI EvaluationAI Evaluation & Observability

No dates yet

We will let you know as soon as the next date is scheduled.