45 free lesson minutes · Start now, no credit card needed
Shiur Prati

Private Evaluating & Testing LLM Apps lessons with AI

Evaluating & Testing LLM Apps with a live private lesson: the teacher talks with you out loud and explains on a shared board, moving at your pace through a clear plan, lesson after lesson.

Live 1:1 lesson with an AI teacher

Teacher: Noa

  • Course updated 11/2026
  • Hebrew, and 5 more languages
  • 3 levels
  • 35 lessons · about 19 hours of lessons

Evaluating & Testing LLM Apps

Evaluating & Testing LLM Apps with a live private lesson: the teacher talks with you out loud and explains on a shared board, moving at your pace through a clear plan, lesson after lesson.

12 lessons · 4 modules · Live private lesson with the teacher on the board

What you'll learn

  • Mental Model: What Evaluating an LLM Means
  • Building an Eval Set
  • Metrics: Correctness and Faithfulness
  • From Results to Decisions
  • LLM-as-Judge
  • Component Evals: RAG and Agents
  • Prompt Regression Testing
  • Red-Team Testing and Capstone
  • Production Monitoring
  • Model Comparison and Cost
  • Bias and Safety
  • Governance and Capstone

What's included

  • Live 1:1 lessons with an AI tutor on a shared board
  • 19 hours of live one-on-one lessons
  • 35 lessons on a shared board, each with a clear goal
  • Practice and a understanding check in every lesson
  • A summary after every lesson, saved in the history
  • A completion certificate for each level
  • Computer, tablet or phone
  • Hebrew, plus 5 more languages

Course plan

3 levels · 12 modules · 35 lessons · 19 hr 5 min total

BeginnersBeginner: Why and How to Evaluate an LLM App4 modules • 12 lessons • 6 hr 20 min
  1. Mental Model: What Evaluating an LLM Means3 lessons • 1 hr 30 min
    1. 1Why Testing an LLM Is HardTrial lessonabout 30 min
    2. 2Anatomy of an LLM App and Where It Failsabout 30 min
    3. 3The Eval Pipeline: From Test Case to Reportabout 30 min
  2. Building an Eval Set4 lessons • 2 hr 5 min
    1. 4Success Criteria and Pass Criteriaabout 30 min
    2. 5Writing Test Cases and Coverageabout 35 min
    3. 6Sourcing Cases: Logs, Synthetic Data, Error Analysisabout 30 min
    4. 7Human Labeling and Rubricsabout 30 min
  3. Metrics: Correctness and Faithfulness3 lessons • 1 hr 35 min
    1. 8Correctness for Closed Answers and Formatabout 30 min
    2. 9Metrics for Open-Ended Textabout 30 min
    3. 10Faithfulness, Groundedness and Hallucinationabout 35 min
  4. From Results to Decisions2 lessons • 1 hr 10 min
    1. 11Reading Results: Slices, Variance, Failure Analysisabout 30 min
    2. 12Capstone: A Small Eval for a Real Featureabout 40 min
IntermediateIntermediate: LLM-as-Judge, Regression and Red-Team Testing4 modules • 12 lessons • 6 hr 40 min

Mapped to exam: ללא מבחן רשמי; המבנה נשען על OWASP Top 10 for LLM Applications ועל NIST AI 600-1 (Generative AI Profile)

  1. LLM-as-Judge4 lessons • 2 hr 10 min
    1. 1When and How an LLM Judges an LLMabout 30 min
    2. 2Writing a Judge Promptabout 35 min
    3. 3Pairwise Comparison and Judge Biasesabout 30 min
    4. 4Validating the Judge Against Human Labelsabout 35 min
  2. Component Evals: RAG and Agents2 lessons • 1 hr 10 min
    1. 5Evaluating RAG: Retrieval vs Generationabout 35 min
    2. 6Evaluating Multi-Turn and Tool-Calling Agentsabout 35 min
  3. Prompt Regression Testing3 lessons • 1 hr 35 min
    1. 7The Regression Gate: Baseline, Versions, Comparisonabout 30 min
    2. 8Is the Improvement Real? Noise and Statisticsabout 35 min
    3. 9Evals in CI: Tiers, Thresholds and Budgetabout 30 min
  4. Red-Team Testing and Capstone3 lessons • 1 hr 45 min
    1. 10Red-Team Mindset: Risk Map of an LLM Appabout 30 min
    2. 11Building a Red-Team Set and Making It Regressionabout 35 min
    3. 12Capstone: Release Plan for a Prompt Changeabout 40 min
AdvancedLLM Evaluation in Production: Monitoring, Model Comparison, Bias and Safety4 modules • 11 lessons • 6 hr 5 min
  1. Production Monitoring4 lessons • 2 hr 5 min
    1. 1From Offline Eval to Production: Tracing and Loggingabout 30 min
    2. 2Online Quality Metrics: Feedback, Sampling and Live Judgesabout 30 min
    3. 3Drift, Alerts and Incident Responseabout 35 min
    4. 4Closing the Loop: Production to Eval Set and A/B Testsabout 30 min
  2. Model Comparison and Cost3 lessons • 1 hr 40 min
    1. 5Comparing Models: Public Benchmarks vs Your Own Evalabout 30 min
    2. 6Cost and Latency as Quality Dimensionsabout 35 min
    3. 7Routing, Caching and Safe Model Migrationabout 35 min
  3. Bias and Safety2 lessons • 1 hr 10 min
    1. 8Bias Testing: Counterfactuals and Disparity Metricsabout 35 min
    2. 9Safety Evaluation: Over-refusal, Under-refusal and Guardrailsabout 35 min
  4. Governance and Capstone2 lessons • 1 hr 10 min
    1. 10Eval Reports and Governance: NIST, EU AI Act, Israelabout 30 min
    2. 11Capstone: Production Evaluation Strategyabout 40 min

What you need

  • No prior knowledge needed for beginner level: the teacher checks what you already know and starts from there
  • For higher levels: what you learned at the level before (you can skip it if you already know it)
  • A computer, tablet or phone with a microphone, and a browser

Who it's for

  • Adults: for work, study, an exam, or just curiosity

Exams and certifications

  • Intermediate: ללא מבחן רשמי; המבנה נשען על OWASP Top 10 for LLM Applications ועל NIST AI 600-1 (Generative AI Profile)
  • AI & Machine Learning

    with Noam

    Live 1:1 lesson with an AI teacherabout 19 hours35 lessons3 levels
  • Using AI Wisely

    with Noam

    Live 1:1 lesson with an AI teacherabout 12 hours24 lessons2 levels
  • Machine Learning

    with Noam

    Live 1:1 lesson with an AI teacherabout 20 hours36 lessons3 levels
  • Deep learning

    with Noa

    Live 1:1 lesson with an AI teacherabout 18 hours34 lessons3 levels
  • Computer Vision

    with Noam

    Live 1:1 lesson with an AI teacherabout 20 hours36 lessons3 levels
  • Generative AI & Prompt Engineering

    with Noa

    Live 1:1 lesson with an AI teacherabout 19 hours36 lessons3 levels
See all courses in Artificial Intelligence

Your teacher

NoaNoaAI private tutor, in a live lesson on a shared board

A private AI teacher who teaches in a live conversation: asks, listens, draws on the board, and lets you solve it yourself with hints. Remembers what you've learned and starts each lesson where you left off.

What's a lesson like?

  • Your child talks with the teacher out loud, like a video call, and asks about anything unclear.

  • The teacher explains, writes and draws on a shared board, and your child solves along on the same board.

  • Snap a photo of a worksheet or homework and work through it, one problem at a time.

  • Parents get a report: study time, what went well and what needs more practice.

Lesson rules

  • Just Learning
  • Security and Defense Only

FAQ

How does a private Evaluating & Testing LLM Apps lesson work?
Pick Evaluating & Testing LLM Apps and a level, then talk with the teacher out loud. The teacher asks what you already know, explains, writes and draws on the board, and lets you solve on your own with hints until it's clear.
How long does the Evaluating & Testing LLM Apps course take?
12 lessons of about 32 minutes at the first level, at your own pace. You can pause, go back, or skip what you already know.
Do I need prior knowledge?
Not for beginner level. At the start of the first lesson the teacher checks what you already know and adjusts the level, and you can jump straight to an advanced level.
How do I start for free?
Sign in with Google or a code sent to your email, enter your child's name and grade, and the lesson starts right away. Every family gets 45 free lesson minutes in any subject, counted by actual lesson time, with no credit card. Then choose a paid plan and cancel anytime.
What device do I need?
Any computer, tablet or phone with a browser and a microphone. You can also install the site as an app on your home screen.

Want to try it?

Sign in, enter your child's name and grade, and the lesson begins. 45 free lesson minutes, no credit card.

Start for free

Shiur Prati with AI: what it is and how it works

Evaluating & Testing LLM AppsThe first 45 minutes are free

Private lessonStart free