◆ LearnAILevel 1.5 · Agents ← Roadmap

Station 9 · Testing & Observability

Testing & watching your agent

Building an agent is one thing. Knowing whether it actually works — and seeing what it's doing when it doesn't — is a whole separate skill. This is the part that separates a demo from something you'd trust.

9 of 10 · ~7 min

What you'll walk away with

Concept 1

Why agents are hard to test

A normal program is deterministic: same input, same output, every time. You write a test that says "given 2 + 2, expect 4," and it either passes or fails forever. Agents break both halves of that promise.

🛩️ Analogy: a black box vs. a glass box

A failing agent you can't see inside is a black box — you know it crashed, but not why. Testing and observability turn it into a glass box. Think of a plane's flight recorder: nobody investigates a crash by re-flying the plane and hoping it happens again. They read the recorder — the exact sequence of what the pilot saw and did. Your agent's trace is that recorder.

Concept 2

How to evaluate an agent

Because one test isn't enough, you layer several. Each catches a different kind of problem — from "a single tool is broken" all the way up to "the answer is technically correct but unhelpful."

Numbers

📈 Track metrics

Did it reach the goal? How many steps did it take? How much did it cost in tokens? Slower and pricier over time is a real regression.

One piece

🔧 Unit-test the tools

Test each tool alone. Does search_web return good results? Does the calculator return the right number? Broken tools mean a broken agent.

Whole flow

🔗 Integration-test the loop

Does the agent chain steps correctly end-to-end — search, then read, then answer — without dropping the thread?

Judgment

🧑‍⚖️ Human-in-the-loop

A person rates the outputs. Some things ("is this answer actually helpful?") only a human can score reliably.

You don't grade this by hand every time. Eval frameworks automate it — you give them a set of test cases and they score runs for you. A few worth knowing by name: LangSmith, Ragas, and DeepEval.

Concept 3

Debugging & monitoring: observability

Observability is a fancy word for "can you see what your agent actually did?" The number-one habit that makes it possible is structured logging & tracing: record every Thought → Action → Observation so you can replay the exact path the agent took.

🔑 The one habit that matters most

Log every step. If you save each Thought, each tool call, and each result the agent got back, then debugging becomes reading instead of guessing. You can see exactly which step went wrong — the bad tool result, the wrong turn — instead of re-running and hoping to catch it.

You could write logs by hand, but observability tools capture and visualize traces for you — every step, its tokens, and its cost, laid out as a timeline. Two common ones: Helicone and LangFuse.

agent_trace — one logged loop
# Goal: "How many gold medals did the host country win at the last Summer Olympics?"

[step 1]  Thought  → "I don't know the latest host or medal count. Search."
          Action   → search_web("most recent Summer Olympics host country")
          Observe  → "Paris, France — 2024"          · 214 tokens · 0.4s

[step 2]  Thought  → "Now I need France's gold-medal total."
          Action   → search_web("France gold medals 2024 Olympics")
          Observe  → "France won 16 gold medals"      · 198 tokens · 0.5s

[step 3]  Thought  → "I have host + count. Answer."
          Action   → finish()                       · 22 tokens

# run summary → steps: 3 · tokens: 434 · cost: $0.006 · goal_reached: true
why this is goldIf the final answer had been wrong, you wouldn't have to guess. You'd scan the trace and spot it instantly — maybe step 1 returned the wrong host, and every step after inherited the mistake. The trace shows you the exact broken link.

Concept 4

The mindset: read the trace first

When an agent misbehaves, the instinct is to immediately start rewriting the prompt. Resist it. The trace already contains the answer to what went wrong — you just have to look.

Don't

Guess and re-run

Tweak the prompt, run again, cross your fingers. With a non-deterministic system you might just get lucky once and never learn the real cause.

Do

Read the trace first

Open the log. Find the first step where reality diverged from what you expected. That step — not the final answer — is your bug.

Checkpoint

Your agent gave a confidently wrong final answer. What's the best first move?

Rewrite the system prompt and run it again to see if it's fixed. Open the trace and find the first step where the agent went off track. Assume the LLM is broken and switch to a different model.

Because agents are non-deterministic and multi-step, re-running can accidentally "pass" without fixing anything. The trace is your flight recorder — reading it shows you the exact Thought or Observation where things went wrong, so you fix the real cause instead of guessing.

See it yourself — 5 minutes

Turn a black box into a glass box

Open any AI tool that shows its steps (ChatGPT with browsing, or Claude/Gemini running a multi-step task) and give it a research question it can't answer from memory:

  1. Ask "How many gold medals did the host country win at the most recent Summer Olympics?" — a two-hop question.
  2. Watch it expand its steps. Write down each Thought → Action → Observation you can see. That's you building a trace by hand.
  3. If the answer is wrong, don't re-ask — scan your trace for the first step that went sideways. Notice how the bug is almost never the last step.

Recap. Agents are hard to test because they're non-deterministic and multi-step, so pass/fail isn't enough. You evaluate them in layers — metrics, unit tests, integration tests, and human review (with frameworks like LangSmith, Ragas, DeepEval). And you make them debuggable with structured logging & tracing — recording every Thought → Action → Observation (Helicone, LangFuse) so a black box becomes a glass box. The habit that ties it together: when it breaks, read the trace first. Next up — the responsibilities that come with letting an agent act on its own.