Debugging AI Agents with LangSmith

LangSmith is an observability and evaluation platform built for LLM applications. To debug and improve AI agent performance with it, you instrument your agent with the LangSmith SDK, capture full execution traces, tag runs by environment, write evaluators against your traces, and iterate on prompts using the prompt hub. The whole loop typically reduces time-to-diagnosis from hours to minutes.

---

AI agents fail quietly. A chain that retrieves the wrong document, a tool call that returns a malformed response, a prompt that drifts across model versions — none of these throw a stack trace you can grep for. They just produce subtly wrong answers, and you find out when a user complains or a downstream system breaks.

This is the core problem LangSmith was built to solve. It sits between your agent and your LLM calls, capturing everything: inputs, outputs, intermediate steps, latency, token counts, and the full reasoning chain your agent walked through to reach a conclusion. When something goes wrong, you don't have to reconstruct what happened from logs. You can watch the replay.

But LangSmith is more than a debugger. Teams that use it well don't just fix bugs faster. They build a systematic feedback loop that connects production failures to prompt improvements to regression tests. That's the part most tutorials skip, and it's where the real performance gains come from.

---

Setting Up Tracing: The First Thing You Actually Need

Before you can debug anything, you need data. LangSmith collects that data through tracing, and getting it running is genuinely fast — fifteen minutes if you already have a LangChain or LangGraph project.

You'll need a LangSmith account (there's a free tier), your API key, and two environment variables:

LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=your_key_here

Set those, and every LangChain or LangGraph run you execute will automatically stream trace data to LangSmith. No additional instrumentation required for basic use.

If you're running a custom agent outside LangChain, you'll use the @traceable decorator from the langsmith Python package to wrap your functions. The decorator captures function inputs and outputs as a trace node, and you can nest decorators to represent a full agent execution as a tree of spans.

One thing worth doing immediately: tag your runs by environment. Add LANGCHAIN_PROJECT=production or LANGCHAIN_PROJECT=staging so you can filter traces without wading through every test run you've ever made. Teams that skip this step early tend to regret it when they're trying to isolate a production regression six weeks later.

---

Reading a Trace: What You're Actually Looking At

A LangSmith trace is a tree. The root is typically your agent's entry point, and the branches are the individual steps: retriever calls, LLM invocations, tool executions, and any sub-chains that ran inside those steps.

For each node, you can see the exact input that went in and the exact output that came out. For LLM calls specifically, you see the full prompt as rendered, the model response, the latency, and the token counts. This is more useful than it sounds.

Consider a retrieval-augmented agent that's giving wrong answers. The trace lets you check three things immediately:

1. Did the retriever pull the right documents?
2. Were those documents actually included in the prompt as expected?
3. Did the model's response reflect the retrieved content, or did it hallucinate past it?

Each of those is a different failure mode with a different fix. Without trace-level visibility, you're guessing which one it is. With a trace, you can see the answer in under a minute.

Latency trees are equally useful. If an agent is taking twelve seconds to respond, the trace will show you exactly which step consumed eight of those seconds. Maybe it's a slow vector search. Maybe a tool call is making an external API request that takes forever. Maybe you're hitting a model with a 4,000-token prompt when a 600-token prompt would work. The trace makes the bottleneck visible.

---

Using Datasets and Evaluators to Move Beyond Manual Review

Manual trace review works when you're debugging a specific failure. It doesn't scale to ongoing quality monitoring. For that, you need evaluators.

In LangSmith, an evaluator is a function that takes a run's inputs and outputs and returns a score or a label. You attach evaluators to a dataset, run your agent against the dataset, and get a structured comparison of how your agent performed across every example.

Building a useful dataset is the part that takes real thought. The best datasets come from production traces. You filter your traces for runs that received negative feedback, runs where the agent took an unexpected number of steps, or runs where latency spiked. Then you curate those into a labeled dataset with expected outputs or pass/fail criteria.

LangSmith ships with a set of off-the-shelf LLM-as-judge evaluators, including ones for correctness, helpfulness, and conciseness. These are useful starting points, but they have real limitations. An LLM judge that scores "correctness" on a customer support agent doesn't know your product, your policies, or what "correct" means in your specific context. Custom evaluators, written in Python against your domain criteria, usually outperform generic ones significantly.

A team at a mid-size logistics company running route optimization agents found that generic evaluators flagged roughly 60% of the failures their domain-specific evaluator caught. The other 40% required knowing that certain output formats were invalid for their dispatch system. The generic judge couldn't see that. A Python function checking for specific field values could.

---

Prompt Versioning and the Improvement Loop

LangSmith includes a prompt hub where you can version, store, and pull prompts by name. This sounds like a minor convenience. In practice, it's the connective tissue of an improvement loop.

Here's the pattern that works:

You identify a failure class through trace review or evaluator results. You write a revised prompt in the LangSmith prompt hub, tagging it as a new version. You run your evaluation dataset against the new prompt. If scores improve across your key metrics without regressing on others, you promote the new version to production and pull it by name in your agent code.

The key discipline is running the full dataset before promoting. It's easy to fix one failure by changing a prompt and introduce three new ones in the process. The dataset is your regression guard.

Teams that skip versioning tend to lose track of what changed when a metric improves or degrades. Three months later, no one can remember whether the latency improvement came from a prompt change or a retriever tuning. LangSmith's version history gives you a record that doesn't depend on anyone's memory.

This systematic improvement cycle is especially important when you're managing complex operational workflows. If you're deploying agents across your operations team, AI Agent Deployment Checklist: Ops Teams provides a framework for ensuring quality at scale, and LangSmith's evaluation infrastructure is a key tool for maintaining it over time.

---

Monitoring in Production: Closing the Loop

Debugging and evaluation are retrospective. Production monitoring is what catches problems before they compound.

LangSmith lets you set up feedback collection directly in your application. Users can thumbs-up or thumbs-down a response, and that feedback gets attached to the trace that generated it. Over time, you build a signal layer on top of your traces: these runs got positive feedback, these got negative feedback, and you can filter by that signal when building your next evaluation dataset.

You can also monitor aggregate metrics over time through LangSmith's monitoring dashboards. If your agent's average latency increases by 30% after a deployment, you'll see it. If a new model version changes your token consumption in unexpected ways, you'll see that too.

The monitoring setup that makes the most sense for most teams is a daily review of runs tagged with negative feedback, a weekly evaluation run against a curated dataset, and a deployment check that runs the dataset against any new prompt version before it goes to production. That cadence is lightweight enough to maintain and rigorous enough to catch most problems early.

For teams managing agents that need to hand off between systems or escalate to humans, AI Agent Handoff Strategies That Actually Work shows where those handoff points are most critical—and LangSmith's tracing makes it straightforward to monitor whether handoffs are happening correctly and with adequate context transfer.

---

What LangSmith Doesn't Do

Being honest about the limitations matters here.

LangSmith is not a testing framework for the business logic around your agent. It doesn't know what your agent is supposed to accomplish in the context of your product. It will show you that the agent took seven steps when it usually takes three. It won't tell you whether that's a problem.

It also doesn't replace good agent architecture. Teams sometimes treat observability as a substitute for design. If your agent is brittle because it has no error handling, no retry logic, and a retriever that returns irrelevant chunks half the time, LangSmith will help you see all of those failures clearly. But it won't fix them for you. For instance, if your agents need reliable access to internal data sources, MCP: Give AI Agents Access to Your Databases addresses how to set up that infrastructure properly—and LangSmith will help you debug when retrieval isn't working as expected.

And for teams that aren't yet running LLM-powered features in production, LangSmith can feel like a solution in search of a problem. The platform's value scales with the complexity and volume of your agent workloads. A simple single-turn chatbot with low traffic probably doesn't need the full evaluation infrastructure. An agentic workflow making multi-step decisions with real consequences definitely does.

---

If you're building or managing AI agents and the debugging process still feels like guesswork, that's a solvable problem. LangSmith gives you the visibility to move from guessing to knowing. Getting the tracing set up takes an afternoon. Building a rigorous evaluation loop takes a few weeks of iteration. The compounding benefit of that investment, in reduced failure rates and faster improvement cycles, is what separates teams that ship reliable agents from teams that ship agents they're nervous about.

Related reading: Choosing an AI Partner for Growing Utah Companies