Guide
Most tools sold as LLM observability were designed for single-turn completions. They record prompts, responses, latency and token spend, and they do that well. An agent workload asks a different question of them: not what the model said, but why a run made the choices it made. Those are not the same requirement, and the gap is where selection goes wrong.
This page is a set of criteria rather than a verdict. Vendor capabilities in this category change monthly, and any table published today is wrong by the time you read it. What does not change is the set of questions that separate a tool built for agent behaviour from one built for prompt logging. Take these four to whatever you are evaluating and let the answers decide.
A single-turn completion has one prompt, one response, and a clear success condition. An agent run has a sequence: it chooses a tool, reads the result, decides what to do next, and repeats until it concludes. The failures that cost you money live in that sequence rather than in any one call.
An agent that loops politely returns success on every step. An agent that calls the right tool and receives an empty result carries on as though it received an answer. An agent working from stale retrieved context produces a confident, wrong output with no error anywhere in the trace. None of those throw, so none of them appear in error rate, and all of them are invisible to a tool that records calls rather than decisions.
That is the property to test for. A tool can be excellent at what it was built for and still be unable to explain your last incident.
These are the same four signals set out in our guide to what to instrument on an agent fleet. That page covers the signals themselves. This one is about whether a product can actually give them to you.
Ask to see a stored run from a fortnight ago, not a live demo. You want the prompt as it was actually sent after templating, every tool the agent chose, the arguments it passed, what came back, and the order of all of it. Many tools store the first and last of those and summarise the middle. A summary is enough to show you that a run happened and not enough to show you why it went wrong.
The follow-up question is retention. A trace you can hold for twenty-four hours is a debugging aid for incidents you notice immediately. Most agent failures are noticed by a customer, later.
This is the question that eliminates the most candidates, and it is worth being specific about what you are asking. A tool call that times out is an error and everything records it. A tool call that returns HTTP 200 with an empty array, or a polite refusal, or yesterday's data, is a success by every transport-level measure and a failure in every way that matters to the run.
So ask whether the product can express that difference at all, and whether expressing it requires you to write the judgement yourself. Some can carry a per-call annotation you supply. That is a reasonable answer. Being told that failures are captured is not, because it answers a narrower question than the one you asked.
A run that calls the same tool with the same arguments nine times has told you something precise about the agent's reasoning, and it will appear in an ordinary dashboard as nine successful calls and a slightly elevated latency. Ask whether the tool can group calls within a run by tool and argument set, and whether it can alert on repetition rather than on failure.
If the answer is that you can build that view from the raw data, establish how much work that is before you buy. Sometimes it is a saved query. Sometimes it is an export pipeline and a warehouse.
The hardest failures to diagnose are the ones where every step executed correctly against wrong information. To see those you need the retrieved context attached to the run: which documents were pulled, what they contained at the time, and how old they were. Ask whether retrieval is a first class part of the trace or something you attach as free text.
This is the criterion most often missing, and its absence is the most expensive, because a wrong answer from correct reasoning is indistinguishable from a model problem until you can see the inputs.
Tooling in this space grew out of prompt engineering, where the unit of interest is one call and the question is whether the output improved. That heritage shows. Prompt and response capture, version comparison, token accounting and evaluation harnesses are generally strong, because they were the original requirement.
Sequence reasoning is generally weaker for the same reason. Grouping by run, expressing semantic rather than transport failure, counting repetition and treating retrieved context as part of the record are newer concerns, and support for them varies more than marketing pages suggest. That is a property of how the category developed rather than a claim about any particular product, and it is the reason to test against your own traces instead of a feature list.
Pick one incident you have already had and could not explain quickly. Reproduce it if you can, or use the logs you kept. Then ask each candidate to answer that specific question from its own stored data. A tool that can reconstruct your worst past incident will handle your next one. A tool that demonstrates beautifully on a two-step example may not.
Do the same with retention and cost together. Full traces on an agent fleet are large, and the pricing that looks reasonable on completions can change substantially when every run carries a multi-step trace with attached context. Get the number for your own volume before you commit.
Every criterion here is about seeing the failure. None of them is about resolving it. Once the trace shows a loop, or a tool returning nothing, or context falling out of the window, somebody still has to decide which of those is the cause and change the architecture accordingly. That work is specific to your framework and it rarely resembles the tutorial.
FleetHelp is built for that step. Your agents file their own tickets against our on-call rotation and we debug the framework rather than hand you another dashboard.