Grafana Dashboards for MCP Agent Observability

An MCP agent runs through an extra protocol layer sitting between the model and the tools it calls, and by default that layer records nothing. A request goes out, a tool response comes back, and everything that happened in between just disappears. No log, no trace, no record of why the model picked the tool it picked.

The tools built for watching regular services don't help here. APM software tracks whether a service is up, how fast it responds, and whether it's throwing errors, but it has no concept of an agent's reasoning chain: which tool got selected, why that one and not another, how many back-and-forth turns it took, or whether the result that came back was actually complete. A tool can time out, or return half a payload, and the agent will just keep going, because nothing at the infrastructure level technically broke. That absence of a technical break is the trap. Adoption of MCP servers is running well ahead of the security and observability work needed to support it: a majority of organizations are already experimenting with these servers, but only a small slice have made it to production, and the reason is the same both times, because the observability gap and the security gap turn out to be one gapc3. The NSA made this concrete in a May 2026 information sheet on MCP security, which recommends logging every tool and model invocation, exact parameters and identities included, a compliance line that can't be met without structured telemetry. That's not a nice-to-have. The NSA's recommendation is a compliance line that can't be met without structured telemetry.

What "observing" an MCP agent means, the five signals that matter

MCP observability isn't application monitoring wearing a new label. It means capturing tool invocations, policy decisions, how sensitive data gets handled, latency, errors, and the identity behind every action an agent takes. Five signal types need to be wired together to get there.

Metrics come first: counts and rates, tool call totals, error rates, latency spread across percentiles, session counts, token usage, how many calls are running at once. Logs sit next to them, structured records of individual invocations, showing which tool got called, by which client, with what parameters, and what came back.

Traces are where it gets interesting. A single agent request might produce spans across a Python client, an MCP server, and a Node.js tool server, all linked together through OpenTelemetry context propagation carried in params._meta using traceparent, tracestate, and baggage keys. Without that propagation, each hop looks like an unrelated event instead of one continuous chain. Profiles round out the infrastructure side, tracking CPU and memory on the MCP server process itself.

Then there's the fifth signal, and it's the newest one: agent sessions, generally available in Grafana Cloud as of July 30, 2026. The idea is to treat the full conversation, the whole session, as telemetry in its own right, an operational signal that belongs in the same store as metrics and traces, so it can be correlated against them and drilled into when something goes wrong, rather than something reconstructed after the fact by scraping logs together. That's a real shift in thinking. It's an operational signal that belongs in the same store as metrics and traces, so it can be correlated against them and drilled into when something goes wrong.

How telemetry gets from an MCP server into Grafana

The path splits into two sides, metrics and traces, instrumented differently but both landing in Grafana.

On the metrics side, the MCP server exposes a Prometheus endpoint at /metrics using the prometheus_client library, and that's it, no external collector or sidecar agent required. Prometheus scrapes that endpoint every five seconds, per the setup documented for the community dashboard. Grafana Cloud's AI Observability integration builds on top of that using the OpenLIT SDK, which auto-generates distributed traces and metrics for each agentic event, covering MCP-specific counters like mcp_requests_total, mcp_tool_calls_total, mcp_errors_total, mcp_client_operation_duration_seconds, and histograms for request and response size.

On the traces side, the OpenTelemetry SDK sits embedded in the application and creates a span for every tool call, exported over OTLP gRPC to whatever backend is listening, Jaeger, Tempo, or Grafana Cloud itself. The mechanism that makes this useful across services is context propagation: OTel carries context so a span that started in a Python client links up with a span in a Node.js tool server, and the result is one coherent trace crossing both language and network boundaries. The client span is always the parent of the server span no matter what transport is in play, and that rule comes from OpenTelemetry's GenAI semantic conventions, now maintained in their own repository with a dedicated MCP section defining spans, attributes, and metrics. Worth flagging early: those conventions are still marked Development status, not stable. Pin the instrumentation version being used and expect attribute names to shift under it.

The MCP specification dated 2026-07-28 deprecated protocol-level Logging, Roots, and Sampling under SEP-2577, and separately deprecated HTTP+SSE, each running on a twelve-month clock before removal, changing what "built-in" logging even means going forward. Teams leaning on protocol-level logging as their observability plan need to move to OTel-based instrumentation before that clock runs out.

The three dashboard tiers Grafana provides

Grafana's answer to all this comes in three tiers, and each one is built to answer a different operational question. They're not competing options. A team can run the community dashboard on self-hosted Grafana, layer the Cloud integration on top for broader context, and add Agent Observability for session-level depth, all at once.

It's self-hosted, Prometheus-backed, open source, and published by Grafana Labs into the public dashboard library. It's organized into ten sections: Overview, Multi-Client Breakdown, Transport Layer, Protocol & Messages, Tool Execution, Agentic & Token Metrics, Rate Limiting, Resource Access, Sessions & Connections, and System Resources. The panels inside cover total tool calls, an error rate gauge, active sessions, in-flight concurrency, handshake success rate, per-client call rates, connection breakdowns by transport (streamable-HTTP, SSE, WebSocket), JSON-RPC latency at p50, p95, and p99, invocation rate per tool, execution time percentiles, error type classification, token usage per tool, a hallucination-detection panel for calls to tools that don't exist, session duration percentiles, and disconnect reasons broken down by client. This tier answers one question above all: is the MCP server healthy right now?

Tier two is the Grafana Cloud AI Observability integration, which installs six pre-built dashboards into a Cloud instance: Agent Observability, GPU Monitoring, GenAI Evaluations, GenAI Observability, MCP Observability, and VectorDB Observability. It pulls LLM metrics, cost data, quality evaluations, vector database performance, MCP implementation details, and GPU usage into one unified view. The end-to-end tracing here shows the full path a request takes, from the agent through the MCP server and out to each tool call, which is what lets a team pinpoint exactly where a bottleneck or a failure is sitting. This tier answers a bigger-picture question: how does the MCP layer fit into the wider AI application stack, and where along that path is cost or latency piling up?

Tier three is Grafana Cloud Agent Observability, generally available since July 30, 2026, and it adds that fifth signal, agent sessions, alongside metrics, logs, traces, and profiles. It ships with built-in analytics views: Activity (generation counts, conversation counts, active agents over time), Performance (latency distributions, time to first token, error rates), Usage and cost (token usage split by model and provider, estimated cost, cache efficiency), Tools (call frequency, execution duration, error rates, usage share per tool), and Quality (evaluation scores, score distributions, quality trends over time). This is the tier for the harder question: what was the agent actually doing when things went wrong, and is quality drifting in a direction nobody's caught yet. Tier 1, Community Dashboard #25252 (MCP Server Observability).

Reading the dashboards: what the panels tell you about agent behavior

A panel is only as useful as the pattern a team knows to look for in it. Take latency: a widening p99 sitting next to a stable p50 means tail latency, a subset of calls is in trouble while the average case looks fine, and filtering by tool_name shows whether that's one misbehaving tool or the MCP layer itself struggling. Checking the 95th and 99th percentile spread is exactly the method for telling apart a slow downstream API from a slow MCP layer.

Errors split the same way. The dashboard separates JSON-RPC error codes from tool execution errors: a JSON-RPC error points at the protocol or the transport, while a tool execution error points at the tool's own logic, or at whatever service downstream of it is misbehavingc33. Silent partial failures don't show up in either error bucket; they are visible only in response size panels, so if a tool's average payload size suddenly drops, that's worth an alert on its own.

The hallucination-detection panel in the Agentic & Token Metrics section tracks calls to tools that don't exist on the server at all, and that's a direct signal something has drifted, either the model's tool-selection behavior has gone sideways, or a tool got removed from the server without anyone updating the agent's context. Session duration and turns-to-completion tell a related story about efficiency: an agent that should wrap a task in two tool calls but keeps taking six is either poorly prompted or stuck retrying after repeated failures. Self-correction rate, tracked in that same Agentic & Token Metrics section, measures how often the agent is cleaning up after its own mistakes, and a high rate there quietly inflates both token usage and cost.

Exemplar markers on histogram panels in Agent Observability link a latency or error spike directly to the conversation that caused it. The dashboard separates JSON-RPC error codes and tool execution errors (a JSON-RPC error points to a protocol or transport problem, while a tool execution error points to the tool's own logic or the downstream service it calls).

What production teams found when they wired this up

Visibility into MCP behavior doesn't just make dashboards prettier. It changes what teams choose to ship and how they check their own workc38.

Inside Grafana Labs, the Tempo squad runs a custom agentic harness that profiles a development environment holding a few terabytes of data, finds candidate optimizations, implements them, and reruns the same baseline query to measure what changed. A human still makes the call on whether to merge, but that call only means anything because before-and-after trace data supports it. Taking the observability layer away turns "did this actually help" into a guess.

Google Cloud's MCP Toolbox for Databases ran into a more concrete diagnostic problem: when a request was slow, nobody could say with confidence whether the culprit was the network, the MCP server, or the database query itself. Working with Agnost AI, the Toolbox team added distributed tracing across STDIO, SSE, and HTTP transports, latency histograms on both server and client SDKs, and an UpDown Counter tracking active sessions at any given moment, which together made the layer of the stack responsible for latency directly visible.

At TeleTracking, a Grafana customer, the shift changed who gets to do root cause analysis at all. Oren Lion, Director of Software Engineering, Productivity Engineering, described the process in Grafana Labs' July 27, 2026 press release as prompts run through the Grafana Assistant side panel that surface anomalies and correlations across signals, cutting down the time it takes to speed things up. The gcx tool took that same access and opened it up to anyone running a coding agent, including engineers who didn't already know where to look. Observable MCP behavior changes what teams decide to ship and how they validate changes.

Where the instrumentation still has gaps

The GenAI semantic conventions for MCP spans remain in flux, so attribute names are going to shift, and any team building dashboards or alerts against them needs to pin their instrumentation version rather than tracking the latest release blind, since a renamed attribute silently breaks a panel. That's not a small caveat. A renamed attribute silently breaks a panel, and unlike a tool timing out, nothing about a broken panel announces itself.

The deprecation clock on protocol-level Logging, Roots, and Sampling, along with HTTP+SSE, gives teams twelve months to move any observability logic still leaning on those protocol features over to OTel-based instrumentation. Waiting until close to that deadline is the wrong move, given how much of the signal chain, traces, metrics, sessions, all depends on propagation working correctly across client and server boundaries in the first place.

None of this closes the loop entirely on its own, either. The five signals, metrics, logs, traces, profiles, and now sessions, only add up to real visibility when they're wired together and someone's actually reading the panels with the failure patterns in mind, tail latency, silent payload shrinkage, hallucinated tool calls, self-correction rates creeping upward. Adoption of MCP is still outrunning the observability and security maturity needed to run it safely in production, and closing that gap requires more than picking one dashboard and calling it done. It's building the instrumentation now, before the deprecation clocks run out and before the next silent failure is the one nobody catches.