Model Context Protocol (MCP) has gone from an Anthropic-internal spec to a de facto standard for agentic AI integrations faster than most infrastructure primitives. We’ve been deploying MCP servers in client environments since early 2025. Here’s what the spec doesn’t tell you.

What Actually Goes Wrong

The spec is well-designed. The implementation surface area is not the problem. The problems are operational:

  • Tools that work in development fail silently against production APIs because of authentication edge cases
  • Version mismatches between the MCP client and server manifest as cryptic JSON-RPC errors
  • A single slow tool blocks the entire agent loop when there’s no timeout
  • No standard tracing format means you’re flying blind when debugging a 15-tool agentic workflow

None of these are unsolvable. All of them have been production incidents.

Authentication: The First Hard Problem

MCP servers connect to your internal systems. That means they need credentials. How those credentials are provisioned, rotated, and scoped is the most consequential decision you’ll make.

What not to do: long-lived API keys stored in environment variables and distributed with the server binary. These rotate infrequently, have broad scopes, and leak through .env files, log lines, and container images.

What we do instead: short-lived credentials fetched at startup from a secrets manager (Vault, AWS Secrets Manager, GCP Secret Manager). Combined with per-tool OAuth scopes where the upstream API supports it:

interface ToolCredential {
  token: string;
  expiresAt: number;
}

class CredentialManager {
  private cache = new Map<string, ToolCredential>();

  async getCredential(toolName: string): Promise<string> {
    const cached = this.cache.get(toolName);
    if (cached && cached.expiresAt > Date.now() + 60_000) {
      return cached.token;
    }
    const credential = await this.fetchFromSecretsManager(toolName);
    this.cache.set(toolName, credential);
    return credential.token;
  }

  private async fetchFromSecretsManager(toolName: string): Promise<ToolCredential> {
    // Fetch from your secrets manager here
    // Return { token, expiresAt }
    throw new Error('Not implemented');
  }
}

Scope credentials to the minimum required. A tool that reads from a database should not have credentials that write to it. If the MCP server is compromised (and at some point, something will be), blast radius matters.

Tool Versioning

MCP tools have input schemas. Those schemas change. Managing breaking changes is harder than in a REST API because the consumer (the LLM) doesn’t have a version header. It just calls the tool name it learned from the schema.

Our convention: never remove or change a tool’s input schema in place. Instead, version the tool name:

{
  "name": "search_documents_v2",
  "description": "Search documents by semantic query. Returns results with citations. Replaces search_documents.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "maxResults": { "type": "number", "default": 10 },
      "includeMetadata": { "type": "boolean", "default": true }
    },
    "required": ["query"]
  }
}

Keep the old tool alive for one major client release cycle, then deprecate it. The LLM will learn the new tool name from the updated system prompt. Just update the prompt alongside the server.

Timeouts: Non-Negotiable

Every tool in your MCP server must have an explicit timeout. There is no default timeout in the MCP spec, and tools that block indefinitely will freeze your entire agent loop.

async function withTimeout<T>(
  promise: Promise<T>,
  timeoutMs: number,
  toolName: string
): Promise<T> {
  const timeout = new Promise<never>((_, reject) =>
    setTimeout(() => reject(new Error(`Tool ${toolName} timed out after ${timeoutMs}ms`)), timeoutMs)
  );
  return Promise.race([promise, timeout]);
}

// Wrap every tool execution
const result = await withTimeout(executeTool(input), 10_000, 'search_documents');

Set timeouts based on P99 latency of the underlying service, not P50. A database query that normally takes 200ms might take 8s under load. Your timeout should be somewhere between those values: generous enough to avoid false positives, tight enough to fail fast when something is genuinely broken.

Observability

This is the biggest gap in current MCP implementations. The spec has no tracing semantics, so every implementation logs differently (or not at all).

We instrument every tool call with OpenTelemetry spans:

import { trace, context, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('mcp-server', '1.0.0');

async function instrumentedToolCall(
  toolName: string,
  input: unknown,
  handler: (input: unknown) => Promise<unknown>
): Promise<unknown> {
  return tracer.startActiveSpan(`mcp.tool.${toolName}`, async (span) => {
    span.setAttributes({
      'mcp.tool.name': toolName,
      'mcp.tool.input_size': JSON.stringify(input).length,
    });

    try {
      const result = await handler(input);
      span.setStatus({ code: SpanStatusCode.OK });
      span.setAttributes({ 'mcp.tool.output_size': JSON.stringify(result).length });
      return result;
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

This gives you: tool call duration, input/output sizes, error rates, and the ability to trace a full agentic workflow as a waterfall of spans in Jaeger, Honeycomb, or whatever backend you use.

Security Posture

An MCP server is an attack surface. The LLM can be induced (via prompt injection in retrieved content) to call tools with attacker-controlled inputs. Think carefully about what your tools can do.

Limit destructive tools. If a tool can delete records, send emails, or execute code, it needs explicit human approval before execution. The MCP spec’s “sampling” primitive is designed for this, but implementation support is still limited. For now, we recommend keeping destructive tools behind a confirmation tool call pattern.

Validate all tool inputs server-side. The LLM can generate inputs that don’t match the declared schema, either through model error or adversarial injection. Parse and validate with Zod or similar before executing:

import { z } from 'zod';

const SearchInput = z.object({
  query: z.string().min(1).max(1000),
  maxResults: z.number().int().min(1).max(50).default(10),
});

function executeSearch(rawInput: unknown) {
  const input = SearchInput.parse(rawInput); // throws ZodError on invalid input
  return performSearch(input.query, input.maxResults);
}

Log all tool calls. Every tool invocation should be logged with the tool name, sanitized input, caller identity, and timestamp. This is your audit trail when something goes wrong.

The Drift Problem

MCP servers evolve. The system prompt that describes available tools to the LLM needs to stay in sync with what the server actually exposes. When they drift, the LLM hallucinates tool calls for tools that don’t exist, or uses stale parameter names.

Build a validation step into your CI pipeline: introspect the running server’s tool list and compare it against what’s documented in your system prompt. Fail the build on mismatch. This sounds obvious and is almost never done.

MCP is still maturing. The patterns above are what’s working for us today in production, not what the spec mandates. Expect the ecosystem to evolve, but the fundamentals of auth, observability, timeouts, and input validation will remain.