Arize-ai/phoenix
Phoenix is an open-source AI observability platform for experimentation, evaluation, and troubleshooting of large language model applications, supporting multiple frameworks and LLM providers with flexible deployment options.
Awesome MCP › Testing & Debugging
mclenhard/mcp-evals
MCP Evals is a testing harness for Model Context Protocol servers. Rather than asserting on fixed outputs, it drives the tools of a server with prompts and asks a language model to grade the responses, which suits tools whose answers vary between runs. Evaluations are declared in one of two forms. The TypeScript form exports an EvalConfig holding a model and a list of EvalFunction objects, each with a name, a description and an async run method that calls the exported grade helper with a model and a prompt. The YAML form is simpler: a model block naming the provider, either openai or anthropic, and the model name, plus a list of evaluations each carrying a name, a description, the prompt to send to the server and an optional expected_result describing the intended behaviour. API keys come from OPENAI_API_KEY or ANTHROPIC_API_KEY unless given in the file. Every evaluation returns a structured result with numeric scores from one to five for accuracy, completeness, relevance, clarity and reasoning, together with a free text comment summarising strengths and weaknesses. The package installs from npm as mcp-evals and runs from the command line through npx mcp-eval, taking the path to the evaluation file, either .ts or .yaml, and the path to the server entry point. The same tool is published as a GitHub Action: added to a workflow triggered on pull requests, it runs the suite and posts the scores as a pull request comment, updating that comment when the branch is pushed again, so a drop in tool quality is visible during review. A second feature, marked alpha, covers observability: calling metrics.initialize before the MCP server starts exposes metrics on a chosen port and can enable OpenTelemetry tracing to a collector endpoint, with a docker-compose file supplied for the monitoring stack. It targets maintainers of MCP servers who want continuous quality signals in CI.
https://github.com/mclenhard/mcp-evals
Also in Testing & Debugging
modelcontextprotocol/inspector
MCP Inspector is a developer tool that provides a visual UI and CLI for testing, debugging, and interacting with Model Context Protocol (MCP) servers, enhancing MCP server development workflows.
MCPJam/inspector
MCPJam Inspector is a developer tool for testing and debugging MCP servers, supporting multiple protocols and LLM interaction, designed to streamline MCP development workflows.
Arize-ai/openinference
OpenInference is an open-source extension of OpenTelemetry that provides comprehensive tracing and observability for AI applications, including support for the Model Context Protocol (MCP).
LSTM-Kirigaya/openmcp-client
OpenMCP is an all-in-one VSCode plugin that integrates development, testing, and management tools for Model Context Protocol (MCP) server debugging and large model interaction.
workbackai/mcp-nodejs-debugger
MCP Node.js Debugger is an MCP server that enables AI coding assistants like Cursor and Claude Code to debug Node.js applications at runtime by setting breakpoints and inspecting runtime state.
modelscope/MCPBench
MCPBench is an open-source evaluation framework for benchmarking Model Context Protocol (MCP) servers across Web Search, Database Query, and GAIA tasks, measuring accuracy, latency, and token consumption under consistent LLM and agent settings.
hyprmcp/jetski
Open source MCP analytics and authentication platform that deploys an OAuth gateway in front of MCP servers and aggregates prompt analytics, generated setup instructions and real time debug logs.