Arize-ai/phoenix
Phoenix is an open-source AI observability platform for experimentation, evaluation, and troubleshooting of large language model applications, supporting multiple frameworks and LLM providers with flexible deployment options.
Awesome MCP › Testing & Debugging
Arize-ai/openinference
OpenInference is an open-source project designed to extend OpenTelemetry by providing comprehensive tracing and observability specifically for AI applications. It supports the Model Context Protocol (MCP), enabling detailed insight into the invocation of large language models (LLMs) and the surrounding application context, such as retrieval from vector stores and the use of external tools like search engines or APIs. The project is transport and file-format agnostic, making it compatible with various data formats including JSON, ProtoBuf, and DataFrames. OpenInference is natively supported by the Arize-ai phoenix platform but can be integrated with any OpenTelemetry-compatible backend. The project includes a specification maintained in markdown files that outline the conventions and protocols for tracing AI applications. It provides a set of instrumentations for popular machine learning SDKs and frameworks across multiple programming languages, with a strong emphasis on Python. The Python libraries cover semantic conventions for tracing LLM applications and instrumentations for various AI SDKs such as OpenAI, LangChain, LlamaIndex, AWS Bedrock, and others. These instrumentations facilitate the integration of OpenInference tracing capabilities into AI workflows, enhancing observability and debugging. OpenInference aims to standardize how AI application telemetry is collected and analyzed, helping developers and organizations monitor the performance and behavior of AI models in production environments. By leveraging OpenTelemetry's extensible framework, OpenInference bridges the gap between traditional application monitoring and the unique needs of AI systems, particularly those involving complex model interactions and external data sources. Overall, OpenInference is a valuable tool for teams looking to implement robust observability for AI applications, ensuring better reliability, transparency, and performance optimization through detailed tracing and context-aware telemetry.
https://github.com/Arize-ai/openinference
Also in Testing & Debugging
modelcontextprotocol/inspector
MCP Inspector is a developer tool that provides a visual UI and CLI for testing, debugging, and interacting with Model Context Protocol (MCP) servers, enhancing MCP server development workflows.
MCPJam/inspector
MCPJam Inspector is a developer tool for testing and debugging MCP servers, supporting multiple protocols and LLM interaction, designed to streamline MCP development workflows.
LSTM-Kirigaya/openmcp-client
OpenMCP is an all-in-one VSCode plugin that integrates development, testing, and management tools for Model Context Protocol (MCP) server debugging and large model interaction.
workbackai/mcp-nodejs-debugger
MCP Node.js Debugger is an MCP server that enables AI coding assistants like Cursor and Claude Code to debug Node.js applications at runtime by setting breakpoints and inspecting runtime state.
modelscope/MCPBench
MCPBench is an open-source evaluation framework for benchmarking Model Context Protocol (MCP) servers across Web Search, Database Query, and GAIA tasks, measuring accuracy, latency, and token consumption under consistent LLM and agent settings.
hyprmcp/jetski
Open source MCP analytics and authentication platform that deploys an OAuth gateway in front of MCP servers and aggregates prompt analytics, generated setup instructions and real time debug logs.
loopwork/Companion
Swift app for macOS, iOS and visionOS that connects to local and remote MCP servers to browse and exercise their prompts, resources and tools while testing and debugging.