Awesome MCP › Testing & Debugging

Arize-ai/phoenix

⭐ 11445 Jupyter Notebook repository created 2022-11-09

Phoenix is an open-source AI observability platform designed to support the experimentation, evaluation, and troubleshooting of large language model (LLM) applications. It offers a comprehensive suite of tools that enable developers and data scientists to trace, benchmark, and optimize their LLM-based systems effectively. The platform supports multiple popular frameworks such as LlamaIndex, LangChain, Haystack, DSPy, and smolagents, as well as various LLM providers including OpenAI, Bedrock, MistralAI, VertexAI, LiteLLM, and Google GenAI, making it highly versatile and vendor-agnostic. Key features of Phoenix include tracing capabilities that utilize OpenTelemetry-based instrumentation to monitor the runtime behavior of LLM applications, allowing users to gain deep insights into model interactions and performance. The evaluation tools leverage LLMs themselves to benchmark application performance through response and retrieval evaluations. Users can create versioned datasets for experimentation, evaluation, and fine-tuning, facilitating systematic improvements and reproducibility. Phoenix also supports experiments to track and evaluate changes in prompts, LLMs, and retrieval methods, helping teams iterate quickly and confidently. The playground feature allows for prompt optimization, model comparison, parameter adjustments, and replaying traced LLM calls, providing an interactive environment for prompt engineering. Additionally, prompt management capabilities enable systematic testing and version control of prompt changes, enhancing collaboration and control over model inputs. The platform is designed for flexible deployment, capable of running on local machines, Jupyter notebooks, containerized environments, or cloud infrastructure. Installation is straightforward via pip or conda, and container images are available on Docker Hub for easy deployment with Docker or Kubernetes. Phoenix's open-source nature and broad compatibility make it a powerful tool for organizations looking to improve the reliability, transparency, and performance of their LLM applications.

https://github.com/Arize-ai/phoenix

agentsai-monitoringai-observabilityaiengineeringanthropicbedrockbenchmarkingcontainer-deploymentcontainerized-deploymentdatasetsdockerdspyevalsevaluationexperimentationexperimentsgoogle-genaihaystackkuberneteslangchainlarge-language-modelslitellmllamaindexllmllm-evalllm-evaluationllmopsllmsmistralaiopen-sourceopenaiopentelemetryprompt-engineeringprompt-managementsmolagentstracingtroubleshootingversion-controlvertexai