phoenix-evals

Phoenix Evals is a framework for creating and executing evaluators that assess AI/LLM application quality. Use it to validate outputs with code-based rules, LLM judges, and human feedback, then integrate evaluations into CI/CD pipelines and production monitoring.

• Build code-based evaluators for deterministic checks (retrieval quality, format validation)
• Create LLM-based evaluators with custom prompts and templates for nuanced assessment
• Run pre-built evaluators for common tasks (RAG, faithfulness, relevance)
• Batch evaluate DataFrames and run experiments across datasets
• Integrate evaluations into pytest, Vitest, and Jest as CI gates
• Validate evaluator accuracy against human labels to ensure >80% TPR/TNR

1. 1.Set up Phoenix server and install language-specific dependencies (Python or TypeScript)
2. 2.Instrument your application with tracing to capture spans and traces
3. 3.Analyze errors in production traces using error analysis and axial coding
4. 4.Define what to evaluate based on observed failures
5. 5.Choose a judge model (code-based, LLM, or pre-built)
6. 6.Build evaluators using code templates or LLM-based templates
7. 7.Validate evaluator accuracy against human labels
8. 8.Run evaluators on datasets or in batch mode