phoenix-evals

Name: phoenix-evals
Author: arize-ai

arize-ai/phoenix

Source View on skills.sh

How to install phoenix-evals

npx skills add https://github.com/arize-ai/phoenix --skill phoenix-evals

Claude Code

Cursor

Windsurf

Cline

Full instructions (SKILL.md)

Source of truth, from arize-ai/phoenix.

name: phoenix-evals description: Build and run evaluators for AI/LLM applications using Phoenix. license: Apache-2.0 compatibility: Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client. metadata: author: oss@arize.com version: "1.0.0" languages: "Python, TypeScript"

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Run experiment	experiments-running-python, experiments-running-typescript
Run evals in a test runner (CI gate)	integrations-pytest, integrations-vitest-jest
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Gating CI: evaluators-{code|llm}-{python|typescript} → integrations-{pytest|vitest-jest} → production-continuous

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`integrations-*`	Run evals from test runners (pytest, Vitest, Jest) as a CI gate
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5

Related skills

More from arize-ai/phoenix and the wider catalog.

phoenix-cli

arize-ai/phoenix

Debug LLM applications using the Phoenix CLI. Fetch traces, analyze errors, structure trace review with open coding and axial coding, inspect datasets, review experiments, query annotation configs, and use the GraphQL API. Use whenever the user is analyzing traces or spans, investigating LLM/agent failures, deciding what to do after instrumenting an app, building failure taxonomies, choosing what evals to write, or asking "what's going wrong", "what kinds of mistakes", or "where do I focus" — even without naming a technique.

833 installs

phoenix-tracing

arize-ai/phoenix

OpenInference semantic conventions and instrumentation for Phoenix AI observability. Use when implementing LLM tracing, creating custom spans, or deploying to production.

829 installs