io.github.cyanheads/evals-mcp-server MCP Server
io.github.cyanheads/evals-mcp-server
What this kind of tool is for
- Hand your agent a structured way to call a model or an inference API.
- Compare outputs from two different providers on the same prompt.
- Wire retrieval, embeddings, or generation into an existing agent workflow.
How to compare servers like this
- Provider coverage:
- Match the tool's supported providers and models to the ones your project uses.
- Cost exposure:
- Model-backed tools spend real API budget; check how usage and limits are configured.
- Maintenance signal:
- The provider landscape shifts weekly; prefer tools updated within the last few months.
- Config surface:
- Prefer tools whose environment-variable or config keys are documented plainly.
This page carries your copy-paste config: pick your MCP client in the install section on this page and use the copy button.
What is the io.github.cyanheads/evals-mcp-server MCP server?
Author verifiable eval records through a draft→review→revise→submit loop with enforced graders.
How to install io.github.cyanheads/evals-mcp-server
Copy-paste configuration for popular MCP clients.
EVALS_DATA_DIRRoot folder for record JSON. The store manages drafts/, submitted/, and exports/ subdirs under it.
EVALS_REQUIRE_CONFIRMATIONWhen 'true', evals_submit_draft requests human confirmation before finalizing.
EVALS_DEFAULT_LICENSEDefault metadata.license applied when a draft omits one.
EVALS_CAPTURE_DIRDirectory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps.
MCP_LOG_LEVELSets the minimum log level for output (e.g., 'debug', 'info', 'warn').
MCP_HTTP_HOSTThe hostname for the HTTP server.
MCP_HTTP_PORTThe port to run the HTTP server on.
MCP_HTTP_ENDPOINT_PATHThe endpoint path for the MCP server.
MCP_AUTH_MODEAuthentication mode to use: 'none', 'jwt', or 'oauth'.
{
"mcpServers": {
"evals-mcp-server": {
"command": "node",
"args": [
"-y",
"@cyanheads/evals-mcp-server",
"run",
"start:stdio"
],
"env": {
"EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
"EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
"EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
"EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
"MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>",
"MCP_HTTP_HOST": "<YOUR_MCP_HTTP_HOST>",
"MCP_HTTP_PORT": "<YOUR_MCP_HTTP_PORT>",
"MCP_HTTP_ENDPOINT_PATH": "<YOUR_MCP_HTTP_ENDPOINT_PATH>",
"MCP_AUTH_MODE": "<YOUR_MCP_AUTH_MODE>"
}
}
}
}Related MCP servers
Convert currencies, get FX rates, and query historical ECB exchange rate data.
Offline, keyless lookup of the US civil aircraft registry — decode N-numbers, search records.
UN FAOSTAT global food & agriculture statistics over a local SQLite mirror, via MCP.
FBI Crime Data Explorer — UCR estimates, NIBRS breakdowns, hate crimes, arrests, and agency data.
FCC broadband availability, coverage analysis, and digital divide data for US geographies.
Search FDIC-insured banks, Call Report financials, peer rankings, failures, deposit market share.
