PluginBench
MCP Server
Active
Apache-2.0

io.github.cyanheads/evals-mcp-server MCP Server

io.github.cyanheads/evals-mcp-server

RepositoryRegistry

What is the io.github.cyanheads/evals-mcp-server MCP server?

Author verifiable eval records through a draft→review→revise→submit loop with enforced graders.

How to install io.github.cyanheads/evals-mcp-server

Copy-paste configuration for popular MCP clients.

transport: stdio
Config generated by PluginBench — verify against the source before use.
Environment / auth
  • EVALS_DATA_DIR

    Root folder for record JSON. The store manages drafts/, submitted/, and exports/ subdirs under it.

  • EVALS_REQUIRE_CONFIRMATION

    When 'true', evals_submit_draft requests a human confirmation where the client supports elicitation.

  • EVALS_DEFAULT_LICENSE

    Default metadata.license applied when a draft omits one.

  • EVALS_CAPTURE_DIR

    Directory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps.

  • MCP_LOG_LEVEL

    Sets the minimum log level for output (e.g., 'debug', 'info', 'warn').

  • MCP_HTTP_HOST

    The hostname for the HTTP server.

  • MCP_HTTP_PORT

    The port to run the HTTP server on.

  • MCP_HTTP_ENDPOINT_PATH

    The endpoint path for the MCP server.

  • MCP_AUTH_MODE

    Authentication mode to use: 'none', 'jwt', or 'oauth'.

Claude Desktop
~/Library/Application Support/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "node",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server",
        "run",
        "start:stdio"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>",
        "MCP_HTTP_HOST": "<YOUR_MCP_HTTP_HOST>",
        "MCP_HTTP_PORT": "<YOUR_MCP_HTTP_PORT>",
        "MCP_HTTP_ENDPOINT_PATH": "<YOUR_MCP_HTTP_ENDPOINT_PATH>",
        "MCP_AUTH_MODE": "<YOUR_MCP_AUTH_MODE>"
      }
    }
  }
}
Cursor
~/.cursor/mcp.json
{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "node",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server",
        "run",
        "start:stdio"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>",
        "MCP_HTTP_HOST": "<YOUR_MCP_HTTP_HOST>",
        "MCP_HTTP_PORT": "<YOUR_MCP_HTTP_PORT>",
        "MCP_HTTP_ENDPOINT_PATH": "<YOUR_MCP_HTTP_ENDPOINT_PATH>",
        "MCP_AUTH_MODE": "<YOUR_MCP_AUTH_MODE>"
      }
    }
  }
}
Windsurf
~/.codeium/windsurf/mcp_config.json
{
  "mcpServers": {
    "evals-mcp-server": {
      "command": "node",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server",
        "run",
        "start:stdio"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>",
        "MCP_HTTP_HOST": "<YOUR_MCP_HTTP_HOST>",
        "MCP_HTTP_PORT": "<YOUR_MCP_HTTP_PORT>",
        "MCP_HTTP_ENDPOINT_PATH": "<YOUR_MCP_HTTP_ENDPOINT_PATH>",
        "MCP_AUTH_MODE": "<YOUR_MCP_AUTH_MODE>"
      }
    }
  }
}
VS Code
.vscode/mcp.json
{
  "servers": {
    "evals-mcp-server": {
      "type": "stdio",
      "command": "node",
      "args": [
        "-y",
        "@cyanheads/evals-mcp-server",
        "run",
        "start:stdio"
      ],
      "env": {
        "EVALS_DATA_DIR": "<YOUR_EVALS_DATA_DIR>",
        "EVALS_REQUIRE_CONFIRMATION": "<YOUR_EVALS_REQUIRE_CONFIRMATION>",
        "EVALS_DEFAULT_LICENSE": "<YOUR_EVALS_DEFAULT_LICENSE>",
        "EVALS_CAPTURE_DIR": "<YOUR_EVALS_CAPTURE_DIR>",
        "MCP_LOG_LEVEL": "<YOUR_MCP_LOG_LEVEL>",
        "MCP_HTTP_HOST": "<YOUR_MCP_HTTP_HOST>",
        "MCP_HTTP_PORT": "<YOUR_MCP_HTTP_PORT>",
        "MCP_HTTP_ENDPOINT_PATH": "<YOUR_MCP_HTTP_ENDPOINT_PATH>",
        "MCP_AUTH_MODE": "<YOUR_MCP_AUTH_MODE>"
      }
    }
  }
}
Claude Code
claude mcp add evals-mcp-server --env EVALS_DATA_DIR=<YOUR_EVALS_DATA_DIR> --env EVALS_REQUIRE_CONFIRMATION=<YOUR_EVALS_REQUIRE_CONFIRMATION> --env EVALS_DEFAULT_LICENSE=<YOUR_EVALS_DEFAULT_LICENSE> --env EVALS_CAPTURE_DIR=<YOUR_EVALS_CAPTURE_DIR> --env MCP_LOG_LEVEL=<YOUR_MCP_LOG_LEVEL> --env MCP_HTTP_HOST=<YOUR_MCP_HTTP_HOST> --env MCP_HTTP_PORT=<YOUR_MCP_HTTP_PORT> --env MCP_HTTP_ENDPOINT_PATH=<YOUR_MCP_HTTP_ENDPOINT_PATH> --env MCP_AUTH_MODE=<YOUR_MCP_AUTH_MODE> -- node -y @cyanheads/evals-mcp-server run start:stdio

Related MCP servers

io.github.cyanheads/obsidian-mcp-server

Active

Read, write, search, and surgically edit Obsidian vault notes, tags, and frontmatter via MCP.

611
TypeScript
Apache-2.0
View repository →

io.github.cyanheads/git-mcp-server

Maintained

Comprehensive Git MCP server with 28 tools for clone, commit, branching, merging, and advanced workflows.

226
TypeScript
Apache-2.0
View repository →

io.github.cyanheads/mcp-ts-core

Active

Agent-native TypeScript framework for building MCP servers with declarative definitions.

145
TypeScript
Apache-2.0
View repository →

io.github.cyanheads/mcp-ts-template

Active

TypeScript template for building MCP servers with declarative tooling, observability, and auth.

145
TypeScript
Apache-2.0
View repository →

io.github.cyanheads/pubmed-mcp-server

Active

Search PubMed/Europe PMC, fetch articles and full text (PMC/EPMC/Unpaywall), citations, MeSH terms.

116
TypeScript
Apache-2.0
View repository →

io.github.cyanheads/clinicaltrialsgov-mcp-server

Active

Search ClinicalTrials.gov — find studies, retrieve results, match patients to eligible trials.

81
TypeScript
Apache-2.0
View repository →