experiment-design
lingzhi227/agent-research-skills
Design structured, progressive experiment plans for research papers with staged baselines, hyperparameter sweeps, and ablation studies.
What is experiment-design?
This skill helps you design systematic experiment plans organized into four progressive stages: initial implementation, baseline tuning, creative research, and ablation studies. Use it when planning experiments for a research paper to ensure rigorous evaluation across datasets, metrics, and hyperparameter configurations.
- Generate 4-stage progressive experiment frameworks (initial implementation, baseline tuning, creative research, ablation studies)
- Plan baselines, datasets, hyperparameter grids, and evaluation metrics in structured JSON format
- Design ablation matrices to systematically analyze component contributions
- Specify multi-seed evaluation for statistical significance
- Output experiment designs compatible with downstream experiment-code and data-analysis skills
How to install experiment-design
npx skills add https://github.com/lingzhi227/agent-research-skills --skill experiment-design- Python 3.x (for design_experiments.py script)
- Basic understanding of your research method or idea to describe
How to use experiment-design
- 1.Describe your research idea, method, or plan as input to the skill
- 2.Run the design_experiments.py script with --method, --task, or --plan flags to generate structured output
- 3.Review the generated JSON output containing stages, baselines, datasets, metrics, and hyperparameter grids
- 4.Use the 4-stage framework to guide your implementation: start with Stage 1 (basic working implementation), progress through Stage 2 (baseline tuning), Stage 3 (creative improvements), and Stage 4 (ablation studies)
- 5.Document each experiment run and generate figures for training curves and comparisons as you execute the plan
Use cases
- Planning experiments for a machine learning research paper before implementation begins
- Designing hyperparameter sweep strategies across multiple datasets
- Creating ablation study matrices to validate novel method components
- Structuring baseline comparisons and creative improvements in a principled progression
- Documenting experiment stages with completion criteria and iteration limits
- ML researchers planning empirical studies
- PhD students designing thesis experiments
- Research engineers structuring paper submissions
- Teams coordinating multi-stage experimental validation
experiment-design FAQ
Stage 1 focuses on basic working implementation with simple datasets. Stage 2 tunes hyperparameters without changing architecture across 2+ datasets. Stage 3 explores creative improvements across 3+ datasets. Stage 4 performs systematic ablation studies on the same datasets as Stage 3.
No. Stage 2 is explicitly for hyperparameter tuning only (learning rate, epochs, batch size). Do not change the model architecture until Stage 3 (Creative Research).
Multi-seed evaluation (default 3 seeds) ensures statistical significance and accounts for random initialization variance in your results.
Stage 1 uses 1 simple dataset for basic correctness. Stage 2 tests on at least 2 datasets. Stages 3 and 4 use at least 3 datasets for comprehensive evaluation.
This skill is downstream from research-planning and idea-generation, and feeds into experiment-code and data-analysis skills for implementation and result analysis.
Full instructions (SKILL.md)
Source of truth, from lingzhi227/agent-research-skills.
name: experiment-design description: Design experiment plans with progressive stages — initial implementation, baseline tuning, creative research, and ablation studies. Plan baselines, datasets, hyperparameter sweeps, and evaluation metrics. Use when planning experiments for a research paper. argument-hint: [idea-or-plan]
Experiment Design
Design structured, progressive experiment plans for research papers.
Input
$0— Research idea, plan, or method description
References
- 4-stage progressive experiment prompts:
~/.claude/skills/experiment-design/references/stage-prompts.md
Scripts
Generate experiment design
python ~/.claude/skills/experiment-design/scripts/design_experiments.py --plan research_plan.json --output experiment_design.json
python ~/.claude/skills/experiment-design/scripts/design_experiments.py --method "contrastive learning" --task classification --format markdown
Generates baselines, ablation matrix, hyperparameter grid, metric selection. Stdlib-only.
4-Stage Progressive Framework (from AI-Scientist-v2)
Stage 1: Initial Implementation
- Focus on getting a basic working implementation
- Use a simple dataset
- Aim for basic functional correctness
- Completion: at least one working (non-buggy) implementation
Stage 2: Baseline Tuning
- Tune hyperparameters (learning rate, epochs, batch size)
- Do NOT change model architecture
- Test on at least TWO datasets
- Completion: stable training curves, improvement over Stage 1
Stage 3: Creative Research
- Explore novel improvements and insights
- Be creative and think outside the box
- Test on at least THREE datasets
- Completion: demonstrated novel improvement
Stage 4: Ablation Studies
- Systematic component analysis
- Each ablation tests a different aspect
- Use same datasets as Stage 3
- Completion: all planned ablations done
Output Format
{
"stages": [
{
"name": "initial_implementation",
"goals": ["Basic working baseline", "Simple dataset"],
"max_iterations": 5,
"completion_criteria": "Working implementation with non-zero accuracy"
}
],
"baselines": ["Method A", "Method B"],
"datasets": ["Dataset1", "Dataset2", "Dataset3"],
"metrics": ["accuracy", "F1", "inference_time"],
"ablation_components": ["component_A", "component_B"],
"hyperparameter_grid": {
"lr": [1e-4, 1e-3, 1e-2],
"batch_size": [32, 64, 128]
},
"num_seeds": 3
}
Rules
- Always start simple (Stage 1) before complex experiments
- Each stage builds on the best result from the previous stage
- Multi-seed evaluation for statistical significance
- Document every experiment run in notes.txt
- Generate figures for training curves and comparisons
Related Skills
- Upstream: research-planning, idea-generation
- Downstream: experiment-code, data-analysis
- See also: paper-assembly
Related skills
More from lingzhi227/agent-research-skills and the wider catalog.
figure-generation
Generate publication-quality scientific figures with matplotlib/seaborn using query expansion, code generation, and VLM visual feedback.
github-research
Discover and analyze GitHub repositories for research topics using deep-research output and multi-source search.
idea-generation
Generate and refine novel research ideas with literature-backed novelty scoring.
latex-formatting
Set up conference LaTeX templates and fix formatting issues for academic paper submission.
literature-review
Conduct comprehensive literature reviews through multi-perspective expert dialogue and systematic paper synthesis.
literature-search
Search academic literature across Semantic Scholar, arXiv, and OpenAlex with structured results and BibTeX export.