novelty-assessment
lingzhi227/agent-research-skills
Assess research idea novelty through systematic literature search and harsh critic evaluation.
What is novelty-assessment?
Rigorously evaluates whether a research idea is novel by conducting multi-round literature searches across academic databases, applying a harsh critic persona to ensure meaningful contribution. Use this before committing time and resources to a research direction.
- Conducts up to 10 rounds of targeted literature searches across Semantic Scholar, arXiv, and OpenAlex
- Identifies core contributions and technical components of your research idea
- Reviews top results with abstracts to assess overlap with existing work
- Applies harsh critic criteria to distinguish meaningful contributions from trivial extensions
- Generates a binary novel/not-novel decision with confidence level and detailed justification
- Maps most similar papers and explains how your idea differentiates from each
How to install novelty-assessment
npx skills add https://github.com/lingzhi227/agent-research-skills --skill novelty-assessment- Python environment with required dependencies
- Access to Semantic Scholar, arXiv, or OpenAlex APIs (or web access)
- Research idea description, title, or JSON file as input
How to use novelty-assessment
- 1.Run the novelty check script with your research idea description and specify max search rounds (up to 10)
- 2.Review the multi-round search results showing top papers and overlap assessments
- 3.Examine the binary decision (novel or not-novel) with confidence level
- 4.Read the justification explaining which papers were found and why the idea is/isn't novel
- 5.Use the most similar papers list and differentiation summary for your related work section
Use cases
- Validate a research idea before starting a literature review or research proposal
- Identify the specific gap your work fills within an existing research area
- Prepare differentiation arguments for positioning your work in a paper's related work section
- Quickly assess whether a research direction is worth pursuing given existing literature
- Discover concurrent or recent submissions that may overlap with your planned work
- PhD students and researchers planning new research directions
- Academic writers preparing literature reviews and related work sections
- Research teams evaluating project proposals before resource allocation
- Conference or workshop paper authors validating contribution novelty
novelty-assessment FAQ
A novel idea must offer a meaningfully different approach, formulation, or insight—not just a minor modification of existing work. The skill applies harsh critic criteria to ensure sufficient contribution for a conference or workshop paper.
Three rounds allow sufficient exploration of the research space across different query angles and databases to confidently assess novelty. More rounds (up to 10) may be needed for complex or interdisciplinary ideas.
Yes, the skill explicitly checks for concurrent and recent arXiv submissions as part of its workflow to catch papers submitted around the same time as your research.
The skill will still conduct systematic searches and may declare your idea novel if no significantly overlapping papers exist after sufficient searching, but confidence may be lower in sparse areas.
Provide a clear description of your idea, including the core contribution and key technical components. You can pass a text description, title, or JSON file containing your idea details.
Full instructions (SKILL.md)
Source of truth, from lingzhi227/agent-research-skills.
name: novelty-assessment description: Assess research idea novelty through systematic literature search. Multi-round search-evaluate loops with harsh critic persona. Binary novel/not-novel decision with justification. Use before committing to a research direction. argument-hint: [idea]
Novelty Assessment
Rigorously assess whether a research idea is novel through systematic literature search.
Input
$0— Research idea description, title, or JSON file
Scripts
Automated novelty check
python ~/.claude/skills/idea-generation/scripts/novelty_check.py \
--idea "Your research idea description" \
--max-rounds 10 --output novelty_report.json
Literature search
python ~/.claude/skills/deep-research/scripts/search_semantic_scholar.py \
--query "relevant search query" --max-results 10
References
- Assessment prompts and criteria:
~/.claude/skills/novelty-assessment/references/assessment-prompts.md
Workflow
Step 1: Understand the Idea
- Identify the core contribution
- List the key technical components
- Determine the research area and subfield
Step 2: Multi-Round Literature Search (up to 10 rounds)
For each round:
- Generate a targeted search query
- Search Semantic Scholar / arXiv / OpenAlex
- Review top-10 results with abstracts
- Assess overlap with the idea
- Decide: need more searching, or ready to decide
Step 3: Make Decision
- Novel: After sufficient searching, no paper significantly overlaps
- Not Novel: Found a paper that significantly overlaps
Step 4: Position the Idea
If novel, identify:
- Most similar existing papers (for Related Work)
- How the idea differs from each
- The specific gap this idea fills
Harsh Critic Persona
Be a harsh critic for novelty. Ensure there is a sufficient contribution
for a new conference or workshop paper. A trivial extension of existing
work is NOT novel. The idea must offer a meaningfully different approach,
formulation, or insight.
Output Format
{
"decision": "novel" | "not_novel",
"confidence": "high" | "medium" | "low",
"justification": "After searching X rounds...",
"most_similar_papers": [
{"title": "...", "year": 2024, "overlap": "..."}
],
"differentiation": "Our idea differs because..."
}
Rules
- Minimum 3 search rounds before declaring novel
- Try to recall exact paper names for targeted queries
- A paper idea is NOT novel if it's a trivial extension
- Consider both methodology novelty AND application novelty
- Check for concurrent/recent arXiv submissions
Related Skills
- Upstream: literature-search, deep-research
- Downstream: idea-generation, research-planning
- See also: related-work-writing
Related skills
More from lingzhi227/agent-research-skills and the wider catalog.
paper-assembly
Orchestrate end-to-end paper pipelines with state management, checkpointing, and phase dependencies.
paper-compilation
Compile LaTeX papers to PDF with automatic error detection, style checking, and citation validation.
paper-revision
Systematically revise papers by mapping reviewer feedback to sections, executing targeted edits, and verifying improvements.
paper-to-code
Convert ML research papers into complete, runnable code repositories via structured planning, analysis, and generation.
paper-writing-section
Write publication-quality sections for academic papers with section-specific guidance and two-pass refinement.
rebuttal-writing
Write evidence-based rebuttals to peer review comments with structured point-by-point responses.