indexion-segment
trkbt10/indexion-skills
Split text into contextual chunks for RAG/embedding pipelines using divergence, TF-IDF, or punctuation strategies.
What is indexion-segment?
Segments documents into meaningful chunks optimized for retrieval-augmented generation and embedding workflows. Choose from window divergence (default), TF-IDF topic detection, punctuation-based, or hybrid strategies to match your content type and use case.
- Split documents using sliding window divergence detection to find natural semantic boundaries
- Detect topic changes via TF-IDF scoring for content-aware segmentation
- Segment by punctuation and sentence boundaries for simple, fast splitting
- Combine NCD and TF-IDF in hybrid mode for improved accuracy on mixed-content documents
- Tune segment size (min/max/target) and divergence thresholds to control granularity
- Output numbered segment files with customizable prefix for downstream processing
How to install indexion-segment
npx skills add https://github.com/trkbt10/indexion-skills --skill indexion-segmentHow to use indexion-segment
- 1.Run `indexion segment <input-file> <output-dir>` with your document and desired output directory
- 2.Review the generated segment files to assess chunk quality and boundaries
- 3.If segments are too large or small, adjust `--target-size` (default 500) and `--min-size`/`--max-size` parameters
- 4.For topic-based splitting, switch to `--strategy=tfidf` or enable `--hybrid` mode for mixed-content documents
- 5.Fine-tune `--threshold` (default 0.42) to control divergence sensitivity; lower values create more segments
- 6.Use `--prefix=NAME` to customize output file naming for integration with downstream tools
Use cases
- Prepare long documents for vector embedding and semantic search in RAG systems
- Split research papers or technical docs into sections for citation-aware retrieval
- Chunk customer support articles into Q&A pairs for chatbot training
- Segment news articles or blog posts by topic for content recommendation
- Break down legal or compliance documents into clause-level chunks for compliance checking
- RAG/LLM pipeline engineers
- Information retrieval and search specialists
- Document processing and data preparation teams
- Embedding and vector database practitioners
- Content management and knowledge base builders
indexion-segment FAQ
Window (default) uses sliding window divergence to detect semantic boundaries; tfidf detects topic changes via term frequency; punctuation splits on sentence/paragraph marks. Window is best for general documents, tfidf for topic-heavy content, punctuation for speed.
Use `--hybrid` when your document mixes different content types (e.g., narrative + tables + code). It combines NCD divergence with TF-IDF weighting for better accuracy. Adjust `--ncd-weight` and `--tfidf-weight` to balance strategies.
Use `--target-size` (ideal chunk size, default 500), `--min-size` (floor, default 100), and `--max-size` (ceiling, default 2000). The algorithm respects these bounds while finding semantic boundaries.
Threshold (default 0.42) controls divergence sensitivity. Lower values create more, smaller segments; higher values merge more content. Use `--adaptive` (default) for automatic threshold adjustment per document.
Yes, use `--prefix=NAME` to change the output prefix (default 'segment'). Files are numbered sequentially (e.g., segment_1.txt, segment_2.txt).
Full instructions (SKILL.md)
Source of truth, from trkbt10/indexion-skills.
name: indexion-segment description: Split text into contextual chunks for RAG/embedding pipelines. Document segmentation and section extraction using window, tfidf, punctuation, or hybrid strategies chosen by intent.
indexion segment
Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.
When to Use
- User needs to chunk text for RAG or embedding pipelines
- User wants to split a document into meaningful sections
- User asks to segment text for processing
- Preparing text for similarity analysis at sub-document level
Usage
# Default window divergence strategy
indexion segment <input-file> <output-dir>
# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>
# Punctuation-based segmentation
indexion segment --strategy=punctuation <input-file> <output-dir>
# Custom segment sizes
indexion segment --min-size=200 --max-size=3000 --target-size=800 document.txt output/
# Custom divergence threshold
indexion segment --threshold=0.5 document.txt output/
# Adaptive threshold mode (default)
indexion segment --adaptive document.txt output/
# Hybrid NCD+TF-IDF mode
indexion segment --hybrid --ncd-weight=0.6 --tfidf-weight=0.4 document.txt output/
# Custom window size
indexion segment --window-size=5 document.txt output/
# Custom output prefix
indexion segment --prefix=chunk document.txt output/
Options
| Option | Default | Description |
|---|---|---|
--strategy=NAME | window | Strategy: window, tfidf, punctuation |
--min-size=INT | 100 | Minimum segment characters |
--max-size=INT | 2000 | Maximum segment characters |
--target-size=INT | 500 | Target segment characters |
--threshold=FLOAT | 0.42 | Divergence threshold |
--window-size=INT | 3 | Window size |
--adaptive | true | Adaptive threshold mode |
--hybrid | false | NCD+TF-IDF hybrid mode |
--ncd-weight=FLOAT | 0.5 | NCD weight in hybrid mode |
--tfidf-weight=FLOAT | 0.5 | TF-IDF weight in hybrid mode |
--prefix=NAME | segment | Output file prefix |
Strategies
| Strategy | Description |
|---|---|
window (default) | Sliding window divergence detection |
tfidf | TF-IDF based topic change detection |
punctuation | Punctuation/sentence boundary based |
Workflow
- Run
indexion segment <input-file> <output-dir>to split text with defaults - Adjust
--thresholdand--target-sizeto tune segmentation granularity - Use
--hybridmode for better accuracy on mixed-content documents
Related skills
More from trkbt10/indexion-skills and the wider catalog.
indexion-readme
Initialize, generate, and assemble project READMEs from templates, doc comments, and configuration.
indexion-sdd
Generate SDD requirements from RFCs/specs and verify implementation conformance with quantitative drift detection.
indexion-refactor
Detect and eliminate code duplication at textual, structural, and conceptual levels using indexion analysis.
indexion-documentation
Assess documentation coverage, detect code-to-doc drift, and visualize module dependencies.
indexion-kgf
Debug KGF specs by inspecting tokenization, parse events, and extracted edges from source files.
indexion-wiki
Maintain project wikis with change tracking, structural linting, and code-to-doc drift detection.