data-throughput-accelerator
affaan-m/ecc
Optimize large-scale data pipelines—ingestion, ETL, warehouse loading—while maintaining correctness and auditability.
What is data-throughput-accelerator?
This skill helps you diagnose and accelerate data movement bottlenecks in ingestion, backfill, export, and synchronization workflows. Use it when you need to move large volumes of data faster without sacrificing accuracy or losing track of what was processed.
- Identify which stage of your pipeline is the actual bottleneck: extraction, network, warehouse load, transformation, or freshness lag
- Apply fast-path heuristics: move compute to data, use warehouse-native operations, skip completed partitions via manifests, and batch small operations
- Design idempotent writes using unique keys, manifests, or replaceable staging to prevent duplicates and enable safe retries
- Benchmark variants systematically: batch sizes, worker counts, SQL patterns, file grouping, and manifest strategies
- Produce hard accounting output showing source files, rows processed, remaining work, runtime, and correctness verification
- Codify the optimized path as a CLI, scheduled job, or runbook with replay evidence for regulated data
How to install data-throughput-accelerator
npx skills add null --skill data-throughput-acceleratorHow to use data-throughput-accelerator
- 1.Read and document your current source, target, and manifest contracts (schemas, partitioning, update logic)
- 2.Measure the backlog: count external files, manifest rows, raw rows, derived rows, and identify min/max timestamps
- 3.Run a safe catch-up or sample benchmark on a subset to establish a baseline
- 4.Test variants: different batch sizes, worker counts, warehouse SQL patterns, file grouping, and manifest update methods
- 5.Compare results and promote only the fastest variant that keeps row counts and timestamps coherent
- 6.Codify the optimized path as a CLI command, scheduled job, or runbook with clear steps
- 7.Execute the codified path and run final accounting to verify source files, rows added, remaining tail, runtime, and correctness gates
Use cases
- Accelerating a daily ETL that lags behind incoming data, causing stale serving tables
- Backfilling a data warehouse with years of historical files while tracking which files have been processed
- Exporting large result sets from a warehouse to cloud storage in parallel batches without duplicating rows
- Synchronizing two tables across systems by identifying and processing only new or changed partitions
- Catching up a manifest-driven pipeline after a processing outage, skipping already-completed work
- Data engineers optimizing ETL and warehouse pipelines
- Analytics engineers backfilling or refreshing large datasets
- Platform engineers managing high-volume data ingestion systems
- Anyone responsible for data correctness and auditability in regulated or customer-impacting contexts
data-throughput-accelerator FAQ
Separate live-tail freshness from backlog catch-up. The bottleneck may not be your processing speed but the rate of incoming data. Measure both independently and optimize the right one.
Use manifests or checkpoints to track completed files and partitions. Query the manifest before each run to skip already-processed work, and update it atomically after successful processing.
Design your writes so they can be safely retried without creating duplicates. Use unique keys to deduplicate, replace entire staging tables instead of appending, or maintain a manifest of processed files that prevents re-insertion.
If your data lives in a warehouse and you're extracting it to transform elsewhere, consider using warehouse-native SQL for joins, aggregations, and appends instead. This avoids network transfer and is often faster.
Document source files discovered, files processed in this run, raw and derived rows added, remaining unprocessed work, total runtime, and a correctness gate (e.g., manifest counts match table max timestamps).
Full instructions (SKILL.md)
Source of truth, from affaan-m/ecc.
name: data-throughput-accelerator description: Use when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness. metadata: origin: ECC tools: Read, Write, Edit, Bash, Grep, Glob
Data Throughput Accelerator
Use this skill when the bottleneck is moving, transforming, or saving lots of data. The goal is not just speed. The goal is faster correct data landing in the right place with proof.
First Distinction
Separate these before optimizing:
- source extraction speed;
- network transfer speed;
- warehouse/load speed;
- transform speed;
- serving-table freshness;
- live tail growth while the job runs.
A pipeline can be "fast" and still appear behind if new data arrives faster than the final catch-up window.
Fast Path Heuristics
- Move compute to where the data already is.
- Prefer warehouse-native scans, joins, and appends for large landed files.
- Use manifests or checkpoints so completed files/partitions are skipped.
- Use partitioning and clustering that match the read and append pattern.
- Batch small files, requests, and writes.
- Make writes idempotent through unique keys, manifests, or replaceable staging.
- Keep raw, derived, and serving tables separately accountable.
Workflow
- Read the current source, target, and manifest contracts.
- Measure backlog: external files, manifest rows, raw rows, derived rows, min/max timestamps, and unprocessed counts.
- Run a safe catch-up or sample benchmark.
- Compare variants: batch size, worker count, warehouse SQL, file grouping, staging shape, and manifest update method.
- Promote only the fastest path that keeps counts and timestamps coherent.
- Codify the path as a CLI, scheduled job, workflow, or runbook.
- Rerun final accounting after the codified path executes.
Accounting Output
Use a hard accounting block:
Data throughput result:
- Source files discovered: 294
- Files processed this run: 294
- Raw rows added: 9,683,598
- Derived rows added: 8,917,585
- Remaining tail: 24 files at readback time
- Runtime: 38.7s
- Correctness gate: manifest counts and table max timestamps match
Guardrails
- Do not delete raw data to make a metric look better.
- Do not skip failed files silently.
- Do not mix historical backfill status with live-tail freshness.
- Do not call a pipeline complete until the target tables and manifest agree.
- For finance, healthcare, regulated, or customer-impacting data, preserve replay evidence and approval gates.
Related skills
More from affaan-m/ecc and the wider catalog.
database-migrations
Safe, reversible database schema changes and zero-downtime migrations for PostgreSQL, MySQL, and major ORMs.
deep-research
Multi-source research reports with citations using firecrawl and exa web search MCPs.
defi-amm-security
Security checklist and hardened patterns for Solidity AMM contracts, liquidity pools, and swap functions.
delivery-gate
Agent skill from affaan-m/ecc.
deployment-patterns
CI/CD pipelines, Docker containerization, and deployment strategies for production web applications.
design-system
Generate, audit, and maintain cohesive design systems; detect visual inconsistencies and AI-generated design patterns.