recsys-pipeline-architect
affaan-m/ecc
Design composable recommendation and ranking pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework.
What is recsys-pipeline-architect?
A spec-and-scaffold skill for building feed, ranking, and recommendation systems that pick top-K items for a user and context. Use this when designing social feeds, content CMSs, RAG rerankers, task prioritizers, search reranking, or any system that needs to rank candidates and serve personalized results.
- Walk through eight-step workflow to clarify use case, identify candidate sources, design hydration and filtering stages, compose scorer chains, and generate runnable scaffolds
- Encode the six-stage pattern (Source → Hydrator → Filter → Scorer → Selector → SideEffect) with clear ordering rationale and parallelization rules
- Surface key trade-offs: single score vs. multi-action prediction, isolated vs. joint candidate scoring, and online vs. offline computation
- Provide reference implementations and cookbooks for filters (12 patterns), scorers (weighted sum, MMR, diversity, position debiasing), and multi-action prediction
- Generate production-ready scaffolds in TypeScript, Go, or Python with test suites that pass
- Enforce hard rules: no benchmark invention, proper attribution to xAI For You algorithm, fire-and-forget side effects, and filter ordering discipline
How to install recsys-pipeline-architect
npx skills add null --skill recsys-pipeline-architect- Familiarity with your target runtime (TypeScript, Go, or Python)
- A scoring function or ML model (the skill provides plumbing around it, not the model itself)
- Understanding of your candidate sources and what metadata is available at each stage
How to use recsys-pipeline-architect
- 1.Clarify your use case: what items are you ranking, what is the input context, and what is your runtime?
- 2.Identify candidate sources (in-network, out-of-network, ML retrieval, trending, etc.) and whether they run in parallel
- 3.List required hydrations: what metadata does each filter and scorer need that the source does not provide?
- 4.Design your filter chain in order of cost: cheap universal filters first (duplicates, self), then expensive user-specific filters (block/mute, eligibility)
- 5.Compose your scorer chain: primary ML scorer, multi-action combiner with weights, diversity reranking, business rules
- 6.Choose your selector: sort by final score, take top K, or use stratified sampling for in-network/out-of-network mix
- 7.Design side effects: cache served IDs, emit impression events, update counters — all async and non-blocking
- 8.Generate the scaffold for your stack and integrate your scoring function and data sources
Use cases
- Build a social media feed that ranks posts by engagement, diversity, and business rules without retraining on each weight change
- Design a RAG reranker that filters irrelevant documents, scores by relevance and recency, and caches results for repeated queries
- Create a task prioritizer that sources tasks from multiple queues, filters by eligibility, scores by urgency and user context, and emits impression events
- Migrate from a single relevance score to multi-action prediction (read, like, share, skip, report) with tunable weights for A/B testing
- Build a notification triage system that hydrates candidates with user preferences, filters by opt-in status, scores by urgency and personalization, and logs impressions
- Backend engineers building recommendation or ranking systems
- ML engineers integrating scoring models into production pipelines
- Product engineers designing feed algorithms or personalization features
- Developers migrating from monolithic rankers to composable, multi-stage pipelines
recsys-pipeline-architect FAQ
Single score is simpler but requires retraining to change behavior. Multi-action (predict P(read), P(like), P(share), etc., then combine with weights) lets you tune behavior at serving time without retraining. Use multi-action if you expect frequent tuning or have conflicting objectives (engagement vs. safety vs. diversity).
Default to independent scoring: it is deterministic, cacheable, and composes with reranking stages. Use joint scoring only if you have a specific reason, such as explicit batch-aware diversity constraints.
Online (request-time) is the default: lower latency budget is 100–300ms and freshness is high. Offline (batch pre-computed) is lower latency but stale. Hybrid (retrieve offline, rank online) is common for large-scale systems.
Cheap before expensive, universal before user-specific. Example: duplicates and self-filters first, then block/mute, then eligibility checks. This minimizes wasted compute on candidates that will be dropped.
No. Side effects must always be fire-and-forget (goroutines, promises without await, asyncio tasks). Blocking side effects will degrade user-facing latency and are an anti-pattern.
Full instructions (SKILL.md)
Source of truth, from affaan-m/ecc.
name: recsys-pipeline-architect description: Design composable recommendation, ranking, and feed pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework popularized by xAI's open-sourced For You algorithm. Use this skill whenever the user is building any system that picks "the top K items for a (user, context)" — social feeds, content CMSs, RAG rerankers, task prioritizers, notification triage, search reranking, ad ranking. metadata: origin: community
recsys-pipeline-architect
A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. It encodes the six-stage pattern — Source → Hydrator → Filter → Scorer → Selector → SideEffect — popularized by xAI's open-sourced For You algorithm (Apache 2.0). This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.
Upstream: https://github.com/mturac/recsys-pipeline-architect
When to Use
- User wants to build any system that picks "the top K items for a user/context"
- User asks "how should I rank X" or describes a feed/personalization problem
- User has a scoring function and needs the pipeline plumbing around it
- User wants to migrate from a single relevance score to multi-action prediction with tunable weights
- User is wrapping an LLM/ML scorer and needs filters, hydrators, side-effects, and a runnable scaffold in their stack (TypeScript / Go / Python)
- Triggers: "recommendation system", "feed algorithm", "ranking pipeline", "for you feed", "candidate pipeline", "content recommender", "pipeline architecture for recsys", "RAG retrieval reranker"
When NOT to Use
- Model architecture work (transformer design, two-tower retrieval, embedding training) — this skill is plumbing around the model, not the model itself
- Pure ML training pipelines — the scoring function is the user's responsibility
- Operating a deployed pipeline (monitoring, autoscaling) — out of scope
The six-stage framework
| # | Stage | Job | Parallel? |
|---|---|---|---|
| 1 | Source | Fetch candidates from one or more origins | Yes — multiple sources run in parallel |
| 2 | Hydrator | Enrich each candidate with metadata needed for filtering and scoring | Yes — independent hydrators run in parallel |
| 3 | Filter | Drop candidates that should never be shown (blocked, expired, duplicate, ineligible) | Sequential — each filter sees fewer items |
| 4 | Scorer | Assign each surviving candidate one or more scores | Sequential — later scorers see earlier scores |
| 5 | Selector | Sort by final score, return top K | Single op |
| 6 | SideEffect | Cache served IDs, log impressions, emit events, update counters | Async — must never block the response |
Why this exact order
- Sources before hydration: know what candidates exist before paying to enrich them
- Hydration before filtering: many filters need metadata the source did not provide
- Filtering before scoring: scoring is the expensive stage; drop the ineligible first
- Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
- Selector after scoring: keeps scoring deterministic and cacheable
- SideEffects last and async: side effects must never block the user response
Workflow when invoked
Walk the user through these eight steps:
- Clarify the use case (one round, three questions): items being ranked? input context? language/runtime?
- Identify the candidate sources: usually in-network (followed/owned/subscribed) + out-of-network (ML retrieval / trending / similar-to-liked)
- List required hydrations: for each filter and scorer, what data does it need that the source did not provide?
- List the filters: duplicate, self, age, block/mute, previously-served, eligibility. Order matters — cheap before expensive.
- Design the scorer chain: primary (ML) → combiner (multi-action with weights) → diversity → business rules
- Selector: sort descending by final score, take top K (or stratified mix for in-network/out-of-network)
- SideEffects: cache served IDs, emit impression events, update counters, log analytics — all fire-and-forget
- Generate the scaffold in the user's stack
Key trade-offs to surface (don't default silently)
1. Single score vs multi-action prediction
- Single score: train one model to predict relevance. To change behavior → retrain.
- Multi-action: predict
P(action)for many actions (read, like, share, skip, report), combine with weights at serving time. To change behavior → change weights. No retraining.
The X For You system uses multi-action with both positive and negative weights. Recommend multi-action when the user expects to tune frequently.
2. Candidate isolation in scoring
- Isolated: each candidate scored independently. Deterministic, cacheable.
- Joint: candidates attend to each other during scoring (e.g., transformer over batch). More expressive but non-deterministic across batches.
Default to isolation. Joint only when there's a specific reason (e.g., explicit batch-aware diversity).
3. Online vs offline
- Request-time (online): pipeline runs on each request. Latency budget: 100–300ms. Default.
- Pre-computed (offline batch): pipeline runs periodically, results cached. Lower latency, lower freshness.
- Hybrid: candidate retrieval offline, ranking online.
Hard rules
- Do not invent benchmark numbers. "How much faster?" → "depends on workload, run it yourself."
- Attribution discipline. When the pattern is referenced, attribute as "popularized by xAI's open-sourced For You algorithm" /
github.com/xai-org/x-algorithm(Apache 2.0). - No trademark use. Do not name the user's artifact "X-like" or use "For You" branding. Pattern is free; brand is not. Suggested naming: "candidate pipeline", "feed pipeline", "ranking pipeline", "recsys pipeline".
- Surface trade-offs. Multi-action vs single, isolation vs joint, online vs offline — never default silently.
- The generated scaffold must run. No pseudocode passing as code.
- Filter order matters. Cheap before expensive. Universal before user-specific.
- Side effects never block. Wrap in fire-and-forget patterns (goroutines / promises without await / asyncio tasks).
Anti-Patterns
- Scoring before filtering (wastes compute on candidates that will be dropped anyway)
- Synchronous side effects (cache writes / impression emits blocking the response)
- A single "relevance" score when the product needs to tune for multiple objectives (engagement vs safety vs diversity vs ads)
- Joint scoring as default (non-deterministic, harder to cache, doesn't compose with reranking stages)
- Generating pseudocode "for illustration" — the scaffold must actually run
Upstream contents
The upstream repository at https://github.com/mturac/recsys-pipeline-architect ships:
- Full
SKILL.mdwith the complete 8-step workflow - 5 load-on-demand reference docs: interfaces in 4 languages (TS/Go/Python/Rust), multi-action scoring pattern, candidate isolation, filter cookbook (12 patterns), scorer cookbook (weighted sum, MMR, diversity penalty, position debiasing)
- 3 runnable example scaffolds, every one green on its test suite:
- Strapi v5 plugin (TypeScript / Jest — 3/3 pass)
- Zentra-compatible pipeline (Go with generics — 3/3 pass)
- PMAI task prioritizer (Python / FastAPI / pytest — 3/3 pass)
- v0.1.0 release tagged
- MIT license; pattern attributed to xAI X For You algorithm (Apache 2.0)
Install via skills.sh: npx skills add mturac/recsys-pipeline-architect
Related skills
More from affaan-m/ecc and the wider catalog.
recursive-decision-ledger
Recursive decision-making with explicit evidence trails and promotion gates for high-dimensional search and stochastic optimization.
redis-patterns
Redis patterns for caching, rate limiting, locks, sessions, and pub/sub in production.
regex-vs-llm-structured-text
Choose regex for structured text (95%+ accuracy), add LLM only for low-confidence edge cases to cut costs by ~95%.
remotion-video-creation
Best practices for building videos in React with Remotion—29 rules covering animations, audio, captions, 3D, and more.
repo-scan
Cross-stack source code audit: classify files, detect embedded libraries, assign refactoring verdicts.
research-ops
Evidence-first research workflow for current facts, comparisons, and recommendations.