cx-cost-optimization

name: cx-cost-optimization description: > Use this skill when the user asks to "check data usage", "list TCO policies", "reduce Coralogix costs", "optimize observability spend", "lower our logging bill", "data budget exceeded", "TCO policy", "retention tier", "archive storage", "ingestion costs", "frequent search vs archive", "why is our bill so high", "spending too much on logs", "data retention settings", "cost analysis", "usage breakdown", "optimize log volume", "control data ingestion", "archive cold data", "billing units", "plan consumption", "daily plan", "overage", "PAYG", "usage anomaly", "usage trend", "cx_data_usage_units", or wants to investigate, analyze, or reduce Coralogix data costs. metadata: version: "0.1.0"

Cost Optimization Skill

Use this skill when investigating or reducing Coralogix data costs. It covers the full cost management lifecycle: measuring current spend, reviewing TCO policies, adjusting retention periods, and configuring archive storage for cold data.

CLI Commands

Key flags:

• All commands support -o json for structured output and -p <profile> for profile selection
• cx usage daily accepts --type processed-gbs|units|evaluation-tokens and --start/--end time filters
• cx usage summary accepts --start/--end time filters
• cx usage logs-count and cx usage spans-count accept --start/--end time filters, defaulting to the last 24h, plus --resolution (default 1h), --subsystem-aggregation, --application-aggregation, and repeated --param KEY=VALUE for API filter query params
• Data usage summary and count endpoints are documented as newline-delimited JSON over Accept: text/event-stream; the CLI handles that transport and normalizes count chunks into .result.logsCount[] or .result.spansCount[].
• cx usage capabilities is the required first step: it returns the labels, measurements, units, and request limits that the public Data Usage Query API currently supports for the selected tenant
• cx usage query is the required second step: submit only a JSON request derived from the immediately preceding capabilities response with exactly one of --query '<json>' or --from-file <path> (--from-file - reads stdin)
• cx tco create/update, cx retentions update, cx archive logs set, cx archive metrics create/update/validate use --from-file <path> (or - for stdin)

Authoritative Billable Usage Queries

For billable totals, quota units, plan consumption, or a supported usage breakdown, use this mandatory two-step workflow:

1. Run cx usage capabilities in the current session.
2. Build and run cx usage query using only that response.

Do not call cx usage query first and do not guess labels, measurement kinds, units, filter values, intervals, or request limits. They are tenant-specific and can change.

# Inspect the currently valid dimensions and constraints
cx usage capabilities -o json

# Submit a capabilities-derived request inline
cx usage query --query '{"daily":{"relativeRange":"DAILY_RELATIVE_RANGE_LAST_7_DAYS"}}' -o json

# Or read the same request from a file or stdin
cx usage query --from-file usage-query.json -o json
printf '%s' '{"daily":{"relativeRange":"DAILY_RELATIVE_RANGE_LAST_7_DAYS"}}' \
  | cx usage query --from-file - -o json

Load references/data-usage-query-api.md before creating the query body. It defines the capability and response schemas, valid interval forms, and limits.

Cost Investigation Workflow

Follow these steps to diagnose and reduce costs:

Step 1: Measure Current Usage

For billable totals, quota units, plan consumption, or supported breakdowns, first follow the mandatory two-step workflow above:

cx usage capabilities -o json
cx usage query --from-file usage-query.json -o json

Build usage-query.json only from the immediately preceding capabilities response. Do not use the legacy commands below as a substitute for an authoritative billing answer.

For a legacy consumption overview and log/span record counts, use:

cx usage summary -o json
cx usage summary --start now-30d -o json
cx usage daily --type processed-gbs --start now-7d -o json
cx usage logs-count --start now-7d --end now -o json
cx usage spans-count --start now-7d --end now -o json

Identify which data types consume the most volume. Use jq to sort:

cx usage summary -o json | jq '[.[] | {name, daily_avg: .avg_daily_gb}] | sort_by(.daily_avg) | reverse'

Step 2: Review TCO Policies

cx tco list -o json
cx tco settings -o json

TCO policies control which logs go to Frequent Search (expensive, fast) vs. Archive (cheap, slower). Check if high-volume, low-value logs are on Frequent Search:

cx tco list -o json | jq '.[] | select(.priority == "LOW") | {name, application, subsystem, archive_retention}'

Step 3: Check Retention Settings

cx retentions list -o json
cx retentions status -o json

Long retention periods increase storage costs. Identify indices with unnecessarily long retention.

Step 4: Check Archive Configuration

cx archive logs get -o json
cx archive metrics get -o json

Verify that archive storage is configured for cold data. If no archive is set up, that's a cost-saving opportunity.

Step 5: Recommend Optimizations

Based on findings, recommend changes in priority order (highest impact first).

Common Optimization Patterns

jq Examples

Usage Analysis

# Top consumers by daily volume
cx usage summary -o json | jq '[.[] | {name, daily_avg: .avg_daily_gb}] | sort_by(.daily_avg) | reverse | .[0:10]'

# Daily trend for the past week
cx usage daily --type processed-gbs --start now-7d -o json | jq '[.[] | {date, gb: .processed_gbs}]'

# Total logs and spans counts
cx usage logs-count --start now-7d --end now -o json | jq '[.result.logsCount[]?.logsCount | tonumber] | add // 0'
cx usage spans-count --start now-7d --end now -o json | jq '[.result.spansCount[]? | ((.successSpanCount | tonumber) + (.errorSpanCount | tonumber) + (.lowSuccessSpanCount | tonumber) + (.lowErrorSpanCount | tonumber) + (.mediumSuccessSpanCount | tonumber) + (.mediumErrorSpanCount | tonumber))] | add // 0'

TCO Policy Analysis

# Policies routing to archive tier
cx tco list -o json | jq '[.[] | select(.archive_retention != null)]'

# Policies by priority
cx tco list -o json | jq 'group_by(.priority) | map({priority: .[0].priority, count: length})'

# Test if a log pattern matches a policy
cx tco test --from-file test-definition.json -o json

Retention Review

# All retention settings
cx retentions list -o json | jq '.[]'

# Check if retention is active
cx retentions status -o json

Archive Status

# Logs archive configuration
cx archive logs get -o json | jq '{active: .active, bucket: .bucket}'

# Metrics archive configuration
cx archive metrics get -o json | jq '{enabled: .enabled, bucket: .bucket}'

Applying Changes

IMPORTANT: NEVER pass --yes without explicit user approval. All write operations across archive, TCO, and retentions require interactive confirmation and the --yes flag to execute non-interactively. Before executing any write operation, describe the exact change to the user and wait for their approval before passing --yes.

Read-only mode: Use --read-only (or CX_READ_ONLY=1) to safely explore cost data without risk of accidental writes. All query commands (usage, tco list/get, retentions list, archive get) work normally in read-only mode.

Agent mode: When running inside an AI agent, cx fails fast on write operations instead of hanging on a stdin prompt. Get user confirmation first, then re-run with --yes.

When modifying TCO policies, retention, or archive:

1. Template from existing: Get the current configuration as JSON, modify it, then apply:

   cx tco get <policy-id> -o json > policy.json
   # Edit policy.json
   cx tco update --from-file policy.json
   

2. Verify after changes: Re-run the diagnosis commands to confirm the change took effect.

3. TCO policy ordering matters: Use cx tco reorder --from-file to set priority order. Policies are evaluated top-to-bottom; the first match wins.

Metrics-Based Cost Analysis

The cx usage API gives summaries, but for billing-accurate analysis, anomaly detection, and breakdown by pillar/feature, query the customer metrics exporter via PromQL.

Key Metrics

Concept-to-Metric Mapping

• Billing / plan usage / consumption -> cx_data_usage_units + cx_data_plan_units_per_day
• Processed bytes / data volume -> cx_data_usage_total
• AI evaluation tokens -> cx_data_usage_tokens_total
• Metric samples -> cx_data_usage_samples_total
• Overage / PAYG -> cx_data_usage_payg_units

Common PromQL Queries

# Today's billable units consumed so far
cx metrics query 'sum(cx_data_usage_units)' --time now -o json

# Units breakdown by pillar
cx metrics query 'sum by (pillar) (cx_data_usage_units)' --time now -o json

# Daily plan quota
cx metrics query 'cx_data_plan_units_per_day' --time now -o json

# Plan consumption percentage
cx metrics query '100 * sum(cx_data_usage_units) / cx_data_plan_units_per_day' --time now -o json

# Units by feature group
cx metrics query 'sum by (feature_group_id) (cx_data_usage_units)' --time now -o json

# PAYG overage (if any)
cx metrics query 'cx_data_usage_payg_units' --time now -o json

UTC-Day Bucketing Rules

All usage metrics accumulate from UTC midnight and reset at 00:00 UTC:

• An instant query during the day returns "today so far"
• For completed-day totals, use the last sample before midnight
• Never subtract values across a UTC midnight boundary
• For weekly/monthly analysis, derive completed daily totals first, then roll up
• Exclude the current partial UTC day when computing trends or averages

Anomaly Detection

When investigating usage anomalies:

1. Compare completed UTC days (exclude current partial day)
2. Break down by: measurement_type -> pillar -> entity_type -> priority -> feature_group_id -> application_name -> subsystem_name
3. Prefer same-weekday comparisons for seasonal traffic
4. Use cx_data_usage_units for billing anomalies, cx_data_usage_total for volume anomalies

Breakdown Labels

Usage metrics support these grouping dimensions: pillar, entity_type, priority, measurement_type, feature_group_id, feature_id, application_name, subsystem_name.

Key Principles

• Measure before changing - always run usage/summary commands before modifying policies
• Use -o json with jq - structured output enables precise analysis
• Verify changes - re-query after every modification to confirm it took effect
• Multi-profile awareness - use -p <profile> or --all-profiles to compare costs across environments
• Template from existing - get current config as JSON before creating or updating
• TCO is the biggest lever - moving logs from Frequent Search to Archive tier has the largest cost impact

Related Skills

• cx-telemetry-querying - investigate what data is being ingested (query logs, metrics, and spans to identify high-volume sources)

Reference Files

• references/data-usage-query-api.md - capabilities and query schemas, interval rules, limits, and response interpretation