Metrics Skill - Analyzing metrics via Coralogix/PromQL

This skill describes how to discover and analyze metrics using your available tools and explains how to use PromQL for that goal.

Core Workflow:

1. Initial Assessment

When a user reports a problem (e.g., "I have production issues"):

• If the problem is vague or broad, ASK 1-2 CLARIFYING QUESTIONS to get specifics
• Focus on: What exactly is failing? When did it start? What symptoms are users seeing?
• ONLY ask clarifying questions if absolutely necessary - prefer to start investigating immediately

2. Problem Decomposition

Once you understand the issue (e.g., "users get HTTP error unavailable for the last 30 minutes"):

• Break down the problem into SMALL, SPECIFIC questions that can each be answered with a single metric query
• Think systematically about layers: Application → Services → Infrastructure → Dependencies
• Generate questions like:
  • "Are there any K8S deployments down in the last 30 minutes?"
  • "What's the p99 latency in the ELB?"
  • "Are there any pods with high restart counts?"
  • "What RDS instances exist and are they healthy?"

3. Metric Discovery

• ALWAYS start by using search_relevant_metrics tool to find metrics related to the subject the user is measuring (e.g. "requests", "errors", "latency") — NOT the dimension they want to slice by (e.g. "user_email", "account_id"). If that metric doesn't carry the dimension, see "Handling a missing filter label" below — but don't go looking there unless a needed label is actually missing.
• Analyze the returned metrics and their descriptions to identify the most appropriate ones
• If no relevant metrics are found, try alternative search terms or broader keywords
• When two metric names differ only by a _count (or _sum) suffix, prefer the base metric. This follows Prometheus naming conventions: _count and _sum are the components a histogram or summary emits, where _count is the number of observations recorded — not the quantity being measured. Treat this as a heuristic, not a hard rule: if the user explicitly wants the number of observations, _count is the correct choice.
• Match the metric to what the user is measuring, not to whichever metric happens to carry their filter label. Identify the action or subject in the user's question (e.g. "requests served", "errors", "logins", "bytes sent") and pick the metric whose description reflects that outcome — even if a different metric is the one that exposes the label they want to slice by. Example: if the user asks "how many requests came from premium accounts?" and http_requests_total measures requests but lacks an account_tier label, while sessions_total has account_tier but counts sessions, still measure with http_requests_total and bring in the label via a join (see "Handling a missing filter label" below). Switching to sessions_total just because it has the label silently changes what you're counting (sessions ≠ requests). However you discover the bridge metric, the dimension filter must be applied inside a single PromQL on()/group_left() join — never by running separate queries and combining the numbers in prose.

4. Query Construction & Execution

• Based on the discovered metrics and labels, construct appropriate PromQL queries
• Use query_promql_instant for current state queries (e.g., "What is the current status?"), or single value queries (e.g., "How many users were active yesterday?")
  • IMPORTANT: You can use the time parameter to retrieve data from absolute dates.
• Use query_promql_range for queries that require comparison of different time periods (e.g., "What changed in the last hour?", "How many users were active per day in the last week?")
  • IMPORTANT: for the 'start' and 'end' parameters, use the 'now' keyword to get the current time, and use relative time like 'now-1h' to get the time 1 hour ago.
  • IMPORTANT: utilize the step parameter to control the resolution, when the resolution is too high, the query will return too many results, which can't be displayed or consumed.
• Start with simpler queries and build complexity as needed. If the query fails, analyze the error message, fix the query, and retry. You may check if an existing dashboard already tracks the same concept — reusing a known-good query is faster and keeps analysis consistent with existing monitoring:
  • search_queries(search_type="semantic", query_text="p99 latency") — find queries by meaning
  • search_queries(search_type="by_field", field_path="k8s_pod_cpu_usage") — find queries referencing a metric name

5. Result Validation & Retry Logic

• If a query returns no results or fails validation:
  1. Analyze what might be wrong (syntax, metric name, labels, time range)
  2. Try alternative metric names or adjust the query structure
  3. Consider broader or narrower filtering criteria
  4. Maximum 5 retry attempts with improvements each time
• If the results return with empty labels, and the question requires a specific label, try refining your query to filter out null values for the label.
  • Example: If the question is "Which deployment has the highest CPU usage?", and you used the query max by (k8s_deployment_name) (rate(k8s_pod_cpu_usage[5m])), you should refine the query to max by (k8s_deployment_name) (rate(k8s_pod_cpu_usage{k8s_deployment_name!=""}[5m]))
• If still no results after retries, explain what was attempted and suggest alternatives

6. Pattern Recognition and Root Cause Analysis

• Look for correlations across different metrics
• Identify temporal patterns, like high loads on a recurring basis, or spikes in error rates, etc.
• Identify potential root causes based on the evidence
• Provide actionable next steps for resolution

Handling a missing filter label (cross-metric joins)

This is an uncommon case — most questions are answered by a single metric, so skip this section entirely when your measurement metric already carries the labels the user wants to filter or group by. Only when a label the user wants is not present on your measurement metric do you need to join to a "bridge" metric that has it.

Terminology (the four roles in the join pattern below):

• Measurement metric — the metric that answers what you're counting/measuring (e.g. requests, tokens, sessions). It drives the value in the result.
• Dimension — the label the user wants to filter or group by (e.g. plan, user_email, account_slug) that is missing from the measurement metric.
• Join key — a label present on both metrics that links their series (typically user_id, account_id, service, pod, tenant_id).
• Bridge metric — a metric that carries both the missing dimension AND the join key; it projects the dimension onto the measurement metric.

Flow:

When is by_label required? by_label discovery (steps 1–2) is OPTIONAL only when a single metric in your earlier search_relevant_metrics results (whichever mode you used) already lists labels that include both (i) the missing dimension and (ii) a proper join key shared with your measurement metric. In that case the bridge is already fully confirmed — skip to step 3 and build the join. OTHERWISE by_label is MANDATORY: if no by_description metric exposes both labels (the common case — by_description ranks by measurement subject, so the bridge usually isn't in those results, or only shows a partial label set), you MUST resolve the bridge with search_relevant_metrics(<dimension>, mode="by_label", join_key_label=<join_key>) before building the join. Do not infer a bridge from a metric that shows only one of the two labels. Step 3 (the join itself) is never optional.

Choosing the join key. The join key must be a label present on both the measurement metric and the bridge. When several shared labels are candidates, pick the one whose entity granularity matches the dimension — the identifier for the same entity the dimension describes:

• user_email (a per-user attribute) → join on user_id, not team_name/account_id (coarser — over-aggregates and misattributes across users) and not pod_name/instance (infra, a different entity — usually won't even co-occur with the dimension).
• Match the noun: user_* dimensions ↔ a user_* key; account_* ↔ an account_* key; route/http_route ↔ an http_*/route key.
• Prefer the most specific (closest-granularity / highest-cardinality) shared identifier. Pass that label as join_key_label so the by_label result is restricted to metrics that carry the dimension AND that key on the same series (guaranteed bridges).

1. Find candidate label names. Call search_labels(<dimension_substring>). This returns label names containing the substring. Use a short, distinctive substring (e.g. email, not the full phrase).
   • search_labels queries the current, account-wide label index (it is intentionally NOT pinned to the investigation's time window, since label names are schema, not data). It reliably surfaces high-cardinality, application-specific labels such as user_email, user_id, route, and tenant_id — trust its results.
   • If search_labels returns nothing for a dimension you're confident exists, the label is almost certainly only present on metrics that aren't currently active (the index reflects the present schema). Don't get stuck — proceed with either:
     • (a) Re-read the search_relevant_metrics output from step 3 — each metric entry already lists its labels. Find the metric that exposes your dimension there.
     • (b) Call search_relevant_metrics(<canonical_label>, mode="by_label") directly with the canonical name (e.g. user_email, user_id, route / http_route). The by_label mode lists the distinct metrics that carry the label (deduplicated server-side), so a real label is confirmed (a wrong guess returns no metrics) — and if the result is truncated, narrow it with the name_filter argument.
2. Find candidate bridge metrics. For each promising label name (usually 1), call search_relevant_metrics(<dimension>, mode="by_label", join_key_label=<join_key>), passing the join key you chose above. With join_key_label set, the result is restricted to metrics whose series carry both the dimension and that join key — every metric returned is therefore a usable bridge (no separate get_metric_labels cross-check needed). A standalone metric is shown by its full name (e.g. requests_total or up). Suffix variants of one logical metric are collapsed into a metric family with their suffixes in parentheses, e.g. http_request_duration (_bucket, _sum, _count); when the bare base metric also exists alongside its variants you'll see base (plain, _total) — the plain tag means base itself is directly queryable. Pick the concrete name when you build the query (_total + rate() for counters, _bucket for histograms).
   • Empty result? It means no single metric carries both the dimension and that join key on the same series. Either (a) retry with a different join-key candidate (another shared label), or (b) drop join_key_label to list every metric exposing the dimension and inspect the candidates' labels with get_metric_labels. If still nothing pairs the dimension with a shared key, no bridge exists (see Caveats).
   • Omitting join_key_label falls back to listing all metrics that expose the dimension. The list may then be truncated for labels carried by many metrics: only when the output ends with a truncation hint, re-call with name_filter set to a metric-name substring to narrow to the family you care about.
3. Build the PromQL join. The canonical pattern:

   <measurement> * on(<join_key>) group_left(<dimension>) <bridge>{<dimension>!=""}
   

   Then add the user's actual filter on <dimension> (e.g. =~, !=).
   • Counting distinct entities (very common): when the user asks "how many distinct X", you must collapse cardinality with an explicit count by. Otherwise count(...) will count series (often dozens per entity due to other labels like account_id, route, status_code) and inflate the answer by an order of magnitude.
     • Pattern: count(count by (<join_key>) (<join expression> > 0))
   • Never stitch in prose. Do NOT answer a cross-metric question by running one query for the dimension population and a separate query for the measurement, then combining the counts in text. The dimension filter MUST live inside one PromQL expression via on(<join_key>) group_left(<dimension>). If your written answer claims "users/accounts who also had activity" but no query you ran contained group_left, you did not actually join — redo it as a single joined expression.
   • Magnitude sanity check. A distinct-entity count produced by a join is bounded by activity on the measurement metric and is therefore ≤ the size of the filtered bridge population. If your result comes out roughly equal to the whole filtered bridge population (e.g. every user who merely opened a chat, every account that merely exists), you almost certainly counted the bridge alone and skipped the join — recompute with the on()/group_left() expression before answering.

Worked example — "How many distinct users on the premium plan made requests over the last 7 days?"

• Subject is "made requests" → http_requests_total (request counter). Dimension is plan (filter).
• Step 3: search_relevant_metrics("requests") returns:
  • http_requests_total (labels: user_id, route, ...) — no plan ✗
  • sessions_total (labels: user_id, plan, ...) — has plan ✓
• Choice: measure with http_requests_total, because the user asked about requests (per-request activity), not sessions. Do NOT switch to sessions_total just because it has plan.
• The filter label plan is missing from the measurement metric → join (see below).
• Join key: http_requests_total exposes user_id and route. The dimension plan is a per-user attribute, so the matching-granularity key is user_id (not route).
• search_labels("plan") → [plan, billing_plan]. (If it returns nothing, fall back to the metric labels you already saw in step 3.)
• search_relevant_metrics("plan", mode="by_label", join_key_label="user_id") → sessions_total, …. Confirms sessions_total carries both plan and user_id, so it's a usable bridge.
• Build the join. Right shape for "distinct users":

  count(
    count by (user_id) (
      increase(http_requests_total[7d])
      * on(user_id) group_left(plan)
      max by (user_id, plan) (sessions_total{plan="premium"})
      > 0
    )
  )
  

  Wrong shape (counts series, not users): count((increase(http_requests_total[7d]) > 0) and on(user_id) sessions_total{plan="premium"}) — typically gives a much larger number.

Caveats — this approach does not always work:

• A bridge metric may not exist. Not every dimension is reachable via a join — the data simply may not be there. If search_relevant_metrics(<dimension>, mode="by_label", join_key_label=<key>) returns nothing for every plausible join key (and a join-key-less by_label shows the dimension only on metrics that share no key with your measurement metric), say so and offer the closest partial answer (or a logs/spans alternative) instead of forcing a join or inventing a metric.
• A matching label name is not proof of a correct join. Two metrics can both have a user_id (or account_id, service, …) whose values come from different sources/ID spaces, so an on(<key>) join occasionally yields empty or misleading results even when it "type-checks". You don't need to verify every join — trust a plausible result and move on. Only when a join comes back empty or clearly implausible should you suspect the key isn't truly shared and reconsider the bridge/key (a quick get_metric_label_values on both sides confirms whether values overlap).

Available Tools:

• search_relevant_metrics: Find metrics related to the user's question. Supports three modes via the mode arg:
  • by_description (default): semantic search over a natural-language description of what to measure.
  • by_name: wildcard metric-name pattern matching (e.g. http_*, *_total).
  • by_label: list which metrics expose an EXACT label name (e.g. user_email), to pick a bridge metric for a PromQL join (use after search_labels). Optional name_filter (a metric-name substring) narrows a truncated result; optional join_key_label (a second label) restricts results to metrics carrying both labels. See "Handling a missing filter label" above for the full workflow.
• query_promql_instant: Use this to query a metric at a specific time
• query_promql_range: Use this to query a metric over a time range
• get_metric_labels: Use this to get the labels of a metric
• get_metric_label_values: Use this to get the values of a specific label for a metric
• search_labels: Find label names matching a substring across all metrics. Cheap discovery call — use when the user mentions a dimension/filter but you don't yet know which metric carries it. Follow up with search_relevant_metrics(<label>, mode="by_label").

Key Principles:

• Be Proactive: Don't wait for the user to ask specific questions - drive the investigation
• Chain Steps: Use answers from one query to inform the next step
• Think Holistically: Consider all layers of the stack (app, network, compute, storage, dependencies)
• Be Systematic: Follow a logical troubleshooting flow from symptoms to root cause
• Explain Your Reasoning: Help the user understand why you're checking specific metrics

Common Investigation Patterns:

For HTTP Errors:

1. Check deployment/pod status
2. Check service health and connectivity
3. Check resource utilization (CPU, memory)
4. Check dependencies (databases, external services)
5. Check network and load balancer metrics

For Performance Issues:

1. Check resource saturation (CPU, memory, disk, network)
2. Check application metrics (latency, throughput, errors)
3. Check scaling metrics (pod counts, autoscaling triggers)
4. Check dependency performance

For Availability Issues:

1. Check pod/deployment readiness and restarts
2. Check node health and capacity
3. Check service discovery and networking
4. Check infrastructure dependencies

PromQL Guidelines

Core Principles

1. Pick the right API

   • Instant queries evaluate an expression at a single timestamp (now, or a given time). Use when the user wants one number or one vector as of a moment, essentially any query that does not require results over different timeframes.
   • Range queries evaluate the expression repeatedly across [start, end] at a given step. Use for time series over a period (e.g., daily active users per day).
     • Important: Note that Range queries evaluate the expression repeatedly at all given step. i.e if you set step=1d, start=now-1d, end=now, and the query is max_over_time(metric[1d]), the query will be evaluated 1 time at now-1d, and 1 time at now, which means it will evaluate 2 days of data.
   • Prioritize Instant Queries over Range Queries for most questions, except for questions that require comparing different time frames.

2. Understand PromQL value types

   • Instant vector (set of series with 1 sample each at eval time)
   • Range vector (series with many samples over a window [t-range, t])
   • Scalar (single number)
   • String (rare)
   • Functions like *_over_time() require a range vector. Aggregations like sum/max/min/avg ... by(...) consume instant vectors.
   • Important: When using *_over_time() functions with Range Queries you must be aware that the query will also be evaluated at the start time, and include the window specified in the *_over_time() function.
   • Example: If you use max_over_time(metric[1d]) with start=now-1d, end=now, step=1d, the query will be evaluated at now-1d and now, and the result will be the max of the metric over the window [now-2d, now]. This is a very common mistake, you must avoid! If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and you use max_over_time(x[7d]) with start=2025-01-01, end=2025-01-07, step=1d, the query will be evaluated at 2025-01-01 and 2025-01-07 - which is wrong! You must use Instant Query to avoid this mistake.

3. Separation of concerns

   • Use *_over_time() for temporal reductions across a window (e.g., max_over_time, avg_over_time, quantile_over_time).
   • Use sum/max/min/avg by (...) for label-set aggregation across series at the eval point.
   • Chain them as needed (temporal reduction first, then label aggregation, or vice versa).

4. Counters vs. gauges

   • Counters (monotonic, suffixed _total) → use rate()/irate() or increase() over a window.
   • Gauges (current value) → use avg_over_time, max_over_time, etc., or plain avg(...) depending on intent.

5. Suffix conventions

   • Canonical: _total (counter), _bucket/_sum/_count (histogram), _sum/_count (summary), _created.
   • Non-standard: _avg, _mean, etc. Prefer computing averages via PromQL unless exporter dictates otherwise.

---

PromQL Tool Usage

Instant Query

Example: absolute max over last 24h (single result)

max by () (max_over_time(http_requests_in_flight[24h]))

Range Query

Example: Absolute max per day over the last 7 days

max by () (max_over_time(metric[1d]))

Set start=now-7d, end=now, step=1d.

IMPORTANT: When using range queries, you must align the step parameter, with any range parameters you use in temporal reduction functions. i.e if you use max_over_time(metric[1d]), you must set step=1d.

PromQL Fundamentals (with examples)

Label Matching & Aggregation

• Matchers: {label="v"}, {label!="v"}, {label=~"re.*"}, {label!~"re"}
• Aggregate by labels to keep them; use without to drop them.

sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)

Temporal Reductions (range → instant)

max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])

Counters: Rates, Increases, Windows

• Per-instance RPS:

rate(http_requests_total[5m])

• Total RPS across fleet:

sum by () (rate(http_requests_total[5m]))

• Events in last day (per user, then count actives):

count( sum by (user_id) (increase(api_call_count[24h])) > 0 )

Histograms & Summaries

• p95 from a histogram:

histogram_quantile(
  0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

• Average from summary parts:

sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))

Max over a Period

• Correct:

max_over_time(metric[4d])

• Aggregate across series (absolute max):

max by () (max_over_time(metric[4d]))

• Aggregate across series (absolute max) per label:

max by (label) (max_over_time(metric[4d]))

• Incorrect: max(metric[4d]) (aggregation cannot take a range vector)

Top-k / Ranking

topk(5, sum by (instance) (rate(http_requests_total[5m]))

---

Common Tasks (ready to adapt)

1. Absolute peak per instance over 7d, then pick the winner

topk(1, max by (instance) (max_over_time(my_metric[7d])))

2. Global CPU usage % (avg across cores & hosts)

avg by () (
  rate(process_cpu_seconds_total[5m])
) * 100

3. Error rate (%) per route

100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (route) (rate(http_requests_total[5m]))

4. Daily active users over a week (time series)

• Expression:

count(count by (user_id) (increase(api_call_count[1d]) > 0))

• Use query_promql_range with step=1d, start=now-6d, end=now. (Note we start from 6 days ago, because the increase function takes the earlier day into account)

---

Performance & Safety Guidelines

• Prefer short windows for rate() (e.g., 1-5m) unless data is bursty or sparse.
• Avoid unbounded fan-out (e.g., joining massive label sets).
• Keep cardinality under control; aggregate early (sum by (...)) when you only need totals.
• Use clamp_max/clamp_min to tame outliers when needed.
• For histograms, always aggregate buckets (sum by (le, ...)) before histogram_quantile.
• Be mindful of counter resets; rate()/increase() handle resets automatically.

---

Frequent Gotchas (and fixes)

• “Why am I getting steps when I only want one number?” You're using query_promql_range. Use query_promql_instant instead.

• “max(metric[...]) errors.” max() can't take a range vector. Use max_over_time(metric[...]), then aggregate with max by () (...) if needed.

• “_over_time(metric[...]) by (label) errors.” _over_time aggregations cannot include a by label clause. Use max_over_time(metric[...]), then aggregate with max by () (...).

• “Avg looks wrong for counters.” Counters need rate()/increase(), not avg_over_time.

• “p95 from summary?” Summaries expose quantiles directly via the quantile label. For histograms, use histogram_quantile on bucket rates.

---

Mini Cheat-Sheet

• Rate of a counter: rate(x_total[5m])
• Increase last 24h: increase(x_total[24h])
• Avg of a gauge over 1h: avg_over_time(x[1h])
• Max over 4d (absolute): max by () (max_over_time(x[4d]))
• Top 5 by RPS: topk(5, sum by (instance) (rate(x_total[5m])))
• p95 latency (histogram): histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m])))
• Filter labels: {env="prod", job=~"api|web"}
• Drop a label in agg: sum without (instance) (x)
• Time travel: expr @ <unix_ts> or expr offset 1h

Related Skills

• If the question is about a specific service/pod/entity's RED metrics, health, or resource saturation (CPU/memory/OOMKilled, JVM heap/GC), load the coralogix-service-catalog skill first — it's a curated, pre-aggregated path for that data rather than hand-writing PromQL. Fall back to this skill for ad-hoc metrics beyond the fixed Service Catalog columns.

MCP tools

query_promql_instant

• query: PromQL expression
• time (optional): RFC-3339/ISO 8601 UTC only; omit for current time. Relative strings like now-1h are not accepted.

query_promql_range

• query: PromQL expression
• start / end: now, relative (now-1h, now-6d), Unix timestamp, or RFC-3339/ISO 8601 UTC
• step: resolution (e.g. 30s, 1h, 1d)
• limit: max samples per series (1–200)

get_datetime

Returns the current UTC time as ISO 8601. Use for an explicit time on instant queries; for range queries you can pass now / now-1h directly to start / end.

search_relevant_metrics

• query: search term, interpreted per mode (description text, name pattern, or EXACT label name)
• mode: by_description (default) / by_name / by_label
• top_k: max results, by_description only (1–100, default 5)
• name_filter (optional, by_label only): metric-name substring to narrow a truncated result
• join_key_label (optional, by_label only): EXACT second label (the shared join key); returns only metrics carrying BOTH query and this label on the same series (guaranteed bridges)

search_labels

• substring: case-insensitive substring matched against label names across all metrics; exact match returned first. Returns one label name per line (with a truncation hint when more match).