AWS X-Ray Skill

Analyze AWS X-Ray traces and service maps with parallel execution and anti-hallucination guardrails.

Relationship to other AWS skills:

• aws-xray/ → X-Ray-specific analysis (traces, service maps, sampling)
• aws/ → "How to execute" (parallel patterns, throttling, output format)

CRITICAL: Parallel Execution Requirement

ALL independent operations MUST run in parallel using background jobs (&) and wait.

#!/bin/bash
export AWS_PAGER=""

for group in $groups; do
  get_group_traces "$group" &
done
wait

Helper Functions

#!/bin/bash
export AWS_PAGER=""

# Get service graph (service map)
get_service_graph() {
  local hours=${1:-1}
  local end_time start_time
  end_time=$(date -u +"%Y-%m-%dT%H:%M:%S")
  start_time=$(date -u -d "$hours hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-${hours}H +"%Y-%m-%dT%H:%M:%S")
  aws xray get-service-graph \
    --start-time "$start_time" --end-time "$end_time" \
    --output text \
    --query 'Services[].[Name,Type,State,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount,ResponseTimeHistogram[0].Average]'
}

# Get trace summaries with filter
get_trace_summaries() {
  local filter=$1 hours=${2:-1}
  local end_time start_time
  end_time=$(date -u +"%Y-%m-%dT%H:%M:%S")
  start_time=$(date -u -d "$hours hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-${hours}H +"%Y-%m-%dT%H:%M:%S")
  aws xray get-trace-summaries \
    --start-time "$start_time" --end-time "$end_time" \
    --filter-expression "$filter" \
    --output text \
    --query 'TraceSummaries[].[Id,Duration,HasFault,HasError,Http.HttpStatus,ResponseTime]' | head -20
}

# List sampling rules
list_sampling_rules() {
  aws xray get-sampling-rules \
    --output text \
    --query 'SamplingRuleRecords[].[SamplingRule.RuleName,SamplingRule.Priority,SamplingRule.FixedRate,SamplingRule.ReservoirSize,SamplingRule.ServiceName,SamplingRule.HTTPMethod,SamplingRule.URLPath]'
}

# List X-Ray groups
list_groups() {
  aws xray get-groups \
    --output text \
    --query 'Groups[].[GroupName,GroupARN,FilterExpression]'
}

Common Operations

1. Service Map Overview

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "1 hour ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-1H +"%Y-%m-%dT%H:%M:%S")
aws xray get-service-graph \
  --start-time "$START" --end-time "$END" \
  --output text \
  --query 'Services[].[Name,Type,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount]'

2. Fault Analysis (5xx Errors)

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "6 hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-6H +"%Y-%m-%dT%H:%M:%S")
aws xray get-trace-summaries \
  --start-time "$START" --end-time "$END" \
  --filter-expression 'fault = true' \
  --output text \
  --query 'TraceSummaries[].[Id,Duration,Http.HttpStatus,Http.HttpURL,ResponseTime]' | head -20

3. Latency Investigation (Slow Traces)

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "1 hour ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-1H +"%Y-%m-%dT%H:%M:%S")
aws xray get-trace-summaries \
  --start-time "$START" --end-time "$END" \
  --filter-expression 'responsetime > 5' \
  --output text \
  --query 'TraceSummaries[].[Id,Duration,ResponseTime,Http.HttpURL]' | sort -k3 -rn | head -10

4. Sampling Rule Review

#!/bin/bash
export AWS_PAGER=""
aws xray get-sampling-rules \
  --output text \
  --query 'SamplingRuleRecords[].[SamplingRule.RuleName,SamplingRule.Priority,SamplingRule.FixedRate,SamplingRule.ReservoirSize,SamplingRule.ServiceName,SamplingRule.HTTPMethod,SamplingRule.URLPath,SamplingRule.Version]'

5. Error Rate by Service

#!/bin/bash
export AWS_PAGER=""
END=$(date -u +"%Y-%m-%dT%H:%M:%S")
START=$(date -u -d "6 hours ago" +"%Y-%m-%dT%H:%M:%S" 2>/dev/null || date -u -v-6H +"%Y-%m-%dT%H:%M:%S")
aws xray get-service-graph \
  --start-time "$START" --end-time "$END" \
  --output text \
  --query 'Services[].[Name,Type,SummaryStatistics.TotalCount,SummaryStatistics.FaultStatistics.TotalCount,SummaryStatistics.ErrorStatistics.TotalCount,SummaryStatistics.OkCount]' \
  | awk '{total=$3; faults=$4; errors=$5; if(total>0) printf "%s\t%s\tTotal:%s\tFaultRate:%.2f%%\tErrorRate:%.2f%%\n", $1, $2, total, faults/total*100, errors/total*100}'

Anti-Hallucination Rules

1. Fault vs Error - In X-Ray, a fault is a 5xx server error. An error is a 4xx client error. Throttle is a 429 specifically. Do not conflate them.
2. Filter expression syntax - X-Ray uses its own filter expression syntax, NOT JMESPath. Use service(), annotation.key, responsetime, fault, error keywords.
3. Trace retention - X-Ray retains trace data for 30 days. Trace summaries are available for 30 days. Full trace data beyond this requires export to S3.
4. Service graph time range - Maximum time range for get-service-graph is 6 hours per call. For longer periods, make multiple calls and aggregate.
5. Sampling affects completeness - X-Ray samples traces. A 5% sampling rate means you see ~5% of actual requests. Do not report trace counts as request counts.

Output Format

Present results as a structured report:

Aws Xray Report
═══════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

• Time range limits: get-trace-summaries supports up to 6 hours per call. For longer analysis, paginate across time windows.
• Group filter expressions: Groups filter traces server-side. Different groups may show different subsets of the same traces.
• Segment documents: Full trace details require batch-get-traces with specific trace IDs. get-trace-summaries is a lightweight summary.
• CloudWatch statistics syntax: Use spaces not commas: --statistics Average Maximum.
• Active tracing vs passive: Active tracing (X-Ray SDK) sends traces. Passive tracing (downstream services) only records if upstream sends trace headers.