Datadog Monitoring

This skill provides access to Datadog for monitoring, logging, and alerting. Use the pup CLI (DataDog/pup) as the primary tool. Fall back to the API directly only for features pup doesn't cover yet.

Setup

pup CLI (preferred)

pup is installed via install.sh. Authenticate with OAuth2 or API keys:

# OAuth2 (preferred — browser-based, auto-refreshing tokens)
export DD_SITE="us3.datadoghq.com"
pup auth login

# Or a personal access token (fallback). pup reads a bearer token from
# DD_ACCESS_TOKEN, not from DD_PAT — DD_PAT is only the name the vault and the
# curl examples below use. Needs pup >= 1.x; 0.9.2 has no bearer auth at all.
export DD_ACCESS_TOKEN="your-personal-access-token"
export DD_SITE="us3.datadoghq.com"

Verify: pup auth status or pup test

API fallback

For features pup doesn't cover, use curl with the PAT:

export DD_PAT="your-personal-access-token"
export DD_SITE="us3.datadoghq.com"

DD_PAT replaces the old DD_API_KEY + DD_APP_KEY pair — one token instead of two. It is loaded on demand from Bitwarden by ~/dotfiles/secrets.zsh; run secrets-load if unset.

On a host with no credentials on it

Some hosts — the ones agents run on unattended — hold no token at all. There, DD_PAT is brokered one command at a time from the vault on my laptop, and a human approves each request:

with-secret DD_PAT --as DD_ACCESS_TOKEN -- \
  pup logs search --query="service:api status:error" --from="1h"

--as because the vault field is DD_PAT and pup reads DD_ACCESS_TOKEN.

The token goes into that one command's environment and is scrubbed out of its output; there is no way to print it, and asking for one is the wrong move. For the API fallback, write the placeholder {{DD_PAT}} — the command runs without a shell, so $DD_PAT in an argument expands to nothing:

with-secret DD_PAT -- curl -sS -H 'Authorization: Bearer {{DD_PAT}}' \
  "https://api.us3.datadoghq.com/api/v2/current_user"

Only pup and curl may use DD_PAT, and curl only against the Datadog API host with a restricted set of flags — no -o, -L, --proxy or --config. That allowlist is enforced on the laptop; if something legitimate is blocked, ask for it to be added rather than looking for a way around it.

When to Use

Use this skill when the user:

• Asks about logs, errors, or application behavior
• Wants to check monitor/alert status
• Needs to investigate an incident
• Asks about metrics or performance
• Mentions "Datadog" or monitoring

pup CLI Reference

Output formats: pup <command> -o json|table|yaml (default: json). Use -y to skip confirmation on destructive ops.

Logs

pup logs search --query="service:my-service status:error" --from="1h"
pup logs list --query="status:error" --from="30m"
pup logs aggregate --query="service:api" --from="1d"

Monitors (Alerts)

pup monitors list
pup monitors get 12345678
pup monitors search --query="status:Alert"

Metrics

pup metrics query --query="avg:system.cpu.user{*}" --from="1h"
pup metrics search --query="avg:system.cpu.user{*}" --from="1h"
pup metrics list --filter="system.*"
pup metrics get METRIC_NAME

Dashboards

pup dashboards list
pup dashboards get abc-123-def
pup dashboards url abc-123-def

Incidents

pup incidents list
pup incidents get abc-123-def
pup incidents attachments abc-123-def

Events

pup events list --from="1d"
pup events search --query="source:my-service" --from="1h"

SLOs

pup slos list
pup slos get abc-123

Traces (APM)

pup traces search --query="service:api" --from="1h"
pup apm services
pup apm dependencies --service=api
pup apm flow-map --service=api

Infrastructure

pup infrastructure hosts list
pup tags list
pup tags get HOSTNAME

Security

pup security rules list
pup security signals list --from="1d"
pup security findings search --query="status:critical"

Other Commands

pup on-call teams                    # On-call team management
pup cases search --query="status:open"  # Case management
pup error-tracking issues search     # Error tracking
pup service-catalog list             # Service catalog
pup audit-logs search --from="1d"    # Audit logs
pup usage summary                    # Usage metering
pup cost projected                   # Cost management
pup synthetics tests list            # Synthetic tests
pup downtime list                    # Scheduled downtimes
pup cicd pipelines list              # CI/CD visibility

API Fallback

pup covers ~45% of Datadog APIs. For features it doesn't support (profiling, containers, processes, session replay, DORA metrics, etc.), fall back to the API directly.

Base URL: https://api.$(printenv DD_SITE)/api/v1 or v2

# Example: API endpoint not covered by pup
curl -s "https://api.$(printenv DD_SITE)/api/v2/ENDPOINT" \
  -H "Authorization: Bearer $(printenv DD_PAT)"

# Same thing on a host where the token is brokered (see Setup)
with-secret DD_PAT -- curl -sS -H 'Authorization: Bearer {{DD_PAT}}' \
  "https://api.us3.datadoghq.com/api/v2/ENDPOINT"

Verified working with the PAT on us3: POST /api/v2/logs/events/search, GET /api/v1/query (metrics), GET /api/v1/dashboard/lists/manual.

Note: GET /api/v1/validate returns 403 with a PAT — that endpoint validates an API key specifically, so a 403 there does NOT mean the PAT is bad. To check a PAT, use GET /api/v2/current_user instead.

Log Query Syntax

Works in both pup logs search --query= and API calls:

Operator	Example	Description
AND	service:api status:error	Both conditions (implicit)
OR	status:error OR status:warn	Either condition
NOT	-status:debug	Exclude matches
Wildcard	service:api-*	Pattern matching
Range	@duration:>1000	Numeric comparisons
Exists	@http.url:*	Field exists

Common filters: service:name, status:error|warn|info|debug, @http.status_code:500, host:hostname, env:production

Hawk/Scout Scan Metrics

Hawk scan jobs emit structured log entries containing @scan_metrics and @scan_progress fields. These are log-based (not Datadog custom metrics) — query them via pup logs search.

Querying scan metrics

# Get scan metrics for a specific job (use --limit 1000 --sort asc for chronological history)
pup logs search \
  --query="inspect_ai_job_id:<JOB_ID> @scan_metrics.tasks_scanning:*" \
  --from="7d" --limit 1000 --sort asc -o json

# Note: default limit is 10. Use --limit 1000 to get more entries.
# Logs are emitted every ~6 seconds by inspect_scout.
# To cover the full timeline, query both --sort asc (oldest 1000) and --sort desc (newest 1000).

Available fields

@scan_metrics.* (from inspect_scout.ScanMetrics):

Field	Description
@scan_metrics.task_count	Total async worker tasks
@scan_metrics.tasks_idle	Workers doing nothing
@scan_metrics.tasks_parsing	Workers parsing transcript files
@scan_metrics.tasks_scanning	Workers running scanners (making API calls)
@scan_metrics.buffered_scanner_jobs	Jobs buffered waiting to be scanned
@scan_metrics.completed_scans	Total completed scan operations
@scan_metrics.memory_usage	Process memory (RSS/USS) in bytes
@scan_metrics.process_count	Number of OS processes
@scan_metrics.batch_pending	Pending batch API requests
@scan_metrics.batch_failures	Failed batch API requests
@scan_metrics.batch_oldest_created	Timestamp of oldest pending batch

@scan_progress.*:

Field	Description
@scan_progress.completed	Number of completed scans
@scan_progress.total	Total scans expected
@scan_progress.percent	Completion percentage

Common tags for filtering

• inspect_ai_job_id:<JOB_ID> — the scan run ID
• inspect_ai_job_type:scan
• inspect_ai_created_by:<user>
• kube_job:<JOB_ID>
• service:runner

Container-level metrics

Standard Datadog container metrics are also available, tagged by kube_job:<JOB_ID>:

pup metrics query --query="avg:container.pid.thread_count{kube_job:<JOB_ID>}" --from="24h"
pup metrics query --query="avg:container.cpu.usage{kube_job:<JOB_ID>}" --from="24h"
pup metrics query --query="avg:container.memory.usage{kube_job:<JOB_ID>}" --from="24h"

Notes

• Datadog site: us3.datadoghq.com
• API rate limits apply — be mindful of query frequency
• Log queries return max 1000 results per request; use pagination for more
• Monitor status values: OK, Alert, Warn, No Data