inspect-vai-pipes
Use when debugging, inspecting, or analyzing Vertex AI Pipeline jobs - checking worker logs, diagnosing 429 errors, confirming parameters, analyzing scores, finding straggler tasks, or estimating comp
SKILL.mdclaude/inspect-vai-pipes/SKILL.md
GitHubInspect Vertex AI Pipelines
Debug and analyze Vertex AI Pipeline jobs using the SDK, Cloud Logging, and GCS artifacts.
Required Inputs from User
- Pipeline display name or run ID (e.g.,
my-pipeline-run-20260319-150709) - GCP project ID (e.g.,
hybrid-vertex) - Region (e.g.,
us-central1) - GCS bucket for pipeline artifacts (if checking evidence reports, logs, progress)
Quick Reference
| Task | Method |
|---|---|
| Find pipeline | aiplatform.PipelineJob.list() + filter by display name |
| Get state | r.state (see state codes below) |
| Get parameters | r.gca_resource.runtime_config.parameter_values |
| Get task details | r.task_details (list of tasks with state, name, execution metadata) |
| Worker logs | Cloud Logging: resource.type="ml_job" |
| Pipeline events | Cloud Logging: resource.type="aiplatform.googleapis.com/PipelineJob" |
| GCS artifacts | Check evidence_reports/, progress.json, log files in pipeline bucket |
Pipeline State Codes
| Code | State | Description |
|---|---|---|
| 1 | QUEUED | Waiting to start |
| 2 | PENDING | Being prepared |
| 3 | RUNNING | Actively executing |
| 4 | SUCCEEDED | Completed successfully |
| 5 | FAILED | Terminated with error |
| 6 | CANCELLING | Cancel in progress |
| 7 | CANCELLED | User cancelled |
| 8 | PAUSED | Paused |
| 9 | SKIPPED | Skipped (caching) |
Task Execution State Codes
| Code | State | Notes |
|---|---|---|
| 1 | PENDING | |
| 2 | RUNNING_DRIVER | |
| 3 | DRIVER_SUCCEEDED | Also seen for lightweight tasks that skip executor |
| 4 | RUNNING_EXECUTOR | |
| 5 | SUCCEEDED | |
| 7 | CANCELLED | Upstream dependency failed |
| 8 | CACHED/SKIPPED | KFP caching hit — task reused previous output |
| 9 | FAILED |
Pipeline DAG Structure (3-Component)
The pipeline uses a ParallelFor over datasets with 3 chained components per table:
for each table in datasets:
get-table-metadata(table) # lightweight BQ metadata, always runs
|
v
scan-table(table, modified_time) # Dataplex DATA_DOCUMENTATION scan, KFP cached
|
v
run-fan-out-wrapper(scan_results) # main worker, caching disabled
When inspecting, look for this chain per worker. If scan-table fails, run-fan-out-wrapper shows state=9 (SKIPPED) since its upstream dependency failed.
Step 1: Find and Inspect the Pipeline
from google.cloud import aiplatform
aiplatform.init(project='PROJECT_ID', location='REGION')
# Find by display name substring
runs = aiplatform.PipelineJob.list()
r = [r for r in runs if 'SEARCH_TERM' in (r.display_name or '')][0]
print(f'Display Name: {r.display_name}')
print(f'Resource Name: {r.resource_name}')
print(f'State: {r.state}') # See state codes above
print(f'Created: {r.create_time}')
# Parameters
params = r.gca_resource.runtime_config.parameter_values
for k, v in sorted(params.items()):
print(f' {k}: {v}')
# Task details — include error messages for quick triage
for t in r.task_details:
err = ''
if t.error and t.error.message:
err = t.error.message[:150]
print(f'{t.task_name}: state={t.state} err={err}')
Common gotcha: PipelineJob objects lack start_time and end_time attributes. Use create_time and check Cloud Logging for actual start/end timestamps.
Display name vs resource name: The display_name and resource_name often differ slightly (e.g., my-pipeline-20260319-150709 vs my-pipeline-run-20260319-150709). Always search with a substring.
Step 2: Query Worker Logs (Cloud Logging)
Worker container logs use resource.type="ml_job". Pipeline orchestration events use resource.type="aiplatform.googleapis.com/PipelineJob".
Get pipeline task events (start/complete times)
gcloud logging read \
'resource.type="aiplatform.googleapis.com/PipelineJob" AND "PIPELINE_RUN_ID"' \
--project=PROJECT_ID --limit=50 --format=json --order=asc
Parse with: e['jsonPayload']['payload']['taskName'], state, startTime, endTime
Get worker execution logs
# Key execution milestones (epochs, scores, errors, timeouts, completions)
gcloud logging read \
'resource.type="ml_job" AND
(jsonPayload.message=~"EPOCH|Critic Score|completed|evidence|Error|429|ReadError|TimeoutError|crashed" OR
jsonPayload.levelname="ERROR") AND
timestamp>="PIPELINE_START_TIME"' \
--project=PROJECT_ID --limit=500 \
--format="value(timestamp,resource.labels.job_id,jsonPayload.message)" --order=asc
Always include resource.labels.job_id in the format string. Each ParallelFor worker runs in a separate container with a unique job_id. You need this to map logs to specific containers/datasets and build per-container progress reports.
Log structure: Worker logs are in jsonPayload.message (not textPayload). The levelname field contains INFO, ERROR, WARNING.
Important: Lightweight component logs (e.g., get-table-metadata, scan-table) often have empty jsonPayload.message. Use textPayload or full JSON format instead:
# For lightweight components, get the job_id from the task error message, then:
gcloud logging read \
'resource.type="ml_job" AND resource.labels.job_id="JOB_ID" AND resource.labels.task_name="workerpool0-0"' \
--project=PROJECT_ID --limit=50 \
--format="value(jsonPayload.message)" --order=asc
# If messages are empty, try JSON format to see all fields:
gcloud logging read \
'resource.type="ml_job" AND resource.labels.job_id="JOB_ID"' \
--project=PROJECT_ID --limit=10 --format=json --order=desc
Filter for specific issues
# 429 rate limit errors (Gemini API + Memory Bank)
jsonPayload.message=~"429|RESOURCE_EXHAUSTED"
# Critic scores
jsonPayload.message=~"Critic Score"
# Epoch transitions
jsonPayload.message=~"EPOCH.*Concurrency"
# Worker task assignments
jsonPayload.message=~"\\[Worker \\d+\\] Executing"
# Memory Bank writes
jsonPayload.message=~"Memory Bank|memory bank"
# GCS uploads (evidence reports, progress)
jsonPayload.message=~"Uploaded.*evidence|Uploaded.*progress"
# Pipeline completion
jsonPayload.message=~"fan_out_insights completed"
Step 3: Check GCS Artifacts
from google.cloud import storage
client = storage.Client(project='PROJECT_ID')
bucket = client.bucket('BUCKET_NAME')
# Evidence reports (per-epoch summaries)
for b in client.list_blobs(bucket, prefix='pipeline_root/evidence_reports/'):
print(f'{b.name} ({b.size} bytes, updated {b.updated})')
# Progress file (live status)
blob = bucket.blob('pipeline_root/evidence_reports/progress.json')
if blob.exists():
import json
print(json.loads(blob.download_as_text()))
# Main log file
blob = bucket.blob('pipeline_root/evidence_reports/react_loop_fanout_log.json')
if blob.exists():
print(f'Log size: {blob.size} bytes')
Artifact paths may vary. Some pipelines write to pipeline_root/evidence_reports/, others to self-evolve/pipeline_root/evidence_reports/. Check both.
Step 4: Analyze Scores
Extract and analyze critic scores from logs:
import re
from collections import Counter
scores, novelty_scores = [], []
persisted, not_persisted = 0, 0
for line in log_lines:
score_m = re.search(r'Critic Score:\s*([\d.]+)', line)
novelty_m = re.search(r'Novelty:\s*([\d.]+)', line)
persist_m = re.search(r'Persist:\s*(True|False)', line)
if score_m:
scores.append(float(score_m.group(1)))
if novelty_m: novelty_scores.append(float(novelty_m.group(1)))
if persist_m: (persisted if persist_m.group(1) == 'True' else not_persisted) + 1
print(f'Total: {len(scores)}, Persisted: {persisted}, Rate: {persisted/len(scores)*100:.1f}%')
print(f'Mean: {sum(scores)/len(scores):.3f}, Max: {max(scores):.3f}, Min: {min(scores):.3f}')
Floor scores (0.10) are NOT always errors. Before assuming floor scores are API failures, check for "Critic scoring failed" or "Skipping critic" in the logs. If those messages are absent, the critic agent legitimately scored the insight at 0.10 — this is a quality signal, not a rate-limiting problem. True API failures return 0.0 (not 0.10) via the error handler. Only diagnose floor scores as a rate-limiting issue when accompanied by actual error log messages.
Common Issues
Massive 429 Errors
Symptom: Logs flooded with HTTP/1.1 429 Too Many Requests or RESOURCE_EXHAUSTED
Root cause: concurrency param too high. Effective parallelism = concurrency × num_datasets. At concurrency=100 with 5 datasets = 500 simultaneous API calls.
Fix: Reduce concurrency to 20-30 per dataset.
Memory Bank RESOURCE_EXHAUSTED
Symptom: Quota exceeded for quota metric 'Memory Bank write requests'
Root cause: Memory Bank write quota is 100 writes/min/region. High concurrency overwhelms the rate limiter.
Fix: Lower concurrency, or increase semaphore delay in react_loop_fanout.py.
Straggler Workers
Symptom: Pipeline running long after most workers complete. One or two tasks still in state 2/4.
Diagnosis: Check which for-loop-1-worker-N tasks are still running. Cross-reference with dataset_item in task execution metadata to identify the slow dataset.
Common causes: Large tables triggering slow Dataplex scans, workers stuck in retry loops from persistent 429s.
Pipeline State SKIPPED (9)
Cause: Vertex AI pipeline caching. If inputs match a previous run, tasks may be skipped entirely.
Fix: Change a parameter value or disable caching in the pipeline definition.
scan-table Component Failures
The scan-table component runs in a lightweight python:3.11 container with only google-cloud-dataplex installed. Common failure modes:
AttributeError: DATA_DOCUMENTATION
- Cause: Pinned
google-cloud-dataplexversion too old.DataScanType.DATA_DOCUMENTATIONwas added in later versions (needs >=2.10.0). - Fix: Update version pin in
deploy_vertex_pipeline.pyscan_tablecomponent'spackages_to_install.
501 Received http2 header with status: 404 on create_data_scan
- Cause: Regional Dataplex endpoint (
{location}-dataplex.googleapis.com:443) does NOT supportDATA_DOCUMENTATIONscan type. Only the global endpoint (dataplex.googleapis.com) works. - Fix: Use
DataScanServiceClient()withoutclient_options(defaults to global endpoint). - Note: The existing
src/dataplex_client.pyalso uses the regional endpoint — this works forDATA_PROFILEbut notDATA_DOCUMENTATIONin some SDK versions.
Cascading failures pattern: When scan-table fails, run-fan-out-wrapper shows state=9 (SKIPPED) and for-loop-1-worker-N shows state=7 (CANCELLED) with error "The DAG failed because some tasks failed: [scan-table]". This is expected — fix the scan-table error and the downstream tasks will run.
KFP Component Version Mismatches
Symptom: Component works locally but fails in pipeline with import/attribute errors.
Root cause: KFP @dsl.component containers install packages at runtime via pip. The pinned versions may differ from your local environment. A python:3.11 base image also has different system packages than your dev machine.
Diagnosis: Check the worker logs for pip install output and the actual traceback. The error usually shows the exact missing attribute or module.
Fix: Match the version pins in packages_to_install to your local working versions. Check with importlib.metadata.version('package-name').
Container Bootstrap Time Variance
Symptom: Some ParallelFor containers start 3-4 minutes after others even though all were submitted simultaneously.
Cause: Vertex AI schedules containers across available resources. Startup time includes image pull, pip install of packages_to_install, and driver initialization. This is normal behavior, not an error.
Impact on analysis: When calculating per-container durations, use each container's first log entry as its start time, not the pipeline's create_time. Otherwise you'll overestimate duration for late-starting containers.
Per-Worker Timeout Patterns
Symptom: Workers killed by asyncio.TimeoutError before completing their LLM call.
Key observations:
- Timeout rates increase in later epochs (Epoch 2 > Epoch 1) because richer context from elite skills and crossover makes prompts larger and LLM responses slower
- Adaptive concurrency (auto-reducing concurrency when timeout rate exceeds a threshold) can mitigate this
- Tuning data points: 180s timeout → ~26% timeout rate; 240s timeout → ~10% timeout rate (with concurrency 30, 4-5 datasets)
- Timeouts are per-worker, not per-epoch — a single worker timing out doesn't affect others in the same epoch
Diagnosis: Search for TimeoutError in worker logs. Group by resource.labels.job_id to identify which containers are most affected. Late-starting containers or those processing larger tables tend to have higher timeout rates.
Epoch Regression (Epoch 2 worse than Epoch 1)
Symptom: Mean scores decrease across epochs instead of improving.
Common causes: 429 errors during epoch 1 corrupted skill evolution feedback; floor scores (0.10) polluting aggregate metrics used for skill selection.
Cross-Validating Source Code
When logs show unexpected behavior, check the source:
| Concern | File to Check |
|---|---|
| Pipeline parameters & defaults | deploy_vertex_pipeline.py (argparse block + deploy_pipeline()) |
| Pipeline DAG & components | deploy_vertex_pipeline.py (get_table_metadata, scan_table, run_fan_out_wrapper, my_pipeline) |
| Component package versions | deploy_vertex_pipeline.py (packages_to_install in each @dsl.component) |
| Precomputed scan passthrough | deploy_vertex_pipeline.py (scan_results param) → react_loop_fanout.py (PRECOMPUTED_SCAN_RESULTS env var) |
| Worker execution logic | react_loop_fanout.py (fan_out_insights()) |
| Concurrency / rate limiting | react_loop_fanout.py (semaphore, delay constants) |
| Critic scoring | src/prompts/critic_scoring.md |
| Skill evolution | src/swarm_manager.py |
| API endpoint defaults | backend/main.py |
| Dataplex data scans (fallback) | src/dataplex_client.py |
Per-Container Progress Reports
To build a per-container progress report, map resource.labels.job_id to datasets:
Step 1: Get job_id → dataset mapping
# Get component metadata logs that show dataset_item assignments
gcloud logging read \
'resource.type="aiplatform.googleapis.com/PipelineJob" AND "PIPELINE_RUN_ID" AND "dataset_item"' \
--project=PROJECT_ID --limit=100 --format=json --order=asc
Alternatively, get task details from the SDK — each for-loop-1-worker-N task's execution metadata contains the dataset_item input value.
Step 2: Query per-container metrics
# Get epoch transitions, scores, and errors per container
gcloud logging read \
'resource.type="ml_job" AND resource.labels.job_id="JOB_ID" AND
(jsonPayload.message=~"EPOCH|Critic Score|completed|TimeoutError|crashed")' \
--project=PROJECT_ID --limit=200 \
--format="value(timestamp,jsonPayload.message)" --order=asc
Step 3: Aggregate per container
For each container, extract:
- Epochs completed (count
EPOCHtransition messages) - Insights generated (count
Critic Scorelines) - Mean/max/min scores (parse score values)
- Timeout rate (count
TimeoutError/ total workers per epoch) - Persist rate (count
Persist: True/ total scored) - Wall-clock duration (first log → last log timestamp)
Estimating Completion Time
- Check how many
for-loop-1-worker-Ntasks have completed vs still running - Note the wall-clock time of completed workers (from Cloud Logging task events)
- Workers processing larger tables or hitting more 429s take proportionally longer
- Account for container bootstrap stagger (3-4 min between first and last container start)
- A 2-epoch run with concurrency 30 on 5 tables typically takes 1-2 hours
- A 2-epoch run with concurrency 100 on 5 tables can take 2-3+ hours (due to 429 backoff)
- Later epochs take longer due to richer context (larger prompts, more timeouts)
Google Cloud Console
Pipeline runs can also be inspected in the console at:
https://console.cloud.google.com/vertex-ai/pipelines/locations/REGION/runs/PIPELINE_RUN_ID?project=PROJECT_ID
The console shows the DAG visualization, task states, logs, and parameter values.