Data
AI Pipeline Agent
An autonomous AI agent that continuously monitors your BigQuery pipelines, detects failures, diagnoses root causes, and creates remediation plans — all powered by Vertex AI.
Organization only
How the agent works
The AI Pipeline Agent runs analysis cycles where it scans your BigQuery job history for failures, stale tables, and anomalous patterns. For each issue found, it uses Gemini to generate a detailed diagnosis including root cause analysis, blast radius assessment, and step-by-step fix instructions.
Agent capabilities
Pipeline ScanningDiscovers all BigQuery jobs, scheduled queries, and recurring patterns. Identifies which are failing or degraded.
Root Cause AnalysisUses Gemini to analyze failure patterns, error messages, and historical context to identify why a pipeline broke.
Impact AssessmentMaps downstream dependencies to show which tables, dashboards, and reports are affected by a failure.
Remediation PlansGenerates concrete fix steps — SQL patches, permission changes, schema fixes — that you can execute directly.
Issue CreationAutomatically creates issues in your Issue Board with full context, severity, and assignment suggestions.
Scheduled RunsConfigure the agent to run on a cron schedule (e.g. every 6 hours) for continuous monitoring.
Running the agent
- Go to AI Pipeline Agent in the sidebar
- Configure which datasets and tables to monitor
- Set sensitivity and notification preferences
- Click Run Analysis for an immediate scan, or set up a cron schedule
- Review generated drafts and plans in the Drafts and Plans tabs
- Approve plans to create issues or apply fixes
Agent cycles
Each run creates a cycle — a timestamped record of what the agent found, analyzed, and recommended. You can browse past cycles to see how pipeline health has changed over time.
The AI Pipeline Agent requires Vertex AI API access and
bigquery.jobs.list permission on the service account for full job history scanning.