Skip to main content
Testing agents before deployment and debugging issues when they arise are essential for building reliable automation. This guide covers validation, debugging techniques, and best practices for iterating on agent configurations.

​
Validating Configuration

Before deploying, validate your configuration to catch errors early.

​
Dry Run

Use --dry-run to validate without actually deploying: 
buster deploy --dry-run
This checks:
  • YAML syntax correctness
  • Required fields present (name, prompt, triggers)
  • Trigger configuration validity
  • Agent file references exist

​
Verbose Output

Add --verbose for detailed validation output: 
buster deploy --dry-run --verbose
Shows:
  • Which files are being processed
  • Configuration parsing results
  • What would be deployed

​
Debug Mode

Enable debug logging for troubleshooting: 
buster deploy --debug
Debug logs are written to ~/.buster/logs/debug.log.

​
Viewing Agent Runs

Every agent execution is logged and visible in the Buster web app.

​
Run Logs

Navigate to the Runs tab to see all agent executions. Each run includes:
What caused the execution:
  • PR number, author, changed files (for PR triggers)
  • Timestamp, branch (for scheduled triggers)
  • DAG ID, error message (for Airflow triggers)
Complete list of files read or modified:
  • File paths and operation types
  • Content before/after for modifications
All queries executed against your warehouse:
  • Full SQL text
  • Execution time
  • Rows returned
  • Any errors
Every tool invocation:
  • Tool name and parameters
  • Output or return value
  • Execution duration
The agent’s decision-making process:
  • Why certain actions were taken
  • How conclusions were reached
Final outcomes:
  • PRs created (with links)
  • Comments posted
  • Files modified
  • Slack messages sent
Any failures encountered:
  • Error messages
  • Failed tool calls
  • Stack traces

​
GitHub Check Runs

Every agent execution creates a GitHub Check Run visible on the PR:
  • Queued — Agent execution started
  • In Progress — Agent is running
  • Completed — Finished with success or failure
Click the check run to see the agent’s output summary (limited to 65,535 characters).

​
Common Issues

​
Agent Didn’t Trigger

Check that your includes patterns match the changed files:
# This won't match .yml files
includes:
  - "models/**/*.sql"

# Include both SQL and YAML
includes:
  - "models/**/*.sql"
  - "models/**/*.yml"
Verify the PR targets a branch in your branches list:
# Only triggers for PRs to main
branches:
  - main

# Triggers for main and develop
branches:
  - main
  - develop
Ensure the Buster GitHub App is installed on the repository:
  1. Go to GitHub Settings → Applications → Installed GitHub Apps
  2. Verify “Buster” is listed and has access to the repository
Confirm your agent was deployed successfully:
buster deploy --verbose
Check that the agent file is referenced in buster.yml:
agents:
  - .buster/agents/my-agent.yml

​
Agent Didn’t Take Expected Action

The agent may have interpreted instructions differently than intended. Make prompts more specific:
# Vague
prompt: Check the PR

# Specific
prompt: |
  For each changed SQL file in this PR:
  1. Read the file contents
  2. Check if it has a corresponding YAML documentation file
  3. If missing, create one with basic structure
  4. Post a comment listing what was created
If your agent needs Slack, ensure it’s enabled:
tools:
  include:
    - slack_tool
Review the agent’s reasoning in the run logs to understand why it made certain decisions. Look for:
  • What files it read
  • What conditions it evaluated
  • What actions it attempted

​
Agent Encountered Errors

Error: Query exceeded timeout
The query took too long. Optimize your SQL or add filters:
prompt: |
  When querying large tables, always add a date filter:
  WHERE created_at >= DATEADD('day', -7, CURRENT_DATE())

Error: File 'path/to/file.sql' does not exist
Verify file paths in your prompt. Use Glob to find files dynamically:
prompt: |
  First, use Glob to find all SQL files matching models/**/*.sql
  Then process each file found.

Error: Failed to push to remote
This can happen if:
  • Branch protection rules block the push
  • Another commit was pushed simultaneously (with auto_commit: true)
Consider using auto_commit: false to create a separate PR instead.
Error: Unable to connect to data source
Check your data source configuration in the Buster platform:
  1. Go to Settings → Data Sources
  2. Verify credentials are correct
  3. Ensure the warehouse is accessible (IP allowlisting, etc.)

​
Iterating on Prompts

Most debugging involves refining the agent’s prompt. Start broad and add specificity based on results.

​
Iteration Strategy

1

Start with a simple prompt

prompt: |
  Update documentation for changed models.
Run the agent and observe what it does.
2

Add specific steps

prompt: |
  Update documentation for changed models.

  For each changed model:
  1. Use RetrieveMetadata to get column statistics
  2. Update the YAML file with descriptions
  3. Run dbt parse to validate
  4. Commit the changes
3

Add error handling

prompt: |
  Update documentation for changed models.

  For each changed model:
  1. Use RetrieveMetadata to get column statistics
  2. Update the YAML file with descriptions
  3. Run dbt parse to validate
  4. If validation passes: commit the changes
  5. If validation fails: post the error as a PR comment
4

Add formatting requirements

prompt: |
  Update documentation for changed models.

  For each changed model:
  1. Use RetrieveMetadata to get column statistics
  2. Update the YAML file following our standards:
     - Model description: purpose, grain, row count
     - Column descriptions: business meaning, data type
     - Note null rates if > 5%
     - Use sentence case
  3. Run dbt parse to validate
  4. If validation passes: commit with message "docs: Update [model] documentation"
  5. If validation fails: post the error as a PR comment

​
Testing Edge Cases

Test your agent with various scenarios:
ScenarioWhat to Test
Empty changesPR with no relevant files
Single fileOne model changed
Many files20+ models changed
Invalid SQLSyntax errors in models
Missing YAMLModel without documentation file
Large tablesModels with millions of rows

​
Best Practices

​
Pre-Deployment Checklist

1

Validate configuration

buster deploy --dry-run --verbose
2

Review agent file references

Ensure all agent paths in buster.yml are correct:
agents:
  - .buster/agents/my-agent.yml  # File must exist
3

Test with a sample PR

Create a test branch, make a small change, and open a PR to verify the agent triggers and behaves as expected.
4

Monitor first runs

Watch the first few executions in the Runs tab. Check:
  • Did it trigger on the right events?
  • Did it access the expected files?
  • Did it produce the desired output?
5

Iterate based on results

Refine the prompt based on actual behavior. Add specificity where the agent made unexpected decisions.

​
Start Simple

Begin with a minimal agent and expand: 
# Start here
name: my_agent
prompt: Document changed models.
triggers:
  - event: pull_request

# Then add specificity as needed

​
Use Descriptive Names

Name agents clearly so run logs are easy to identify: 
# Good
name: pr_documentation_updater

# Avoid
name: agent1

​
Log Important Context

Include diagnostic output in your prompts: 
prompt: |
  Before making changes, list:
  - Which files you're processing
  - What metadata you retrieved
  - What changes you're making
  
  This helps with debugging if something goes wrong.

​
Getting Help

Creating Agents

Complete agent configuration reference

Triggers & Scheduling

Trigger types and configuration

Tools & Permissions

Available tools and how to use them

Examples

Working agent examples