GitHub Actions Integration for Autotest.ing

Integrate autotest.ing directly into your GitHub Actions workflow to enable autonomous AI agent testing on every Pull Request or push.

By adding our intelligent agent to your pipeline, you can catch functional regressions and visual anomalies before they merge — without maintaining brittle selectors.

Prerequisites

Before configuring the workflow, ensure you have:

Autotest.ing Project ID — Found in your Project Settings.
API Key — Generated from the Settings → API Keys section.

Setting Up GitHub Secrets

Navigate to your repository on GitHub.
Go to Settings → Secrets and variables → Actions.
Click New repository secret.
Name: AUTOTEST_API_KEY
Value: Your copied API key.

Integration Strategies

We recommend two primary integration patterns: Pull Request Checks (fast, blocking) and Nightly Regression (comprehensive, non-blocking).

Scenario 1: The Pull Request Check (Blocking)

This workflow runs on every PR to main. It executes a "Sanity" suite using the AI agent. If the agent detects a critical failure, the PR check will fail, preventing the merge.

            📄 .github/workflows/autotest-pr.yml
            yaml
          

name: Autotest.ing AI Check
on:
  pull_request:
    branches: [ "main" ]

jobs:
  ai-verification:
    name: AI Agent Verification
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js environment
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Autotest CLI
        run: npm install -g @autotest/cli

      - name: Run AI Agent
        env:
          AUTOTEST_API_KEY: ${{ secrets.AUTOTEST_API_KEY }}
        # The agent automatically detects GITHUB_SHA and GITHUB_REF
        run: |
          autotest run \
            --project-id "proj_12345" \
            --suite "sanity" \
            --wait \
            --output-format junit \
            --output-file autotest-results.xml

      - name: Publish Test Report
        uses: mikepenz/action-junit-report@v4
        if: always() # Always run to see results even if tests fail
        with:
          report_paths: 'autotest-results.xml'

💡 Insight

We use mikepenz/action-junit-report to visualize the results. This is a crucial "delighter" feature that elevates the integration beyond simple console logs — test results appear as rich annotations directly in the PR.

Scenario 2: Nightly Full Regression (Non-Blocking)

AI agents perform best when given time to explore. A nightly job can run a comprehensive exploration without slowing down developers during the day.

            📄 .github/workflows/autotest-nightly.yml
            yaml
          

name: Nightly AI Exploration
on:
  schedule:
    - cron: '0 2 * * *' # Runs at 2 AM UTC

jobs:
  deep-exploration:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }

      - name: Run Deep Exploration Agent
        env:
          AUTOTEST_API_KEY: ${{ secrets.AUTOTEST_API_KEY }}
        run: |
          npx @autotest/cli run \
            --project-id "proj_12345" \
            --mode exploratory \
            --timeout 30m \
            --non-blocking

Advanced: Posting Comments to PRs

The autotest.ing agent can act as a reviewer by posting screenshots and insights directly to the PR conversation. To enable this, pass the GITHUB_TOKEN.

            📄 Workflow step snippet
            yaml
          

      - name: Run with PR Comments
        env:
          AUTOTEST_API_KEY: ${{ secrets.AUTOTEST_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: autotest run --comment-on-pr

Troubleshooting

⚠ Error: 401 Unauthorized

Verify that the AUTOTEST_API_KEY is set in Secrets, not Variables. Environment variables and repository secrets are configured in different sections of GitHub settings.

⚠ Error: Process exited with code 1

The agent found a bug! Check the "Test Report" tab or the console logs for the video link. This is expected behavior when regressions are detected.

⚠ Timeout

If the agent takes longer than 60 minutes, GitHub will kill the job. Set timeout-minutes: 120 in the job configuration to extend the limit.

Ready to integrate?

Connect your GitHub repo and run your first AI regression test in under 5 minutes.

← All Integrations