GitLab CI Integration for Autotest.ing

Leverage GitLab's powerful CI/CD pipelines to trigger autotest.ing agents. Test Merge Requests and deploy previews automatically.

Prerequisites

Set up a masked variable to securely store your API key:

Go to Settings → CI/CD → Variables.
Add AUTOTEST_API_KEY.
Important: Check the "Mask variable" option to prevent the key from appearing in logs.

Configuration Patterns

Pattern 1: The Standard Pipeline

Add this stage to your .gitlab-ci.yml. It uses the official Node.js Docker image to execute the agent.

                        📄 .gitlab-ci.yml
                        yaml
                    

stages:
  - build
  - test
  - deploy

ai_autotest:
  stage: test
  image: node:20-alpine
  variables:
    # Optional: Override default timeout
    AUTOTEST_TIMEOUT: "15m"
  before_script:
    - npm install -g @autotest/cli
  script:
    - echo "Starting AI Agent..."
    - autotest run --project-id "proj_12345" --ci gitlab --output junit --file report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
    expire_in: 1 week
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

💡 Deep Insight

The use of artifacts:reports:junit is critical here. GitLab has a native "Tests" widget in the Merge Request view that parses this XML. By implementing this, autotest.ing feels native to GitLab users — test results appear directly in the MR UI.

Pattern 2: Testing Review Apps

GitLab's "Review Apps" feature deploys a dynamic environment for every branch. autotest.ing is perfect for validating these ephemeral environments.

                        📄 .gitlab-ci.yml — Review App Testing
                        yaml
                    

ai_review_app_test:
  stage: verify
  image: node:20-alpine
  needs: ["deploy_review_app"] # Wait for deployment to finish
  script:
    # dynamic URL from previous stage or environment variable
    - autotest run --url $CI_ENVIRONMENT_URL --project-id "proj_12345"
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: verify

Handling Flakiness

AI agents are probabilistic. If you experience occasional false positives, use GitLab's retry keyword:

                        📄 Retry configuration
                        yaml
                    

ai_autotest:
  script: autotest run...
  retry:
    max: 2
    when: script_failure

💡 Tip

The retry keyword with when: script_failure ensures only genuine test failures trigger retries, not infrastructure issues like Docker pull failures.

Ready to integrate?

Connect your GitLab repo and run your first AI regression test in minutes.

← All Integrations