← All posts

Automated Testing

Test automation reporting across multiple CI systems

Send JUnit XML from Jenkins, GitHub Actions, GitLab and Azure DevOps to one Qase test run and get a single test automation report.

Qase Team

Qase Team

Nov 5, 20269 min read
Four different sources streaming particles into one central stacked panel, representing test results from multiple CI systems in a single report

Ask a team that runs tests in three CI systems whether the release is ready, and someone opens three browser tabs. Jenkins has the backend suite. GitHub Actions runs Playwright on every pull request. The mobile team's pipeline lives in Azure DevOps because that's where it started four years ago, and moving it has never been worth the trouble. Each tab is green or red about its own slice. The real answer gets pieced together in a Slack thread.

This guide is the fix we'd set up ourselves: every CI sends its JUnit XML to the same Qase test run, each CI keeps its own section inside that run, and the release ends with one report you can link to. It covers GitHub Actions, Jenkins, GitLab CI and Azure DevOps. If you only use two of those, skip the other sections. The setup is the same idea every time.

Why does test automation reporting break when results live in more than one CI?

Each CI keeps its own report format and its own history, nothing joins them, and the release pass/fail ends up assembled by hand.

Every CI system treats test results as an artifact of a single run. Jenkins gives you a build page. GitHub Actions gives you a log and maybe an uploaded HTML report. GitLab shows a test tab on the pipeline. Azure DevOps has its own Tests view. All of them answer "what happened in this run?" reasonably well. None of them can answer "what happened across all our pipelines for release 2.14?", because none of them know the others exist.

So the history fragments too. If a checkout test fails in Jenkins on Tuesday and a related flaky test shows up in GitHub Actions on Thursday, you'd only connect them if the same person happened to see both.

The good news is that almost every test runner already speaks one common format. Playwright, pytest, JUnit, Cypress, Maven Surefire and Gradle can all write JUnit XML. That shared format is what makes a single destination possible. Each team keeps running its tests the way it does today, and each pipeline adds one step that sends its XML to the same place.

What you need before you start

The list is short:

  • A Qase project and its code. The code is the short prefix you see on test case IDs, like SHOP.
  • A Qase API token. Create one in your workspace settings and store it as a secret in each CI (we'll call it QASE_TOKEN throughout).
  • JUnit XML output from each CI job. Most runners need one flag or one config line. For Playwright, add a JUnit reporter in playwright.config.ts:
reporter: [["list"], ["junit", { outputFile: "results.xml" }]],

If you want to try this before committing to anything, the Free plan allows 5,000 API-submitted results a month and 150 requests a minute. That's plenty for a walkthrough with a couple of real pipelines.

Step 1: Create one Qase test run for the release

Everything below depends on a single run ID that every CI shares. Get that right and the rest is mostly copy and paste.

You can create the run in the Qase UI and copy its ID from the URL. Or use the Qase CLI, qasectl, which is what the rest of this guide uses anyway. Install it with Go or Homebrew:

go install github.com/qase-tms/qasectl@latest
# or
brew tap qase-tms/tap && brew install qasectl

There's also a Docker image, ghcr.io/qase-tms/qase-cli:latest, which is handy in CI because there's nothing to install.

Then create the run:

qasectl testops run create \
  --project SHOP \
  --token "$QASE_TOKEN" \
  --title "Release 2.14" \
  --output qase.env

That writes a file containing one line, QASE_TESTOPS_RUN_ID=123. The number is your run ID.

Now the part that takes a little thought: getting that ID to four different CI systems. Within one CI it's easy (a job output, an artifact, a variable). Across systems, the simplest thing that works is to treat the run ID as a release input. Whoever cuts the release creates the run, then starts each pipeline with the ID as a parameter. If your release process is already scripted, add the run create call to that script and pass the ID along with the version number.

One rule matters here. The run has to stay open until the last CI reports. A completed run is locked, so no job should complete it except the final step in Step 7.

Step 2: GitHub Actions

Qase maintains official actions in the qase-tms/gh-actions repo: run-create, run-complete and report, among others. For a shared run you only need report.

Here's a Playwright workflow that takes the run ID as an input when you start it:

name: e2e

on:
  workflow_dispatch:
    inputs:
      qase_run_id:
        description: Qase run ID for this release
        required: true

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test

      - name: Report to Qase
        if: always()
        uses: qase-tms/gh-actions/report@v1
        with:
          token: ${{ secrets.QASE_TOKEN }}
          project: SHOP
          id: ${{ inputs.qase_run_id }}
          format: junit
          path: results.xml
          batch: 100

Look at if: always(). Without it, a failing test step stops the job and the upload never runs, so Qase only hears about the green builds. That's the opposite of what you want.

If the run is created inside the same workflow (using run-create), swap the input for ${{ steps.qase-run-create.outputs.id }}.

Step 3: Jenkins

Jenkins uses the same CLI in a shell step. We couldn't find a Jenkins plugin worth recommending instead, and the CLI is the route Qase documents, so install qasectl on the agent (or run the Docker image) and call it from the pipeline.

This example is a Maven build that takes the run ID as a build parameter:

pipeline {
  agent any
  parameters {
    string(name: 'QASE_RUN_ID', description: 'Qase run ID for this release')
  }
  stages {
    stage('Test') {
      steps {
        sh 'mvn test'
      }
    }
  }
  post {
    always {
      withCredentials([string(credentialsId: 'qase-token', variable: 'QASE_TOKEN')]) {
        sh '''
          qasectl testops result upload \
            --project SHOP \
            --token "$QASE_TOKEN" \
            --id "$QASE_RUN_ID" \
            --format junit \
            --path target/surefire-reports/
        '''
      }
    }
  }
}

A few notes:

  • --path takes a file or a folder. Surefire writes one XML file per test class, so pointing it at the folder picks them all up.
  • The upload is in post { always { } } for the same reason as if: always() in GitHub Actions.
  • Keep the token in Jenkins credentials. That way it never appears in the Jenkinsfile or in the repo.

Step 4: GitLab CI

In GitLab, the cleanest setup is a separate publish job that runs the Qase CLI image and picks up the XML as an artifact from the test stage:

stages: [test, report]

api-tests:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    paths: [report.xml]

qase-upload:
  stage: report
  image:
    name: ghcr.io/qase-tms/qase-cli:latest
    entrypoint: [""]
  when: always
  script:
    - >
      qasectl testops result upload
      --project SHOP
      --token "$QASE_TOKEN"
      --id "$QASE_RUN_ID"
      --format junit
      --path report.xml

Two lines do the quiet work here. artifacts: when: always keeps report.xml even when tests fail, and when: always on the upload job runs it after a failed stage. The entrypoint: [""] override is there because the image's entrypoint is qasectl itself, and GitLab needs a shell to run the script.

Set QASE_TOKEN as a masked CI/CD variable. Pass QASE_RUN_ID when you trigger the pipeline.

Step 5: Azure DevOps

Azure Pipelines already reads JUnit XML through its PublishTestResults task, so if your pipeline uses it, the XML file already exists. Keep that task if people use the Azure Tests tab. Then upload the same file to Qase with a script step.

There's nothing Azure-specific to install. It's the same CLI call, run through Docker:

steps:
  - script: ./gradlew test
    displayName: Run tests

  - task: PublishTestResults@2
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: '**/build/test-results/test/*.xml'

  - script: |
      docker run --rm -v "$(Build.SourcesDirectory):/work" \
        ghcr.io/qase-tms/qase-cli:latest \
        testops result upload \
        --project SHOP \
        --token "$QASE_TOKEN" \
        --id "$(QASE_RUN_ID)" \
        --format junit \
        --path /work/build/test-results/test/
    displayName: Upload results to Qase
    condition: always()
    env:
      QASE_TOKEN: $(QaseToken)

Azure doesn't expose secret variables to scripts automatically, which is why the env: block maps QaseToken in by hand. If the upload says the token is missing, check this first.

Step 6: Keep each CI as its own section in the run

At this point every result lands in one run, which is already better than four tabs. But if all four CIs dump results into the same flat list, it's hard to tell the Jenkins backend suite from the GitHub Playwright suite.

The fix is one flag. --suite on qasectl testops result upload sets a root suite for that upload, so each CI gets its own top-level folder in the run:

qasectl testops result upload ... --suite "Jenkins"
qasectl testops result upload ... --suite "GitLab CI"
qasectl testops result upload ... --suite "Azure DevOps"

Add it to the commands in Steps 3 to 5.

GitHub Actions needs one change. When we checked in November 2026, the report action had no suite input. If you want GitHub results in their own folder too, replace the action with the CLI image:

- name: Report to Qase
  if: always()
  run: |
    docker run --rm -v "$PWD:/work" ghcr.io/qase-tms/qase-cli:latest \
      testops result upload --project SHOP --token "$QASE_TOKEN" \
      --id "${{ inputs.qase_run_id }}" --format junit \
      --path /work/results.xml --suite "GitHub Actions"
  env:
    QASE_TOKEN: ${{ secrets.QASE_TOKEN }}

Some teams use a Qase reporter (the Playwright, pytest or Cypress plugin) instead of uploading XML. That works in the same run. Set QASE_TESTOPS_RUN_ID to the shared run ID, set QASE_ROOT_SUITE (or rootSuite in qase.config.json) to name its section, and set run.complete to false. That last one matters: by default, a reporter completes the run when its tests finish, and the first job to finish would lock the run before the others report.

Step 7: Complete the run and share the report

When the last pipeline has reported, complete the run:

qasectl testops run complete --project SHOP --token "$QASE_TOKEN" --id 123

Put this wherever "the release testing is done" actually happens for your team. That might be a final job in whichever CI runs last, a step in the release script, or a person clicking Complete in the UI after the manual checks. Just make sure only one place does it.

Then open the run dashboard. You'll see:

  • pass/fail broken down by suite, so each CI is its own row
  • failed tests with their attachments and error output
  • how long each test took

Two more things make the report useful to people outside the test team. First, turn on the run's public report link (or set showPublicReportLink in a reporter config), so a product manager or release manager without a Qase login can open it. Second, link the run to the release ticket.

Add the Jira issue under External Issues on the run dashboard, or set externalLink with type jiraCloud in a reporter config. Now the results travel with the release.

That's the whole setup. The first time through probably takes an afternoon, and most of it goes into getting the run ID to each pipeline. After that, the answer to "are we good to ship?" is a link.

Further reading: Flaky tests | A guide to continuous testing | Automated test management in Qase | Qase CLI reference

Ship quality software faster with Qase