Configuring Test Coverage

Overview

The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to Code Climate.

Getting Started

A few important notes when setting up test coverage reporting with Code Climate:

• You must instrument test coverage yourself as part of your CI build (in one of our supported languages and formats).
• Test Coverage reporting in Code Climate relies on a universal test reporter which is distributed as a pre-built binary (cc-test-reporter). It gathers the output of your test suite, transforms the output into a generalized format, then uploads the resulting file to Code Climate.
• The test reporter uses a variety of subcommands. The simplest use case requires only 2 subcommands (cc-test-reporter before-build, cc-test-reporter after-build). More complex use cases (such as parallel builds or multiple test suites) require additional subcommands (cc-test-reporter format-coverage, cc-test-reporter sum-coverage, cc-test-reporter upload-coverage).
• We require two ENV vars (GIT_COMMIT_SHA and GIT_BRANCH) to be included in your test reports. These are included by default on most CI tools.

To begin setting up test coverage reporting with Code Climate, jump to our Quick Guide below. For those running tests in parallel, head to our section on for parallel tests.

Supported Languages & Formats
Quick Guide
Parallel Tests
Multiple Test Suites
Working with a Docker container
Running tests within a subdirectory
Locations of Pre-built Binaries
List of subcommands
CI Examples
Troubleshooting and Getting Help

Supported Languages and Formats

Quick Guide

In your CI's config file or CI's in-app config options:

1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.

2. Fetch the test reporter pre-built binary.

3. Run the before-build command before running your test suite to notify Code Climate of a pending report.

4. Execute your test suite.

5. Run the after-build command after your test suite completes to send a formatted report or notify Code Climate of a failing build.

For examples, click the link for your specific CI provider below:

• Travis
• CircleCI
• Solano
• Codeship
• Semaphore
• Jenkins

Parallel Tests

The Code Climate test reporter can join parallelized test reports, combining them into one individual test report that can be submitted to Code Climate.

Configure your CI to store partial results from each parallel run. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage, sum-coverage, and upload-coverage commands to combine the results and upload them as one complete test report.

1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings in Code Climate.

2. Fetch the test reporter pre-built binary.

3. Run before-build before running your test suite to notify of a pending report.

4. Execute your test suite.

5. Run format-coverage on the reports from each parallelized instance.

6. Run sum-coverage to combine the parallelized test reports into one unified report.

7. Run upload-coverage to upload the combined test report to Code Climate.

For examples, click the link for your specific CI provider below:

• CircleCI
• See the README on GitHub for detailed instructions.

Multiple Test Suites

The Code Climate test reporter can join multiple test reports together from separate test suite processes.

Configure your CI to temporarily store the reports from each separate test suite. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage, sum-coverage, and upload-coverage commands to combine the results and upload them as one complete test report.

1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.

2. Fetch the test reporter pre-built binary.

3. Run before-build before running your test suite to notify of a pending report.

4. Execute your test suite.

5. Run format-coverage on the reports from each individual test suite.

6. Run sum-coverage to combine multiple test reports into one unified report.

7. Run upload-coverage to upload the combined test report to Code Climate.

Note: the after-build command is not used with parallel or multi-suite tests.

For examples, click the link for your specific CI provider below:

• CircleCI
• Travis CI

Running tests within a subdirectory

You may need to specify a prefix value if your tests are run outside of your project's root directory.

For example, if you're running tests within a docker container and your app code is located at /usr/src/app, you'd use:

• cc-test-reporter format-coverage --prefix /usr/src/app

Locations of Pre-built Binaries

The test report is distributed as a pre-built binary named cc-test-reporter. You can fetch the pre-built binary from the following URLs:

Linux

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64

• https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-amd64

Linux ARM64

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-arm64

• https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-arm64

Linux netcgo (recommended if you're using a VPN)

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-netcgo-linux-amd64

• https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-netcgo-linux-amd64

OS X

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-darwin-amd64

• https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64

Windows

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-windows-amd64

• https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64

Verifying Binaries

Along with the binaries you can download a file with a SHA 256 checksum for the given version and a GPG detached signature. Check this docs for further details.

📘

Running test-reporter on a Linux host using a VPN

If you're running the test-reporter on a Linux host that uses a VPN, use the netcgo-linux binary. More details here.

📘

If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.

List of subcommands

format-coverage - formats test report from local test suite into generalized format, readable by Code Climate. Supports the following flags:

• -t or --input-type simplecov | lcov | coverage.py | gcov | clover - Identifies the input type (format) of the COVERAGE_FILE

• -o PATH or --output PATH - Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.

• -p PATH or --prefix PATH - The prefix to remove from absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. Defaults to current working directory.

• COVERAGE_FILE - Path to the coverage file to process. Defaults to searching known paths where coverage files could exist and selecting the first one found.

sum-coverage - combines test reports from multiple sources (i.e. multiple test suites or parallelized CI builds) into one test report which is readable by Code Climate. Supports the following flags:

• -o PATH or --output PATH - Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.

• -p NUMBER or --parts NUMBER - Expect NUMBER payloads to sum. If this many arguments are not present, command will error. This ensures you don't accidentally sum incomplete results.

upload-coverage - uploads formatted, singular test report to Code Climate API. Supports the following flags:

• -i PATH or --input PATH - Read payload from PATH. If - is given, the payload will be read from stdin. Defaults to coverage/codeclimate.json.

• -r ID or --id ID - The reporter identifier to use when reporting coverage information. The appropriate value can be found in your Repository Settings page on codeclimate.com. Defaults to the value in the CC_TEST_REPORTER_ID environment variable. The uploader will error if a value is not found.

• -e URL or --endpoint URL - For use only with Code Climate: Enterprise. The endpoint to upload coverage information to. Defaults to the value in the CC_TEST_REPORTER_COVERAGE_ENDPOINT environment variable, or a hard-coded default (currently "https://api.codeclimate.com/v1/test_reports").

after-build - combines format-coverage and upload-coverage. Supports the following flags:

• --exit-code $EXIT_CODE - $EXIT_CODE should be the exit code of your test suite process. Some CI system expose this as an environment variable; for others, you may need to manually capture $? to provide it to after-build later. Providing this will prevent sending test coverage results for failed tests.

CI Examples

📘

We support any CI provider

Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.

Below, you'll find an examples of how to configure our test reporter with some common CI providers.

• Travis CI
• Solano CI
• Circle CI
• Codeship CI
• Jenkins
• GitHub Actions
• GitHub Actions with XCode
• More Examples

Troubleshooting and Getting Help

• Test Coverage Troubleshooting Tips
• Troubleshooting branch names
• My diff-coverage and total-coverage statuses are hanging on my PR
• Working with a docker container?

Apex

📘

Apex

If you're using Apex, you will be able to see coverage information by using the tool apexcov before using the test reporter.

For example, you can use it like this:

- apexcov
- ./cc-test-reporter format-coverage -t lcov ./coverage/lcov.info
- ./cc-test-reporter upload-coverage

For more details and examples, go to the apexcov documentation.

Jacoco

📘

Jacoco

Most often users see this message with Java projects when the path to source code can't be inferred:
Error: could not find any viable formatter. available formatters: simplecov, lcov, coverage.py, clover, gocov, gcov, cobertura, jacoco

---

For these cases, instead of after-build, use:

1. format-coverage, which includes:

• JACOCO_SOURCE_PATH: the path to Java source files (a new environment variable)
• coverage file: to path to JaCoCo coverage XML file (the first argument)

AND

2. upload-coverage

---

Example and more details in this conversation.

---

When setting JACOCO_SOURCE_PATH some users may receive the following error: could not find coverage info for source files. To move forward, remove the group tag from your jacoco.xml. The group tag is not currently supported.

📘

Jacoco group tag

When using a group tag in your jacoco.xml, you might need to adjust your source path like this:

(thanks to https://github.com/seancarroll/jfluentvalidation/commit/9a9e7c97049a5db3441f9811698f8d9a1a72a17e)

AWS Codebuild

📘

One way to add needed information to your AWS CodeBuild build via environment variables for the Code Climate test reporter:

Add the following command to the install or pre_build phase of your buildspec:

curl -fsSL https://raw.githubusercontent.com/easybib/aws-codebuild-extras/master/install >> extras.sh && . ./extras.sh

Or for better readability, break the installation into two steps. For example in the install phase:

phases:
  install:
    commands:
      - echo Installing codebuild-extras...
      - curl -fsSL https://raw.githubusercontent.com/easybib/aws-codebuild-extras/master/install >> extras.sh
      - . ./extras.sh

Thanks to https://github.com/easybib/aws-codebuild-extras for this example!

Azure Pipelines

📘

One way to add needed information to your Azure Pipelines is to specify environment variables for the Code Climate test reporter:

Add the following script:

- script: |
      curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
      chmod +x ./cc-test-reporter
      ./cc-test-reporter format-coverage -t lcov -o coverage/codeclimate.json coverage/lcov.info
      ./cc-test-reporter upload-coverage
    displayName: Upload coverage to Code Climate
    env:
      CC_TEST_REPORTER_ID: $(frontend-codeclimate-reporter-id)
      GIT_BRANCH: $(Build.SourceBranchName)
      GIT_COMMIT: $(Build.SourceVersion)

Thanks to @kpapadatos for the above example!

A different customer provided this context:
Azure sets up two variables BUILD_SOURCEBRANCH and BUILD_SOURCEVERSION with information from GIT. The branch is checked out as refs/remote/${PRNumber}/merge or refs/heads/master and this is put into the source branch variable.

In the end I had to write a script to fetch the PR from GitHub via the API by parsing the number out of the source branch variable and using that to get the actual branch name back and writing that into an environment variable of GIT_BRANCH.

However, for some reason, cc-test-reporter wasn’t able to pick this up and was thus looking in the GIT repo for the value which, thanks to the way Azure scripts the branch checkout, was coming through in a detached head state! In the end, I had to force the use of the right environmental variable and this sorted it.

Get more help - contact us!

❗️

Need more help?

1. Check Test Coverage Troubleshooting Tips
2. Open an issue on the test reporter project
3. Check example scripts on the test reporter project
4. Contact Support

👍

Opening an issue or contacting Support?

Please Include:

1. Repo name
2. CI name
3. If you use single, parallel or multi-suite builds
4. A copy of your CI configuration file
5. Output from your CI when using the --debug flag in your CI configuration
6. Any relevant screenshots of your setup

PS: Make sure that you're able to run coverage results locally with the basic requirements, outside of Code Climate's test reporter.