Configuring Test Coverage

Overview

The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to Code Climate.

Getting Started

A few important notes when setting up test coverage reporting with Code Climate:

  • You must instrument test coverage yourself as part of your CI build (in one of our supported languages and formats).
  • Test Coverage reporting in Code Climate relies on a universal test reporter which is distributed as a pre-built binary (cc-test-reporter). It gathers the output of your test suite, transforms the output into a generalized format, then uploads the resulting file to Code Climate.
  • The test reporter uses a variety of subcommands. The simplest use case requires only 2 subcommands (cc-test-reporter before-build, cc-test-reporter after-build). More complex use cases (such as parallel builds or multiple test suites) require additional subcommands (cc-test-reporter format-coverage, cc-test-reporter sum-coverage, cc-test-reporter upload-coverage).
  • We require two ENV vars (GIT_COMMIT_SHA and GIT_BRANCH) to be included in your test reports. These are included by default on most CI tools.

To begin setting up test coverage reporting with Code Climate, jump to our Quick Guide below. For those running tests in parallel, head to our section on for parallel tests.

Supported Languages & Formats
Quick Guide
Parallel Tests
Multiple Test Suites
Working with a Docker container
Running tests within a subdirectory
Locations of Pre-built Binaries
List of subcommands
CI Examples
Troubleshooting and Getting Help

Supported Languages and Formats

Language
Format / Coverage Tool

Ruby

JavaScript

lcov (generated by Istanbul e.g.)

PHP

Go

Python

Java

Swift

Cobertuna (using native Xcode coverage and slather e.g.)

gcov (generated by SwiftCov e.g.)

C/C++

Scala

Apex

Quick Guide

  1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.

  2. Fetch the test reporter pre-built binary.

  3. Run the before-build command before running your test suite to notify Code Climate of a pending report.

  4. Execute your test suite.

  5. Run the after-build command after your test suite completes to send a formatted report or notify Code Climate of a failing build.

For examples, click the link for your specific CI provider below:

Parallel Tests

The Code Climate test reporter can join parallelized test reports, combining them into one individual test report that can be submitted to Code Climate.

Configure your CI to store partial results from each parallel run. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage, sum-coverage, and upload-coverage commands to combine the results and upload them as one complete test report.

  1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings in Code Climate.

  2. Fetch the test reporter pre-built binary.

  3. Run before-build before running your test suite to notify of a pending report.

  4. Execute your test suite.

  5. Run format-coverage on the reports from each parallelized instance.

  6. Run sum-coverage to combine the parallelized test reports into one unified report.

  7. Run upload-coverage to upload the combined test report to Code Climate.

For examples, click the link for your specific CI provider below:

Multiple Test Suites

The Code Climate test reporter can join multiple test reports together from separate test suite processes.

Configure your CI to temporarily store the reports from each separate test suite. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage, sum-coverage, and upload-coverage commands to combine the results and upload them as one complete test report.

  1. Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.

  2. Fetch the test reporter pre-built binary.

  3. Run before-build before running your test suite to notify of a pending report.

  4. Execute your test suite.

  5. Run format-coverage on the reports from each individual test suite.

  6. Run sum-coverage to combine multiple test reports into one unified report.

  7. Run upload-coverage to upload the combined test report to Code Climate.

Note: the after-build command is not used with parallel or multi-suite tests.

For examples, click the link for your specific CI provider below:

Running tests within a subdirectory

You may need to specify a prefix value if your tests are run outside of your project's root directory.

For example, if you're running tests within a docker container and your app code is located at /usr/src/app, you'd use:

  • cc-test-reporter format-coverage --prefix /usr/src/app

Locations of Pre-built Binaries

Running test-reporter on a Linux host using a VPN

If you're running the test-reporter on a Linux host that uses a VPN, use the netcgo-linux binary. More details here.

If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.

List of subcommands

format-coverage - formats test report from local test suite into generalized format, readable by Code Climate. Supports the following flags:

  • -t or --input-type simplecov | lcov | coverage.py | gcov | clover - Identifies the input type (format) of the COVERAGE_FILE

  • -o PATH or --output PATH - Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.

  • -p PATH or --prefix PATH - The prefix to remove from absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. Defaults to current working directory.

  • COVERAGE_FILE - Path to the coverage file to process. Defaults to searching known paths where coverage files could exist and selecting the first one found.

sum-coverage - combines test reports from multiple sources (i.e. multiple test suites or parallelized CI builds) into one test report which is readable by Code Climate. Supports the following flags:

  • -o PATH or --output PATH - Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json.

  • -p NUMBER or --parts NUMBER - Expect NUMBER payloads to sum. If this many arguments are not present, command will error. This ensures you don't accidentally sum incomplete results.

upload-coverage - uploads formatted, singular test report to Code Climate API. Supports the following flags:

  • -i PATH or --input PATH - Read payload from PATH. If - is given, the payload will be read from stdin. Defaults to coverage/codeclimate.json.

  • -r ID or --id ID - The reporter identifier to use when reporting coverage information. The appropriate value can be found in your Repository Settings page on codeclimate.com. Defaults to the value in the CC_TEST_REPORTER_ID environment variable. The uploader will error if a value is not found.

  • -e URL or --endpoint URL - The endpoint to upload coverage information to. Defaults to the value in the CC_TEST_REPORTER_COVERAGE_ENDPOINT environment variable, or a hard-coded default (currently "https://codeclimate.com/test_reports").

after-build - combines format-coverage and upload-coverage. Supports the following flags:

  • --exit-code $EXIT_CODE - $EXIT_CODE should be the exit code of your test suite process. Some CI system expose this as an environment variable; for others, you may need to manually capture $? to provide it to after-build later. Providing this will prevent sending test coverage results for failed tests.

CI Examples

We support any CI provider

Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.

Below, you'll find an examples of how to configure our test reporter with some common CI providers.

Troubleshooting and Getting Help

Apex

If you're using Apex, you will be able to see coverage information by using the tool apexcov before using the test reporter.

For example, you can use it like this:

- apexcov
- ./cc-test-reporter format-coverage -t lcov ./coverage/lcov.info
- ./cc-test-reporter upload-coverage

For more details and examples, go to the apexcov documentation.

Jacoco

Most often users see this message with Java projects when the path to source code can't be inferred:
Error: could not find any viable formatter. available formatters: simplecov, lcov, coverage.py, clover, gocov, gcov, cobertura, jacoco


For these cases, instead of after-build, use:

  1. format-coverage, which includes:
    • JACOCO_SOURCE_PATH: the path to Java source files (a new environment variable)
    • coverage file: to path to JaCoCo coverage XML file (the first argument)

AND

  1. upload-coverage

Example and more details in this conversation.

Need more help?

  1. Check Test Coverage Troubleshooting Tips
  2. Open an issue on the test reporter project
  3. Check example scripts on the test reporter project
  4. Contact Support

Opening an issue or contacting Support?

Please Include:

  1. Repo name
  2. CI name
  3. If you use single, parallel or multi-suite builds
  4. A copy of your CI configuration file
  5. Output from your CI when using the --debug flag in your CI configuration
  6. Any relevant screenshots of your setup

PS: Make sure that you're able to run coverage results locally with the basic requirements, outside of Code Climate's test reporter.