Configuring Test Coverage
Overview
The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to Code Climate.
Getting Started
A few important notes when setting up test coverage reporting with Code Climate:
- You must instrument test coverage yourself as part of your CI build (in one of our supported languages and formats).
- Test Coverage reporting in Code Climate relies on a universal test reporter which is distributed as a pre-built binary (
cc-test-reporter
). It gathers the output of your test suite, transforms the output into a generalized format, then uploads the resulting file to Code Climate. - The test reporter uses a variety of subcommands. The simplest use case requires only 2 subcommands (
cc-test-reporter before-build
,cc-test-reporter after-build
). More complex use cases (such as parallel builds or multiple test suites) require additional subcommands (cc-test-reporter format-coverage
,cc-test-reporter sum-coverage
,cc-test-reporter upload-coverage
). - We require two ENV vars (
GIT_COMMIT_SHA
andGIT_BRANCH
) to be included in your test reports. These are included by default on most CI tools.
To begin setting up test coverage reporting with Code Climate, jump to our Quick Guide below. For those running tests in parallel, head to our section on for parallel tests.
Supported Languages & Formats
Quick Guide
Parallel Tests
Multiple Test Suites
Working with a Docker container
Running tests within a subdirectory
Locations of Pre-built Binaries
List of subcommands
CI Examples
Troubleshooting and Getting Help
Supported Languages and Formats
Language | Format / Coverage Tool |
---|---|
Ruby | SimpleCov Note: Versions Simplecov 0.18 and Simplecov 0.19 are not supported. We offer support for Simplecov 0.17 and earlier and for Simplecov 0.20+ |
JavaScript | lcov (generated by Istanbul e.g.) |
PHP | Clover |
Go | Cover |
Python | CoveragePy |
Java | Cobertura and Jacoco (note**) |
Swift | Cobertuna (using native Xcode coverage and slather e.g.) gcov (generated by SwiftCov e.g.) |
C/C++ | gcov |
Scala | Cobertura |
Apex | Apex (note*) |
Kotlin | Cobertura and Jacoco. (note**) |
Quick Guide
In your CI's config file or CI's in-app config options:
-
Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.
-
Fetch the test reporter pre-built binary.
-
Run the
before-build
command before running your test suite to notify Code Climate of a pending report. -
Execute your test suite.
-
Run the
after-build
command after your test suite completes to send a formatted report or notify Code Climate of a failing build.
For examples, click the link for your specific CI provider below:
Parallel Tests
The Code Climate test reporter can join parallelized test reports, combining them into one individual test report that can be submitted to Code Climate.
Configure your CI to store partial results from each parallel run. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage
, sum-coverage
, and upload-coverage
commands to combine the results and upload them as one complete test report.
-
Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings in Code Climate.
-
Fetch the test reporter pre-built binary.
-
Run
before-build
before running your test suite to notify of a pending report. -
Execute your test suite.
-
Run
format-coverage
on the reports from each parallelized instance. -
Run
sum-coverage
to combine the parallelized test reports into one unified report. -
Run
upload-coverage
to upload the combined test report to Code Climate.
For examples, click the link for your specific CI provider below:
- CircleCI
- See the README on GitHub for detailed instructions.
Multiple Test Suites
The Code Climate test reporter can join multiple test reports together from separate test suite processes.
Configure your CI to temporarily store the reports from each separate test suite. We recommend syncing and fetching files from S3 as one approach. Alternately, you can use a temporary folder in your CI build. Then, use the test reporter's format-coverage
, sum-coverage
, and upload-coverage
commands to combine the results and upload them as one complete test report.
-
Set your repo's test reporter ID as an environment variable to identify your repo. This can be found on your Repo Settings page in Code Climate.
-
Fetch the test reporter pre-built binary.
-
Run
before-build
before running your test suite to notify of a pending report. -
Execute your test suite.
-
Run
format-coverage
on the reports from each individual test suite. -
Run
sum-coverage
to combine multiple test reports into one unified report. -
Run
upload-coverage
to upload the combined test report to Code Climate.
Note: the after-build
command is not used with parallel or multi-suite tests.
For examples, click the link for your specific CI provider below:
Running tests within a subdirectory
You may need to specify a prefix value if your tests are run outside of your project's root directory.
For example, if you're running tests within a docker container and your app code is located at /usr/src/app
, you'd use:
cc-test-reporter format-coverage --prefix /usr/src/app
Locations of Pre-built Binaries
The test report is distributed as a pre-built binary named cc-test-reporter
. You can fetch the pre-built binary from the following URLs:
Linux
-
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-amd64
Linux ARM64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-arm64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-linux-arm64
Linux netcgo (recommended if you're using a VPN)
-
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-netcgo-linux-amd64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-netcgo-linux-amd64
OS X
-
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-darwin-amd64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64
Windows
-
https://codeclimate.com/downloads/test-reporter/test-reporter-latest-windows-amd64
-
https://codeclimate.com/downloads/test-reporter/test-reporter-0.11.1-darwin-amd64
Verifying Binaries
Along with the binaries you can download a file with a SHA 256 checksum for the given version and a GPG detached signature. Check this docs for further details.
Running test-reporter on a Linux host using a VPN
If you're running the test-reporter on a Linux host that uses a VPN, use the netcgo-linux binary. More details here.
If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.
List of subcommands
format-coverage
- formats test report from local test suite into generalized format, readable by Code Climate. Supports the following flags:
-
-t
or--input-type
simplecov | lcov | coverage.py | gcov | clover - Identifies the input type (format) of the COVERAGE_FILE -
-o PATH
or--output PATH
- Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json. -
-p PATH
or--prefix PATH
- The prefix to remove from absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. Defaults to current working directory. -
COVERAGE_FILE
- Path to the coverage file to process. Defaults to searching known paths where coverage files could exist and selecting the first one found.
sum-coverage
- combines test reports from multiple sources (i.e. multiple test suites or parallelized CI builds) into one test report which is readable by Code Climate. Supports the following flags:
-
-o PATH
or--output PATH
- Output to PATH. If - is given, content will be written to stdout. Defaults to coverage/codeclimate.json. -
-p NUMBER
or--parts NUMBER
- Expect NUMBER payloads to sum. If this many arguments are not present, command will error. This ensures you don't accidentally sum incomplete results.
upload-coverage
- uploads formatted, singular test report to Code Climate API. Supports the following flags:
-
-i PATH
or--input PATH
- Read payload from PATH. If - is given, the payload will be read from stdin. Defaults to coverage/codeclimate.json. -
-r ID
or--id ID
- The reporter identifier to use when reporting coverage information. The appropriate value can be found in your Repository Settings page on codeclimate.com. Defaults to the value in theCC_TEST_REPORTER_ID
environment variable. The uploader will error if a value is not found. -
-e URL
or--endpoint URL
- For use only with Code Climate: Enterprise. The endpoint to upload coverage information to. Defaults to the value in the CC_TEST_REPORTER_COVERAGE_ENDPOINT environment variable, or a hard-coded default (currently "https://api.codeclimate.com/v1/test_reports").
after-build
- combines format-coverage
and upload-coverage
. Supports the following flags:
--exit-code $EXIT_CODE
-$EXIT_CODE
should be the exit code of your test suite process. Some CI system expose this as an environment variable; for others, you may need to manually capture$?
to provide it toafter-build
later. Providing this will prevent sending test coverage results for failed tests.
CI Examples
We support any CI provider
Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.
Below, you'll find an examples of how to configure our test reporter with some common CI providers.
- Travis CI
- Solano CI
- Circle CI
- Codeship CI
- Jenkins
- GitHub Actions
- GitHub Actions with XCode
- More Examples
Troubleshooting and Getting Help
- Test Coverage Troubleshooting Tips
- Troubleshooting branch names
- My diff-coverage and total-coverage statuses are hanging on my PR
- Working with a docker container?
Apex
Apex
If you're using Apex, you will be able to see coverage information by using the tool apexcov before using the test reporter.
For example, you can use it like this:
- apexcov - ./cc-test-reporter format-coverage -t lcov ./coverage/lcov.info - ./cc-test-reporter upload-coverage
For more details and examples, go to the apexcov documentation.
Jacoco
Jacoco
Most often users see this message with Java projects when the path to source code can't be inferred:
Error: could not find any viable formatter. available formatters: simplecov, lcov, coverage.py, clover, gocov, gcov, cobertura, jacoco
For these cases, instead of
after-build
, use:
format-coverage
, which includes:
JACOCO_SOURCE_PATH
: the path to Java source files (a new environment variable)coverage file
: to path to JaCoCo coverage XML file (the first argument)AND
upload-coverage
Example and more details in this conversation.
When setting
JACOCO_SOURCE_PATH
some users may receive the following error:could not find coverage info for source files
. To move forward, remove thegroup
tag from yourjacoco.xml
. Thegroup
tag is not currently supported.
Jacoco
group
tagWhen using a group tag in your jacoco.xml, you might need to adjust your source path like this:
(thanks to https://github.com/seancarroll/jfluentvalidation/commit/9a9e7c97049a5db3441f9811698f8d9a1a72a17e)
AWS Codebuild
One way to add needed information to your AWS CodeBuild build via environment variables for the Code Climate test reporter:
Add the following command to the install or pre_build phase of your buildspec:
curl -fsSL https://raw.githubusercontent.com/easybib/aws-codebuild-extras/master/install >> extras.sh && . ./extras.sh
Or for better readability, break the installation into two steps. For example in the install phase:
phases:
install:
commands:
- echo Installing codebuild-extras...
- curl -fsSL https://raw.githubusercontent.com/easybib/aws-codebuild-extras/master/install >> extras.sh
- . ./extras.sh
NAME | VALUE |
---|---|
CI | true |
CODEBUILD | true |
CODEBUILD_GIT_AUTHOR | Committer Name |
CODEBUILD_GIT_AUTHOR_EMAIL | [email protected] |
CODEBUILD_GIT_BRANCH | branch name |
CODEBUILD_GIT_COMMIT | commit hash |
CODEBUILD_GIT_COMMITTED_AT | commit timestamp |
CODEBUILD_GIT_MESSAGE | commit message |
CODEBUILD_GIT_TAG | git tag |
CODEBUILD_PROJECT | project |
CODEBUILD_PULL_REQUEST | Pull request number |
GIT_BRANCH | Alias to CODEBUILD_GIT_BRANCH for CodeClimate |
GIT_COMMIT_SHA | Alias to CODEBUILD_GIT_COMMIT for CodeClimate |
GIT_COMMITTED_AT | Alias to CODEBUILD_GIT_COMMITTED_AT for CodeClimate |
Thanks to https://github.com/easybib/aws-codebuild-extras for this example!
Azure Pipelines
One way to add needed information to your Azure Pipelines is to specify environment variables for the Code Climate test reporter:
Add the following script:
- script: |
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
chmod +x ./cc-test-reporter
./cc-test-reporter format-coverage -t lcov -o coverage/codeclimate.json coverage/lcov.info
./cc-test-reporter upload-coverage
displayName: Upload coverage to Code Climate
env:
CC_TEST_REPORTER_ID: $(frontend-codeclimate-reporter-id)
GIT_BRANCH: $(Build.SourceBranchName)
GIT_COMMIT: $(Build.SourceVersion)
Thanks to @kpapadatos for the above example!
A different customer provided this context:
Azure sets up two variables BUILD_SOURCEBRANCH
and BUILD_SOURCEVERSION
with information from GIT. The branch is checked out as refs/remote/${PRNumber}/merge
or refs/heads/master
and this is put into the source branch variable.
In the end I had to write a script to fetch the PR from GitHub via the API by parsing the number out of the source branch variable and using that to get the actual branch name back and writing that into an environment variable of GIT_BRANCH.
However, for some reason, cc-test-reporter
wasn’t able to pick this up and was thus looking in the GIT repo for the value which, thanks to the way Azure scripts the branch checkout, was coming through in a detached head state! In the end, I had to force the use of the right environmental variable and this sorted it.
Get more help - contact us!
Need more help?
- Check Test Coverage Troubleshooting Tips
- Open an issue on the test reporter project
- Check example scripts on the test reporter project
- Contact Support
Opening an issue or contacting Support?
Please Include:
- Repo name
- CI name
- If you use single, parallel or multi-suite builds
- A copy of your CI configuration file
- Output from your CI when using the
--debug
flag in your CI configuration- Any relevant screenshots of your setup
PS: Make sure that you're able to run coverage results locally with the basic requirements, outside of Code Climate's test reporter.
Updated over 1 year ago