Search results for "{{ search.query }}"

No results found for "{{search.query}}". 
View All Results

Configuring Test Coverage

Public Beta

Our new test reporter, documented below, is in Public Beta, and subject to breaking changes in the future.

If you prefer not to use our Public Beta, you are welcome to (continue to) use our soon to be deprecated older test reporters.

Overview

The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to the Code Climate API. It can also combine multiple test coverage report payloads created from parallel test processes within a parallelized CI build.

Code Climate uses test coverage reports to provide test coverage annotations on codeclimate.com and via the GitHub browser extension. If configured, Code Climate will also send test coverage changes to enabled integrations.

Supported Languages and Formats

The test reporter currently supports the following test coverage languages and formats:

Language
Format / Coverage Tool

Ruby

JavaScript

lcov (generated by Instanbul e.g.)

PHP

Go

Python

Java

Swift

gcov (generated by SwiftCov e.g.)

C/C++

Scala

More languages and formats are coming soon.

Installation

If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.

Usage

Before using the test reporter, you should visit your Code Climate repo settings page to fetch the repo's test reporter ID. This value is required by the test reporter to correctly identify which repo the test report should be submitted for.

Before running your test suite, run the before-build command to notify Code Climate of a pending report.

cc-test-reporter before-build

After your test suite completes, run the after-build command to send a formatted report or notify Code Climate of a failing build. Some CI providers expose the exit code as an environment variable. Otherwise, if you're executing the test reporter within the same step, you may use $? to pass the last exit code to the reporter.

cc-test-reporter after-build --exit-code $?

Parallel Tests and Multiple Test Suites

The Code Climate test reporter can join multiple test reports together from separate test suite processes. Uploading partial test coverage data after a batch of tests and fetching all partial test coverage data after all batches complete is a DIY process. We recommend syncing and fetching files from S3 as one approach.

See the README on GitHub for additional instructions.

CI Examples

We support any CI provider

Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.

Below, you'll find an examples of how to configure our test reporter with some common CI providers.

Troubleshooting

The test coverage reporter works by:

  • taking a report (generated by a support coverage tool), then
  • transforming it into a generalized format, then
  • submitting it to the Code Climate API.

If you run into an issue, follow the steps outlined below:

  1. If there's an issue with the reporting (or with Code Climate), remove Code Climate from the equation.
  2. Look at coverage results locally. Are they what you expect?
  3. Are your results what you expect, regardless of Code Climate? How are they different while looking at Code Climate?

For more detailed setup instructions, please visit the test reporter repo on Github.

To assist with troubleshooting, use the --debug flag to output debug messages during operation.

Common Errors

Error Message

Description

Resolution

Error: you must supply a CC_TEST_REPORTER_ID ENV variable or pass it via the --id/-r flag

The test reporter is unable to find your test reporter ID. This value either needs to configured within the environment or passed directly as a CLI argument.

export CC_TEST_REPORTER_ID=<your token> cc-test-reporter after-build --exit-code $?

OR

export CC_TEST_REPORTER_ID=<your token> cc-test-reporter after-build --id <your token> --exit-code $?

Error: file not found

This error is raised with the reporter is unable to find a file referenced with the test report. Does that file exist within your git repository? Was your test suite run within a different filesystem (such as in a docker container)?

You may need to specify a prefix value. For example, if you're running the tests within a docker container and your app code is located at /usr/src/app, the reporter run outside of the docker container will not be able to find files at the same absolute path. Pass --prefix /usr/src/app to instruct the test reporter to strip the unknown base path.

(Travis) Error: “bundler: command not found: rspec”

Change bundle exec rspec in the .travis.yml to bundle exec rake

(Cobertura) could not find any files in search paths for cobertura

cc-test-reporter looks for your coverage info within your project folder. Cobertura stores this information outside the project folder.

Use low-level commands to indicate where the coverage data is located, then upload to Code Climate. Example:
./cc-test-reporter format-coverage -t cobertura path/to/coverage_data ./cc-test-reporter upload-coverage

Contact Support

For help with the configuration of this test reporter, please contact support. To help expedite the troubleshooting process, please include the following items in your support ticket:

  • Name of your repo
  • Name of your CI
  • Are you running tests in single or parallel builds?
  • A copy of your CI configuration file
  • Output from your CI (after including the --debug flag in your CI configuration)
  • A copy of your codeclimate.json (this can be generated inside your coverage folder by running ./cc-test-reporter format-coverage)
  • Any relevant screenshots of your setup

Configuring Test Coverage