Search results for "{{ search.query }}"

No results found for "{{search.query}}". 
View All Results

Configuring Test Coverage

Public Beta

Our new test reporter, documented below, is in Public Beta, and subject to breaking changes in the future.

If you prefer not to use our Public Beta, you are welcome to (continue to) use our soon to be deprecated older test reporters.

Overview

The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to the Code Climate API. It can also combine multiple test coverage report payloads created from parallel test processes within a parallelized CI build.

Code Climate uses test coverage reports to provide test coverage annotations on codeclimate.com and via the GitHub browser extension. If configured, Code Climate will also send test coverage changes to enabled integrations.

Requirements

Code Climate's test reporter relies on third party (typically open source) tools to generate local test coverage reports.

In order to use the Code Climate test reporter, please first ensure you're able to generate test coverage in a supported format locally.

If you're having issues generating local coverage, please contact the maintainers of the test coverage library you're using (e.g. SimpleCov, Instanbul)

Supported Languages and Formats

The test reporter currently supports the following test coverage languages and formats:

Language
Format / Coverage Tool

Ruby

JavaScript

lcov (generated by Istanbul e.g.)

PHP

Go

Python

Java

Swift

gcov (generated by SwiftCov e.g.)

C/C++

Scala

More languages and formats are coming soon.

Installation

If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.

Usage

Before using the test reporter, you should visit your Code Climate repo settings page to fetch the repo's test reporter ID. This value is required by the test reporter to correctly identify which repo the test report should be submitted for.

Before running your test suite, run the before-build command to notify Code Climate of a pending report.

cc-test-reporter before-build

After your test suite completes, run the after-build command to send a formatted report or notify Code Climate of a failing build. Some CI providers expose the exit code as an environment variable. Otherwise, if you're executing the test reporter within the same step, you may use $? to pass the last exit code to the reporter.

cc-test-reporter after-build --exit-code $?

Parallel Tests and Multiple Test Suites

Code Climate assists with coverage for parallel testing and multiple suites by offering lower level commands to join together partial test coverage results and then post the final result to Code Climate.

In cases where test suites are being run across multiple VMs, you will also need upload partial test coverage data after a batch of tests and fetch all partial test coverage data after all batches complete. While our reporter does not handle this for you, there are a variety of relatively simple options for performing this. We recommend syncing and fetching files from S3 as one approach.

See the README on GitHub for additional instructions.

CI Examples

We support any CI provider

Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.

Below, you'll find an examples of how to configure our test reporter with some common CI providers.

For more examples, check out the test reporter repo on Github.

Common Errors & Troubleshooting

If you run into an issue, follow the steps outlined below:

  • If there's an issue with the reporting (or with Code Climate), remove Code Climate from the equation.
  • Look at coverage results locally. Are they what you expect?
  • Are your results what you expect, regardless of Code Climate? How are they different while looking at Code Climate?

For more detailed setup instructions, please visit the test reporter repo on Github.

To assist with troubleshooting, use the --debug flag to output debug messages during operation.

For more information on common errors with our test reporter, refer to our doc on Troubleshooting Test Coverage.

FAQ

Q: When Code Climate is down, it causes my builds to fail because I can't upload my coverage reports. How can I fix this?

A: We're looking to update our reporter to account for this. In the meantime, we recommend the following workarounds: after-build || echo “Skipping CC coverage upload” or upload-coverage || echo “Skipping CC coverage upload”

Q: Where should I put the --debug statement?

A: If you're running your tests in single builds, use after-build --debug. For tests run in parallel, please use both format-coverage --debug and upload-coverage --debug.

Contact Support

For help with the configuration of this test reporter, please contact support. To help expedite the troubleshooting process, please include the following items in your support ticket:

  • Name of your repo
  • Name of your CI
  • Are you running tests in single or parallel builds?
  • A copy of your CI configuration file
  • Output from your CI (after including the --debug flag in your CI configuration)
  • A copy of your codeclimate.json (this can be generated inside your coverage folder by running ./cc-test-reporter format-coverage)
  • Any relevant screenshots of your setup

Configuring Test Coverage