Search results for "{{ search.query }}"

No results found for "{{search.query}}". 
View All Results

Configuring Test Coverage

Overview

The Code Climate test coverage reporter takes a supported test coverage report, transforms it into a generalized format, and submits it to the Code Climate API. It can also combine multiple test coverage report payloads created from parallel test processes within a parallelized CI build.

Code Climate uses test coverage reports to provide test coverage annotations on codeclimate.com and via the GitHub browser extension. If configured, Code Climate will also send test coverage changes to enabled integrations.

Supported Languages & Formats
Requirements
Single Tests
Parallel and Multiple Test Suites
Locations of Pre-built Binaries
CI Examples
Troubleshooting and Getting Help

Supported Languages and Formats

Language
Format / Coverage Tool

Ruby

JavaScript

lcov (generated by Istanbul e.g.)

PHP

Go

Python

Java

Swift

gcov (generated by SwiftCov e.g.)

C/C++

Scala

Requirements

Code Climate's test reporter relies on third party (typically open source) tools to generate local test coverage reports.

  • You'll find these tools listed in the Supported Languages and Formats table.

  • When troubleshooting test coverage submission, please first ensure you're able to generate test coverage in a supported format locally.

  • If you're having issues generating local coverage, please contact the maintainers of the test coverage library you're using (e.g. SimpleCov, Istanbul).

Single and Parallel Tests

SINGLE TESTS

The basic steps needed for sending test coverage results include:

How to: (single test payloads)

  1. Use the repo's test reporter ID from your Code Climate repo settings page as an environment variable to identify your repo.

  2. Fetch the test reporter pre-built binary.

  3. Run the before-build command before running your test suite to notify Code Climate of a pending report.

  4. Run the after-build command after your test suite completes to send a formatted report or notify Code Climate of a failing build.

Using Circle CI as an example:

machine:
  environment:
    CC_TEST_REPORTER_ID: ABC123

dependencies:
  post:
    - curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
    - chmod +x ./cc-test-reporter

test:
  pre:
    - ./cc-test-reporter before-build

  post:
    - ./cc-test-reporter after-build --exit-code $EXIT_CODE

PARALLEL AND MULTIPLE TEST SUITES

The Code Climate test reporter can join multiple test reports together from separate test suite processes. Uploading partial test coverage data after a batch of tests and fetching all partial test coverage data after all batches complete is a DIY process. We recommend syncing and fetching files from S3 as one approach.

Formatting and Summing Partial Results

Parallel test support requires the ability for you to:

  • upload and download partial test coverage data to/from shared storage (ex: AWS S3),
  • and to use the following sub-commands:
Sub-commands
  • format-coverage

run commands after each batch has completed

  • sum-coverage

run commands after all tests have completed

  • upload-coverage

upload the single result to Code Climate

See the README on GitHub for detailed instructions.

For example:

$ ./cc-test-reporter format-coverage --output "coverage/codeclimate.$N.json"
$ aws s3 sync coverage/ "s3://my-bucket/coverage/$BUILD_NUMBER"


# Where:
# $N should be a unique identifier for that batch of tests
# $BUILD_NUMBER should be the build number provided by your CI.
$ aws s3 sync "s3://my-bucket/coverage/$BUILD_NUMBER" coverage/
$ cc-test-reporter sum-coverage --output - --parts $PARTS coverage/codeclimate.*.json | \
  cc-test-reporter upload-coverage --input -
  
  
# Where:
# $PARTS should be the number of payloads to sum.

And, as with single payload reporting, you'll need to:

  1. Use your repo's test reporter ID from your Code Climate repo settings page.
  2. Fetch the test reporter pre-built binary.
  3. Run the before-build command before running your test suite to notify of a pending report.

After those steps:

  • you'll use the format-coverage, sum-coverage, and upload-coverage sub commands.
  • (the after-build command is not used with parallel or multi-suite tests.)

Locations of Pre-built Binaries

If you'd like to pin to a specific version of the test reporter, you should refer to the releases on GitHub.

CI Examples

We support any CI provider

Our test reporter can run on any CI provider, even if you don't see a specific example here. If you'd like help setting up the test reporter on a different CI provider, feel free to send us a message or open an issue on GitHub.

Below, you'll find an examples of how to configure our test reporter with some common CI providers.

Troubleshooting and Getting Help

Need more help?

Opening an issue or contacting Support?

Please Include:

  1. Repo name
  2. CI name
  3. If you use single, parallel or multi-suite builds
  4. A copy of your CI configuration file
  5. Output from your CI when using the --debug flag in your CI configuration
  6. Any relevant screenshots of your setup

PS: Make sure that you're able to run coverage results locally with the basic requirements, outside of Code Climate's test reporter.

Configuring Test Coverage