Setting Up Test Coverage

We accept test coverage data from virtually any location, including locally run tests and your continuous integration (CI) service. We also have a few different test reporters including ones for Ruby, JavaScript, PHP, and Python. This help doc covers:

How we display Test Coverage
Important FYI's
How to setup our Test Coverage feature
How to exclude files and directories from your payloads
Test Coverage Troubleshooting Tips

How we display Test Coverage

While we only have a few officially supported test coverage reporters at the moment, nothing on our server side prevents receiving test coverage for any particular language.

At a high-level, you'll call our test reporter, which will generate a test coverage payload and post it to Code Climate. Once it's been passed to us, we will display your test coverage data alongside our code quality and security analysis. For example:

  • The Feed page displays your repository's overall test coverage percentage.
  • The Coverage page displays your coverage data, file-by-file.
  • A code quality page highlights code (at line-by-line level) based on test coverage. Green shading represents lines that have been tested, while red shading represents lines that have not.

Important Considerations

Across the many different testing frameworks, setups, and environments, there are lots of variables at play. Before setting up test coverage, it's important to understand what we do and do not currently support:

  • Test coverage reporters exist for Ruby, PHP, Python and JavaScript: While in theory there's no restriction from posting test coverage for any particular language, officially supported clients only exist at the moment for Ruby, JavaScript and PHP.

  • Default branch: The coverage percentage shown on your Dashboard is reflective of the last payload we received from your default branch.

  • Pull requests: If you've enabled our GitHub pull request integration, as soon as we receive and analyze your test report payload, we'll update the PR with test coverage details giving you insight into the proposed changes.

If you're using a CI, make sure that you've instructed the CI to build pushes so that we receive test coverage payloads for your PR's.

  • Single payload: We currently only support a single test coverage payload per commit. If you run your tests in multiple steps, or via parallel tests, Code Climate will only process the first payload that we receive. If you are using a CI, be sure to check if you are running your tests in a parallel mode.
    Note: There is one exception to this rule. We've specifically built an integration with Solano Labs to support parallel tests.

  • One language per repository: If you've configured Code Climate to analyze multiple languages in the same repository (e.g., Ruby and JavaScript), we can nonetheless only process test coverage information for one of these languages. We'll process the first payload that we receive.

  • Invalid File Paths: By default, our test reporters expect your application to exist at the root of your repository. If this is not the case, the file paths in your test coverage payload will not match the file paths that Code Climate expects. If you are working with our Ruby test reporter, we have a work-around to this issue.

How To

The step-by-step set up instructions vary based on which of our test reporters you are using.

For an organization-owned repository, you must be in an organization's Owners group to perform the actions described below. Alternatively, if you added the repository in question by clicking Add Open Source Repo, you must have GitHub-rights to administer the repository.

  1. From your Dashboard, mouse over the repository's name and click Settings.
  1. Select the Test Coverage tab.
  1. Your test coverage token will be displayed on the page. Follow the instructions for the test reporter you are using:

Important: If you are running your tests via a CI service that we've directly integrated with (Semaphore, Solano Labs, and Travis CI), scroll to the bottom of your repository's Test Coverage tab, and select your CI. Perform the steps listed on the page.

Excluding Files and Directories

Your repository's overall coverage percentage is calculated according to the coverage payload that we receive. For example, if your repository contains 100 files, but the project's testing framework only passes us coverage data for 95 files, the missing five files will have no effect on your repository's overall coverage percentage. On the other hand, if a file is mentioned in your coverage payload, but that file has been excluded from Code Climate's analysis, the file will still be factored into your repository's overall coverage percentage.

Our goal is to analyze only non-compiled, non-vendored, and non-test/spec code. As a result, code in certain directories (such as test, spec, and vendor) is typically ignored (though this can vary from language to language). We also auto-exclude any folder whose name ends in cache (for all languages; this only applies to folders, not files). Right now there is not a way to modify these default exclusions.

Exclusions applied in your .codeclimate.yml file have no affect on test coverage. These exclusions only affect our analysis.

Ruby

You can exclude specific files or directories from your test coverage payload, by applying the following configuration to your spec_helper.rb/test_helper.rb file.

SimpleCov.start do
  add_filter "/vendor/" # Ignores any file containing "/vendor/" in its path.
  add_filter "/lib/myfile.rb" # Ignores a specific file.
end

Other Languages

To exclude specific files or directories from your Code Climate test coverage information, exclusions should be applied directly within your project's testing framework. This should result in the files/directories not being included in the stand-alone file that is passed to our reporter. For example, if you were using PHPUnit to generate a clover.xml file, you'd configure your phpunit.xml file to apply exclusions. The step-by-step exclusions steps depend on the testing framework that you are using.

Test Coverage Troubleshooting Tips

Trouble setting up test coverage or not seeing the results you expect?

  • Take a look at this help doc for additional troubleshooting tips.

Setting Up Test Coverage