Codeship CI Test Coverage

📘

Examples

For more example scripts, head to the test reporter repo on GitHub.

Codeship currently offers two different CI products. For docs on configuring Code Climate test reporting with each of these products, see below.

• Codeship Basic
• Codeship Pro

Codeship Basic

Adding Reporter ID

• To start, you need to add your CC_TEST_REPORTER_ID to your to your project’s environment variables.

You can do this by navigating to Project Settings and then clicking on the Environment tab.

Project Configuration

Once your Code Climate project ID is loaded via your environment variables, you will want to install Code Climate via your setup commands:

curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > "${HOME}/bin/cc-test-reporter" &&
chmod +x "${HOME}/bin/cc-test-reporter"

Next, you will need to add special Code Climate commands before and after your test commands:

Before your tests:

cc-test-reporter before-build

After your final tests have run:

cc-test-reporter after-build --exit-code $?

📘

Using --prefix

It may be necessary to use the --prefix flag (docs). The prefix is necessary to remove absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run.

Example: cc-test-reporter after-build --exit-code $? --prefix /home/rof/src/github.com/user_or_org_name/repo_name

Parallel Test Coverage

Code Climate supports parallel test reports by uploading the partial result to an external service, such as S3.

To use Code Climate with parallel reporting you will need to add a command at the end of your test commands, in each parallel pipeline that you run tests in - as well as a new command at the end of your build.

Here are Code Climate’s example scripts for doing so.

At the end of each parallel pipeline:

./cc-test-reporter format-coverage --output "coverage/codeclimate.$N.json"
aws s3 sync coverage/ "s3://my-bucket/coverage/$CI_BUILD_NUMBER"

Note that you will need to modify the S3 path (or provide an alternative storage path), as well as set the $N value value by manually declaring separate parallel test pipeline IDs.

At the end of your build itself, you will not need to use the after-build call, but instead need to complete the parallel coverage reports in one of two ways.

• As a command, run via the custom-script deployment option. This means code coverage for parallel testing will only run on branches you have configured deployments.

• As an additional test step placed at the end of one of your parallel test pipelines. This method will require additional logic to be written to pause the script while it queries your external storage service for the existence of the appropriately-named coverage reports for all the additional pipelines, so that it doesn’t erroneously combine coverage reports for pipelines that are still in progress.

The code to use to end the parallel coverage report is:

cc-test-reporter sum-coverage --output - --parts $PARTS coverage/codeclimate.*.json | \
    ./cc-test-reporter upload-coverage

Note that you will need to manually $PARTS to reflect the number of parallel threads.

All of these commands will work best when executed from script files.

Common Issues

1. "Successful Build, Even Though Tests Failed"

Note: Because of a bug with version 0.8.x of simplecov, tests are reported as successful, even though they actually failed. This is caused by simplecov overriding the exit code of the test framework. According the the issue report on GitHub this won’t be fixed in the 0.8 release. Please either use versions prior to 0.8 or higher than 0.9.

Codeship Pro

Adding Reporter ID

To start, you need to add your CC_TEST_REPORTER_ID to the encrypted environment variables that you encrypt and include in your codeship-services.yml file.

Project Configuration

Once your Code Climate project ID is loaded via your environment variables, you will need to install Code Climate into one of your services via your Dockerfile by using the following command:

curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > "$/usr/local/bin/cc-test-reporter" &&
chmod +x "/usr/local/bin/cc-test-reporter"

Next, you will need to add a couple additional commands to your pipeline via your codeship-steps.yml file.

Before your test commands:

- name: codeclimate_pre
  service: YOURSERVICE
  command: cc-test-reporter before-build

After your final test commands:

- name: codeclimate_post
  service: YOURSERVICE
  command: cc-test-reporter after-build --exit-code $?

📘

Using --prefix

It may be necessary to use the --prefix flag (docs). The prefix is necessary to remove absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run.

Example: cc-test-reporter after-build --exit-code $? --prefix /home/rof/src/github.com/user_or_org_name/repo_name

Parallel Test Coverage

Code Climate supports parallel test reports by uploading the partial result to an external service, such as S3.

To use Code Climate with parallel reporting you will need to add another command after your individual tests, and after all tests have completed, in your codeship-steps.yml file.

Here are example scripts for doing so.

After each parallel test command you’ll run a new script:

- type: parallel
  steps:
    - service: YOURSERVICE
      command: tests1.sh
    - service: YOURSERVICE
      command: tests2.sh

Note that we’re using script files to run our tests, so that we can execute the tests and export the coverage report as one command. This is because each step uses a new container, so the coverage report will not persist if the commands are separated. Inside the new tests.sh files, you will have:

# your test commands go here

./cc-test-reporter format-coverage --output "coverage/codeclimate.$N.json"
aws s3 sync coverage/ "s3://my-bucket/coverage/$CI_COMMIT_ID"

Note that you will need to modify the S3 path (or provide an alternative storage path), as well as set the $N value by manually declaring separate pipeline IDs.

Next, at the end of your build itself you will not use the after-build call, but instead as a new test command placed after your normal tests:

- name: codeclimate_assemble_results
  service: YOURSERVICE
  command: codeclimate-assemble.sh

Inside the codeclimate-assemble.sh file, you will have:

cc-test-reporter sum-coverage --output - --parts $PARTS coverage/codeclimate.*.json cc-test-reporter upload-coverage

Note that you will need to manually $PARTS to reflect the number of parallel threads.