Codeship CI Test Coverage

📘

Examples

For more example scripts, head to the test reporter repo on GitHub.

Codeship currently offers two different CI products. For docs on configuring Code Climate test reporting with each of these products, see below.

Codeship Basic

Adding Reporter ID

  • To start, you need to add your CC_TEST_REPORTER_ID to your to your project’s environment variables.

You can do this by navigating to Project Settings and then clicking on the Environment tab.

Project Configuration

Once your Code Climate project ID is loaded via your environment variables, you will want to install Code Climate via your setup commands:

chmod +x "${HOME}/bin/cc-test-reporter"```

Next, you will need to add special Code Climate commands before and after your [test commands](https://documentation.codeship.com/basic/quickstart/getting-started/):

**Before your tests:**

`cc-test-reporter before-build`

**After your final tests have run:**

`cc-test-reporter after-build --exit-code $?`


[block:callout]
{
  "type": "info",
  "title": "Using `--prefix`",
  "body": "It may be necessary to use the `--prefix` flag ([docs](https://github.com/codeclimate/test-reporter/blob/master/man/cc-test-reporter-format-coverage.1.md#-p---prefix-path)). The prefix is necessary to remove absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. \n\nExample: `cc-test-reporter after-build --exit-code $? --prefix /home/rof/src/github.com/user_or_org_name/repo_name`"
}
[/block]




[block:api-header]
{
  "title": "Parallel Test Coverage"
}
[/block]

Code Climate supports parallel test reports by uploading the partial result to an external service, such as S3.

To use Code Climate with parallel reporting you will need to add a command at the end of your test commands, in each [parallel pipeline](https://documentation.codeship.com/basic/builds-and-configuration/parallel-tests/) that you run tests in - as well as a new command at the end of your build.

Here are Code Climate’s example scripts for doing so.

At the end of each parallel pipeline:

```./cc-test-reporter format-coverage --output "coverage/codeclimate.$N.json"
aws s3 sync coverage/ "s3://my-bucket/coverage/$CI_BUILD_NUMBER"```

Note that you will need to modify the S3 path (or provide an alternative storage path), as well as set the $N value value by manually declaring separate parallel test pipeline IDs.

At the end of your build itself, you will not need to use the `after-build` call, but instead need to complete the parallel coverage reports in one of two ways.

- As a command, run via the [custom-script deployment option](https://documentation.codeship.com/basic/continuous-deployment/deployment-with-custom-scripts/). This means code coverage for parallel testing will only run on branches you have configured deployments.

- As an additional test step placed at the end of one of your parallel test pipelines. This method will require additional logic to be written to pause the script while it queries your external storage service for the existence of the appropriately-named coverage reports for all the additional pipelines, so that it doesn’t erroneously combine coverage reports for pipelines that are still in progress.

The code to use to end the parallel coverage report is:

```cc-test-reporter sum-coverage --output - --parts $PARTS coverage/codeclimate.*.json | \
    ./cc-test-reporter upload-coverage```

Note that you will need to manually `$PARTS` to reflect the number of parallel threads.

All of these commands will work best when executed from script files.


[block:api-header]
{
  "title": "Common Issues"
}
[/block]

1.  "Successful Build, Even Though Tests Failed"

**Note**: Because of a bug with version 0.8.x of simplecov, tests are reported as successful, even though they actually failed. This is caused by simplecov overriding the exit code of the test framework. According the the [issue report](https://github.com/colszowka/simplecov/issues/281) on GitHub this won’t be fixed in the 0.8 release. Please either use versions prior to 0.8 or higher than 0.9.


[block:api-header]
{
  "title": "Codeship Pro"
}
[/block]




[block:api-header]
{
  "title": "Adding Reporter ID"
}
[/block]

To start, you need to add your `CC_TEST_REPORTER_ID` to the encrypted environment variables that you encrypt and include in your [codeship-services.yml](https://documentation.codeship.com/pro/builds-and-configuration/services/) file.


[block:api-header]
{
  "title": "Project Configuration"
}
[/block]

Once your Code Climate project ID is loaded via your environment variables, you will need to install Code Climate into one of your services via your Dockerfile by using the following command:

```curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > "$/usr/local/bin/cc-test-reporter" &&
chmod +x "/usr/local/bin/cc-test-reporter"```

Next, you will need to add a couple additional commands to your pipeline via your [codeship-steps.yml](https://documentation.codeship.com/pro/builds-and-configuration/steps/) file.

**Before your test commands:**

```- name: codeclimate_pre
  service: YOURSERVICE
  command: cc-test-reporter before-build```

**After your final test commands:**

```- name: codeclimate_post
  service: YOURSERVICE
  command: cc-test-reporter after-build --exit-code $?```



[block:callout]
{
  "type": "info",
  "body": "It may be necessary to use the `--prefix` flag ([docs](https://github.com/codeclimate/test-reporter/blob/master/man/cc-test-reporter-format-coverage.1.md#-p---prefix-path)). The prefix is necessary to remove absolute paths in coverage payloads, to make them relative to the project root. This is usually the directory in which the tests were run. \n\nExample: `cc-test-reporter after-build --exit-code $? --prefix /home/rof/src/github.com/user_or_org_name/repo_name`",
  "title": "Using `--prefix`"
}
[/block]




[block:api-header]
{
  "title": "Parallel Test Coverage"
}
[/block]

Code Climate supports parallel test reports by uploading the partial result to an external service, such as S3.

To use Code Climate with parallel reporting you will need to add another command after your individual tests, and after all tests have completed, in your `codeship-steps.yml` file.

Here are example scripts for doing so.

After each parallel test command you’ll run a new script:
  • type: parallel
    steps:
    • service: YOURSERVICE
      command: tests1.sh
    • service: YOURSERVICE
      command: tests2.sh

Note that we’re using script files to run our tests, so that we can execute the tests and export the coverage report as one command. This is because each step uses a new container, so the coverage report will not persist if the commands are separated. Inside the new `tests.sh` files, you will have:

your test commands go here

./cc-test-reporter format-coverage --output "coverage/codeclimate.$N.json"
aws s3 sync coverage/ "s3://my-bucket/coverage/$CI_COMMIT_ID"```

Note that you will need to modify the S3 path (or provide an alternative storage path), as well as set the $N value by manually declaring separate pipeline IDs.

Next, at the end of your build itself you will not use the after-build call, but instead as a new test command placed after your normal tests:

- name: codeclimate_assemble_results
  service: YOURSERVICE
  command: codeclimate-assemble.sh```

Inside the `codeclimate-assemble.sh` file, you will have:

```cc-test-reporter sum-coverage --output - --parts $PARTS coverage/codeclimate.*.json cc-test-reporter upload-coverage```

Note that you will need to manually `$PARTS` to reflect the number of parallel threads.

 

Updated about a year ago

Codeship CI Test Coverage


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.