Test Coverage Troubleshooting Tips
Test Reporter: Troubleshooting and Getting Help
Having trouble setting up test coverage for your repo?
In addition to the the Configuring Test Coverage help doc, these topics provide additional troubleshooting context:
Basic requirements
Client-side considerations
Self help from technical documentation
Contact Support
Command Line Interface (CLI)
Common Questions
Specific error messages from the reporter
Common GitHub scenarios
Diff-coverage and total-coverage statuses are hanging
Generating test coverage for branches, but not seeing coverage results in GitHub
Need more help?
Open an issue or contact Support! Include the information listed here. ⬅️
Basic requirements:
- Do you have coverage reporting configured for one of our supported languages?
- Are you using the correct CC_TEST_REPORTER_ID?
- Are you using single or multiple test coverage setups? Setup varies for each.
- Are you running your tests outside of your root directory? You may need to use the
--prefixoption. - Are you working with a Docker container? Check out this doc.
Client-side considerations:
- Coverage data is based on the data that you send us via your coverage payloads, which is created client-side.
- Try removing Code Climate from the equation:
(1) Look at coverage results locally. Are they what you expect?
(2) Are your results what you expect, regardless of Code Climate?
(3) How are they different while looking at Code Climate?
Self help from technical documentation
- Output debug messages to your CI using the
--debugflag with your cc-test-reporter. - View uploaded reports (with potential errors) in the Code Climate UI. These can be found under the Recent Reports section at the following URL:
codeclimate.com/repos/repo#/settings/test_reporter - Search the test reporter GitHub project for specific error messages.
- Read the test reporter's README for detailed instructions and low level usage commands.
- Check out the sample/working test config files on the test reporter GitHub project here.
- Ask a question and open an issue on the test reporter repo itself.
Contact Support
Opening an Issue or contacting Support?
Please include:
- Repo name
- CI name
- If you use single, parallel or multi-suite builds
- A copy of your CI configuration file
- Output from your CI when using the
--debugflag in your CI configuration- Any relevant screenshots of your setup
Common Questions - FAQs
| Common Problems and FAQs | Reason and Action |
|---|---|
| ------ I see a "successfully sent" message in my CI, but no results show in Code Climate's UI. | ------ 1. In order to coverage info in the Code Climate UI, you'll need to send a test report for your default branch. If you've only setup the test reporter on a feature branch, then reports won't be uploaded for the default branch. If you're successfully uploaded reports, but still not seeing coverage info in the UI, it's most likely because you haven't yet sent a test report for the default branch. Coverage info for non-default branches is visible in GitHub using the browser extension. 2. Possibly due to malformed payloads. Head to codeclimate.com/repos/repo#/settings/test_reporter to view a list of uploaded reports and potential errors. Add the --debug flag to view additional output.(note: the Test Coverage Settings page is only available to users that have Owner/Admin permissions for the repo on GitHub.) 3. Make sure that you're sending ENV (not Git) values. See this doc 4. Check to see if you're pinning to a specific test coverage reporter version. |
| ------ Where should I put the --debug flag? | ------ 1. If you're running your tests in single builds, use after-build --debug.2. For tests run in parallel, please use both format-coverage --debug and upload-coverage --debug. |
| ------ What CI-specific environmental variables and calls should I use? | ------ 1. Although the same basic variables are needed, we've found that each CI's naming of variables and calls can be quite different. 2. Check our CI sample config files in our help docs here as well as in the test reporter project. 3. See this doc. |
| ------ Coverage results from one repo are showing on a different repo. | ------ 1. You might be using the incorrect CC_TEST_REPORTER_ID.2. Check the test reporter id used in your test coverage config file or CI configurations. |
| ------ When Code Climate is down, it causes my builds to fail because I can't upload my coverage reports. How can I fix this? | ------ 1. We're looking to update our reporter to account for this. In the meantime, we recommend the following workaround: after-build || echo “Skipping CC coverage upload” or upload-coverage || echo “Skipping CC coverage upload”You can also use: || true |
| ------ My total-coverage or diff-coverage statuses are hanging in my PR, but my CI said the coverage info was "successfully sent". | ------ 1. See this doc. |
Specific error messages from the reporter
| Error Message | Description | Resolution |
|---|---|---|
Error: you must supply a CC_TEST_REPORTER_ID ENV variable or pass it via the --id/-r flag | The reporter is unable to find your repo's test reporter ID. This value either needs to configured within the environment or passed directly as a CLI argument. | export CC_TEST_REPORTER_ID=<your token> cc-test-reporter after-build --exit-code $?OR export CC_TEST_REPORTER_ID=<your token> cc-test-reporter after-build --id <your token> --exit-code $? |
------Error: file not found | ------ The reporter is unable to find a file referenced with the test report. Does that file exist within your git repository? Was your test suite run within a different filesystem (such as in a docker container)? | ------ You may need to specify a prefix value. - For example, if you're running the tests within a docker container and your app code is located at /usr/src/app, the reporter run outside of the docker container will not be able to find files at the same absolute path.- Pass --prefix /usr/src/app to instruct the test reporter to strip the unknown base path. |
------Invalid path part | ------ Most often, this is related to: 1. The reporter not able to find your test coverage results to upload them to Code Climate. 2. The reporter encountering a file that it can't process | ------ For #1, adding the -- prefix option to make the path mentioned relative to the project root.For #2, try excluding the mentioned file from your test coverage payloads. If you need the coverage results of that file, contact us and we'll help. |
------Error: not find any viable formatter. available formatters: simplecov, lcov, coverage.py, clover, gocov, gcov, cobertura, jacoco | ------ The reporter is unable to find a formatter. | ------ This is often seen with Java projects, when the the path to source code can't be inferred. Solution: Instead of after-build, please use:1) format-coverage, which includes:- JACOCO_SOURCE_PATH: the path to Java source files (a new environment variable)- coverage file: to path to JaCoCo coverage XML file (the first argument)upload-coverage AND 2) upload-coverage |
Invalid certificate | When using the --insecure flag, any batch request will be made using HTTP, but the the main endpoint will still use the URL specified CC_TEST_REPORTER_COVERAGE_ENDPOINT. | Change the endpoint from https:export CC_TEST_REPORTER_COVERAGE_ENDPOINT=https://codeclimate.corporate.domain.com/api/v1/test_reportsTo http:export CC_TEST_REPORTER_COVERAGE_ENDPOINT=http://codeclimate.corporate.domain.com/api/v1/test_reports |
Error: json: cannot unmarshal object into Go struct field input.coverage of type []formatters.NullInt | This error happens when using SimpleCov newer versions (simplecov >= 0.19) and the reporter is not able to parse the default SimpleCov formatter, you need to set up the JSON one. | Ensure that the CC_TEST_REPORTER_ID variable is present in the environment when you run the test suite. This will force SimpleCov to automatically use the JSON formatter along with the HTML formatter.References: - this comment in the SimpleCov README - this comment in an issue on CodeClimate |
Updated 4 months ago