GitHub Pull Requests

Code Climate provides a few different ways for users to view code quality and test coverage analysis results on a pull request.

Pull Request Review Comments

Code Climate has an associated Github App that, once installed, supports two types of automated review comments: Summary Comments and Inline Issue Comments.

Summary Comments

Summary comments are comments left on the pull request summarizing the issues introduced by the pull request, as well as the test coverage changes.

There are several configuration options for how this comment should behave:

  • Delete (default): For every push to the pull request, the original comment will be deleted and a new one will be posted. This will move the summary comment to the bottom of the pull request and will create a new notification via GitHub.
  • Update: For every push to the pull request, the original comment will be updated. This will not create new notifications via GitHub.
  • Once: A summary comment should be left once when the pull request is opened. Future pushes to the pull request will not modify the comment.

To enable this functionality, please see Configuring Summary Comments.

Inline Issue Comments

Inline issue comments are pull request comments left as part of a GitHub review. Any new issues related to lines changed in the diff will be added as comments to the Pull Request. Once addressed, the comments will collapse using the native GitHub behavior.

There are two configuration options for how this comment should behave:

  • Additional (default): For every push to the pull request, if there are new issues introduced by the changes, another review will be left with comments for the new issues. This will create a new notification via GitHub.
  • Once: A review should be left the first time issues are detected. Afterward, if new pushes to the pull request introduce additional issues, they will not be left as a new review.

In addition, the maximum number of comments left per review is configurable (default: 5).

To enable this functionality, please see Configuring Inline Issue Comments.

Installing the GitHub App

To make use of automated pull request comments, first the CodeClimate GitHub app must be installed.

This integration is configured via a separate GitHub app to utilize several advantages of GitHub Apps compared to OAuth applications:

  • Comments will be left by a single bot user, rather than a user in the GitHub organization
  • Installations are tied to a GitHub organization, rather than a specific user
  • A minimal set of permissions is required

For more information about GitHub Apps, please see the GitHub documentation.

Show me how

Note

Only admins of the GitHub organization are authorized to install and configure GitHub apps. These steps must be completed by a GitHub admin.

  1. Navigate to the CodeClimate GitHub App. There is also a link from the repository settings page.
  1. Click "Install" and select which GitHub organization you would like to have automated pull request comments enabled for.

The app can be installed for all current and future repositories in the GitHub organization:

Or for a select list of repositories in the GitHub organization:

  1. Verify each of the installed repositories has been added on Code Climate.

Configuring Summary Comments

  1. Verify that the GitHub app has been installed for the repo and the repo has been added to Code Climate as described above.

  2. Visit the repo settings page and navigate to GitHub settings.

  3. Enable the Summary Comments feature, choose the desired strategy, and click save.

  1. The next time a pull request is opened for one of the installed repositories, a summary comment will be posted based on the selected strategy.

  2. To add or remove this functionality for other repositories in the GitHub organization, the app installation can be managed from the GitHub organization settings page where the app has been installed under Settings -> Installed GitHub Apps.

Configuring Inline Issue Comments

  1. Verify that the GitHub app has been installed for the repo and the repo has been added to Code Climate as described above.

  2. Visit the repo settings page and navigate to GitHub settings.

  3. Enable the Inline Issue Comments feature, choose the desired strategy, and click save.

  1. The next time a pull request is opened for one of the installed repositories, a summary comment will be posted based on the selected strategy.

  2. To add or remove this functionality for other repositories in the GitHub organization, the app installation can be managed from the GitHub organization settings page where the app has been installed under Settings -> Installed GitHub Apps.

Pull Request Statuses

If you have our GitHub PR integration enabled, when opening up a pull request, we'll run our 10-point maintainability assessment and post a status update in GitHub notifying you of new code quality issues that have been introduced.

Once you've setup test coverage reporting, you can use our test coverage statuses to ensure that all PRs meet your team's specific coverage standards.

In addition to our in-house maintainability checks, you can optionally enable open source plugins to run checks against your code.

Our browser extension ensures code quality and test coverage data is constantly visible throughout your development workflow, so your team can get, discuss, and act on all of this information without leaving GitHub. By clicking over to Files Changed in GitHub, you'll be able to view the static analysis and test coverage results right in-line.

Teams should use their judgment to validate and invalidate issues found by Code Climate. For these cases, Code Climate allows developers to mark any issue with one of 3 different statuses:

  • confirmed - a legitimate issue that should be addressed
  • invalid - is a false-positive
  • wontfix - a legitimate issue, but it does not need to be addressed at this time

For more information on changing issue statuses, please see this doc.

Ineligible Pull Requests

Though we hope to expand our pull request integration in the future, note that we currently do NOT support:

  • Pull requests against branches other than your default branch.
  • Pull requests from repositories not hosted on GitHub.com. Our PR integration currently does not support pull requests from SCM's such as Bitbucket or GitHub Enterprise.
  • For PR integration with on-prem SCM's, check out Code Climate: Enterprise.

Show me how

  1. From your Repo Dashboard, navigate to Repo Settings.
  1. Select the GitHub tab.
  1. Scroll down to Pull request status updates and click Install.

Note

To communicate with GitHub, the oauth token of the user installing the integration will be used. To change to a new token, the integration can be disabled and re-installed.

  1. Once installed, the setting will show a green check mark indicating that it is active.
  1. Make sure that the GitHub webhook has been installed. This notifies Code Climate when pull requests are opened and updated. The webhook can be installed or re-installed via the same GitHub settings page under Connections > Webhook on GitHub.
  1. To disable the integration or rotate the GitHub token being used, click Edit next to Pull request status updates. Flip the toggle to off and click Save. The integration can then be re-installed.
  1. Once the PR integration is enabled, our test coverage statuses will be enabled by default. However, you'll need to configure test coverage in order to see those statuses populate.

Does Code Climate require Read/Write access to my GitHub repositories?

Though optional, there are a number of benefits to providing Code Climate with read/write access to your repositories. These benefits include the ability to:

In short, this is the best overall Code Climate experience and the one that we recommend.

However, if this is not an option for you, it's possible to limit Code Climate to having read-only access. Keep in mind that there are a few extra setup steps to configure this, as described below. In addition, you will unfortunately not be able to use all of our features -- this is not by design but is primarily a result of the way GitHub permissions are handled.

GitHub Linking

When signing up for Code Climate, you have the option of creating either a GitHub-linked user or a stand-alone user. Since GitHub-linking automatically grants us the Repositories permission in GitHub, if you're looking to restrict Code Climate to read-only access, you'll want to skip linking and instead create a stand-alone Code Climate user by not authenticating with GitHub.

Are you already GitHub-linked? If so, you can remove this link at anytime.

SSH Keys

There are two different ways to add a private repository to Code Climate. To limit us to having read-only access, ensure you use the method which requires manually entering the repository's URI. During this process, you'll be prompted to handle an SSH key. If you add our public key directly to the repository in GitHub, that will grant Code Climate read/write access. To instead grant us read-only access, add the key to a GitHub machine user that has read-only access to the repository. Doing so will, in turn, restrict Code Climate to having read-only access.

Note: Machine users can only be created on GitHub organization accounts, not on GitHub user accounts (what's the difference?). This, unfortunately, means that if your repository is owned by a user account in GitHub, it's not possible to limit Code Climate's access to being read-only.

GitHub Integrations

We have two GitHub integrations: one for GitHub pull requests and another for GitHub Issues. Since both integrations require you to create a GitHub personal access token, ensure that the token does not have read/write access:

  • GitHub Pull Requests: The token should be generated by any user that has GitHub permissions to access the repository (it is not necessary to generate the token as a GitHub machine user). When generating the token, ensure that only the repo:status scope is selected.

  • GitHub Issues: The token should be generated by a GitHub machine user that has read-only access to the repository. When generating the token, ensure the repo scope is selected.

GitHub Pull Requests