Building a Code Climate Engine

Whether you've started working on an engine or are just curious what goes into making a new analysis tool work on the Code Climate Platform, this document is for you.

What is a Code Climate Engine?

In order to understand how to build a Code Climate engine, we thought it would be helpful to start with a working definition:

A Code Climate Engine is a Docker Image that invokes a program which parses a config file and analyzes files, potentially producing formatted output representing issues in those files.

Within that definition are some bolded words which could use some elaboration.

Before we dig in, this is a good time to state that that the design, packaging, and operation of an engine is guided by the Code Climate Spec, and Quality Assurance steps are outlined in the Code Climate Engine QA Spreadsheet.

  • Docker Image - all Code Climate engines are packaged as Docker Images. If you're not familiar with Docker and how it works, we recommend checking out their website - there's lots of great information there. We use Docker because it allows us to let people use a variety of implementations for their engines and still package them in a repeatable, specification compliant way. Notably, when a Docker Image is created for a Code Climate Engine, we run the same image wherever we offer analysis - locally on your computer in the CLI, in our cloud on codeclimate.com, or behind the firewalls of enterprises with Code Climate Enterprise. Pretty neat.
  • invokes a program - Inside your Docker Image, your engine will invoke a program (often just a simple script) which will do all of the work necessary to be specification compliant.
  • parses a config file - The first thing your program will do is parse a config file which is detailed in the Code Climate Spec and will contain information that is necessary to run the engine properly. The configuration a user provides with their .codeclimate.yml file ends up in this config file, and includes data like which files to analyze, how to configure the engine, etc.
  • analyze files - Once the config file is parsed and the engine has a record of which files to analyze, it can iterate over them and do it's job. Your engine can either wrap an existing tool or be its own novel tool, it doesn't really matter - as long as you comply with the Spec, your engine should function properly.
  • formatted output - The output produced when an issue is found in a file to be analyzed is strictly controlled both in how the issues are formatted and how they are sent to STDOUT for consumption by various Code Climate services. Check the Spec for more information.
  • issues - Issues found with analyzed files are to be assigned categories, and given short and long descriptions which are described in the Code Climate spec. Issues are what users see when they run Code Climate, so they are subject to a high quality bar for content and accuracy.

How to Get Started

Once you've read the "How to build an engine" blog post, looked at the Spec, seen the QA spreadsheet, and maybe even browsed around a few example engines, you still may not know what a good approach is to getting started.

Here are some tips that we've given to a number of engine authors. We think they're good inspiration for getting started with what can seem like a daunting task. If they help, let us know! If they don't, hey, let us know too! We'd love to make this list as complete as possible.

Concerning Docker

Docker is an essential aspect of our platform - the whole thing basically runs on it. To that end, you need to have a fully functioning Docker environment running locally, and you need to understand the basics of making a Docker Image to correctly build a Code Climate Engine. That being said, it can wait.

You can get started with the basics of your engine, like making a simple script, studying engines, etc., before you worry about making your own Dockerfile, and besides, in all likelihood you'll be able to copy an existing engine's implementation for most of what you need.

So if the Docker work is scaring you off, don't let it. Support for Docker is becoming better and better on all widely deployed Desktop environments, educational material is abundant, and much of the work ahead you has been done for you.

Start with a simple script

Once you're ready to build your engine, we recommend as a first step that you build a simple script that can accomplish the following:

  • Accept two parameters - config_file_path and workspace_path
  • Use the config_file_path parameter to find and parse a file named config.json, and store its keys and values in a data structure
  • Create a list of files to analyze using the include_paths key from the parsed config.json file and the workspace_path parameter
  • Invoke the tool your engine is wrapping with the list of files to analyze
  • Output issues that have been found in JSON format to STDOUT

Parsing a config, executing the tool, and formatting the output comprise the majority of the functionality of a Code Climate engine. Once you have a script which can accomplish these things, you're most of the way there!

Study some other engines, and copy what you can

All Code Climate engines are Open Source. Feel free to fork any of the engines available in our list of engines and use it as the basis for your own work.

If you're building engines in pretty much any major programming language, we have examples for you to be inspired by. There are even libraries in certain languages for making writing engines even easier.

When you're ready to test with the CLI

The Code Climate CLI ships with a few handy tricks for making engine development easier:

  • The --dev flag - when you want to test your engine with the CLI, add it to the .codeclimate.yml file of a project, and run codeclimate analyze --dev. The --dev flag will allow the CLI to check locally for engine Docker Images, letting you specify engines that aren't generally available.
  • The -f json flag - if your issues are breaking the standard output or you want to see parts that aren't represented there, you can make the CLI output JSON, which shows all available fields
  • CODECLIMATE_DEBUG=1 codeclimate analyze The CLI looks for the presence of an environment variable called CODECLIMATE_DEBUG and outputs extra useful debugging information if it finds it. This can save you lots of time!

If you've come across any other good tips, let us know!

When you want to see how your issues will render on codeclimate.com

The Code Climate CLI has an HTML output flag that will show you rendered markdown for issue descriptions and content. To use the HTML formatter, run codeclimate analyze -f html > out.html and inspecting out.html in a browser.

Use the community

Join our Code Climate Developer Slack Channel! You should get an invitation automatically when you join the Developer Program.

Building a Code Climate Engine