Getting Started

CI/CD by Example in Python

A simple demonstration of CI/CD in Python with poetry and pre-commit hooks

By: Edward Krueger and Douglas Franklin

We’ll cover static analysis as a step in continuous integration and continuous delivery (CI/CD) with an example in Python.

CI/CD pipelines allow rapid and automated code deployments. They automate as much of the development process as possible. CI/CD pipelines may include formatting, enforcing good style, and alerting a developer to bugs and running tests. They ensure code can be integrated into a codebase and ultimately deliver code into production.

Some CI/CD tasks can be achieved with static analysis. Static analysis is often accomplished through pre-commit hooks.

What is Static Analysis?

Static analysis is a method of ensuring code quality where code is examined without executing it. Some examples of static analysis tools are formatters, linters and static vulnerability analyzers.

Formatters automatically style code to improve readability and consistency. They mostly make changes in whitespace; they do not modify how code is executed.

Linters are similar to a spell-checker for written documents. They look for defects and bugs in code. They remove ‘fluff,’ hence the name, like unused imports and variables. Linters can also detect some bugs. For example, if a variable is used but never defined, the linter will emit a warning. Linters also provide recommendations for style and best practices.

Static vulnerability analysis tools warn you if you are using a package version that has known security vulnerabilities. This prevents security-related issues from being an afterthought. An example of this is when GitHub warns you that your requirements.txt contains a dependency with a known security vulnerability. Static vulnerability analysis warnings are pulled from CVE, a website and company dedicated to tracking these vulnerabilities.

Static analysis tools have many benefits; one is discovering bugs before they are released into production. An advantage of static analysis is that it doesn't have to execute your code, making the process quick.

Implementing static analysis tools presents such minor overhead for developers there is no reason not to use them.

Static analysis can be easily implemented and accomplished with pre-commit hooks.

What are pre-commit hooks?

Pre-commit-hooks are a type of Git-hooks. Git-hooks trigger when a command occurs in a Git repository — such as a commit or a push. A “pre-commit hook” runs when a git commitis executed.

Pre-commit hooks are often used to run static analysis, so code is linted and formatted before it is committed. Pre-commit hooks can be used for continuous integration and to prepare code for deployment, aid in the build process, dependency management and testing. There are upsides and downsides to testing with pre-commit hooks. The upside is you never get code pushed that fails tests. The downside is that you may be okay with having code that fails some tests for works in progress. Most people would not put tests as pre-commit hooks and these more advanced hooks are out of the scope of this article.

Our CI/CD pipeline

We use a very simple ‘CI/CD pipeline’ in the repository below. Our pre-commit hooks do some static analysis. You can observe them in the .pre-commit-config.yaml . Our custom pre-commit hook, create-requirements ensures that our requirements.txt is in sync with our poetry toml and poety.lock, i.e., our local virtual environment.

This automates updates to requirements.txt so all we need to update is the poetry toml. If we fail to update our requirements.txt any change to or dependencies can cause our changes not to appear, or in the worst case, crash our app.

This custom hook acts as a continuous delivery tool, especially since we’ve enabled automatic deployments from our Github master branch. Its purpose is to update our requirements.txt automatically before we deploy, so that Heroku can build from this file.

For more information on this custom hook or how to write your own with Bash, give this a read:

In certain situations, an app crashing for a few minutes could be catastrophic for a business, for your reputation as a developer, or both. If your app is a simple internal dashboard to measure KPIs, and your boss decides to check in on the performance right when you break it, this is not a good look. In the case of a customer-facing app, crashed minutes mean lost customers and lost revenue.

These situations are meant to highlight the importance of using simple tools that prevent this sort of thing from happening. With a good CI/CD pipeline, most mistakes can be caught before they are ever pushed into production.

Additionally, while customers and bosses may not care how the code works, developers do. This is another reason to use static analysis tools and a CI/CD pipeline. If your code is poorly written, it will be hard to test and expand on. These changes, implanted well or poorly over time, lead to more or less technical debt.

Static Analysis, CI/CD, and Heroku deployment

As seen in the image below, we selected ‘enable automatic deploys’ for developing our application. By selecting this option on Heroku, new code is continuously deployed to our application when pushed to our Git repository. This means each push to the master branch is a push into production.

This is why it is important to have a continuous integration ‘service’ set up, as the warning above the button suggests. For us, this service is our linting and formatting pre-commit hooks and our custom hook that ensures our requirements.txt is up to date.

Our repository’s pre-commit-config.yaml, the bash code for create-requirements and enabling automatic deploys on Heroku, together represent our simple CI/CD pipeline.

Together these three pieces ensure that our code is properly formatted and linted and that we don’t crash our app by pushing changes that aren't supported by the repositories requirements.txt file.

Conclusion

There is no reason for developers to push code to GitHub without using simple static analysis tools as pre-commit hooks. This helps ensure your code is clean and has fewer bugs with little to no overhead. Prescient developers working on more significant production codebases will almost always be using a CI/CD pipeline to automate the linting, formatting, testing, and integration/delivery of their code changes.

From self-taught developers looking for work to programming academics publishing their work on GitHub, all the way to the career software engineers, these are crucial tools to ensure you aren't making easy mistakes. So pip intstall pre-commit and check out their documentation and hooks to get started!

If that is too much to configure, don’t! Use Ed’s template repo. Click the green button on GitHub to get started (see image below).