using <a href="https://numpydoc.readthedocs.io/en/latest/format.html">Numpy’s Docstring style</a> for the docstring. VisualStudio Code has an <a href="https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring">extension</a> which auto-generates large parts of the doc-string in Numpy’s style. Make sure to invest in proper docstring documentation and add the <code>Examples</code> section too, as this will be very useful later on when we’ll generate documentation for ReadTheDocs .</li><li>We’re<i> </i>using<i> Black’s</i> and isort’s formatting style. Run <code> poetry run black src/</code> to format the core package to <i>Black’s</i> style and <code> poetry run isort src/</code> to automatically sort the imports.</li><li>To use our function <code>square(x)</code> we can import it through:
<code>>>> from my_package.calc import square</code> . Make sure to install <code>my_package</code> first with: <code> poetry install</code></li></ul><h1 id="5a2a">Creating tests</h1><p id="deb2">To have a professional package means every<b> </b>function must have its own tests. We’ll use pyt<a href="https://docs.pytest.org/en/7.1.x/">est</a> in combination with <a href="https://hypothesis.readthedocs.io/en/latest/">Hypothesis</a> for testing.</p><p id="9ff4">First let’s add pytest and Hypothesis as test dependencies with a oneliner. Make sure your inside the poetry shell: <code> poetry shell</code> .</p><p id="f401"><code> poetry add pytest hypothesis --group test</code></p><p id="07df">pytest will look inside the <i>tests</i> folder for files with <i>test_</i> in its name, and tests functions inside those files with <i>test_</i> in their name.</p><p id="7afe">Let’s create a folder in <i>tests</i> called <i>test_calc</i></p><p id="dac9"><code> mkdir tests/test_calc</code></p><p id="d26c">Inside that folder let’s create a file called <i>test_square.py</i></p><p id="6957"><code>$ touch tests/test_calc/test_square.py</code></p><p id="3c90">Your tests folder should now have this structure:</p><figure id="e50a"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*66I8wAyN1HADG3u2LZlX-w.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="5f3f">Let’s create some simple tests. In a professional package, both the ‘happy flow’ and the ‘unhappy flow’ of functions should be tested. This means that tests should not only test for expected values, but also for unexpected values and values that should generate an error.</p><h2 id="bc11">Testing the happy flow</h2><p id="b908">To test for how the <code>square</code> function should work, we should only use integers and floats as arguments for parameter <code>x</code> . For this we can use <code>hypothesis.strategies</code> . Hypothesis’ strategies generate values for specific types, e.g. characters, integers, datetimes etc. Using <code>hypothesis.strategies.one_of(), hypothesis.strategies.integers()</code>and<code>hypothesis.strategies.floats()</code> we can generate integers and floats to insert as the argument of <code>x</code> into <code>square</code> . Using the <code>assert</code> statement we check whether the output of <code>square(x)</code> is equal to <code>x * x</code> .</p>
<figure id="7a12">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/03bd20c01128f21c9b7332825e4b8ccb.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="61f1">To run pytest:</p><p id="7ff6"><code>$ poetry run pytest</code></p><p id="a7b7">If everything goes ‘well’, pytest will inform us that a test has failed. If you’re having issues make sure that you’ve ran <code>$ poetry install</code> before.</p><h2 id="88b3">The power of Hypothesis</h2><p id="bccc">Even though the test <code>assert square(x) == x * x</code> seems very trivial, as <code>square(x)</code> simply returns <code>x * x</code> , Hypothesis can find an input that will cause the test to fail. This is because <code>nan</code> is of type <code>float</code> , but <code>nan</code> has the property of never being equal to <code>nan</code> . This is exactly why Hypothesis is such a powerful package for testing. It will try to insert edge cases that will make your code break, which forces you to create robust code. If you’d written yourself examples of integers and floats, you’d probably had never tested for a <code>nan</code> value.</p><p id="7a75">For now, let’s assume we don’t want our <code>square()</code> function to multiply <code>nan</code> values, so we’ll add a check for <code>nan</code> into our function. Of course, make sure to update the docstring too.</p>
<figure id="70e6">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/22fa7cc818f693001557138734a2f7a0.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="1bdf">We’ll also change <code>hypothesis.strategies.floats()</code> to <code>hypothesis.strategies.floats(allow_nan=False)</code> to prevent Hypothesis from generating <code>nan</code> values, as we wanted to only test for the happy flow. All tests should pass now.</p><h2 id="f66e">Testing for errors</h2><p id="fcfa">It’s almost just as important to test for errors as it is for expected input. Two test functions are added to test for errors: <code>test_for_nan()</code> and <code>test_for_invalid_types()</code> :</p>
<figure id="c6e6">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/fbbd855b21af07dee2466791d02cafce.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="fe61"><code>@pytest.mark.parametrize()</code> let’s you manually add different inputs to the test function’s parameter. Using parametrize for <code>test_for_invalid_types(x)</code> we insert all kinds of different data types to check if <code>square(x)</code> will raise a <code>TypeError</code> for it. Now <code>$ poetry run pytest</code> should still say all tests have passed!</p><h2 id="596f">Testing for style</h2><p id="8c2f">Next to testing the code, we also want to test that our code adheres to style formatting rules. For this we’ll use <a href="https://flake8.pycqa.org/en/latest/">flake8</a>. Let us first add <code>flake8</code> as a test dependency:</p><p id="8e55"><code>$ poetry add flake8 --group test</code></p><p id="2e65">Let’s define some rules for <code>flake8</code> , by creating a .<i>flake8</i> file in the <i>my-package</i>/ folder:</p><p id="7016"><code>$ touch .flake8</code></p><p id="eb31">Add the following lines to <i>.flake8</i>:</p>
<figure id="ab7b">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/c9c073dc2768b6ba1f8496fd013f17c9.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="0202">To completely dive into <code>flake8</code> is beyond the scope of this article. This file e.g. specifies that some errors thrown by <code>fake8</code> should be ignored and that the maximum length of a line of code should be 79. This is a convention from Python’s styling guide <a href="https://peps.python.org/pep-0008/">PEP 8</a>.</p><p id="c813">In the beginning we installed Black to automatically format our code, so let’s run that now:</p><p id="31ca"><code>$ poetry run black src/ tests/</code></p><p id="8ff9">To test our code on style let’s run</p><p id="ad52"><code>$ poetry run flake8 src/ tests/</code></p><figure id="3484"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Rh1fWaL1tOVkHTxSX4EDCg.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="f3ca">As you can see, during development I’ve imported <code>cmath.isnan</code> and <code>typing.Type</code> while I’m not actually using that in the final code. <code>flake8</code> warns for that! Let’s adjust calc.py to adhere to <code>flake8</code> :</p>
<figure id="3c42">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/f8a1b1806e4a273e56e6bb46181d3fb5.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="c8a0">Still, we’re left with ‘E501 line too long’ in <i>tests/test_calc/test_square.py. </i>But didn’t we install <i>Black </i>to automatically format our .<i>py</i> files the right way? Yes, but by itself <i>Black</i> doesn’t use a max line length of 79, so we’ll add some settings of <i>Black</i> to the <i>pyproject.toml</i> file:</p><div id="6bc1"><pre><span class="hljs-section">[tool.black]</span>
<span class="hljs-attr">line-length</span> = <span class="hljs-number">79</span>
<span class="hljs-attr">exclude</span> = <span class="hljs-string">'''
/(
.git
| .hg
| .mypy_cache
| .tox
| .venv
| _build
| buck-out
| build
| dist
| docs
)/
'''</span></pre></div><p id="b75c">Now if we run <code> poetry run black src/ tests/</code> <i>Black</i> will automatically reformat <i>tests/test_calc/test_square.py</i> for us and <code> poetry run flake8 src/ tests/</code> should generate no more errors.</p><h1 id="33ae">Checkpoint</h1><p id="55e5">Before continuing let’s make sure we are all on the same page at this stage. This should be your directory structure at the current moment:</p><figure id="f200"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*QTxYhON6xZdyy2bdkxQvAA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="9ff8">To make sure the formatting of the files is right run:</p><div id="ad49"><pre>poetry <span class="hljs-built_in">run</span> black src/ tests/ poetry <span class="hljs-built_in">run</span> isort src/ tests/
$ poetry <span class="hljs-built_in">run</span> flake8 src/ tests/</pre></div><p id="201c">A few scrolls above you can see how the final version of <i>src/my_package/calc.py. </i>should look like.</p><p id="a450">How your <i>pyproject.toml</i> file should look like by now (except for fields like <code>author</code>) :</p>
<figure id="a386">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/ec5e67b10e704c9a2ccf74255102bb43.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="435c">How your <i>tests/test_calc/test_square.py </i>should look like:</p>
<figure id="32ce">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/c517a83c2e0897359c4982a8e8d634ed.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="0ef6">Super! At this moment package has:</p><ul><li>A waterproof function <code>square(x)</code> which can be imported through
<code>>>> from my_package.calc import square</code></li><li>Solid tests for <code>square(x)</code> by using pytest and Hypothesis for testing for wanted , unwanted and edge case inputs.</li><li>A consistent style of coding using <i>Black</i>, isort and flake8.</li></ul><p id="dfa0">This is a great moment to save the changes we’ve made. We’ll do this using git.</p><h2 id="17e2">Save changes using git</h2><p id="b993">Before saving the changes we need to add a <i>.gitignore </i>file to our <i>my-package </i>folder(<code>touch .gitignore </code>) which will automatically ignore some files which should only be kept locally. <a href="https://www.toptal.com/developers/gitignore/">gitignore.io</a> is a website which automatically generates a .<i>gitignore </i>file for us based on preferences. Go to <a href="https://gitignore.io">https://gitignore.io</a> and generate a <i>.gitignore </i>file.</p><figure id="6a4e"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*jfXne7EtIninI-9VtIlPLg.png"><figcaption>Since I’m using macOS and VSCode I’ve selected these settings. [screenshot by Author]</figcaption></figure><p id="262b">Press <i>Create</i> and copy all the text to your <i>.gitignore </i>file. Now we can save our current state of the package using the following commands: (I’ve added the branch details here to clarify we’re making these changes on the <i>.v0.1.0_initalize_package </i>branch).</p><div id="9667"><pre><span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package) git add .
<span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package) git commit -m <span class="hljs-string">"created square(), with tests and formatting"</span></pre></div><h1 id="38f5">Setting up a test automation</h1><p id="e5fd">We know our package is properly tested and styled at this moment, but what if we make some changes to the code and forget to run Black, isort and flake8? Or we make some changes to <code>square(x)</code> which would case some tests to fail, but we forget to run <code> poetry run pytest</code> ? Or we ourselves are very careful, but a colleague or a friend that wants to contribute to the package isn’t? For that we’ll set up test automation.</p><h2 id="c8fa">Pre-commit hooks for styling.</h2><p id="3fdf">The standard workflow of adding your local files and directories to Github is: <code> git add .</code> <code> git commit -m "commit message"</code> <code> git push</code> . We can use pre-commit hooks to check for adhering to our style preferences before being able to commit code. To do this we have to add <a href="https://pre-commit.com/">pre-commit </a>as a development dependency:</p><p id="d87b"><code> poetry add pre-commit --group dev</code></p><p id="8323">The settings of pre-commit should be written in <i>.pre-commit-config.yaml </i>:</p><p id="8b9e"><code>$ echo "" > .pre-commit-config.yaml</code> (we’re using <code>echo "" ></code> instead of <code>touch</code> here as <code>touch</code> will generate an error.</p><p id="6ec4">Copy the following settings to <i>.pre-commit-config.yaml </i>:</p>
<figure id="4543">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/fddbf3bbe20212b17600fc487aa8a8e2.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="37c2">Run <code>$ poetry run pre-commit install</code> to install the pre-commit hooks. Then run <code>$ poetry run pre-commit autoupdate</code> to possibly update any packages. When we run pre-commit locally for the first time it will generate some erros:</p><p id="289b"><code>$ poetry run pre-commit run --all-files</code></p><p id="7bda">pre-commit will automatically add some settings to <i>pyproject.toml </i>and now run the above command again.</p><p id="3eae">If all went well the output should look like this:</p><figure id="ec55"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*PI8ENrJgkXlPSCc3hwXtWA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="93ae">Great! As soon as we’ve pushed this to Github it will make sure anyone who wants to commit code has to adhere to our specified style rules.</p><h2 id="cac9">Autorun pytest on Github on pull requests</h2><p id="68ea">Our worries that someone (ourselves) can add code in the wrong style have been taken care of. But what if someone (or ourselves) adds faulty changes to <code>square(x)</code>?</p><p id="1ae1">Using <a href="https://github.com/features/actions">Github Actions</a> we can make sure that any code that is being pushed to Github will automatically be tested through pytest. To setup such a workflow we have to create a <i>test-packge.yaml</i> file in <i>.github/workflows </i>directory in our <i>my-package </i>folder:</p><div id="f17b"><pre><span class="hljs-meta prompt_">$ </span><span class="language-bash"><span class="hljs-built_in">mkdir</span> .github</span>
<span class="hljs-meta prompt_"> </span><span class="language-bash"><span class="hljs-built_in">mkdir</span> .github/workflows</span>
<span class="hljs-meta prompt_"> </span><span class="language-bash"><span class="hljs-built_in">touch</span> .github/workflows/test-package.yaml</span></pre></div><p id="5953">Copy the following code into <i>.github/workflows/test-package.yaml :</i></p>
<figure id="01cb">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/32212b1fdee7b3e8b313616409092fea.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="8a66">This will make sure the package will be tested for Python 3.10 on the latest versions of Ubuntu and macOS when code is being pushed to the main branch of the Git repository or a pull request is made. The cache will help the tests be quicker the next times, if possible. Please refer <a href="https://jacobian.org/til/github-actions-poetry/">here </a>for more info on this workflow.</p><p id="6a64">That’s it! Our package now contains automatic testing!</p><h2 id="28bb">Create the pull request</h2><p id="d828">Let’s push our code to Github to save the changes</p><div id="ad08"><pre><span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package) $ git add .
<span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package) git commit -m <span class="hljs-string">"added pre-commit and workflow for automatic testing"</span>
<span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package) git push --set-upstream origin v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_package</pre></div><p id="d6f6">We use <code>--set-upstream origin v0.1.0_initialize_package</code> because that branch is not yet known on Github, we created it locally.</p><p id="b386">Go on <a href="https://github.com">https://github.com</a> to your <i>my-package </i>repository and click on <i>Pull requests </i>and then on <i>Compare & pull request </i>or <i>New pull request</i> and select the <i>v0.1.0_initialize_package </i>branch.</p><figure id="2299"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*56PeQ7gL8K_PxLRaLKDG6A.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="4ac5">Edit the Pull request (<i>Write, Assignees, L
Options
abels, Projects, Milestone)</i> to look like this:</p><figure id="eeaa"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*I-M3f0IPyv48rJtwhSYrqA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="0456">Resolves #1 Resolves #2 Resolves #3 refers to the issues we’ve created at the start of this article. When you type # you’ll get to choose which issues are linked to this pull request. When the pull accept is requested then automatically theses issues on the Github Project will be closed.</p><p id="72be">When you click <i>Create pull request </i>a request will be made to merge the branch to the main branch and because of .<i>github/workflows/test-package.yaml </i>tests from our <i>tests/</i> folder will now start to run. After a few minutes (the next time should be quicker because we implemented caches in the workflow) the workflow should have passed all tests!</p><figure id="1637"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*T6N0WfdusYLQ-TRKv_WfsA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="b74d">Now let’s press <i>Merge pull request </i>to push our once local code to the main branch on Github. Great!</p><h1 id="446f">Create documentation on ReadTheDocs</h1><p id="3625">Let’s create a new branch for the documentation part that starts from the current state of the main branch:</p><div id="4b93"><pre><span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_initialize_project) git switch -c <span class="hljs-string">"v0.1.0_create_docs"</span> main</pre></div><p id="e266">First, let’s install <a href="https://www.sphinx-doc.org/en/master/">Sphinx</a> and <a href="https://sphinx-extensions.readthedocs.io/en/latest/sphinx-autobuild.html">sphinx-autobuild</a> which will form the the building blocks for our documentation. We’ll create the group <code>doc</code> for the packages we’ll install for creating documentation.</p><p id="19f0"><code> poetry add sphinx sphinx-autobuild --group doc</code></p><p id="52b3">To initialize the process of making our documentation to publish on ReadTheDocs run</p><p id="9f07"><code> poetry run sphinx-quickstart docs</code></p><p id="2a6f">Please answer the first question with <b>y</b>, all other questions speak for themselves.</p><div id="e0d2"><pre><span class="hljs-meta prompt_">> </span><span class="language-bash">Separate <span class="hljs-built_in">source</span> and build directories (y/n): y</span></pre></div><p id="1b65">After running you should see a <i>docs/ </i>folder appear. Inside <i>docs</i>/<i>source </i>there are two very important files: <i>index.rst </i>and <i>conf.py. index.rst </i>is your the index page of your documentation website and <i>conf.py </i>contains the settings such as which theme and extensions to use.</p><p id="ff91">To see our documentation so far let’s run the following command <b>in a new terminal:</b></p><div id="09ce"><pre><span class="hljs-meta prompt_"> </span><span class="language-bash">poetry shell</span>
<span class="hljs-meta prompt_"> </span><span class="language-bash">poetry run sphinx-autobuild docs/source/ docs/build/html</span></pre></div><p id="1532">In the terminal you should find a link to probably port <i>localhost:8000. </i>Clicking on it should show you this webpage:</p><figure id="7473"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*qJj16MoorYg-OWwSUziFIg.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="81e4">There it is! The start of our ReadTheDocs website! Notice that if you make changes to the <i>index.rst </i>the website re-renders automatically.</p><h2 id="fabb">Professionalizing the documentation</h2><p id="d5e7"><i>.rst </i>is the extension of reStructuredText, a quite old but still used markdown language. However, since <i>markdown </i>gained quite some popularity, is more modern and is more commonly used nowadays, we’ll switch to <i>markdown </i>using (because we’ll use <code>rst2myst</code> only once we’ll not add it as a doc dependency):</p><div id="a225"><pre><span class="hljs-symbol"></span> pip install rst-to-myst
<span class="hljs-symbol"></span> rst2myst convert docs<span class="hljs-comment">/**/</span>*.rst</pre></div><p id="9162">Now you should see an <i>index.md </i>inside <i>docs/source/. </i>This means we can safely remove the <i>.rst </i>version:</p><p id="1ae0"><code> rm docs/source/index.rst</code></p><p id="5318">To be able for Sphinx to interpret <i>markdown </i>instead of reStructuredText, we’ll have to install M<a href="https://myst-parser.readthedocs.io/en/latest/">ySt Parser</a>:</p><p id="2e46"><code> poetry add myst-parser --group doc</code></p><p id="e8a6">For Sphinx to be able to use the myst_parser we have to add in to the list of extensions in <i>docs/source/conf.py:</i></p><div id="581b"><pre><span class="hljs-attr">extensions</span> = [
<span class="hljs-string">"myst_parser"</span>,
]</pre></div><p id="28bc">Now rerun:</p><div id="8de5"><pre> poetry <span class="hljs-keyword">run</span><span class="language-bash"> sphinx-autobuild docs/source docs/build/html</span></pre></div><p id="6c06">You should see the exact same website as before, even though we now use <i>markdown </i>instead of<i> reStructuredText.</i></p><p id="0a20">Let’s add some more extensions! Install these as follows:</p><p id="821f"><code>$ poetry add nbsphinx sphinx-autoapi sphinx-rtd-theme --group doc</code></p><p id="b10a">Change your <i>docs/source/conf.py </i>to: (but change the name etc.)</p>
<figure id="86de">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/795ab7fce2dfd5d2258b2bf8022030d9.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="89de">You might have noticed that in the bottom of <i>conf.py</i> we changed the html_theme to <code><i>"</i>sphinx_rtd_theme"</code> . Which is a very classic theme for ReadTheDocs. You can pick any theme you like, just use <code>$ poetry add <theme> --group doc</code> and change the <code>html_theme</code> in <i>conf.py </i>to the name of the theme. You can find the best Sphinx themes <a href="https://sphinx-themes.org/">here</a>.</p><p id="6e30">Run <code>$ poetry run sphinx-autobuild docs/source docs/build/html</code></p><p id="ebf9">You should now see a webpage which looks something like this:</p><figure id="c2f4"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*IQ1DXmIgyOAcrsTMrTKmhQ.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="63af">If you’re familiar with looking up documentation for Python you should recognize this often used theme! What’s very important to notice is that now in <b>Contents </b>we have an <i>API Reference</i>. This is automatically generated by the package we just added: <a href="https://sphinx-autoapi.readthedocs.io/en/latest/">Sphinx AutoAPI</a>. And this is where we’re going to find out why we spent all this effort in creating a proper docstring of <code>square(x)</code> . Click on <i>API Reference </i>and navigate to <i>my_package.calc </i>. Here you should see the following:</p><figure id="ab9b"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Qw7eTlK8KR3LDzpuE3d8ww.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="38c9">Without writing a single extra line of documentation besides the function’s docstring, we have beautiful documentation of <code>square(x)</code> !</p><h2 id="ff84">Adding Jupyter Notebooks to the Documentation</h2><p id="1e04">Jupyter Notebooks have become very popular over the past decade. In these notebooks, you can add both Python code and <i>markdown </i>in the same file. Because we installed <a href="https://nbsphinx.readthedocs.io/en/0.8.9/">nbsphinx</a> we can host Jupyter Notebooks on our documentation website. Let’s start by creating a <i>usage</i> notebook<i> </i>in <i>docs/notebooks/source:</i></p><div id="2cb6"><pre><span class="hljs-meta prompt_">$ </span><span class="language-bash"><span class="hljs-built_in">mkdir</span> docs/source/notebooks</span>
<span class="hljs-meta prompt_">$ </span><span class="language-bash"><span class="hljs-built_in">touch</span> docs/source/notebooks/usage.ipynb</span></pre></div><p id="4cc6">Create the following Notebook: (notice here that I wrote <code>pip install my-package-tomergabay</code> as the name<code> my-package</code> is already taken on <a href="https://pypi.org">https://pypi.org</a>.</p>
<figure id="0ac0">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/bcd0eed47dd3df826126ba89f37486ef.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="0b9f">If you have a problem with creating a Notebook you can also just copy-paste the raw code from <a href="https://gist.github.com/sTomerG/d8835937c7bfe1588e30c5338653323f">here</a> into <i>docs/source/notebooks/usage.ipynb.</i></p><p id="869a">We have to link this new file to the <i>toctree </i>in<i> docs/source/index.md </i>to make sure it’s findable by adding the line: <code>notebooks/usage</code></p><div id="38ea"><pre><span class="hljs-code">````</span>{toctree}
<span class="hljs-meta">:caption:</span> <span class="hljs-emphasis">'Contents:'</span>
<span class="hljs-meta">:maxdepth:</span> 2</pre></div><div id="55c1"><pre>notebooks/usage
<span class="hljs-string">``</span><span class="hljs-string">`</span></pre></div><p id="279a">There is a chance you might get an error that Pandoc wasn’t found if you now run <code> poetry run sphinx-autobuild docs/source docs/build/html </code>Install Pandoc with <code> brew install pandoc</code> or <a href="https://pandoc.org/installing.html">click</a> here for installation instructions for Windows or Linux, and rerun the <code>sphinx-autobuild </code>command above again.</p><figure id="5017"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*YKQfReK_M0fsQtErKEAUmA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="ee5f">Now you should see the <i>Usage</i> page on your website too! You can add more <i>.md or .ipynb </i>files if you like<i>. </i>Just make sure to place them inside the <i>docs/source</i> folder and add them to the <i>toctree </i>in <i>docs/source/index.md</i></p><p id="7e5f">Congratulations! That’s all the basics you need to know to create a beautiful documentation website for your Python package! If you want to have a deeper understanding of all the communicating parts within the documentation, I’d recommend to watch <a href="https://youtu.be/qRSb299awB0">this video</a>, which is where I first learned about Sphinx and publishing to ReadTheDocs.</p><h2 id="cc5e">Publishing to ReadTheDocs</h2><p id="5e78">Before publishing to ReadTheDocs we have to do five things:</p><ol><li>Create an account on <a href="https://readthedocs.org/">ReadTheDocs</a>.</li><li>Create a <i>.readthedocs.yaml </i>configuration file</li><li>Create a <i>rtd_requirements.txt </i>in <i>docs/</i></li><li>Push our code to Github and merge it with the main branch.</li><li>Create a release version of our package on Github.</li></ol><p id="8550"><b>Create an account on ReadTheDocs
</b>Go to <a href="https://readthedocs.org/">https://readthedocs.org/</a> and sign up if you don’t have an account yet, it’s free!</p><p id="08c2"><b>Create a .readthedocs.yaml configuration file</b></p><p id="25b3"><code>$ touch .readthedocs.yaml</code></p><p id="a43f">And add the following code:</p>
<figure id="bc87">
<div>
<div>
<iframe class="gist-iframe" src="/gist/sTomerG/70e9027c330096a6ce1655aca0533619.js" allowfullscreen="" frameborder="0" height="undefined" width="undefined">
</div>
</div>
</figure></iframe></div></div></figure><p id="5c1c"><b>Create a <i>rtd_requirements.txt </i>in <i>docs/</i></b></p><div id="aed9"><pre>$ poetry ex<span class="hljs-keyword">port</span> <span class="hljs-comment">--with doc -f requirements.txt --output docs/rtd_requirements.txt</span></pre></div><p id="b1b0"><b>Push our code to Github and merge it with the main branch.</b></p><div id="230a"><pre><span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_create_docs) $ git add .
<span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_create_docs) git commit -m <span class="hljs-string">"ready for ReadTheDocs"</span>
<span class="hljs-attribute">git</span>:(v0.<span class="hljs-number">1</span>.<span class="hljs-number">0</span>_create_docs) git push </pre></div><p id="ef4f">Create and merge the pull request on Github. All tests should still pass!</p><figure id="6b71"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*EqHz2Xvwmgo6k0dvZoQIqw.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="5853"><b>Create a release version of our package.</b></p><p id="521c">If you haven’t added text in the <i>README.md</i> yet, make sure to do so! Also add a <i>LICENSE</i> file (the file should literately be named LICENSE), in the same folder as the <i>README</i>, which you can pick from <a href="https://choosealicense.com/">https://choosealicense.com/</a> (I chose MIT). Then everything should be ready for the first release version of our package:</p><figure id="b91b"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*tAlwgFimjzhMUv4eQdCpzA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="f125">Then add the tag <i>v0.1.0:</i></p><figure id="c38f"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*HX8Yg0jBdp10LoQPZOPQdg.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="7cd2">Add a name for the release, add a description and click <i>Publish release.</i></p><figure id="da1e"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*tQ2MXcKEFoogDsdp5i59Dw.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="e0d7">You’ve now released the first version of your package!</p><h1 id="2e10">Publish to ReadTheDocs</h1><p id="1af0">Go to <a href="https://readthedocs.org">https://readthedocs.org</a> and sign up.</p><p id="cc3f">Click on the top of right on your name when you’re logged in and on <i>My Projects </i>and click on <i>Import a Project:</i></p><figure id="2ee2"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*GgjND-eqmk8N8Lpu7A8Dxw.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="dfb2">Find your package in the list and click on the <i>+ </i>sign. You’ll see the default settings for your package, there is no need to change anything here, just click <i>Next </i>and then <i>Build version</i>. After a minute or so the build should be ready and you can click on <i>view docs </i>to see the documentation of your package on ReadTheDocs!</p><figure id="b81c"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Q0ja8I76sDyaVmQnadg7bA.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="32e1">If you want to specifically build a ReadTheDocs for this version you can go to your project and click on <i>Versions </i>and activate version <i>v0.1.0. </i>The build process will start automatically.</p><figure id="f43c"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*JP164a89x77mdJeHjX4Ygg.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="7ef5">There rests us only one thing now and that is to publish our package to PyPi!</p><h1 id="4688">Publishing to PyPi</h1><p id="c747">If you want to make your package open source and thus installable for everyone using PyPi follow these steps:</p><p id="92dc">Go back to the terminal and make sure your virtual environment is activated with <code> poetry shell</code> and that you are on the main branch with the last updates: <code> git switch main && git pull</code></p><p id="8943">If don’t have an account yet on PyPi create it <a href="https://pypi.org/account/register/">here</a>.</p><p id="4e2a">Before continuing please make sure your <i>pyproject.toml </i>is how you want it to be, especially section <code>[tool.poetry]</code> . The README will appear on the homepage of your project on PyPi, so make sure useful information is in the README (e.g. the link to your ReadTheDocs page). Also make sure your code is still working as wanted with <code> poetry run pytest</code>.</p><p id="11af">Every time you want to make an adjustment to an already published package on PyPi you have to increase the version number according to the <a href="https://semver.org/">semantic versioning rules</a>.</p><p id="3213">Build the required files for publishing to PyPi with</p><p id="a9cc"><code> poetry build</code></p><p id="399b">and publish it to PyPi with:</p><p id="033f"><code> poetry publish -u <username> -p <password></code></p><p id="9ac6">That’s all! Your package is now installable for everyone through:</p><p id="6312"><code> pip install <package name></code></p><p id="adb0">Check your package webpage by visiting <a href="https://pypi.org/project/my-package-tomergabay/">https://pypi.org/project/<package_name>/</package_name></a> or search for your package on <a href="https://pypi.org/">https://pypi.org/</a>.</p><h1 id="29ca">Conclusion</h1><p id="9770">Congratulations! You’ve taken all steps to set up a professional (open source) Python package that contains uniformly formatted code, test-automation, documentation on ReadTheDocs and is published on PyPi! All your initial <i>Todos </i>can be moved to <i>Done</i> now on the Projects page:</p><figure id="8174"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*pEpgCRjGj6GIiLtrRyUL1A.png"><figcaption>[screenshot by Author]</figcaption></figure><p id="d12c">You can continue developing your Python package from here onwards yourself. If you want to re-use this structure for another project in the future you can also clone <a href="https://github.com/sTomerG/my-package">this repository</a> and follow its instructions in its README so that you don’t have to go through each step of this article again.</p><p id="fc00">Feel free to contact me if you have any questions, or ask your question in the comment section!</p><div id="c2f9" class="link-block">
<a href="https://readmedium.com/mlearning-ai-submission-suggestions-b51e2b130bfb">
<div>
<div>
<h2>Mlearning.ai Submission Suggestions</h2>
<div><h3>How to become a writer on Mlearning.ai</h3></div>
<div><p>medium.com</p></div>
</div>
<div>
<div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*6xCb1sNpjadaSBuVLPTFQQ.png)"></div>
</div>
</div>
</a>
</div></article></body>
How to start any professional Python Package project
Including test automation, creating documentation on ReadTheDocs and publishing to PyPi.
Virtually everyone who uses Python makes use of PyPi, the Python Package Index where open-source code can be downloaded from (which is consulted when using $ pip install). Famous packages as pandas and Numpy are all on PyPi. It’s actually really easy to publish your own project to PyPi. However, to simply write some code and publish it to PyPi isn’t a professional or sustainable approach. The following problems can arise:
Lack of or absence of documentation can cause other people to not know how to use your package, or you yourself forget it after a few weeks, months or years.
Lack of or absence of (automated) tests can cause broken code to enter your package unnoticed when making changes.
Changes to your package’s dependencies can cause your code to break.
Badly or inconsistently formatted code can make the code hard to read.
In this article we will tackle all the steps anyone should take when wanting to create a professional (open source) package. This includes e.g. project management, dependency management, automated testing, creating documentation and publishing it on ReadTheDocs, and publishing the package to PyPi.
This article follows a step-by-step approach, which results in that anyone with some Python experience can follow along. Because we’ll tackle many subjects of professional package development we won’t go into the detail of every area, but I leave references for anyone who’d want to dive deeper into a certain aspect.
You can use this repository as a reference for the files we’re creating in this article. To have a proper understanding of all aspects of a professional Python package I strongly encourage you to follow along with the article instead of just cloning the Github repo. You can always do that at the end :).
Go to https://github.com and create a new repository (or sign up for an account if you don’t have one yet, it’s free). Please make sure to use the same settings as pictured below. Of course, you can choose a different name for your repository, which should be the same as the name you want for your package. I’ll use my-package for the name of the package we’ll create throughout this article. It’s a consensus to use hyphens instead of underscores here.
[screenshot by Author]
In the Repository of my-project, click on Issues, and on the right of your screen there is a button called Milestones. Click on Milestones and create a new one:
[screenshot by Author]
Milestones in Python packages are often linked to the new release of a (sub)version of a package. As this will be our first version of our package, let’s use the semantic version convention by calling it 0.1.0 , also see: https://semver.org/
Given a version number MAJOR.MINOR.PATCH, increment the:
1. MAJOR version when you make incompatible API changes
2. MINOR version when you add functionality in a backwards compatible manner
3. PATCH version when you make backwards compatible bug fixes
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
Setup a Github Project
Next to having the Github Repository where we’ll push our code, we want to create a Project, where we can track what issues we’re working on and what we’d like to work on in the future. Go to your account, click on Projects and click on New project.
[screenshot by Author]
Click on Board and then Create, you can change to a different layout at any time.
[screenshot by Author]
Now, let’s rename the project in the top left to my-project and let’s add some draft items in the bottom left.
[screenshot by Author]
Our Todo list on this board is exactly what we’ll cover in the rest of this article:
[screenshot by Author]
We can convert the drafts to issues of my-project by clicking on them. Make sure to convert the draft to an issue of my-project, give it the right labels, and add it to the v0.1.0 milestone. A draft should end up looking like this:
[screenshot by Author]
Which, if you’ve edited all your drafts, should result in a Todos that look something like the screenshot below. Click on the arrow next to View 1 to make the labels visible.
[screenshot by Author]
Now let’s link the Github Project to our Github Repository:
[screenshot by Author]
That’s it! Our Github is now ready for a professional-level developed project. If you or someone else finds a bug or wants to suggest a feature, you can add that on the project board to track who is working on what and when. For more functionalities of Github Projects please take a look here.
Setup virtual environment using pyenv
Before we start our project we want to choose a Python version to develop our project in. With pyenv you can choose any Python version, regardless of what you’ve installed globally on your laptop, that’s why we’re using it here.
Install pyenv (for installation on other OS systems than macOS click here).
Choose a Python version to install, which will be the Python version you’ll develop your package in.
$ pyenv install 3.10.6
Create a virtual environment to develop your package in:
$ pyenv virtualenv 3.10.6 my_package
Activate your virtual environment:
$ pyenv shell my_package
Check if the version is correct:
$ python --version
Python 3.10.6
Congratulations! You now have your virtual environment set up for your package.
Setup Poetry and link the repo to Github
Poetry is a Python package which makes packaging and dependency management easy. This saves us a lot of hassle with dependencies, versioning and publishing.
Setup Poetry
We’ll continue where we left. Make sure your virtual environment is activated.
$ pyenv shell my_package
First let’s upgrade pip:
$ pip install --upgrade pip
Then install Poetry:
$ pip install poetry
Initialize a new package with: (use the same name as on Github, with hyphens and not underscores)
$ poetry new my-package --src
Created package my_package in my-package
Enter the newly created directory:
cd my-package
[screenshot by Author]
This is how your directory should look like. All the code for the core of the package will be coded in src/my_package. Tests of the code should be written in the tests folder. In pyproject.toml is all the meta info about the reposiory, from name to author to dependencies. It is steadily replacing the long used setup.py and setup.cfg structure. The README.md still empty, this is what a user will read at first when bumping into your package on Github or PyPi. Usually there are installation instructions and a basic usage explanation in the README.MD.
For now that’s all we need from poetry.
Link the repo to GitHub
Go back to the terminal. In the my-package folder (note: not src/my_package) initialize git with:
$ git init -b main
Your terminal should look now something like this:
git:(main) $
To add the current files created by poetry to GitHub:
Now your local repository is linked to Github, which allows you to submit any local changes to Github.
Creating a simple package
Let’s first switch to a new git branch. For now it’s enough to know that each new feature of your package should ideally be created on it’s own branch, before being merged to the main branch, when the feature’s code is final. For more information about Git branching please read this article.
Because this gets a bit long, I’ll shorten the terminal back to just $
Activate the poetry virtual environment
When developing we’ll use poetry’s virtual environment that will activate (or create if it doesn't exist yet) for our package through:
$ poetry shell
Adding Black and isort as a dependencies.
For this project we’ll use Black and isort to format our code. Black automatically formats Python code in a structured way to create consistency throughout your entire project. isort is a package that sorts imports at the top of your Python file in a structured and consistent way.
To add Black andisort as a dependency to the package we’ll use poetry. However, it’s important to note that only people developing and contributing to the package need to have Black and isort installed. End users won’t need them since they’ll only use the code, and not change it. To add Black and isort as a dependency for developers use:
$ poetry add black isort --group dev
This should add the following two lines to your pyproject.toml file (exact versions can differ):
Next to that poetry has added Black and isort as dependencies for developers, it has also automatically installed it for us in poetry’s virtual environment. In general there is no need for pip installcommands with poetry.
To have Black and isort align add the following two lines to pyproject.toml:
From inside the my-project folder on your laptop create a .py file called calc.py inside the my_project/src folder. You can also use the touch command to create the file:
$ touch src/my_package/calc.py
Inside calc.py create a function called square which returns the square of its input.
There are some important things to notice here:
We’re using type hinting Union[int, float]to imply which value types are accepted as input and to imply which value type should be returned.
We’re using Numpy’s Docstring style for the docstring. VisualStudio Code has an extension which auto-generates large parts of the doc-string in Numpy’s style. Make sure to invest in proper docstring documentation and add the Examples section too, as this will be very useful later on when we’ll generate documentation for ReadTheDocs .
We’reusing Black’s and isort’s formatting style. Run $ poetry run black src/ to format the core package to Black’s style and $ poetry run isort src/ to automatically sort the imports.
To use our function square(x) we can import it through:
>>> from my_package.calc import square . Make sure to install my_package first with: $ poetry install
Creating tests
To have a professional package means everyfunction must have its own tests. We’ll use pytest in combination with Hypothesis for testing.
First let’s add pytest and Hypothesis as test dependencies with a oneliner. Make sure your inside the poetry shell: $ poetry shell .
$ poetry add pytest hypothesis --group test
pytest will look inside the tests folder for files with test_ in its name, and tests functions inside those files with test_ in their name.
Let’s create a folder in tests called test_calc
$ mkdir tests/test_calc
Inside that folder let’s create a file called test_square.py
$ touch tests/test_calc/test_square.py
Your tests folder should now have this structure:
[screenshot by Author]
Let’s create some simple tests. In a professional package, both the ‘happy flow’ and the ‘unhappy flow’ of functions should be tested. This means that tests should not only test for expected values, but also for unexpected values and values that should generate an error.
Testing the happy flow
To test for how the square function should work, we should only use integers and floats as arguments for parameter x . For this we can use hypothesis.strategies . Hypothesis’ strategies generate values for specific types, e.g. characters, integers, datetimes etc. Using hypothesis.strategies.one_of(), hypothesis.strategies.integers()andhypothesis.strategies.floats() we can generate integers and floats to insert as the argument of x into square . Using the assert statement we check whether the output of square(x) is equal to x * x .
To run pytest:
$ poetry run pytest
If everything goes ‘well’, pytest will inform us that a test has failed. If you’re having issues make sure that you’ve ran $ poetry install before.
The power of Hypothesis
Even though the test assert square(x) == x * x seems very trivial, as square(x) simply returns x * x , Hypothesis can find an input that will cause the test to fail. This is because nan is of type float , but nan has the property of never being equal to nan . This is exactly why Hypothesis is such a powerful package for testing. It will try to insert edge cases that will make your code break, which forces you to create robust code. If you’d written yourself examples of integers and floats, you’d probably had never tested for a nan value.
For now, let’s assume we don’t want our square() function to multiply nan values, so we’ll add a check for nan into our function. Of course, make sure to update the docstring too.
We’ll also change hypothesis.strategies.floats() to hypothesis.strategies.floats(allow_nan=False) to prevent Hypothesis from generating nan values, as we wanted to only test for the happy flow. All tests should pass now.
Testing for errors
It’s almost just as important to test for errors as it is for expected input. Two test functions are added to test for errors: test_for_nan() and test_for_invalid_types() :
@pytest.mark.parametrize() let’s you manually add different inputs to the test function’s parameter. Using parametrize for test_for_invalid_types(x) we insert all kinds of different data types to check if square(x) will raise a TypeError for it. Now $ poetry run pytest should still say all tests have passed!
Testing for style
Next to testing the code, we also want to test that our code adheres to style formatting rules. For this we’ll use flake8. Let us first add flake8 as a test dependency:
$ poetry add flake8 --group test
Let’s define some rules for flake8 , by creating a .flake8 file in the my-package/ folder:
$ touch .flake8
Add the following lines to .flake8:
To completely dive into flake8 is beyond the scope of this article. This file e.g. specifies that some errors thrown by fake8 should be ignored and that the maximum length of a line of code should be 79. This is a convention from Python’s styling guide PEP 8.
In the beginning we installed Black to automatically format our code, so let’s run that now:
$ poetry run black src/ tests/
To test our code on style let’s run
$ poetry run flake8 src/ tests/
[screenshot by Author]
As you can see, during development I’ve imported cmath.isnan and typing.Type while I’m not actually using that in the final code. flake8 warns for that! Let’s adjust calc.py to adhere to flake8 :
Still, we’re left with ‘E501 line too long’ in tests/test_calc/test_square.py. But didn’t we install Black to automatically format our .py files the right way? Yes, but by itself Black doesn’t use a max line length of 79, so we’ll add some settings of Black to the pyproject.toml file:
Now if we run $ poetry run black src/ tests/Black will automatically reformat tests/test_calc/test_square.py for us and $ poetry run flake8 src/ tests/ should generate no more errors.
Checkpoint
Before continuing let’s make sure we are all on the same page at this stage. This should be your directory structure at the current moment:
[screenshot by Author]
To make sure the formatting of the files is right run:
$ poetry run black src/ tests/
$ poetry run isort src/ tests/
$ poetry run flake8 src/ tests/
A few scrolls above you can see how the final version of src/my_package/calc.py. should look like.
How your pyproject.toml file should look like by now (except for fields like author) :
How your tests/test_calc/test_square.py should look like:
Super! At this moment package has:
A waterproof function square(x) which can be imported through
>>> from my_package.calc import square
Solid tests for square(x) by using pytest and Hypothesis for testing for wanted , unwanted and edge case inputs.
A consistent style of coding using Black, isort and flake8.
This is a great moment to save the changes we’ve made. We’ll do this using git.
Save changes using git
Before saving the changes we need to add a .gitignore file to our my-package folder($ touch .gitignore ) which will automatically ignore some files which should only be kept locally. gitignore.io is a website which automatically generates a .gitignore file for us based on preferences. Go to https://gitignore.io and generate a .gitignore file.
Since I’m using macOS and VSCode I’ve selected these settings. [screenshot by Author]
Press Create and copy all the text to your .gitignore file. Now we can save our current state of the package using the following commands: (I’ve added the branch details here to clarify we’re making these changes on the .v0.1.0_initalize_package branch).
git:(v0.1.0_initialize_package) $ git add .
git:(v0.1.0_initialize_package) $ git commit -m "created square(), with tests and formatting"
Setting up a test automation
We know our package is properly tested and styled at this moment, but what if we make some changes to the code and forget to run Black, isort and flake8? Or we make some changes to square(x) which would case some tests to fail, but we forget to run $ poetry run pytest ? Or we ourselves are very careful, but a colleague or a friend that wants to contribute to the package isn’t? For that we’ll set up test automation.
Pre-commit hooks for styling.
The standard workflow of adding your local files and directories to Github is: $ git add .$ git commit -m "commit message"$ git push . We can use pre-commit hooks to check for adhering to our style preferences before being able to commit code. To do this we have to add pre-commit as a development dependency:
$ poetry add pre-commit --group dev
The settings of pre-commit should be written in .pre-commit-config.yaml :
$ echo "" > .pre-commit-config.yaml (we’re using echo "" > instead of touch here as touch will generate an error.
Copy the following settings to .pre-commit-config.yaml :
Run $ poetry run pre-commit install to install the pre-commit hooks. Then run $ poetry run pre-commit autoupdate to possibly update any packages. When we run pre-commit locally for the first time it will generate some erros:
$ poetry run pre-commit run --all-files
pre-commit will automatically add some settings to pyproject.toml and now run the above command again.
If all went well the output should look like this:
[screenshot by Author]
Great! As soon as we’ve pushed this to Github it will make sure anyone who wants to commit code has to adhere to our specified style rules.
Autorun pytest on Github on pull requests
Our worries that someone (ourselves) can add code in the wrong style have been taken care of. But what if someone (or ourselves) adds faulty changes to square(x)?
Using Github Actions we can make sure that any code that is being pushed to Github will automatically be tested through pytest. To setup such a workflow we have to create a test-packge.yaml file in .github/workflows directory in our my-package folder:
Copy the following code into .github/workflows/test-package.yaml :
This will make sure the package will be tested for Python 3.10 on the latest versions of Ubuntu and macOS when code is being pushed to the main branch of the Git repository or a pull request is made. The cache will help the tests be quicker the next times, if possible. Please refer here for more info on this workflow.
That’s it! Our package now contains automatic testing!
We use --set-upstream origin v0.1.0_initialize_package because that branch is not yet known on Github, we created it locally.
Go on https://github.com to your my-package repository and click on Pull requests and then on Compare & pull request or New pull request and select the v0.1.0_initialize_package branch.
[screenshot by Author]
Edit the Pull request (Write, Assignees, Labels, Projects, Milestone) to look like this:
[screenshot by Author]
Resolves #1 Resolves #2 Resolves #3 refers to the issues we’ve created at the start of this article. When you type # you’ll get to choose which issues are linked to this pull request. When the pull accept is requested then automatically theses issues on the Github Project will be closed.
When you click Create pull request a request will be made to merge the branch to the main branch and because of .github/workflows/test-package.yaml tests from our tests/ folder will now start to run. After a few minutes (the next time should be quicker because we implemented caches in the workflow) the workflow should have passed all tests!
[screenshot by Author]
Now let’s press Merge pull request to push our once local code to the main branch on Github. Great!
Create documentation on ReadTheDocs
Let’s create a new branch for the documentation part that starts from the current state of the main branch:
git:(v0.1.0_initialize_project) $ git switch -c "v0.1.0_create_docs" main
First, let’s install Sphinx and sphinx-autobuild which will form the the building blocks for our documentation. We’ll create the group doc for the packages we’ll install for creating documentation.
$ poetry add sphinx sphinx-autobuild --group doc
To initialize the process of making our documentation to publish on ReadTheDocs run
$ poetry run sphinx-quickstart docs
Please answer the first question with y, all other questions speak for themselves.
> Separate source and build directories (y/n): y
After running you should see a docs/ folder appear. Inside docs/source there are two very important files: index.rst and conf.py. index.rst is your the index page of your documentation website and conf.py contains the settings such as which theme and extensions to use.
To see our documentation so far let’s run the following command in a new terminal:
$ poetry shell$ poetry run sphinx-autobuild docs/source/ docs/build/html
In the terminal you should find a link to probably port localhost:8000. Clicking on it should show you this webpage:
[screenshot by Author]
There it is! The start of our ReadTheDocs website! Notice that if you make changes to the index.rst the website re-renders automatically.
Professionalizing the documentation
.rst is the extension of reStructuredText, a quite old but still used markdown language. However, since markdown gained quite some popularity, is more modern and is more commonly used nowadays, we’ll switch to markdown using (because we’ll use rst2myst only once we’ll not add it as a doc dependency):
Change your docs/source/conf.py to: (but change the name etc.)
You might have noticed that in the bottom of conf.py we changed the html_theme to "sphinx_rtd_theme" . Which is a very classic theme for ReadTheDocs. You can pick any theme you like, just use $ poetry add <theme> --group doc and change the html_theme in conf.py to the name of the theme. You can find the best Sphinx themes here.
Run $ poetry run sphinx-autobuild docs/source docs/build/html
You should now see a webpage which looks something like this:
[screenshot by Author]
If you’re familiar with looking up documentation for Python you should recognize this often used theme! What’s very important to notice is that now in Contents we have an API Reference. This is automatically generated by the package we just added: Sphinx AutoAPI. And this is where we’re going to find out why we spent all this effort in creating a proper docstring of square(x) . Click on API Reference and navigate to my_package.calc . Here you should see the following:
[screenshot by Author]
Without writing a single extra line of documentation besides the function’s docstring, we have beautiful documentation of square(x) !
Adding Jupyter Notebooks to the Documentation
Jupyter Notebooks have become very popular over the past decade. In these notebooks, you can add both Python code and markdown in the same file. Because we installed nbsphinx we can host Jupyter Notebooks on our documentation website. Let’s start by creating a usage notebookin docs/notebooks/source:
Create the following Notebook: (notice here that I wrote pip install my-package-tomergabay as the name my-package is already taken on https://pypi.org.
If you have a problem with creating a Notebook you can also just copy-paste the raw code from here into docs/source/notebooks/usage.ipynb.
We have to link this new file to the toctree in docs/source/index.md to make sure it’s findable by adding the line: notebooks/usage
````{toctree}
:caption:'Contents:':maxdepth: 2
notebooks/usage
```
There is a chance you might get an error that Pandoc wasn’t found if you now run $ poetry run sphinx-autobuild docs/source docs/build/html Install Pandoc with $ brew install pandoc or click here for installation instructions for Windows or Linux, and rerun the sphinx-autobuild command above again.
[screenshot by Author]
Now you should see the Usage page on your website too! You can add more .md or .ipynb files if you like. Just make sure to place them inside the docs/source folder and add them to the toctree in docs/source/index.md
Congratulations! That’s all the basics you need to know to create a beautiful documentation website for your Python package! If you want to have a deeper understanding of all the communicating parts within the documentation, I’d recommend to watch this video, which is where I first learned about Sphinx and publishing to ReadTheDocs.
Publishing to ReadTheDocs
Before publishing to ReadTheDocs we have to do five things:
Create and merge the pull request on Github. All tests should still pass!
[screenshot by Author]
Create a release version of our package.
If you haven’t added text in the README.md yet, make sure to do so! Also add a LICENSE file (the file should literately be named LICENSE), in the same folder as the README, which you can pick from https://choosealicense.com/ (I chose MIT). Then everything should be ready for the first release version of our package:
[screenshot by Author]
Then add the tag v0.1.0:
[screenshot by Author]
Add a name for the release, add a description and click Publish release.
[screenshot by Author]
You’ve now released the first version of your package!
Click on the top of right on your name when you’re logged in and on My Projects and click on Import a Project:
[screenshot by Author]
Find your package in the list and click on the + sign. You’ll see the default settings for your package, there is no need to change anything here, just click Next and then Build version. After a minute or so the build should be ready and you can click on view docs to see the documentation of your package on ReadTheDocs!
[screenshot by Author]
If you want to specifically build a ReadTheDocs for this version you can go to your project and click on Versions and activate version v0.1.0. The build process will start automatically.
[screenshot by Author]
There rests us only one thing now and that is to publish our package to PyPi!
Publishing to PyPi
If you want to make your package open source and thus installable for everyone using PyPi follow these steps:
Go back to the terminal and make sure your virtual environment is activated with $ poetry shell and that you are on the main branch with the last updates: $ git switch main && git pull
If don’t have an account yet on PyPi create it here.
Before continuing please make sure your pyproject.toml is how you want it to be, especially section [tool.poetry] . The README will appear on the homepage of your project on PyPi, so make sure useful information is in the README (e.g. the link to your ReadTheDocs page). Also make sure your code is still working as wanted with $ poetry run pytest.
Every time you want to make an adjustment to an already published package on PyPi you have to increase the version number according to the semantic versioning rules.
Build the required files for publishing to PyPi with
$ poetry build
and publish it to PyPi with:
$ poetry publish -u <username> -p <password>
That’s all! Your package is now installable for everyone through:
Congratulations! You’ve taken all steps to set up a professional (open source) Python package that contains uniformly formatted code, test-automation, documentation on ReadTheDocs and is published on PyPi! All your initial Todos can be moved to Done now on the Projects page:
[screenshot by Author]
You can continue developing your Python package from here onwards yourself. If you want to re-use this structure for another project in the future you can also clone this repository and follow its instructions in its README so that you don’t have to go through each step of this article again.
Feel free to contact me if you have any questions, or ask your question in the comment section!