This context provides practical tips to organize and tidy up a Data Science workflow in Jupyter Notebook, including using separate notebooks for each major task, using markups for expressive documentation, using pickles, creating user-defined functions, and ensuring the notebook is run top-down.
Abstract
The context discusses the importance of keeping a Data Science workflow clean, organized, and easy-to-understand in Jupyter Notebook. It provides five practical tips to achieve this, starting with using separate notebooks for each major task to make the overall workflow easier to see and find things. The second tip is to use markups for expressive documentation, allowing for more expressive documentation compared to code comments. The third tip is to use pickles, which allow saving objects as .pkl files, such as trained models or machine learning pipelines. The fourth tip is to create user-defined functions and save them in a module, which can reduce copy-pasting and make the notebook less cluttered. The final tip is to ensure the notebook is run top-down, which can prevent out-of-order cells.
Bullet points
Use separate notebooks for each major task
Use markups for expressive documentation
Use pickles to save objects as .pkl files
Create user-defined functions and save them in a module
Ensure the notebook is run top-down
Organise your Jupyter Notebook with these tips
Practical tips to tidy up your Data Science workflow
Data scientists do lots of exploration and experimentation. Jupyter Notebook (Notebook from here onwards) is a great tool for exploring and experimenting. However, things can get cluttered and messy quickly when using Notebook. Keeping your workflow clean, organised and easy-to-understand is an important skill and will serve you well in your professional career. In this post, I will share a few of my tips on how to keep your Notebook and workflow more organised.
📍 Tip 1. Use separate Notebooks for each major task
Instead of putting your workflow from exploratory data analysis to modelling in one giant Notebook, it’s good to separate them out into a few logical pieces and name each Notebook descriptively. Let’s look at an example:
If we look at this example, we have a couple of Notebooks. From the name, we can see the sequence of Notebooks. For instance, we can guess that 3_base_models.ipynb precedes 4_gbm_model.ipynb while 4_gbm_model.ipynb and 4_rfc_model.ipynb are probably exploring two options in parallel. If we renamed these two to 4A_gbm_model.ipynb and 4B_rfc_model.ipynb, then we can hint an order in the name in a slightly different way as well.
You can have more or less number of Notebooks than the example. That is, the exact number of Notebook is not important. The key idea here is to have logical major tasks (it’s up to you how you define this) spread out, so it’s easier to see the overall workflow and find things. For instance, when writing production-ready code for deployment, we just have to refer to 5_final_pipeline.ipynb.
📍 Tip 2. Use markups for expressive documentation
Markdown cells in Notebook allow us to be more expressive in how we document our workflow compared to code comments. We could use headers of different font size, ordered or unordered list of items, tables, mathematical formulas, checkboxes, hyperlinks, images and many more with Markdown. Here’s a one-minute introduction to Markdown on its commonly used syntax:
You can learn more about Markdown from here and here.
📍 Tip 3. Use pickles
Pickles are not only delicious, but when used in the context of Python, they allow us to save objects as .pkl files. The process of saving the object is referred to as pickling.
One popular use of pickling in Data Science is saving trained models or machine learning pipelines into .pkl files. In the following example, we can see that there are couple of models saved in pickle:
This, for instance, will allow us to load the trained models or pipelines in different Notebooks for extended analysis of the predictions.
Another good use for pickles is to save objects like lists, dictionaries and the like. For instance, if we assigned manual selection of categorical features to a list in one Notebook and plan to also use it in another Notebook, we can write the list to a pickle file and load it in other Notebooks. This way if we manually make changes to the list, then we only have to do it in one place. Here’s an example code to write or load a pickle file named categorical.pkl where categorical refers to an object in Python (e.g. list).
📍 Tip 4. Create user-defined functions and save it in a module
You may have heard of the DRY principle: Don’t Repeat Yourself. If you haven’t heard of this software engineering principle before, it is about “not duplicating a piece of knowledge within a system”. One of my interpretation of this principle in Data Science is to create functions to abstract away the reoccurring tasks to reduce copy pasting. You can even use classes if it makes sense in your case.
Here are the suggested steps for this tip:
Create a function
Ensure the function has an intuitive name
Document the function with docstring
(Ideally) Unit test the function
Save the function in a .py file (.py file is referred as module)
Import module in Notebook to access the function
Use the function in Notebook
Let’s try to contextualise these steps with examples:
Here’s a simple way to assess if a function has an intuitive name: If you think a colleague who hasn’t seen the function before could roughly guess what the function does just by looking at its name, then you are on the right track. When documenting these functions, I have adapted a few different styles in a way it made more sense to me. While these examples serve as a working example function for Data Science, I highly encourage you to check out official guides such as below to learn the best practices in naming and documentation conventions, style guides and type hints:
You can even browse through modules in well-established package’s Github repository to get inspirations.
If you saved these functions in a helpers.py file and imported helpers module (by the way, Python module just means a .py file) in our Notebook with import helpers, you can access the documentation by writing the function name followed by Shift + Tab:
If you have many functions, you could even categorise and put them in separate modules. If you take this approach, you may even want to create a folder containing all the modules. While putting stable code into a module makes sense, I think it is fine to keep experimental functions in your Notebook.
If you implement this tip, you will soon notice that your Notebook start to look less cluttered and more organised. In addition, using functions will make you less prone to silly copy paste mistakes.
Unit testing was not covered in this post as it deserves its own section. If you would like learn about unit testing for Data Science, this PyData talk may be a good starting point.
📍 Tip 5. Make sure Notebook is run top-down
It’s not uncommon to be tweaking parts of your code anywhere in your Notebook when experimenting. However, once you are done experimenting, clean it up, restart the kernel and ensure cells in your Notebook ran top-down once such that you won’t have out of order cells like this:
Voila❕ These were my tips for organising your Jupyter Notebook workflow.✨
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. I hope you will get to use these tips to have a more organised workflow. If you are interested, here are links to some of my posts: