Building a bitcoin dataset

Summary

The website content details the process of creating a comprehensive Bitcoin dataset for analysis, using Python and various data sources.

Abstract

The author of the content embarked on a journey to build a reliable Bitcoin dataset after failing to find existing datasets that met their needs. Utilizing Python with PyCharm, pandas, and numpy, they structured the dataset to facilitate analysis. Data was sourced from Blockchair for Bitcoin blockchain information and from BraveNewCoin's liquid index via TradingView for historical price data, with a workaround found on Reddit to export TradingView data into CSV format. The dataset was refined by merging daily price data with blockchain data, resampling blockchain data to match the daily timeframe, and ensuring data quality using Pandas Profiling for statistical summaries. The author acknowledges the challenges faced, such as preserving column headers during data resampling, and emphasizes the importance of starting with clean, complete, and reliable data for analysis and model building.

Opinions

The author believes in the necessity of having a custom dataset tailored to specific analysis needs.
Python is favored for its ability to handle large datasets beyond the capacity of spreadsheet software like Excel.
Blockchair is highlighted as a valuable resource for accessing Bitcoin blockchain data.
The author appreciates the community's contributions, such as the Reddit user's web application for exporting TradingView data to CSV.
Data integrity and structure are considered crucial for reliable analysis and subsequent model building.
The author finds Pandas Profiling to be a convenient tool for quickly assessing the quality and characteristics of a dataset.
Encountered programming challenges are seen as obstacles to be solved, contributing to the overall learning process and dataset quality.

Building a bitcoin dataset

Good analysis starts with clean complete and good data. I have been looking for good datasets on bitcoin for a while, but couldn’t find anything that matched with what I was looking for. I decided to create my own dataset that I can use for different purposes. I wanted a dataset with reliable data, so my journey started with finding the right sources and instruments.

Build in Python

When datasets start to exceed the dimensions of google sheets or Excel, you’d better start looking for different solutions. I decided to use Python. More specifically, I’m working with PyCharm to write and evaluate the code and using the pandas and numpy package that provide appropriate tooling to first build the dataset and later analyse it. An easy way to build datasets is by using the DataFrame data structure functionality and read the CSV-files into a dataframe. Now that we have found a good way to get all the data in one place where we can easily analyse, it’s time to actually collect the data.

Getting all bitcoin blockchain data

A great source to get all the bitcoin blockchain data you’re looking for is www.blockchair.com. In the bitcoin/blocks section of the site you can select what data you’d like to export to a csv file and download it from there.

Blockchair.com: a great place to get your bitcoin blockchain data in csv format

Collecting bitcoin price data

Finding reliable bitcoin price data that goes back to the early days of bitcoin is a challenge. A good source for the data is the BraveNewCoin liquid index from TradingView. Unfortunately, TradingView doesn’t offer CSV export functionality, but I figured there must be more people around dealing with that same wish. (The wish to convert TradingView data to a downloadable CSV file). I stumbled on a reddit post from someone who made a web based application to generate a csv file from a privately published TradingView chart. Great! Now I was able to get all historic bitcoin price data as of October 2010 from a reliable source.

Shoutout to the hero who made this webapp

Blending all the data together

The collected price data is on a daily timeframe, while the bitcoin blockchain data is based on blockheight. In order to merge the data, I had to resample the bitcoin blockchain data to daily data, where I used the max for the blockheight and the average difficulty for all blocks on a given day. After blending all the data together, you want to know if it all looks good enough to proceed with further analysis. The Pandas Profiling is a very convenient package to do exactly that. It’s comparable to the df.describe functionality of pandas itself, but a profile reports offers just a little more. It let’s you for instance generate an html output file, which enables you to review all your data with the greatest ease by showing you a summary of the statistics on your dataset.

Summary statistics on the dataset provided by the ProfileReport package

In the blink of an eye, you can check the quality of your data and correlation values in the set.

Encountered challenges

Every programming or scripting language has its own challenges in terms of syntax. One issue I encountered was that the headers of my columns disappeared after resampling the data. From the dataframe where I stored timestamps and blockheight (df_tsbh), I had to go to a version based on daily data while preserving all the column headers of the dataframe. Here’s how I solved the issue.

#Blockheight by day
func = {
    'blockheight': df_tsbh['blockheight'].resample('1d').max()
}
df_daily = pd.DataFrame(func)

Building a bitcoin dataset

Build in Python

Getting all bitcoin blockchain data

Collecting bitcoin price data

Blending all the data together

Encountered challenges

Final Remark

References