Convert a large CSV file to JSON using Python

# Summary

The web content describes a Python script for converting a large CSV file into a keyed JSON file for easier data manipulation in various applications.

# Abstract

The article discusses a practical problem encountered by the author, who needed to transform a substantial CSV file obtained from the Google Ads API into a JSON format that is more versatile and readable. After online conversion tools failed due to the file's size, the author resorted to writing a Python script to accomplish the task. This script, `csv_to_json.py`, can be executed from the command line and is designed to handle large files, automatically creating the necessary directory structure for the output JSON file.

# Opinions

- The author expresses a preference for JSON over CSV for data storage and manipulation due to its readability and ease of use in various programming environments.
- The author believes that JSON files are more suitable for tasks such as storing data in a database column, importing into HTML documents, or integrating with front-end frameworks like VueJS and React.
- The author's initial attempt to use online CSV to JSON converters was unsuccessful due to the file size, leading to the conclusion that a custom script was necessary for handling large datasets.
- The author advocates for the "if the mountain won't come to Muhammad, then Muhammad must go to the mountain" approach, suggesting a hands-on solution when available tools are insufficient.

I’ve been playing around with Python and the Google Ads API for some keyword planning automation. I ended up with a large CSV file that I needed to convert to a keyed JSON file and wrote a quick python script to that I figured I’ll share with you.

Now, while I know it is very easy to use the CSV file in Python, I wanted to convert that file into a keyed JSON file instead.

Call me biased, but coming from the world of PHP and JavaScript, I personally think JSON files are easier on the eyes and can be utilized in other places where CSV would be to complicated to implement . Whether it is storing it into a database column (I do not recommend that), importing it into a HTML document, or even loading it into VueJS, React etc. You’ll get my point.

Anyways, I first tried a couple of those CSV to JSON converter sites that roam the internet. But quickly realized they all crash due to the file being to big. So next I figured, “if the mountain won’t come to Muhammad, then Muhammad must go to the mountain”. So I wrote a quick python script that you can use in the command line to convert a CSV file into a keyed JSON file.

import csv
import json
import sys
import os


def csv_to_json(csv_file, json_file):
    """
    Convert a CSV file to a JSON file
    """
    # Open the CSV
    with open(csv_file, 'r') as csv_file:
        reader = csv.DictReader(csv_file)
        data = list(reader)

    # Convert to JSON
    json_data = json.dumps(data, indent=4)

    # create parent directories if not exists
    if not os.path.exists(os.path.dirname(json_file)):
        os.makedirs(os.path.dirname(json_file))

    # Write JSON to a file
    with open(json_file, 'w') as json_file:
        json_file.write(json_data)


def main():
    files = sys.argv[1].split(' ')
    csv_to_json(files[0], files[1])


if __name__ == "__main__":
    main()

Convert a large CSV file to JSON using Python

How you use it