Reading and Writing Files in Python using Pandas

Reading and Writing Files in Python using Pandas

When working with labeled and time series data in Python, the pandas library provides a powerful and flexible way to handle the data. In addition to offering statistical methods and data visualization tools, pandas also supports reading and writing data from various file formats such as Excel and CSV.

In this tutorial, you will learn how to read and write files using pandas, along with working with different file types and efficiently handling big data.

Reading and Writing CSV Files with Pandas

To read a CSV file using pandas, you can use the read_csv() method.

import pandas as pd

# Read a CSV file
df = pd.read_csv('file.csv')

To write data to a CSV file, you can use the to_csv() method.

# Write to a CSV file
df.to_csv('new_file.csv', index=False)

Reading and Writing Excel Files with Pandas

You can also read and write Excel files using pandas.

# Read an Excel file
df = pd.read_excel('file.xlsx')

# Write to an Excel file
df.to_excel('new_file.xlsx', index=False)

Working With Different File Types

Pandas supports various file formats including JSON, HTML, and SQL. Here’s how you can work with these file types:

Working With JSON Files

# Read a JSON file
df = pd.read_json('file.json')

# Write to a JSON file
df.to_json('new_file.json')

Working With HTML Files

# Read an HTML file
dfs = pd.read_html('file.html')

# Write to an HTML file
df.to_html('new_file.html')

Working With SQL

import sqlite3

# Read from a SQL database
conn = sqlite3.connect('file.db')
query = "SELECT * FROM table"
df = pd.read_sql(query, conn)

# Write to a SQL database
df.to_sql('new_table', conn, index=False)

Working With Big Data

When working with large datasets, pandas provides methods to efficiently handle big data. For example, you can use the chunksize parameter to process data in chunks.

# Read a large CSV file in chunks
chunk_size = 1000
for chunk in pd.read_csv('big_file.csv', chunksize=chunk_size):
    process_data(chunk)

By leveraging the pandas library, you can effectively read and write data from various file formats and efficiently handle big datasets in Python.

In conclusion, pandas provides a comprehensive set of tools for reading and writing files, making it a versatile and powerful library for data manipulation and analysis in Python.