Data Science with Python

3 Practical Differences Between astype() and to_datetime() in Pandas

The differences you need to know for effective data analytics

Choose the correct data type conversion method for time-efficient data analysis!

In my last two articles, you can explore the tricks and tips for working with date-time or time-series data in Python and Pandas.

When working with time-series data in Pandas, you can use either pandas.Series.astype() or pandas.to_datetime() to convert date-time strings to datetime64[ns] data type. Both these methods return exactly the same output.

However, there is a significant difference in their performance, flexibility, and the way they handle errors. And choosing the correct method for data type conversion will be easier when you understand these differences.

In this article, you’ll learn about these 3 practical differences between pandas.Series.astype() and to_datetime() methods. Here is a quick overview of the topics you can explore in this article —

· Performance Differences Between astype() and to_datetime() · Handling of Dates and Time · Error Handling

Let’s get started!

The comparison of two methods or functions in the programming is incomplete without comparing their efficiency. And one of the best methods to compare efficiency is in terms of time.

Performance Differences Between astype() and to_datetime()

The performance of the method helps you understand how efficiently and quickly that method works, i.e. in this case converts the data type to datetime64[ns].

It can be a critical aspect when you are working on an analytics project and processing a really huge amount of data.

One of the simplest ways to measure performance is execution time. The method that takes the lowest time to execute will be certainly time-efficient, and you can say it performs better than others.

Let’s use the same example as my previous article — Read the dummy dates sales data in a DataFrame df!

df = pd.read_csv("Dummy_dates_sales.csv")
df.head()

It is a simple, 100 x 2 dataset containing the date-time column Dates and the integer column Sales. I created this dataset using the Python package Faker and you can take it from my GitHub repo for Free!

As the pandas has detected values in the Dates column as strings — you need to convert them into the date-time data type.

You can use the magic function — %%timeit — in Jupyter-Notebook as shown below, which gives you the execution time for the code mentioned in the cell.

As you can see in the above picture, %%timeit runs the code 1000 times and takes the average of the execution time for each of the methods astype() and to_datetime().

Clearly,

pandas.Series.astype() is 1.87x faster than to_datetime().

Moreover, Series.astype() can change the data type of multiple columns in one go, which is not the case in pandas.to_datetime(). And that’s where you come across the next key difference.

Handling of Dates and Time

In contrast to the pandas.to_datetime(), pandas.Series.astype() is not intended to convert the data type of date-time values.

So if you use it for converting data type into datetime64[ns] it will simply try to convert the date-time string as it is into a date-time value.

Let’s take the following example — where dates in the DataFrame are present in YYYY-MM-DD format.

df = pd.DataFrame({"Dates": ["2022-12-25", "2021-12-01", "2022-08-30"]})

To convert values from the Dates column to the datetime64[ns] data type, you can use both methods astype() and to_datetime() as shown below.

df["NewDate_using_astype()"] = df["Dates"].astype("datetime64[ns]")
df["NewDate_using_to_datetime()"] = pd.to_datetime(df["Dates"])

df.info()
df.head()

Both methods worked well.

However, now if you change the format of the values in the Dates column and make it in YYYY-DD-MM format, the method pandas.Series.astype() will throw a ValueError.

df = pd.DataFrame({"Dates": ["2022-25-12", "2021-01-12", "2022-30-08"]})
df["NewDate_using_astype()"] = df["Dates"].astype("datetime64[ns]")

As the method, .astype() converts the date-time string as it is into a date-time value. It expects the input date string in YYYY-MM-DD format. That’s why the value error shows ‘month must be in 1..12’.

Whereas, when you use to_datetime() to convert such date-string to the data type datetime64[ns], you can use its optional parameter — format. It helps you to specify in which date format an input date string is present.

df = pd.DataFrame({"Dates": ["2022-25-12", "2021-01-12", "2022-30-08"]})
df["NewDate_using_to_datetime()"] = pd.to_datetime(df["Dates"],
                                                   format='%Y-%d-%m')
df.info()
df.head()

As seen in the above output, pandas.to_datetime() works perfectly and converts the date string into the date-time value with no error.

Well, data transformation is not only about performance and flexibility, but it also includes error handling. And these two methods handle the errors differently.

Error Handling

Although the method pandas.to_datetime() is slower than .astype(), it is better for handling errors in data type conversion.

Sometimes the input data or the time-series data which you want to convert into date-time format contains some unwanted characters. It can be anything like missing a month, day, or year or some random character at the month’s place, as shown in the below example.

df1 = pd.DataFrame({"Dates": ["2022-12-25", "2021-12-20", "2022-12-b", "2023-07-15", "2020- -31"]})
df1.info()
df1.head()

It is a simple DataFrame with 5 values in the Dates column, out of which two date values are invalid. The dates-strings in the Dates column are in YYYY-MM-DD format.

In one of the invalid values, a character ‘b’ is present instead of the day number, and in another invalid date string, the month number is missing.

Now, let’s see how pandas.Series.astype() and pandas.to_datetime() deals with these invalid date strings.

# Using pandas.Series.as_type()
df1["Dates"] = df1["Dates"].astype("datetime64[ns]")
df1.info()

## Output
TypeError: Unrecognized value type: <class 'str'>
ParserError: Unknown string format: 2022-12-b

# Using pandas.to_datetime()
df1["Dates"] = pd.to_datetime(df1["Dates"])
df1.info()

## Output
TypeError: Unrecognized value type: <class 'str'>
ParserError: Unknown string format: 2022-12-b

Both methods, by default, raise the TypeError and ParserError.

One way to avoid these errors is to use the errors parameter available in both methods. You can assign — ignore — to this parameter to ignore these errors and get the original date string in the output, as shown here —

df1["Dates-astype-ignore"] = df1["Dates"].astype("datetime64[ns]", errors='ignore')
df1["Dates-to_datetime-ignore"] = pd.to_datetime(df1["Dates"], errors='ignore')

df1.info()
df1.head()

Ignoring the errors returns the input as it is, and so the output data type remains unchanged, i.e. string/object.

So this won’t help you understand exactly at which places you have invalid date strings. And that’s where another optional parameter of to_datetime() is useful.

While using pandas.to_datetime() you can assign — coerce — to the errors parameter. As a result, pandas will convert all the valid date strings to datetime64[ns] format, and all the invalid date strings will be set to NaT .

NaT stands for Not-a-Time (similar to NaN: Not-a-Number) which simply says the pandas can’t store the value as a date-time value.

df1["Dates-to_datetime-coerce"] = pd.to_datetime(df1["Dates"], errors='coerce')
df1.info()
df1.head()

This solves your two problems —

1. The data type of the valid date string is changed to datetime64[ns]
2. It helps you to locate the invalid date strings.

You can get the location of the invalid date strings simply using df1[“Dates-to_datetime-coerce”].isna()

Altogether,

I hope you found this article useful and learned the differences between pandas.Series.astype() and pandas.to_datetime() for changing the data type of time-series data.

Although the method astype() can convert the data type of multiple columns in one go, it is always better to use the method to_datetime() to convert the data type of time-series data.

Knowing the correct method to convert data types can save you time while working with time series data in Pandas.

Interested in reading more stories on Medium??

💡 Consider Becoming a Medium Member to access unlimited stories on Medium and daily interesting Medium digest. I will get a small portion of your fee and No additional cost to you.

💡 Be sure to Sign-up to my Email list to never miss another article on data science guides, tricks, and tips, SQL, and Python.

Thank you for reading!

Here are a few more resources to help you work efficiently with such date-time data in Python and Pandas.