Python for Financial Analysis Series — Python Tools Day 5

It’s all about essential time series analysis in this session, from log returns, rolling based statistics to technical analysis. We will also learn how to use IEX (one of the most popular financial data cloud) to retrieve historical daily prices.

IMPORTANT UPDATE

Since Microsoft Azure Notebooks has moved to GitHub driven, which may be challenging for a beginner to start with. I have put all the material under Google Colab. It is a very similar Jupyter Notebook environment and you may check it out by click here.

All my code and files are shared and you may click here to open.

Data API (IEX Cloud)

Before we start, we need the time series data. One of the most common time-series data is historical prices. More precisely, it’s historical bars, i.e. open, high, low and close. In today’s session, we will use the historical daily bars of SPY (Biggest S&P 500 ETF) as an example. We will request the data dynamically using one of the popular financial data APIs, IEX Cloud.

IEX Cloud offers API to access a huge amount of financial data, from simple stock market data to corporate actions, fundamentals, company researches, FX, commodity, economics and crypto. It also provides premium data such as Machine Learning Stock Rankings.

Such a good product won’t be free, but for our learning purpose, we just need to create an account and skip any subscriptions for free testing data.

We can register for free testing data!

Let’s go to the register page and create a free account: https://iexcloud.io/cloud-login#/register/

When asked to select a plan, please skip (ignore). We don’t have to have any plan to test the API with sample data.

Once done the register, we can log in and go to the Console. First, we need to turn the “SANDBOX TESTING” on, then to go to the “API Tokens” tab to copy the “PUBLISHABLE” token.

That’s all set!

One small note about API vs SDK. In general, API (Application Programming Interface) is a kind of service to provide certain data or function. On the other hand, the SDK (Software Development Kit) is a language-specific package/library (on top of the API) to provide easy ways to code with specific languages. For example, we have IEX Cloud data API and it has Python SDK.

Download Data

Back to Microsoft Azure Notebooks, in our project, we need to install the IEX Cloud Python SDK for data download. To do so, we can use the command “!pip install”.

“pyEx” is the Python SDK for IEX Cloud APIs. Python won’t be useful without open source packages, for example, Pandas, which we have been using in the last four sessions. The reason why we could import Pandas is that Microsoft Azure Notebooks has already installed it. If the package is not installed as default, we need to use PIP to install it (PIP is a Python package management tool). We can use the command “pip install” with the package name (optionally with version) to install the package. In most managed Jupyter environments, such as Microsoft Azure Notebooks and Google Colab, we can use “!pip install”. In our example, we installed “pyEX”.

Next step let’s import the “pyEX”.

We then need to initialize the connection for data downloading.

Notice that you need your API token and the version is the sandbox. Once the connection is ready, we can download the data.

The function “chartDF” takes the ticker “SPY” with time frame “5y” and returns the DataFrame of the historical daily bars. If you would like to explore more functions, please check the document link below.

There are many columns we won’t use today, so let’s only take open, high, low and close.

We did the copy here because we will do many computations based on this sub DataFrame. We also sorted by the index (dates), as we need the time series in ascending order. The API may have done the job, but it’s good to make sure.

Period Percentage Change and Log Return

Pandas is a good tool as it provides us with all Excel can do (and more) in an easy way. In Excel, we could calculate row by row percentage change by referencing the ranges and doing the simple math, then fill down the formulas. In most programming languages, we have to do a loop. With Pandas, it is easier!

In our example, we took the column “close”, then apply the function “pct_change”, which is used to create a new column “pct_chg_daily”. Notice that the first value of the column is NaN as the function “pct_change” starts from the second row then automatically rolling down. This is also why we need the time series in ascending order. We could drop the NaN by using the function “dropna”.

In many financial analyses, we need log returns, i.e. we take the log of each number then calculate net change. To do so, we need another package Numpy, which provides mathematical and statistical functions. Let’s import the package and give it a short name “np”.

We now can use the function “np.log” to the column “close”, then apply the net change function “diff”.

Like the function “pct_change”, “diff” starts from the second row and calculate the net change on a rolling basis.

What if we want to calculate 5 days percentage changes? One of the easiest ways is to use the function “shift”.

The function “shift” will take the previous value down by the number of rows given as the input. Here we created one new column “close_5d” with the shifted values, for example, 216.62 is the close on 2015–06–22, which is moved down to align with the date 2015–06–26.

We then can do the simple math for the percentage change.

Rolling

Some time-series computation is based on a rolling window. For example, if we need a simple moving average, we need to take a window (a sub-series) to calculate the average, then roll it down with the fixed size of the window. We can easily achieve so with the function “rolling”.

Here we used the function “rolling” with the column “close”. We set the window size to be 5, so it will take the first 5 days to calculate the average with the function “mean”, then move down one day with another 5 days window. We then created a new column “SMA5”.

The function “rolling” works with the function “apply” too for more advanced computations. For example, what if we would like to do the linear regression on the rolling basis and get the slope of the regression line, i.e. rolling slope. We can use “rolling” with the function “apply”.

To use the function “apply” we need to define a custom function that we would use first. We created the function “regression”. Like other function for “apply”, it should take one input and return one value as output. Within the function, we used the “np.ployfit” function (remember np is the short name we gave to import NumPy), which does the regression. Three parameters needed: “x”, “y” and “degree” (with value 1, it does linear regression). Here the little trick we did is creating an integer series given the length of the “y” series (like 0, 1, 2, 3, 4…).

Correlation

Correlation is a common thing to look at when you work with time-series data. For example, the correlation matrix helps us to adjust holdings, as we don’t like concentration risk. The rolling correlation could be a good market indicator for trading. Let’s have a look at how we can do so with DataFrame.

To do so, let’s prepare another time series data for “IBM”. First, download the daily bars of “IBM”.

Second, we can join the column “close” from both DataFrames. Here we used the function “join”. It is like function “merge” (we learnt it in the last session) to combine two DataFrames. With “merge” we need to specify the matching column (the parameter “on”), but with “join” we match the index if we leave the parameter “on” as default.

We used the “inner” join (you may check the last session for detail). We need the dates to be aligned for both time series. We then applied the left and right suffix to avoid the conflict column name “close”.

Third, we compute the log return series based on the joined DataFrame and keep the log return columns only.

Finally, we can calculate the correlation matrix by just calling the function “corr”.

For singe correlation number we can do the following.

Take the column “ret_spy” and apply the function “corr” with another column “ret_ibm”.

If we would like to calculate the rolling 30 days correlation between “IBM” and “SPY”, we could use the function “rolling”.

Take the column “ret_spy” and apply the function “rolling” to create the rolling window. Then use the function “corr” with another column “ret_ibm”.

Technical Analysis

The final topic of this session is about Technical Analysis. This is one of the common analysis of trading. We could use Numpy and Pandas to create the study, but since we have the package already, why not use it directly. Let’s start with installing the package “ta” and import it, as it’s not included as default.

The exponential moving average is the base for many other indicators, so even we would like to create our studies, we can use the package “ta” and use those basic building components. Let’s do one example to calculate 20 days exponential moving average.

With the first line of code, we created the exponential moving average indicator, which is under the module “trend”. The package “ta” groups the studies into different categories. The indicator created (variable “ema20”) is an object, where we can call the function “ema_indicator” to create the time series and add one more column “ema20” into our original DataFrame.

Let’s have a look at another example, Ichimoku.

Again, the first line created the Ichimoku study object. We then used two functions “ichimoku_base_line” and “ichimoku_conversion_line” to get the base and conv columns into our DataFrame.

For more details, we can check the documentation.

Great job! We’ve done the essential time series based analysis with Pandas and Numpy. We also learnt a new package “ta” for technical analysis. Next session will be our last session with the series “Python Tools”. We will check out essential charting with Matplotlib.