A Complete Introduction To Time Series Analysis (with R):: ARMA processes (Part I)
Perhaps one of the most famous and best-studied approaches to working with time series, still widely used today is the ARMA(p,q) models and its derivatives. As you can guess, these essentially introduce a generalization of the AR(1) and MA(1) processes that we have previously seen. Before we start, let’s introduce some useful operators that will allow us to simplify our notation.
Autoregressive and Moving-average Operators
Simply put, these operators are no more than polynomials defined in the way above. Now, we are fully equipped to define the ARMA process.
ARMA(p,q) processes
A stationary process is said to be ARMA(p,q), denoted {X_t} ~ ARMA(p,q), if it satisfies, for all t :
where B is the backshift operator, and Phi and Theta are the operators we defined above.
Clearly, we can also write the ARMA(p,q) process as
.In this form, we can see that our process is modeled as having dependency not only q steps on past noise, but also on p observations in the past. We will switch notations and use the operators expressions when convenient. You can verify that both expressions are indeed equivalent!
AR(p) and MA(q) processes
Now that we understand the shape of an ARMA(p,q) process, we can immediately see two trivial cases:
In other words,
- The AR(p) process means that, in addition to X_{t} , being explained by some random noise, all the covariance/correlation in the process can be explained by the p previous lags.
- The MA(p) process is saying that X_{t} does not only depend on the current noise, but also on past noise.
- The ARMA(p,q) model implies that even if we adjust for autocorrelation on previous observations, there is still a dependence on past noise.
Shifted ARMA processes
A slightly more general definition of the ARMA(p,q) process goes like this:
In other words, we can simply substract the mean to obtain an actual ARMA process.
Causality, Invertibility and Stationarity of ARMA(1,1) process
So far, we have implied that the ARMA(p,q) process is stationary. However, when is this actually true? In other words, what conditions are required to ensure stationarity of an ARMA process? Further, how can we know whether our process only depends on observations from the past (that is, it is a causal process), and that we can actually invert such process (solve in terms of Z_{t})? It turns out that we have to restrict the values of the coefficients of the polynomials that define the ARMA process to ensure this. To illustrate, we will first explore the causality, invertibility and stationarity of the ARMA(1,1).
Causal Processes
A time series X_{t} is said to be causal if it only depends on the past, that is if {X_{t}} is a linear process of the form
where
Indeed, a causal process is a linear process whose coefficients do not take on negative indices. Now, let’s take a look at an example.
Causality of the ARMA(1,1) process
We consider the ARMA(1,1) process, defined as
which implies that
Then, for what values of phi and theta will the process be causal and stationary?
That is,
Now, notice that
, which implies that
i.e., it looks as if the operator Chi taht we defined is the (left) inverse operator of Phi! Armed with this fact, we have that
i.e.
satisfying
and so
which happens to be a sort of MA(infinity) process, which only depends on the past! Notice that in order for this process to be stationary, we need the series to be summable, therefore we can see that the process is causal and stationary if
Non-causal ARIMA(1,1) process
A natural question would be: there exists a process that is not causal, but stationary? The funny thing is, such processes are actually possible, but I doubt they are vert useful. We will see what could go wrong. Once again, consider the ARIMA(1,1) process:
with
. What would happen if we placed the restriction
? How can we define the Chi operator in this case? Consider:
So we can define in this case
One can show as before that
holds, and so we have that
Once again, by applying the definitions and simplifying the expressions, it can be shown that this is equivalent (you should give it a try) to
, and so our observation X_{t} depends on MA noise of the future!!! (awkward…). If you don’t feel comfortable with this, don’t worry, I don’t feel either. It turns out that this process is stationary, but is not causal!
Remark
If the absolute value of phi is equal to one, the series obviously diverges, so the process is not even stationary. In short, we have
Invertibility of the ARMA(1,1) process
Let’s now investigate what makes a process invertible, illutrating with the ARMA(1,1). Once again, we have
with
. We will now investigate which values of theta will make our process invertible.
that is,
Note that this expression is summable iff
Then, we have that
That is, we have correctly defined an inverse for the Theta operator. With this knowledge, we then must have
where
that is,
Using similar methods as before, we can show that when the absolute value of theta is less than 1, Z_{t}can be expressed as
Therefore, the ARMA(1,1) is invertible whenever the expression is summable and we can express Z_{t} as a function of the X_{t}’s. If , on the contrary,
the ARMA(1,1) process is said to be non-invertible, as once again the expression would result to be
which implies that the noise at time t depends on observations from the future???? (more than awkward…).
Next time
And that’s it for now!! In the next article, we will be generalizing these ideas to the more general ARMA(p,q) process. For this purpose, we will make use of complex numbers, for which you can find a review here. Until next time!
