Time Series Feature Selection: Which Methods to Apply?

Time series data has a natural temporal ordering, which makes feature selection for time series forecasting uniquely challenging compared to other types of data.

Feature selection for time series forecasting often requires additional steps like detrending, deseasonalizing, and ensuring that all features are relevant for the predictive modeling task considering the time dimension. Here’s a deeper dive into the concepts you mentioned:

Detrending

Detrending is the process of removing long-term trends from the data. In time series, trends can be caused by various factors like technology growth, inflation, or increasing population. These trends can overshadow the seasonal patterns and noise that might be more predictive of future observations. Detrending makes the series stationary, which is a common assumption in many time series forecasting models.

Deseasonalizing

Deseasonalizing involves removing the effects of seasonal factors that influence a time series. A “season” can be anything from a day-of-week effect in web traffic data to annual sales cycles in retail. By removing these effects, models can focus on the underlying patterns. There are various methods to deseasonalize data, including differencing at seasonal intervals or using models like Seasonal Decomposition of Time Series (STL) that explicitly model and remove seasonal effects.

Relevance Considering Time Dimension

Ensuring that all features are relevant considering the time dimension means acknowledging that the importance of features can change over time. For instance, the price of a commodity might be a strong predictor for a company’s stock price, but this relationship could weaken over time due to market changes. When selecting features, one must ensure they continue to have a predictive relationship with the target variable at the forecast horizon.

To address the relevance of features in a time series context, you can use techniques like:

• Windowing: Transforming the time series into a supervised learning problem by using a rolling window of lagged values as input features. This technique captures the temporal dependencies in the data.
• Time-based Feature Engineering: Creating features that capture the time-based characteristics such as hour of day, weekday vs. weekend, holiday effects, etc. These features can help a model account for periodic changes in patterns.
• Dynamic Feature Selection: Periodically reevaluating and selecting features as new data becomes available. This process can identify changes in the predictive power of features over time.

Stationarity

Many feature selection techniques assume that the relationships between variables are consistent over time. However, time series data often violates this assumption due to trends, cycles, and other forms of non-stationarity. Therefore, tests for stationarity (like the Augmented Dickey-Fuller test) can be used before feature selection to ensure that the data does not have unit roots and the relationships between variables are stable.

Cointegration

When dealing with multiple time series, if non-stationary series are combined to predict another, it’s crucial to check if these series have a long-term equilibrium relationship, which is known as cointegration. If such a relationship exists, error correction models (ECMs) can be used to include this information in the model, providing a way to use non-stationary predictors without differencing them.

By applying these steps, you create a foundation for more accurate and reliable feature selection in time series analysis. It allows models to capture the true underlying patterns in the data without being misled by artificial effects or changing relationships.

Here are several techniques commonly used for feature selection

Filter Methods

• Correlation Coefficient: Using Pearson, Spearman, or Kendall correlation to measure the linear relationship between each feature and the target variable.
• Variance Threshold: Removing features whose variance does not meet some threshold.
• Chi-squared Test: Applying statistical tests to select features that have the strongest relationship with the output variable.

Wrapper Methods

• Recursive Feature Elimination (RFE): Recursively removing attributes and building a model on those attributes that remain.
• Sequential Feature Selection Algorithms: These include forward selection, backward elimination, and stepwise selection, where features are added or removed one by one based on model performance.

Embedded Methods

• Lasso Regression (L1 Regularization): Penalizes the absolute size of coefficients, effectively reducing some coefficients to zero, thus performing feature selection.
• Ridge Regression (L2 Regularization): Penalizes the square size of coefficients but does not necessarily eliminate them, often used in combination with Lasso.
• Elastic Net: Combines penalties of Lasso and Ridge.

Dimensionality Reduction Techniques

• Principal Component Analysis (PCA): Projects the data into a lower-dimensional space with the aim of preserving variance.
• Linear Discriminant Analysis (LDA): Used as a dimensionality reduction technique in the pre-processing step for pattern-classification and machine learning applications.

Tree-based Methods

• Random Forests: Can be used to rank features by importance based on how much they improve the purity of the node.
• Gradient Boosting Machines (GBM): Similar to Random Forest, GBM can also provide a feature importance score.

Information Gain and Mutual Information

• Mutual Information: Measures the amount of information one can obtain from one random variable given another.

Domain Knowledge-Based Feature Selection

• Sometimes, features are selected based on domain knowledge and the specific context of the problem.

Time Series Specific Methods

• Autocorrelation and Partial Autocorrelation: Identifying the lagged variables that have significant correlation with the current observation.
• Cointegration and Error Correction Models (ECM): In the context of non-stationary time series that are cointegrated, ECM can be used to select features.

Feature Importance from Model

• Models like XGBoost and LightGBM provide built-in methods to assess feature importance.

Which methods to apply?

Determining which methods to apply for feature selection in time series forecasting involves several considerations:

1. Stationarity and Time Series Specific Preprocessing: Before applying filter, wrapper, or embedded methods, ensure your data is prepared for analysis. This means checking for and ensuring stationarity, detrending, deseasonalizing, and performing any necessary transformations. This step is crucial because the subsequent feature selection methods rely on the assumption that the data does not have changing statistical properties over time.
2. Filter Methods: As a first step, filter methods are typically quick and do not involve building models. These methods can help you to quickly eliminate features that are unlikely to be informative.
3. Wrapper Methods: These are more computationally intensive, as they involve creating models. However, they can be more effective in finding the best subset of features since they take into account feature interactions.
4. Embedded Methods: These methods perform feature selection as part of the model training process and can be more efficient than wrapper methods. For example, Lasso can be particularly useful when you have many features because it can shrink the less important ones to zero.
5. Tree-based Methods: These can be used as either a primary feature selection method or to validate the selections made by previous methods. They are particularly useful because they naturally capture nonlinear relationships and interactions between features.
6. Cointegration: This is a step for multivariate time series where the focus is on the long-term equilibrium relationships between features, which is crucial when using integrated series.
7. Model-Based Feature Importance: Once a model is built, whether it’s a simple linear model or a complex machine learning model, you can use it to assess feature importance. This can give you insights into which features are most predictive.
8. Dynamic Feature Selection: As new data comes in, features that were once important may lose their predictive power, and vice versa. Therefore, feature selection can be an ongoing process.
9. Domain Knowledge: At any stage, domain knowledge can guide the feature selection process. For instance, it might suggest the inclusion or exclusion of certain features or the transformation of variables.
10. Iterative Process: The nature of feature selection is such that you may need to go back to earlier steps as you learn more about the model’s performance or as new data becomes available.

Feature selection is both an art and a science. It’s about balancing the insights gained from statistical methods with computational efficiency and domain expertise. The sequence and methods you choose should be guided by the specific context of your problem, the nature of your data, and the goals of your analysis. It’s common to try multiple approaches and compare the results to determine which method is most effective for your particular situation.

Please leave comments specifying the aspects or topics you would like me to explore in future articles.