Building the Ultimate Trading Data Pipeline — Bonus: Automate the computation of max and min…

Summary

The article discusses the creation of an efficient trading data pipeline that includes the automation of computing minimum and maximum historical trading values using database views.

Abstract

The article "Building the Ultimate Trading Data Pipeline" provides insight into the process of enhancing trading strategies by analyzing historical price data. It emphasizes the importance of identifying key levels such as support and resistance through historical minimum and maximum values. The author explains the benefits of different data storage methods and introduces a solution to streamline data management by using database views. These views, min_historical_view and max_historical_view, are designed to dynamically calculate and store the minimum and maximum historical closing values for each trading symbol, respectively. The article also includes a note on the educational purpose of the content, clarifying that it does not serve as personal investment advice.

Opinions

The author suggests that the choice of financial data storage is critical and should be based on factors such as the nature of the data, intended use cases, frequency of access, and specific requirements.
The use of database views for minimum and maximum historical values is presented as an efficient method to manage large datasets without the need to regenerate various indicators for the entire historical dataset.
The article implies that leveraging database views can significantly improve the performance of trading data analysis by providing quick access to essential historical values.
The author indicates that table views are a powerful feature in databases, as they are virtual tables that are automatically updated based on the underlying data.
Acknowledging the educational intent, the author stresses that the article is meant for informational or instructional purposes and not as a source of personal investment advice.

Building the Ultimate Trading Data Pipeline — Bonus: Automate the computation of max and min historical values

Trading strategies often rely on analyzing historical price data to identify patterns, trends, and key levels, including minimum or maximum historical values.

For example, historical minimum values indicate support, and maximum values signify resistance levels for traders.

The generation and storage of our finantial data in a database or other formats, such as Parquet files, depend on various factors including the nature of the data, the intended use cases, frequency of access, and specific requirements. Different formats and storage methods offer distinct advantages, and choosing the right one is crucial for efficient data management.

Given the scale of our datasets and the desire to circumvent the generation of various indicators for the entire historical dataset, we can leverage the need into two database views: one dedicated to minimum historical values and another to maximum historical values:

CREATE VIEW min_historical_view
AS
SELECT
  date,
  symbol,
  close,
  MIN(close) OVER (PARTITION BY symbol ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS min_historical_close
FROM
  daily_bars;


CREATE VIEW max_historical_view
AS
SELECT
  date,
  symbol,
  close,
  MAX(close) OVER (PARTITION BY symbol ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS max_historical_close
FROM
  daily_bars;

Remember that a table view is a virtual table in a database that is automatically calculated based on the data present in other tables.

We can now run queries to observe, for instance, the maximum closing value up to the date of each respective row.

The article, it’s concise and straightforward, but provides a glimpse into the considerations surrounding the selection of techniques for persisting data intended for future analysis. We’ll see how in an upcoming series of articles.

👏 Did you like the story? Give 1 to 50 claps to show your support! Your claps really helps me out and motivates me to keep creating valuable content. Thank you for your support! 👏

Thank you for being part of our community! Before you go:

If you liked the story feel free to clap 👏 and follow the author.

Learn How To Develop Your Trading Bots 👉 here.

Join our Premium Discord Server👉 here.

*Note that this article does not provide personal investment advice and I am not a qualified licensed investment advisor. All information found here is for entertainment or educational purposes only and should not be construed as personal investment advice.