From LSTM to PatchTST: Rethinking Time-Series Forecasting for Crypto
K1
Published on Sep 29, 2025
At 1DES, time-series forecasting is a critical component of how we build intelligence into our trading automation systems — especially for fast-moving markets like BTCUSDT. Our current models are primarily based on LSTM (Long Short-Term Memory) networks, and they’ve delivered stable results over time.
However, recent advancements in deep learning — particularly Transformer architectures — have opened the door to new possibilities. In this two-part series, we’ll explore what makes Transformers powerful for time-series tasks and why we’re planning to experiment with a specific variant: PatchTST.
This first part is all about theory: what Transformers are, how attention works, and why PatchTST stands out among the crowd. In the second part, we’ll build, benchmark, and compare PatchTST to our existing LSTM setup.
🔍 What Are Transformers?
Transformers are deep learning architectures originally developed for Natural Language Processing (NLP). Introduced in the 2017 paper “Attention Is All You Need”, Transformers replaced the need for recurrent neural networks (like LSTM) by introducing a new mechanism: self-attention.
Unlike RNNs that process data step-by-step, Transformers take in an entire sequence at once. Each time step in the input can attend (or “look at”) all others in the sequence. This massively parallel structure allows Transformers to:
- Model long-range dependencies
- Learn global structure in the data
- Train significantly faster than RNN-based models
Transformers have revolutionized NLP, powering models like BERT and GPT. More recently, the same ideas have been adapted for time-series forecasting.
🧠 What Is Attention (and Self-Attention)?
The core idea behind attention is contextual weighting. In self-attention, every point in the sequence decides how much attention to pay to every other point.
Example: If you're forecasting BTCUSDT and see a price spike, a self-attention model can learn that a similar spike a few hours ago may be relevant. It doesn’t have to wait step-by-step to reach that point — it connects the dots directly.
Attention helps the model:
- Recognize repeating patterns even when far apart
- Understand feature interactions (e.g., price + volume + indicators)
- Assign more weight to important historical windows
This mechanism is the backbone of all Transformer-based models.
⚙️ Types of Transformers for Time-Series
Vanilla Transformer
A direct adaptation from NLP. Useful as a baseline but inefficient for longer sequences.
Informer
Introduces ProbSparse attention, improving efficiency by focusing on the most relevant time steps.
Autoformer
Decomposes input into trend and seasonal components — useful for long-term, periodic data.
PatchTST
Inspired by Vision Transformers, it divides a time-series into patches (small blocks of time), processes them in parallel, and applies self-attention over patch representations.
🚀 Why We’re Trying PatchTST
PatchTST has shown state-of-the-art performance across several public time-series forecasting benchmarks. It offers:
- Great performance on multivariate data like OHLCV + indicators
- Fast training and inference due to patch-based representation
- The ability to detect both local structures (short-term signals) and global dependencies (macro trends)
Given its success in research and practical relevance to financial data, we’re going to put it to the test.
In the next post, we’ll implement PatchTST using TensorFlow/Keras, run benchmarks against our current LSTM-based models, and share performance insights.
Stay tuned!