Demand Forecasting with Deep Learning: Does an LSTM Earn Its Complexity?用深度學習做需求預測：LSTM 值得它的複雜度嗎？

This article continues the case study introduced in:

• Forecasting iPhone 17 Pro Demand: From Statistics to Deep Learning,
• Demand Forecasting with Statistical Models: How Far Can Demand History Alone Take Us?,
• Demand Forecasting with Supervised Learning: Does More Context Help?.

Statistical models provided a strong baseline for one store–SKU, while a global XGBoost scaled forecasting to 120 series but gained little from additional business context. We now test whether a global LSTM can learn directly from ordered demand sequences and improve enough to justify its complexity.

This article is structured as follows:

1. Why move beyond engineered features?
2. From engineered rows to ordered sequences
3. How an LSTM creates memory
4. Architecture Design
   • Building leakage-safe sequences
   • LSTM layer setup
   • Learned identity embeddings
   • Scaling without future information
5. Training choices
6. Does the improvement justify the complexity?
   • Model comparison
   • Model selection
7. Operational recommendation
8. Conclusion: complexity must earn its cost

Why move beyond engineered features?

Tabular supervised-learning models such as XGBoost do not receive demand history as an ordered sequence. We must first convert that history into features such as lag_7, lag_28, rolling_mean_7, and rolling_std_28. These features give the model a form of memory, but they also determine what information that memory contains.

This creates several limitations:

• The temporal representation is predefined. A seven-day lag tells the model to examine the same weekday last week because we decided that relationship was important.
• Historical detail is compressed. A 28-day mean summarises an entire month with one number, potentially hiding the order and shape of changes within that period.
• Longer relationships require more engineering. Extending memory means creating, validating, and maintaining additional lag, rolling, trend, and interaction features.
• A fixed feature set may not capture every series equally well. One product may respond to the previous weekend, while another may depend on a gradual multi-week change that the selected summaries do not preserve.
• More temporal features increase maintenance. Every additional lag, rolling statistic, and interaction must be defined, validated, and reproduced consistently during training and forecasting.

These limitations motivate a different question:

Can a global sequence model learn a shared, nonlinear representation of demand history without relying on predefined statistical structures or handcrafted temporal summaries?

From engineered rows to ordered sequences

A Long Short-Term Memory network, or LSTM, reads the previous demand observations in order. As it moves through the sequence, it updates an internal memory that can retain useful patterns and discard less relevant information. The resulting hidden state becomes a learned representation of recent demand.

Like Holt–Winters and ARIMA, an LSTM preserves the order of historical observations. However, statistical models estimate parameters within predefined structures—such as level, trend, seasonality, and linear lag relationships—whereas an LSTM learns a nonlinear hidden representation from data.

An LSTM is still a supervised-learning model: historical inputs are paired with known targets during training. The change is not from supervised to unsupervised learning. It is from tabular supervised learning with engineered memory to sequence-based supervised learning with learned memory.

XGBoost and the LSTM see the same forecasting problem through different data representations.

For a target date $t$, one LSTM training example looks back 28 days:

The first input is the ordered 28-day demand sequence. The vector $u_t$ contains information known for the target date: calendar position, store status, promotion, discount, price ratio, competitor activity, marketing, and weather. The identifiers $s$ and $k$ represent the store and SKU; the model converts them into learned embeddings.

Applying this transformation to every eligible target date across all 120 store–SKU series creates the global sequence dataset. Each series contributes examples to the same model, while the chronological training, validation, and test boundaries remain unchanged.

The shared LSTM therefore learns from 70,800 training sequences rather than the history of only one store–SKU. Its recurrent parameters can capture temporal behaviour shared across the portfolio without maintaining a separate network for every series.

The LSTM therefore does not eliminate feature engineering entirely. Tomorrow's planned promotion or weather forecast cannot be inferred reliably from past demand, so forecast-time context must remain explicit. What changes is how the historical demand signal is represented.

How an LSTM creates memory

A standard recurrent neural network repeatedly updates a hidden state as it reads a sequence. That gives it memory, but gradients can shrink or grow as they pass through many time steps. Learning longer relationships then becomes difficult.

An LSTM introduces a separate cell state and gates that regulate information flow. At day $t$, the model receives the current input $x_t$, the previous hidden state $h_{t-1}$, and the previous cell state $c_{t-1}$.

The cell state $c_t$ is the LSTM's longer-running internal memory. The hidden state $h_t$ is the exposed representation passed to the next sequence step and, eventually, to the forecasting layers. In the equations below, $[h_{t-1},x_t]$ joins the previous hidden state and current input into one vector. The matrices $W$ and vectors $b$ are parameters learned during training.

The forget gate decides how much existing memory to retain:

The input gate decides how much new information should be written to memory:

The model separately creates the candidate memory $\widetilde c_t$. This is a proposal for what the new information could be, based on the current input $x_t$ and the previous hidden state $h_{t-1}$:

The candidate is not yet the updated cell state. The input gate $i_t$ controls how much of it is added, while the forget gate $f_t$ controls how much of the previous memory remains:

In other words, $\widetilde c_t$ asks what could be added, $i_t$ decides how much to add, and $c_t$ is the resulting updated memory.

Finally, the output gate controls what memory contributes to the hidden state:

Why sigmoid for gates and tanh for memory?

Sigmoid and tanh perform different jobs inside an LSTM. The sigmoid function produces values between zero and one, making it suitable for controlling how much information passes through a gate. A value near zero blocks information, while a value near one allows most of it through. The forget, input, and output gates therefore use sigmoid as a soft control mechanism rather than an all-or-nothing switch.

The tanh function produces values between $-1$ and $1$. This allows candidate memory to contain positive or negative internal signals—for example, evidence that the recent demand level is increasing or decreasing. Tanh bounds both the candidate memory and the transformed cell state exposed through $h_t$; the internal cell state $c_t$ itself is not restricted to this range.

In short:

Sigmoid controls how much information passes; tanh represents and transforms the information itself.

Why element-wise multiplication?

The symbol $\odot$ represents element-wise multiplication. An LSTM's cell state contains multiple memory components, and each gate produces one control value for every corresponding component. Element-wise multiplication therefore lets the model retain, remove, or expose different parts of its memory independently.

For example, suppose the previous cell state is:

and the forget gate produces:

Then:

Almost all of the first component is retained, 10% of the second remains, and only 1% of the third survives. A dot product would instead collapse these components into one number and prevent component-wise gating.

Element-wise multiplication allows each gate to act like an independent volume control for every component of the LSTM's memory.

In demand forecasting, the cell state might retain a sustained weekly pattern, reduce the influence of an isolated spike, or update when the recent level changes for several consecutive days. We do not tell the LSTM to calculate lag_7 or rolling_mean_28; it learns a hidden representation that is useful for reducing the forecasting loss.

That flexibility is also a limitation. A hidden state is harder to explain than a named lag or rolling mean, and there is no guarantee that the representation learned from training will remain useful in the next demand regime.

Architecture Design

The architecture concatenates four representations rather than adding them numerically:

Here, $\Vert$ denotes vector concatenation. The deliberately compact model contains:

• A 28-day demand sequence with one value at each step.
• One LSTM layer with 32 hidden units.
• Seventeen explicit target-day context features.
• Four-dimensional store embeddings and three-dimensional SKU embeddings.
• A dense layer with 32 units, ReLU activation, and 10% dropout.
• One non-negative demand forecast after reversing the scaling transformation.

Together, these components give the global LSTM 6,435 trainable parameters. The objective is not to build the largest neural network, but to test whether a compact learned sequence representation can outperform a strong tabular benchmark.

Building leakage-safe sequences

The experiment retains the same chronological boundaries used in the previous article. The final 112 target dates form consecutive 56-day validation and test windows.

The figure shows one representative series, but the global model applies the same target-date boundaries to all 120 series. Every sample contains the 28 demand observations preceding its target date. The target itself never appears inside the sequence.

LSTM layer setup

The recurrent layer receives only the standardised demand sequence. For a batch containing $B$ examples, its input shape is:

The 28 positions represent consecutive historical days, while the final dimension contains one demand value per day. Target-day context and identity embeddings do not enter the recurrent layer; they are added after the sequence has been encoded.

The LSTM layer is configured as:

self.lstm = nn.LSTM(
    input_size=1,
    hidden_size=32,
    batch_first=True,
)

As the LSTM reads the 28 observations, it updates its hidden and cell states at every step. The implementation keeps only the final hidden state:

_, (hidden_state, cell_state) = self.lstm(sequence)
sequence_representation = hidden_state[-1]

For a batch of $B$ sequences, sequence_representation has shape:

This 32-dimensional vector summarises the complete 28-day sequence. The cell state helps maintain memory while the sequence is processed, but it is not passed directly to the forecasting head.

The sequence representation is then concatenated with the 17 target-day context features, four-dimensional store embedding, and three-dimensional SKU embedding:

The resulting 56-dimensional vector enters the dense forecasting network. A single layer with 32 hidden units keeps the architecture deliberately compact: it provides capacity for nonlinear temporal patterns while limiting training cost and overfitting risk.

Learned identity embeddings

A shared recurrent model can learn temporal behaviour across the portfolio, but it must still distinguish one store or product from another. The same recent demand pattern may imply a different forecast for a high-volume flagship store than for a smaller regional store. The global LSTM handles such differences through learned identity embeddings.

Unlike the XGBoost pipeline, which explicitly one-hot encodes store and SKU identities, the LSTM receives each identity as an integer index.

Store_A → 0
Store_B → 1
Store_C → 2
...

That index directly selects one row from a trainable embedding. An embedding is a learned lookup table: selecting a store or SKU retrieves a small vector of trainable values. In this model, every store receives a four-dimensional vector and every SKU receives a three-dimensional vector

store_embedding = nn.Embedding(
    num_embeddings=n_stores,
    embedding_dim=4,
)

For $N$ stores, the model creates a matrix:

The store and SKU vectors are concatenated with the learned sequence representation and target-day business context before entering the dense forecasting layer. The recurrent parameters remain shared across all series, while the embeddings give the downstream network information it can use to make identity-specific adjustments.

Embeddings are updated through backpropagation whenever their store or SKU appears in a training sequence. Identities that require similar forecasting adjustments may develop similar vectors. This does not prove that two stores or products are operationally or economically similar: the vectors encode only what helps reduce the forecasting loss.

Embeddings also create a cold-start limitation. A completely new store or SKU has no learned vector. A production system therefore needs a strategy such as a shared unknown-identity embedding, metadata-based features, or retraining after sufficient history becomes available. Accuracy must still be monitored by series because shared training and embeddings do not guarantee equal performance across the portfolio.

Scaling without future information

Demand levels differ substantially across stores and products. Training one global model on raw units could cause high-volume series to dominate the loss. Therefore, we standardise demand separately for each series:

The mean $\mu_{s,k}$ and standard deviation $\sigma_{s,k}$ are calculated using training dates only. Context features are also standardised with a scaler fitted only on the training window. Validation and test data are transformed with those locked statistics.

After prediction, the transformation is reversed to return forecasts to demand units. Negative predictions are clipped to zero because unit demand cannot be negative.

This is an easy place to create leakage. Computing a series mean or standard deviation from all 730 days would allow the future validation and test distributions to influence training. Sequence models do not make point-in-time data discipline less important; they make another preprocessing pipeline that must be protected.

Training choices

After fixing the architecture and input pipeline, three training choices remain especially important: the loss function, gradient control, and the number of training epochs. Together, they determine how the model responds to unusually large errors, how stable each parameter update remains, and how long learning continues before the model begins to overfit the training period.

Huber loss

Daily demand contains occasional large spikes. Squared-error loss gives those observations rapidly increasing influence, which can encourage the network to chase unusual events. Huber loss—implemented as SmoothL1Loss—behaves approximately like squared error near zero and more like absolute error for large residuals.

The notebook configures and applies it as follows:

LOSS_FUNCTION = nn.SmoothL1Loss(beta=0.5)

prediction = model(sequence, context, store_index, sku_index)
loss = LOSS_FUNCTION(prediction, target)

The beta=0.5 threshold marks the transition from the quadratic to the linear part of the loss. Because demand is standardised before training, this threshold is expressed in scaled-demand units.

This does not remove spikes or declare them unimportant. It limits how strongly one extreme residual can dominate an optimisation step.

Gradient Clipping

Training an LSTM requires backpropagation through all 28 sequence steps. Repeated recurrent operations can occasionally produce very large gradients, leading to unstable parameter updates—a problem known as exploding gradients.

After calculating the gradients, the notebook limits their combined norm:

loss.backward()
nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0,
)
optimizer.step()

The function calculates the combined Euclidean norm of all model gradients:

If the norm is at most 1, the gradients remain unchanged. If it exceeds 1, all gradients are scaled proportionally:

For example, a total norm of 5 causes every gradient to be multiplied by $1/5$. This preserves the overall update direction while reducing its magnitude.

Gradient clipping does not alter the model weights, predictions, or loss directly. It limits only the update passed to the optimiser, and it addresses exploding gradients rather than vanishing gradients.

Huber loss and gradient clipping therefore perform complementary roles. Huber loss limits the influence of individual large residuals, while gradient clipping protects the complete optimisation step when the combined gradient becomes unusually large.

The threshold of 1.0 is a training hyperparameter rather than a universal rule. A threshold that is too high may provide little protection, while one that is too low can slow learning. It should therefore be checked through validation and, ideally, gradient-norm monitoring.

Select training duration through validation

Validation loss is checked after every epoch, and training stops after six epochs without improvement. The epoch with the lowest validation loss is locked before the test window is opened.

Validation selects only one training epoch for the global LSTM. After that point, training loss continues to fall while validation loss gradually rises. More optimisation improves the fit to the training period but makes the model less useful for the next temporal regime. That is a textbook form of temporal overfitting.

The result is a useful reminder:

Lower training loss does not mean better forecasting of the future.

Early stopping acts as regularisation by limiting how long the model can adapt to training-specific structure. The selected epoch count is not a universal property of LSTMs; it belongs to this architecture, dataset, optimiser, and validation period.

After model selection, a fresh global model is initialised and trained on 77,520 development sequences—the combined training and validation windows—for the single locked epoch. This gives the final model access to every target available before the test boundary. The untouched test window still contributes no information to training duration or model fitting.

Does the improvement justify the complexity?

Deep learning earns its place only if learned temporal representations improve held-out forecasting enough to justify its cost.

Model Comparison

The primary comparison is global LSTM against the global supervised-learning models from the previous article. All results aggregate the same 6,720 test rows across 120 store–SKU series.

This is a comparison of deployable forecasting systems rather than a controlled experiment that changes only the temporal representation. The LSTM also uses a different loss, learned identity embeddings, and a sequence-specific preprocessing pipeline. The result tells us which complete system forecasted better in this holdout; it does not isolate the causal effect of learned memory alone.

Global LSTM achieves the lowest WAPE at 26.44%, improving on the strongest XGBoost result by approximately 0.17 percentage points. It also records the lowest MAE, while full-context XGBoost retains the lowest RMSE.

The result supports a narrow conclusion: learned sequence representations are competitive and slightly better on the article's primary metric. It does not support the stronger conclusion that deep learning decisively replaces tabular forecasting. A 0.17-point difference from one 56-day window is too small to separate durable improvement from regime-specific variation without additional rolling evaluation.

Model selection

Model selection is not only an accuracy ranking.

In this CPU experiment, global-LSTM model selection takes roughly 14 seconds, compared with about one second for XGBoost. The absolute times are small, but the relative difference signals additional cost when tuning expands across architectures, lookback windows, random seeds, and rolling validation periods.

The LSTM removes handcrafted lag and rolling features, but it does not remove the forecasting pipeline. Production must still maintain:

• a correctly ordered 28-day window for every series;
• training-only demand and context scalers;
• store and SKU identity mappings;
• point-in-time business context;
• model weights, embedding tables, and architecture versions;
• monitoring for drift, unstable training, and forecast bias.

Neither model automatically produces calibrated uncertainty. Quantile loss, conformal calibration, or a probabilistic forecasting model would be required before prediction intervals could support service levels or safety stock.

Operational recommendation

Keep history, calendar, and identity XGBoost as the production benchmark and run the global LSTM as a challenger. The LSTM's 0.17-percentage-point WAPE advantage is promising, but it is not enough evidence for an immediate replacement.

Before promoting the LSTM:

• repeat the comparison across additional rolling validation and test windows;
• test multiple random seeds to measure training variability;
• translate forecast differences into stockouts, excess inventory, and service levels;
• monitor bias at panel, store, SKU, and store–SKU levels;
• verify that the sequence and context pipeline can be reproduced reliably;
• add calibrated uncertainty if forecasts will determine safety stock.

In a daily workflow, the trained weights can remain frozen while each series' 28-day demand window and target-day context are refreshed. Weight retraining can happen on a slower schedule and should be triggered by a defined calendar or drift rule rather than by every new observation.

The global model should produce one demand forecast for every store–SKU. Those forecasts are not order quantities. Replenishment still needs stock on hand, confirmed inbound inventory, lead time, pack sizes, service targets, and a chosen uncertainty buffer.

Conclusion: complexity must earn its cost

The progression from XGBoost to LSTM is not simply a move from an old algorithm to a newer one. It changes how the model represents time.

• Tabular supervised learning receives engineered temporal summaries.
• The LSTM receives an ordered sequence and learns a hidden representation.
• Global training supplies enough related sequences for representation learning to become plausible.
• Embeddings let one model share parameters while adjusting for store and SKU.
• Early stopping remains essential because additional optimisation can reduce future accuracy.

The global LSTM wins the panel WAPE table by a narrow margin. Whether it wins the forecasting system depends on what that margin is worth.

For this case study, the honest answer is that deep learning has become competitive, not indispensable. XGBoost remains the benchmark to beat, while the LSTM must prove that its learned memory creates stable operational value across more than one held-out window.