Recurrent Neural Networks (RNNs, LSTMs, GRUs)

A recurrent neural network applies a state update repeatedly across a sequence. At time $t$, it reads input $x_t$ and the previous hidden state $h_{t-1}$ to produce $h_t$. The shared weights let the same model process different sequence lengths. This is useful when order and carried state matter, such as a sensor stream or a sequence of events.

An RNN does not remember everything it has seen. The state has finite capacity, and learning what to retain is difficult when the relevant event lies many steps back. LSTM and GRU cells introduce gates that make retention and replacement learnable. They ease some long-dependency problems; they do not guarantee perfect memory.

Imagine reading a log one record at a time. A plain RNN rewrites a compact note after each record. An LSTM keeps a separate cell state and learns three decisions: forget part of the old note, write a candidate update, and expose part of the result. A GRU combines retention and writing with fewer gate families and a single hidden state.

The unrolled computation is still sequential across time. Batching many sequences can improve hardware use, but each step's state depends on the previous step. A Transformer uses attention to connect positions differently and is often easier to parallelize during training; this does not make recurrence obsolete for every streaming or stateful task.

A plain RNN uses a shared transition:

Backpropagation through time sends a distant gradient through a product of state-transition Jacobians. The product can shrink or grow, depending on weights, activations, and the trajectory. A simple LSTM form instead computes gates and an additive cell update:

Holding $f_t$ near one and $i_t$ near zero can carry cell information forward. That is a possible learned behavior, not an unconditional gradient guarantee: gates, saturation, truncation, and finite state still matter. For one common GRU convention, $h_t=(1-z_t)\odot n_t+z_t\odot h_{t-1}$; here $z_t$ is the fraction of old state retained. Libraries may use different gate names or algebraic rearrangements, so check their equations before comparing parameters.

PyTorch returns a sequence output and final state. LSTM's final state is a pair; GRU's is one tensor. These shapes are worth checking before adding a classifier or carrying state into another batch.

1import torch
2from torch import nn
3
4x = torch.randn(4, 7, 5)  # batch, time, input features
5gru = nn.GRU(input_size=5, hidden_size=8, batch_first=True)
6lstm = nn.LSTM(input_size=5, hidden_size=8, batch_first=True)
7
8gru_steps, gru_final = gru(x)
9lstm_steps, (lstm_hidden, lstm_cell) = lstm(x)
10assert gru_steps.shape == lstm_steps.shape == (4, 7, 8)
11assert gru_final.shape == lstm_hidden.shape == lstm_cell.shape == (1, 4, 8)

For independent examples, initialize state anew. For a continuous stream, carry state only across the intended sequence boundary; detach it when using truncated backpropagation through time. Otherwise, one user's history can leak into another example or a computation graph can grow without bound. Pad or pack variable-length batches so padding steps do not silently become data.

1. Mechanism: Why does an RNN share weights across time? (The same transition processes each step while hidden state carries prior information; sharing also permits variable sequence length.)
2. Gradient: Why can a plain RNN miss a distant dependency? (The backward signal contains products of time-step Jacobians that can shrink or grow.)
3. Gates: What does the LSTM cell update buy? (An additive carry path with learned forget and input gates, which can ease retention without guaranteeing it.)
4. Implementation: What differs in the returned state of PyTorch GRU and LSTM? (GRU returns a hidden tensor; LSTM returns hidden and cell tensors.)
5. Systems: When would you still test recurrence against attention? (Streaming state, latency, context length, data scale, and hardware can change the tradeoff. Benchmark the actual task.)

Without notes, write the plain RNN update and the LSTM cell update. Name one way gradients can fail across time and one way a stateful data loader can leak information. Then explain why a gate near one helps retention without promising unlimited memory.

Sources and further reading

• Elman, Finding Structure in Time (1990): recurrent context as dynamic memory.
• Bengio, Simard, and Frasconi, Learning Long-Term Dependencies with Gradient Descent is Difficult (1994): analysis of distant-gradient difficulty.
• Hochreiter and Schmidhuber, Long Short-Term Memory (1997): gated cell-state method.
• Cho et al., Learning Phrase Representations using RNN Encoder-Decoder (2014): GRU-style recurrent encoder-decoder.
• PyTorch recurrent layer documentation: current framework state shapes and gate conventions.