Bidirectional RNN | BiLSTM | Bidirectional LSTM | Bidirectional GRU

Introduction to Bidirectional RNNs
📌 The session continues the Deep Learning playlist, focusing on Bidirectional RNNs (BiRNN) after covering Vanilla RNN, LSTM, GRU, and Deep RNNs.
➡️ BiRNNs address limitations in unidirectional RNNs where future inputs might be necessary to correctly interpret past or current data points.

Motivation for Bidirectional RNNs
📌 Unidirectional RNNs only rely on past information (e.g., output at time step $t$ depends on $x_1$ through $x_t$).
📌 A key motivation is tasks like Named Entity Recognition (NER), where context from future words determines the entity type (e.g., "Amazon" as an organization vs. a river location).
📌 BiRNNs overcome this by processing input from both directions: left-to-right (forward) and right-to-left (backward).

Architecture and Mechanics of BiRNNs
📌 A BiRNN consists of two separate RNNs: a forward RNN and a backward RNN.
📌 At each time step $t$, the outputs from both the forward hidden state ($h_t^{\text{forward}}$) and the backward hidden state ($h_t^{\text{backward}}$) are concatenated to form the final output $y_t$.
📌 The forward hidden state calculation uses $h_{t-1}^{\text{forward}}$, while the backward hidden state calculation uses $h_{t+1}^{\text{backward}}$.

Mathematical Formulation
📌 The forward hidden state equation is the standard RNN equation: $h_t^{\text{forward}} = \text{activation}(W^{\text{forward}} h_{t-1}^{\text{forward}} + U^{\text{forward}} x_t + b^{\text{forward}})$.
📌 The backward hidden state calculation involves looking ahead: $h_t^{\text{backward}} = \text{activation}(W^{\text{backward}} h_{t+1}^{\text{backward}} + U^{\text{backward}} x_t + b^{\text{backward}})$.
📌 The final output $y_t$ is derived by concatenating these two states: $y_t = \sigma(W_y [h_t^{\text{forward}}; h_t^{\text{backward}}] + b_y)$.

Implementation in Keras and Extensibility
📌 Implementing BiRNN is straightforward in Keras using the `Bidirectional` wrapper around any standard RNN layer (SimpleRNN, LSTM, or GRU).
📌 Using the wrapper effectively doubles the weights and biases because it initializes two independent recurrent layers.
📌 The BiRNN concept is applicable to advanced cells; for instance, applying it to LSTM creates a BiLSTM (more commonly used than plain BiRNN).

Application Areas and Drawbacks
📌 BiRNNs generally yield better results in tasks requiring full context, such as NER, Part-of-Speech (POS) Tagging, and Machine Translation.
📌 They also show performance improvements in Sentiment Analysis and Time Series Forecasting (stock price, weather prediction).
📌 Drawback 1 (Complexity/Overfitting): Doubling the parameters increases training time and raises the risk of overfitting, necessitating regularization techniques like dropout.
📌 Drawback 2 (Latency): BiRNNs are unsuitable for tasks like real-time speech recognition because they require the entire sequence to be available before processing can begin, introducing latency.

Key Points & Insights
➡️ Use Bidirectional RNNs when the context from future data points is crucial for accurately processing the current point, as seen in NER tasks.
➡️ Implement BiRNNs easily in Keras by wrapping existing layers (like LSTM or GRU) inside the `Bidirectional` layer.
➡️ Be cautious with BiRNNs in real-time streaming applications due to inherent latency caused by the need to process the input sequence both forwards and backwards simultaneously.
➡️ BiLSTM is generally the preferred variant over a plain BiRNN in modern NLP applications.

📸 Video summarized with SummaryTube.com on Mar 10, 2026, 15:59 UTC