The Wayback Machine - https://web.archive.org/web/20240930183330/https://www.geeksforgeeks.org/lstm-derivation-of-back-propagation-through-time/
Open In App

LSTM – Derivation of Back propagation through time

Last Updated : 27 Dec, 2021
Summarize
Comments
Improve
Suggest changes
Like Article
Like
Save
Share
Report
News Follow

LSTM (Long short term Memory ) is a type of RNN(Recurrent neural network), which is a famous deep learning algorithm that is well suited for making predictions and classification with a flavour of the time. In this article, we will derive the algorithm backpropagation through time and find the gradient value for all the weights at a particular timestamp. 
As the name suggests backpropagation through time is similar to backpropagation in DNN(deep neural network) but due to the dependency of time in RNN and LSTM, we will have to apply the chain rule with time dependency. 

Let the input at time t in the LSTM cell be xt, the cell state from time t-1 and t be ct-1  and ct and the output for time t-1 and t be ht-1 and ht . The initial value of ct and ht at t = 0 will be zero. 

Step 1 : Initialization of the weights . 

Weights for different gates are : 
Input gate : wxi, wxg, bi, whj, wg , bg

Forget gate : wxf, bf, whf 

Output gate : wxo, bo, who

Step 2 : Passing through different gates . 

  
Inputs: xt and ht-i , ct-1  are given to the LSTM cell 
      Passing through input gate: 
       
          Zg = wxg *x + whg * ht-1 + bg 
          g = tanh(Zg)
          Zj = wxi * x + whi * ht-1 + bi 
          i =  sigmoid(Zi) 
          
          Input_gate_out = g*i 
           
      Passing through forget gate:  
           
          Zf = wxf * x + whf *ht-1 + bf 
          f = sigmoid(Zf) 
              
      Forget_gate_out = f 
       
      Passing through the output gate:  
              
      Zo  = wxo*x +  who * ht-1 + bo 
      o = sigmoid(zO) 
    
      Out_gate_out = o

Step 3 : Calculating the output ht and current cell state ct. 

  Calculating the current cell state ct :
          ct = (ct-1 * forget_gate_out) + input_gate_out 

Calculating the output gate ht:
          ht=out_gate_out * tanh(ct)

Step 4 : Calculating the gradient through back propagation through time at time stamp t using the chain rule. 

  Let the gradient pass down by the above cell be: 
      E_delta  = dE/dht   
      
      If we are using MSE (mean square error)for error then,
      E_delta=(y-h(x))
      Here y is the original value and h(x) is the predicted value.     
              
  Gradient with respect to output gate  
          
             dE/do = (dE/dht ) * (dht /do) = E_delta * ( dht / do) 
                dE/do =  E_delta * tanh(ct) 
      
  Gradient with respect to ct         
      dE/dct = (dE / dht )*(dht /dct)= E_delta *(dht /dct) 
                dE/dct  =   E_delta   * o * (1-tanh2 (ct))        

  Gradient with respect to input gate dE/di, dE/dg 
           
      dE/di = (dE/di ) * (dct / di)  
             dE/di =  E_delta   * o * (1-tanh2 (ct)) * g 
      Similarly,  
      dE/dg =  E_delta   * o * (1-tanh2 (ct)) * i 
       
  Gradient with respect to forget gate  
           
          dE/df =  E_delta   * (dE/dct ) * (dct / dt) t
          dE/df =  E_delta   * o * (1-tanh2 (ct)) *  ct-1  

  Gradient with respect to ct-1  
           
          dE/dct =  E_delta   * (dE/dct ) * (dct / dct-1) 
          dE/dct =  E_delta   * o * (1-tanh2 (ct)) * f  
 
  Gradient with respect to output gate weights:
    
    dE/dwxo   =  dE/do *(do/dwxo) = E_delta * tanh(ct) * sigmoid(zo) * (1-sigmoid(zo) * xt
    dE/dwho   =  dE/do *(do/dwho) = E_delta * tanh(ct) * sigmoid(zo) * (1-sigmoid(zo) * ht-1
    dE/dbo   =  dE/do *(do/dbo) = E_delta * tanh(ct) * sigmoid(zo) * (1-sigmoid(zo)

   Gradient with respect to forget gate weights:
    
    dE/dwxf  =  dE/df *(df/dwxf) = E_delta * o * (1-tanh2 (ct)) * ct-1 * sigmoid(zf) * (1-sigmoid(zf) * xt
    dE/dwhf =  dE/df *(df/dwhf) = E_delta * o * (1-tanh2 (ct)) *  ct-1 * sigmoid(zf) * (1-sigmoid(zf) * ht-1
    dE/dbo  =  dE/df *(df/dbo) = E_delta * o * (1-tanh2 (ct)) *  ct-1 * sigmoid(zf) * (1-sigmoid(zf) 

   Gradient with respect to input gate weights:
    
    dE/dwxi  =  dE/di *(di/dwxi) = E_delta * o * (1-tanh2 (ct)) * g * sigmoid(zi) * (1-sigmoid(zi) * xt
    dE/dwhi =  dE/di *(di/dwhi) = E_delta * o * (1-tanh2 (ct)) * g * sigmoid(zi) * (1-sigmoid(zi) * ht-1
    dE/dbi  =  dE/di *(di/dbi) = E_delta * o * (1-tanh2 (ct)) * g *  sigmoid(zi) * (1-sigmoid(zi)
    
    dE/dwxg  =  dE/dg *(dg/dwxg) = E_delta * o * (1-tanh2 (ct)) * i * (1?tanh2(zg))*xt
    dE/dwhg  =  dE/dg *(dg/dwhg) = E_delta * o * (1-tanh2 (ct)) * i * (1?tanh2(zg))*ht-1
    dE/dbg  =  dE/dg *(dg/dbg)  = E_delta * o * (1-tanh2 (ct)) * i * (1?tanh2(zg))

Finally the gradients associated with the weights are, 

Using all gradient, we can easily update the weights associated with input gate, output gate, and forget gate 
 


author
sahishanu
News
Improve
Previous Article
Long Short Term Memory Networks Explanation
Next Article
Text Generation using Recurrent Long Short Term Memory Network

Similar Reads

Back Propagation through time - RNN
Introduction: Recurrent Neural Networks are those networks that deal with sequential data. They predict outputs using not only the current inputs but also by taking into consideration those that occurred before it. In other words, the current output depends on current output as well as a memory element (which takes into account the past inputs). Fo
5 min read
Back Propagation with TensorFlow
This article discusses how backpropagation works in TensorFlow, one of the most popular deep-learning libraries. Let's learn about what is backpropagation and the other attributes related to it. BackpropagationBack propagation is a fundamental technique used in the training of neural networks which helps in optimizing the weights and biases of a mo
8 min read
Deep Neural net with forward and back propagation from scratch - Python
This article aims to implement a deep neural network from scratch. We will implement a deep neural network containing two input layers, a hidden layer with four units and one output layer. The implementation will go from scratch and the following steps will be implemented. Algorithm:1. Loading and visualizing the input data2. Deciding the shapes of
6 min read
Understanding of LSTM Networks
This article talks about the problems of conventional RNNs, namely, the vanishing and exploding gradients, and provides a convenient solution to these problems in the form of Long Short Term Memory (LSTM). Long Short-Term Memory is an advanced version of recurrent neural network (RNN) architecture that was designed to model chronological sequences
9 min read
Emotion Detection using Bidirectional LSTM
Emotion Detection is one of the hottest topics in research nowadays. Emotion-sensing technology can facilitate communication between machines and humans. It will also help to improve the decision-making process. Many Machine Learning Models have been proposed to recognize emotions from the text. But, in this article, our focus is on the Bidirection
10 min read
LSTM Based Poetry Generation Using NLP in Python
One of the major tasks that one aims to accomplish in Conversational AI is Natural Language Generation (NLG) which refers to employing models for the generation of natural language. In this article, we will get our hands on NLG by building an LSTM-based poetry generator. Note: The readers of this article are expected to be familiar with LSTM. In or
7 min read
What is LSTM - Long Short Term Memory?
LSTM excels in sequence prediction tasks, capturing long-term dependencies. Ideal for time series, machine translation, and speech recognition due to order dependence. The article provides an in-depth introduction to LSTM, covering the LSTM model, architecture, working principles, and the critical role they play in various applications. What is LST
10 min read
Minimizing Signal Propagation Time in a Binary Tree
Given a binary tree and a source node "start" which transmits a network signal. The signal propagates to neighboring nodes every second, the task is to determine the minimum amount of time required for the entire tree to receive the signal. Examples: Input: 11 / \ 12 13 / \ 14 15 / \ / \ 21 22 23 24Start = 14Output: 3 Input: 4 / \ 3 1 \ 2Start = 1O
12 min read
Sine Rule with Derivation, Example and Implementation
Given angles(in degrees) A, C, and the side c, corresponding to the figure below, the task is to find the remaining two sides a and b. Examples: Input: A = 45, C = 35, c = 23Output:28.35 39.49Explanation: a is 28.35 and b is 39.49 Input: A = 45, C = 45, c = 10Output:1014.14 Approach: The idea is to use Sine rule. It states that the sides of any tri
5 min read
Affinity Propagation in ML | To find the number of clusters
Affinity Propagation creates clusters by sending messages between data points until convergence. Unlike clustering algorithms such as k-means or k-medoids, affinity propagation does not require the number of clusters to be determined or estimated before running the algorithm, for this purpose the two important parameters are the preference, which c
3 min read
Range Update without using Lazy Propagation and Point Query in a Segment Tree
Given an array arr[] consisting of N 0s and a 2D array Q[][] consisting of queries of the following two types: 1 L R X: Increment all the elements in the range [L, R] by X.2 X: Print elements at Xth index of the array. Input: arr[] = { 0, 0, 0, 0, 0 }, Q[][] = { { 1, 0, 2, 100 }, { 2, 1 }, { 1, 2, 3, 200 }, { 2, 2 }, { 4 } } Output: 100 300 0 Expla
13 min read
Decision Boundary of Label Propagation Vs SVM on the Iris Dataset
In machine learning, understanding decision boundaries is crucial for classification tasks. The decision boundary separates different classes in a dataset. Here, we'll explore and compare decision boundaries generated by two popular classification algorithms - Label Propagation and Support Vector Machines (SVM) - using the famous Iris dataset in Py
9 min read
Lazy Propagation in Segment Tree
Segment tree is introduced in previous post with an example of range sum problem. We have used the same "Sum of given Range" problem to explain Lazy propagation How does update work in Simple Segment Tree? In the previous post, update function was called to update only a single value in array. Please note that a single value update in array may cau
15+ min read
Lazy Propagation in Segment Tree | Set 2
Given an array arr[] of size N. There are two types of operations: Update(l, r, x) : Increment the a[i] (l <= i <= r) with value x.Query(l, r) : Find the maximum value in the array in a range l to r (both are included).Examples: Input: arr[] = {1, 2, 3, 4, 5} Update(0, 3, 4) Query(1, 4) Output: 8 After applying the update operation in the giv
15+ min read
Affinity Propagation
Affinity Propagation is a clustering algorithm that is commonly used in Machine Learning and data analysis. Unlike other traditional clustering algorithms which require specifying the number of clusters beforehand, Affinity Propagation discovers cluster centres and assigns data points to clusters autonomously. It is particularly useful when the num
12 min read
Lazy Propagation in Python
Segment tree is the data structure that can be used for solving the range queries efficiently. In Segment tree, a single value update in array may cause multiple updates in Segment Tree as there may be many segment tree nodes that have a single array element in their ranges. Using Lazy Propagation, we can handle the updates in a much faster way. In
9 min read
Range Operations and Lazy Propagation for Competitive Programming
In competitive programming, mastering range operations and understanding lazy propagation techniques is a crucial skill set for efficiently handling complex problems involving large datasets. Range operations involve performing actions, such as queries or updates, over a specific range of elements in a data structure, offering a powerful tool for s
15+ min read
Time difference between expected time and given time
Given the initial clock time h1:m1 and the present clock time h2:m2, denoting hour and minutes in 24-hours clock format. The present clock time h2:m2 may or may not be correct. Also given a variable K which denotes the number of hours passed. The task is to calculate the delay in seconds i.e. time difference between expected time and given time. Ex
5 min read
Minimum time required by n cars to travel through all of the m roads
Given m roads and n cars. The cars are numbered from 1 to n. You are also given an array arr[] of size m, each road has a value arr[i] - the index of a car that runs fast on this road. If a car is fast on a road, then it travels across the road in 1 hour, else it takes 2 hours (if not proficient) to travel through this road. Find out the minimum ti
13 min read
Time saved travelling in shortest route and shortest path through given city
Given a matrix mat[][] of size N * N, where mat[i][j] represents the time taken to reach from ith city to jth city. Also, given M queries in the form of three arrays S[], I[], and D[] representing source, intermediate, and destination respectively. The task is to find the time taken to go from S[i] to D[i] using I[i] city and the time that can be s
10 min read
Front and Back Search in unsorted array
Given an unsorted array of integers and an element x, find if x is present in array using Front and Back search. Examples : Input : arr[] = {10, 20, 80, 30, 60, 50, 110, 100, 130, 170} x = 110; Output : Yes Input : arr[] = {10, 20, 80, 30, 60, 50, 110, 100, 130, 170} x = 175; Output : No A simple solution is to perform linear search. The linear sea
5 min read
Back-Face Detection Method
When we project 3-D objects on a 2-D screen, we need to detect the faces that are hidden on 2D. Back-Face detection, also known as Plane Equation method, is an object space method in which objects and parts of objects are compared to find out the visible surfaces. Let us consider a triangular surface that whose visibility needs to decided. The idea
3 min read
Tree, Back, Edge and Cross Edges in DFS of Graph
Consider a directed graph given in below, DFS of the below graph is 1 2 4 6 3 5 7 8. In below diagram if DFS is applied on this graph a tree is obtained which is connected using green edges. Tree Edge: It is an edge which is present in the tree obtained after applying DFS on the graph. All the Green edges are tree edges. Forward Edge: It is an edge
11 min read
Check if it is possible to get back to 12'0 clock only by adding or subtracting given seconds
Given N seconds. The task is to check if it is possible to start from the 12'0 clock and get back to 12 only by adding or subtracting the given seconds. We need to use all given seconds exactly once, we can either add an element or subtract it.Examples: Input: a[] = {60, 60, 120} Output: YES Add the first two seconds and subtract the last one to ge
7 min read
Sum of shortest distance on source to destination and back having at least a common vertex
Given a directed weighted graph and the source and destination vertex. The task is to find the sum of shortest distance on the path going from source to destination and then from destination to source such that both the paths have at least a common vertex other than the source and the destination. Note: On going from destination to source, all the
15+ min read
Maximum sum path in a matrix from top to bottom and back
Given a matrix of dimension N * M. The task is find the maximum sum of path from arr[0][0] to arr[N - 1][M - 1] and back from arr[N - 1][M - 1] to arr[0][0]. On the path from arr[0][0] to arr[N - 1][M - 1], you can traverse in down and right directions and on the path from arr[N - 1][M - 1] to arr[0][0], you can traverse in up and left directions.
9 min read
Number of ways in which an item returns back to its initial position in N swaps in array of size K
Given two numbers K and N, the task is to find the number of ways such that an item at position i returns back to its initial position in an array of length K in N steps, where, in each step, the item can be swapped with any other item in K Examples: Input: N = 2, K = 5 Output: 4 Explanation: For the given K, lets assume there are 5 positions 1, 2,
8 min read
In how many ways the ball will come back to the first boy after N turns
Four boys are playing a game with a ball. In each turn, the player (who has the ball currently) passes it to a different player randomly. Bob always starts the game. The task is to find in how many ways the ball will come back to Bob after N passes.Examples: Input: N = 3 Output: 6 Here are all the possible ways: Bob -> Boy1 -> Boy2 -> Bob
3 min read
Difference between Tree edge and Back edge in graph
Tree Edge: It is an edge that is present in the tree obtained after performing DFS on the graph. All the Green edges are tree edges as shown in the below image. Back Edge: It is an edge (u, v) such that v is an ancestor of node u but not part of the DFS Traversal of the tree. Edge from 5 to 4 is a back edge. The presence of a back edge indicates a
1 min read
Minimum deletions from front or back required to remove maximum and minimum from Array
Given an array arr[] consisting of integers. The task is to find minimum deletions required to remove the initial minimum and maximum element from arr[].NOTE: Deletion can be performed either from the front or back of the array. Examples: Input: arr[] = {5, 7, 2, 4, 3}Output: 3Explanation: Initial minimum = 2, Initial maximum = 7Deleting first 3 fr
6 min read
Article Tags :
Practice Tags :
We use cookies to ensure you have the best browsing experience on our website. By using our site, you acknowledge that you have read and understood our Cookie Policy & Privacy Policy
Lightbox