Misplaced Pages

#654345

53-450: An LSTM unit is typically composed of a cell and three gates : an input gate , an output gate , and a forget gate . The cell remembers values over arbitrary time intervals, and the gates regulate the flow of information into and out of the cell. Forget gates decide what information to discard from the previous state, by mapping the previous state and the current input to a value between 0 and 1. A (rounded) value of 1 signifies retention of

106-447: A × {\displaystyle \times } symbol represent an element-wise multiplication between its inputs. The big circles containing an S -like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum. Peephole convolutional LSTM. The ∗ {\displaystyle *} denotes the convolution operator. An RNN using LSTM units can be trained in

159-518: A , b , β ) = a ⊙ Swish β ( b ) {\displaystyle {\begin{aligned}\mathrm {ReGLU} (a,b)&=a\odot {\text{ReLU}}(b)\\\mathrm {GEGLU} (a,b)&=a\odot {\text{GELU}}(b)\\\mathrm {SwiGLU} (a,b,\beta )&=a\odot {\text{Swish}}_{\beta }(b)\end{aligned}}} where ReLU, GELU, and Swish are different activation functions (see this table for definitions). In transformer models, such gating units are often used in

212-583: A learning algorithm). 2005: Daan Wierstra, Faustino Gomez, and Schmidhuber trained LSTM by neuroevolution without a teacher. Mayer et al. trained LSTM to control robots . 2007: Wierstra, Foerster, Peters, and Schmidhuber trained LSTM by policy gradients for reinforcement learning without a teacher. Hochreiter, Heuesel, and Obermayr applied LSTM to protein homology detection the field of biology . 2009: Justin Bayer et al. introduced neural architecture search for LSTM. 2009: An LSTM trained by CTC won

265-547: A simplified variant of the forget gate LSTM called Gated recurrent unit (GRU). (Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber, 2015) used LSTM principles to create the Highway network , a feedforward neural network with hundreds of layers, much deeper than previous networks. Concurrently, the ResNet architecture was developed. It is equivalent to an open-gated or gateless highway network. A modern upgrade of LSTM called xLSTM

318-501: A supervised fashion on a set of training sequences, using an optimization algorithm like gradient descent combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight. A problem with using gradient descent for standard RNNs

371-509: A vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory". 2018: OpenAI used LSTM trained by policy gradients to beat humans in the complex video game of Dota 2, and to control a human-like robot hand that manipulates physical objects with unprecedented dexterity. 2019: DeepMind used LSTM trained by policy gradients to excel at the complex video game of Starcraft II . Aspects of LSTM were anticipated by "focused back-propagation" (Mozer, 1989), cited by

424-402: A weighted sum. i t , o t {\displaystyle i_{t},o_{t}} and f t {\displaystyle f_{t}} represent the activations of respectively the input, output and forget gates, at time step t {\displaystyle t} . The 3 exit arrows from the memory cell c {\displaystyle c} to

477-501: Is also commutative . For two matrices A and B of the same dimension m × n , the Hadamard product A ⊙ B {\displaystyle A\odot B} (sometimes A ∘ B {\displaystyle A\circ B} ) is a matrix of the same dimension as the operands, with elements given by For matrices of different dimensions ( m × n and p × q , where m ≠ p or n ≠ q ),

530-514: Is called broadcast multiplication and also denoted with a .* b , and other operators are analogously defined element-wise, for example Hadamard powers use a .^ b . But unlike MATLAB, in Julia this "dot" syntax is generalized with a generic broadcasting operator . which can apply any function element-wise. This includes both binary operators (such as the aforementioned multiplication and exponentiation, as well as any other binary operator such as

583-437: Is defined as: C = A ⊘ B C i j = A i j B i j {\displaystyle {\begin{aligned}{C}&={A}\oslash {B}\\C_{ij}&={\frac {A_{ij}}{B_{ij}}}\end{aligned}}} Most scientific or numerical programming languages include the Hadamard product, under various names. In MATLAB ,

SECTION 10

#1732904799655

636-600: Is positive-semidefinite. This is known as the Schur product theorem, after Russian mathematician Issai Schur . For two positive-semidefinite matrices A and B , it is also known that the determinant of their Hadamard product is greater than or equal to the product of their respective determinants: det ( A ⊙ B ) ≥ det ( A ) det ( B ) . {\displaystyle \det({A}\odot {B})\geq \det({A})\det({B}).} Other Hadamard operations are also seen in

689-548: Is published by a team leaded by Sepp Hochreiter (Maximilian et al, 2024). One of the 2 blocks (mLSTM) of the architecture are parallelizable like the Transformer architecture, the other ones (sLSTM) allow state tracking. 2004: First successful application of LSTM to speech Alex Graves et al. 2001: Gers and Schmidhuber trained LSTM to learn languages unlearnable by traditional models such as Hidden Markov Models. Hochreiter et al. used LSTM for meta-learning (i.e. learning

742-430: Is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to lim n → ∞ W n = 0 {\displaystyle \lim _{n\to \infty }W^{n}=0} if the spectral radius of W {\displaystyle W} is smaller than 1. However, with LSTM units, when error values are back-propagated from

795-408: Is the cell state. h t − 1 {\displaystyle h_{t-1}} is not used, c t − 1 {\displaystyle c_{t-1}} is used instead in most places. Each of the gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of

848-526: Is the most commonly used version of LSTM nowadays. (Gers, Schmidhuber, and Cummins, 2000) added peephole connections. Additionally, the output activation function was omitted. (Graves, Fernandez, Gomez, and Schmidhuber, 2006) introduce a new error function for LSTM: Connectionist Temporal Classification (CTC) for simultaneous alignment and recognition of sequences. (Graves, Schmidhuber, 2005) published LSTM with full backpropagation through time and bidirectional LSTM. (Kyunghyun Cho et al., 2014) published

901-485: Is used in highway networks , which were designed by unrolling an LSTM. Channel gating uses a gate to control the flow of information through different channels inside a convolutional neural network (CNN). Hadamard product (matrices) In mathematics , the Hadamard product (also known as the element-wise product , entrywise product or Schur product ) is a binary operation that takes in two matrices of

954-504: The Matrix class ( a.cwiseProduct(b) ), while the Armadillo library uses the operator % to make compact expressions ( a % b ; a * b is a matrix product). In GAUSS , and HP Prime , the operation is known as array multiplication. In Fortran , R , APL , J and Wolfram Language ( Mathematica ), the multiplication operator * or × apply the Hadamard product, whereas

1007-629: The ICDAR connected handwriting recognition competition. Three such models were submitted by a team led by Alex Graves . One was the most accurate model in the competition and another was the fastest. This was the first time an RNN won international competitions. Gating mechanism In neural networks , the gating mechanism is an architectural motif for controlling the flow of activation and gradient signals . They are most prominently used in recurrent neural networks (RNNs), but have also found applications in other architectures. Gating mechanisms are

1060-569: The SymPy symbolic library, multiplication of array objects as either a*b or a@b will produce the matrix product. The Hadamard product can be obtained with the method call a.multiply_elementwise(b) . Some Python packages include support for Hadamard powers using methods like np.power(a, b) , or the Pandas method a.pow(b) . In C++, the Eigen library provides a cwiseProduct member function for

1113-1506: The feedforward modules . For a single vector input, this results in: GLU ⁡ ( x , W , V , b , c ) = σ ( x W + b ) ⊙ ( x V + c ) Bilinear ⁡ ( x , W , V , b , c ) = ( x W + b ) ⊙ ( x V + c ) ReGLU ⁡ ( x , W , V , b , c ) = max ( 0 , x W + b ) ⊙ ( x V + c ) GEGLU ⁡ ( x , W , V , b , c ) = GELU ⁡ ( x W + b ) ⊙ ( x V + c ) SwiGLU ⁡ ( x , W , V , b , c , β ) = Swish β ⁡ ( x W + b ) ⊙ ( x V + c ) {\displaystyle {\begin{aligned}\operatorname {GLU} (x,W,V,b,c)&=\sigma (xW+b)\odot (xV+c)\\\operatorname {Bilinear} (x,W,V,b,c)&=(xW+b)\odot (xV+c)\\\operatorname {ReGLU} (x,W,V,b,c)&=\max(0,xW+b)\odot (xV+c)\\\operatorname {GEGLU} (x,W,V,b,c)&=\operatorname {GELU} (xW+b)\odot (xV+c)\\\operatorname {SwiGLU} (x,W,V,b,c,\beta )&=\operatorname {Swish} _{\beta }(xW+b)\odot (xV+c)\end{aligned}}} Gating mechanism

SECTION 20

#1732904799655

1166-463: The sigmoid activation function . Replacing σ {\displaystyle \sigma } with other activation functions leads to variants of GLU: R e G L U ( a , b ) = a ⊙ ReLU ( b ) G E G L U ( a , b ) = a ⊙ GELU ( b ) S w i G L U (

1219-565: The 3 gates i , o {\displaystyle i,o} and f {\displaystyle f} represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell c {\displaystyle c} at time step t − 1 {\displaystyle t-1} , i.e. the contribution of c t − 1 {\displaystyle c_{t-1}} (and not c t {\displaystyle c_{t}} , as

1272-418: The Hadamard operator can be used for enhancing, suppressing or masking image regions. One matrix represents the original image, the other acts as weight or masking matrix. It is used in the machine learning literature, for example, to describe the architecture of recurrent neural networks as GRUs or LSTMs . It is also used to study the statistical properties of random vectors and matrices. According to

1325-440: The Hadamard product is expressed as "dot multiply": a .* b , or the function call: times(a, b) . It also has analogous dot operators which include, for example, the operators a .^ b and a ./ b . Because of this mechanism, it is possible to reserve * and ^ for matrix multiplication and matrix exponentials, respectively. The programming language Julia has similar syntax as MATLAB, where Hadamard multiplication

1378-543: The Hadamard product is undefined. For example, the Hadamard product for two arbitrary 2 × 3 matrices is: ( A ⊗ B ) ⊙ ( C ⊗ D ) = ( A ⊙ C ) ⊗ ( B ⊙ D ) , {\displaystyle (A\otimes B)\odot (C\otimes D)=(A\odot C)\otimes (B\odot D),} where ⊗ {\displaystyle \otimes }  is Kronecker product , assuming A {\displaystyle A} has

1431-459: The Kronecker product), and also unary operators such as ! and √ . Thus, any function in prefix notation f can be applied as f.(x) . Python does not have built-in array support, leading to inconsistent/conflicting notations. The NumPy numerical library interprets a*b or a.multiply(b) as the Hadamard product, and uses a@b or a.matmul(b) for the matrix product. With

1484-452: The LSTM for quicktype in the iPhone and for Siri. Amazon released Polly , which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology. 2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks. Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus , incorporating

1537-484: The LSTM paper. Sepp Hochreiter's 1991 German diploma thesis analyzed the vanishing gradient problem and developed principles of the method. His supervisor, Jürgen Schmidhuber , considered the thesis highly significant. An early version of LSTM was published in 1995 in a technical report by Sepp Hochreiter and Jürgen Schmidhuber , then published in the NIPS 1996 conference. The most commonly used reference point for LSTM

1590-1991: The LSTM. Compared to the LSTM, the GRU has just two gates: a reset gate and an update gate . GRU also merges the cell state and hidden state. The reset gate roughly corresponds to the forget gate, and the update gate roughly corresponds to the input gate. The output gate is removed. There are several variants of GRU. One particular variant has these equations: R t = σ ( X t W x r + H t − 1 W h r + b r ) Z t = σ ( X t W x z + H t − 1 W h z + b z ) H ~ t = tanh ⁡ ( X t W x h + ( R t ⊙ H t − 1 ) W h h + b h ) H t = Z t ⊙ H t − 1 + ( 1 − Z t ) ⊙ H ~ t {\displaystyle {\begin{aligned}\mathbf {R} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xr}+\mathbf {H} _{t-1}\mathbf {W} _{hr}+\mathbf {b} _{r})\\\mathbf {Z} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xz}+\mathbf {H} _{t-1}\mathbf {W} _{hz}+\mathbf {b} _{z})\\{\tilde {\mathbf {H} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xh}+(\mathbf {R} _{t}\odot \mathbf {H} _{t-1})\mathbf {W} _{hh}+\mathbf {b} _{h})\\\mathbf {H} _{t}&=\mathbf {Z} _{t}\odot \mathbf {H} _{t-1}+(1-\mathbf {Z} _{t})\odot {\tilde {\mathbf {H} }}_{t}\end{aligned}}} Gated Linear Units (GLUs) adapt

1643-444: The activation of the memory cell c {\displaystyle c} at time step t − 1 {\displaystyle t-1} , i.e. c t − 1 {\displaystyle c_{t-1}} . The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes c t {\displaystyle c_{t}} . The little circles containing

Long short-term memory - Misplaced Pages Continue

1696-2347: The centerpiece of long short-term memory (LSTM). They were proposed to mitigate the vanishing gradient problem often encountered by regular RNNs. An LSTM unit contains three gates: The equations for LSTM are: I t = σ ( X t W x i + H t − 1 W h i + b i ) F t = σ ( X t W x f + H t − 1 W h f + b f ) O t = σ ( X t W x o + H t − 1 W h o + b o ) C ~ t = tanh ⁡ ( X t W x c + H t − 1 W h c + b c ) C t = F t ⊙ C t − 1 + I t ⊙ C ~ t H t = O t ⊙ tanh ⁡ ( C t ) {\displaystyle {\begin{aligned}\mathbf {I} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xi}+\mathbf {H} _{t-1}\mathbf {W} _{hi}+\mathbf {b} _{i})\\\mathbf {F} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xf}+\mathbf {H} _{t-1}\mathbf {W} _{hf}+\mathbf {b} _{f})\\\mathbf {O} _{t}&=\sigma (\mathbf {X} _{t}\mathbf {W} _{xo}+\mathbf {H} _{t-1}\mathbf {W} _{ho}+\mathbf {b} _{o})\\{\tilde {\mathbf {C} }}_{t}&=\tanh(\mathbf {X} _{t}\mathbf {W} _{xc}+\mathbf {H} _{t-1}\mathbf {W} _{hc}+\mathbf {b} _{c})\\\mathbf {C} _{t}&=\mathbf {F} _{t}\odot \mathbf {C} _{t-1}+\mathbf {I} _{t}\odot {\tilde {\mathbf {C} }}_{t}\\\mathbf {H} _{t}&=\mathbf {O} _{t}\odot \tanh(\mathbf {C} _{t})\end{aligned}}} Here, ⊙ {\displaystyle \odot } represents elementwise multiplication . The gated recurrent unit (GRU) simplifies

1749-399: The corresponding input sequences. CTC achieves both alignment and recognition. Sometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution or by policy gradient methods, especially when there is no "teacher" (that is, training labels). Applications of LSTM include: 2015: Google started using an LSTM trained by CTC for speech recognition on Google Voice. According to

1802-448: The current state allows the LSTM network to maintain useful, long-term dependencies to make predictions, both in current and future time-steps. LSTM has wide applications in classification , data processing , time series analysis tasks, speech recognition , machine translation , speech activity detection, robot control , video games , and healthcare . In theory, classic RNNs can keep track of arbitrary long-term dependencies in

1855-416: The definition of V. Slyusar the penetrating face product of the p × g matrix A {\displaystyle {A}} and n -dimensional matrix B {\displaystyle {B}} ( n > 1) with p × g blocks ( B = [ B n ] {\displaystyle {B}=[B_{n}]} ) is a matrix of size B {\displaystyle {B}} of

1908-458: The equations for the forward pass of an LSTM cell with a forget gate are: where the initial values are c 0 = 0 {\displaystyle c_{0}=0} and h 0 = 0 {\displaystyle h_{0}=0} and the operator ⊙ {\displaystyle \odot } denotes the Hadamard product (element-wise product). The subscript t {\displaystyle t} indexes

1961-415: The exploding gradient problem. The intuition behind the LSTM architecture is to create an additional module in a neural network that learns when to remember and when to forget pertinent information. In other words, the network effectively learns which information might be needed later on in a sequence and when that information is no longer needed. For instance, in the context of natural language processing ,

2014-572: The forget gate f {\displaystyle f} or the memory cell c {\displaystyle c} , depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, c t ∈ R h {\displaystyle c_{t}\in \mathbb {R} ^{h}} is not just one unit of one LSTM cell, but contains h {\displaystyle h} LSTM cell's units. See for an empirical study of 8 architectural variants of LSTM. The compact forms of

2067-2149: The form: A [ ∘ ] B = [ A ∘ B 1 A ∘ B 2 ⋯ A ∘ B n ] . {\displaystyle {A}[\circ ]{B}=\left[{\begin{array}{c | c | c | c }{A}\circ {B}_{1}&{A}\circ {B}_{2}&\cdots &{A}\circ {B}_{n}\end{array}}\right].} If A = [ 1 2 3 4 5 6 7 8 9 ] , B = [ B 1 B 2 B 3 ] = [ 1 4 7 2 8 14 3 12 21 8 20 5 10 25 40 12 30 6 2 8 3 2 4 2 7 3 9 ] {\displaystyle {A}={\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix}},\quad {B}=\left[{\begin{array}{c | c | c }{B}_{1}&{B}_{2}&{B}_{3}\end{array}}\right]=\left[{\begin{array}{c c c | c c c | c c c }1&4&7&2&8&14&3&12&21\\8&20&5&10&25&40&12&30&6\\2&8&3&2&4&2&7&3&9\end{array}}\right]} then A [ ∘ ] B = [ 1 8 21 2 16 42 3 24 63 32 100 30 40 125 240 48 150 36 14 64 27 14 32 18 49 24 81 ] . {\displaystyle {A}[\circ ]{B}=\left[{\begin{array}{c c c | c c c | c c c }1&8&21&2&16&42&3&24&63\\32&100&30&40&125&240&48&150&36\\14&64&27&14&32&18&49&24&81\end{array}}\right].} where ∙ {\displaystyle \bullet } denotes

2120-489: The gating mechanism for use in feedforward neural networks , often within transformer -based architectures. They are defined as: G L U ( a , b ) = a ⊙ σ ( b ) {\displaystyle \mathrm {GLU} (a,b)=a\odot \sigma (b)} where a , b {\displaystyle a,b} are the first and second inputs, respectively. σ {\displaystyle \sigma } represents

2173-415: The information, and a value of 0 represents discarding. Input gates decide which pieces of new information to store in the current cell state, using the same system as forget gates. Output gates control which pieces of information in the current cell state to output, by assigning a value from 0 to 1 to the information, considering the previous and current states. Selectively outputting relevant information from

Long short-term memory - Misplaced Pages Continue

2226-545: The input sequences. The problem with classic RNNs is computational (or practical) in nature: when training a classic RNN using back-propagation , the long-term gradients which are back-propagated can "vanish" , meaning they can tend to zero due to very small numbers creeping into the computations, causing the model to effectively stop learning. RNNs using LSTM units partially solve the vanishing gradient problem , because LSTM units allow gradients to also flow with little to no attenuation. However, LSTM networks can still suffer from

2279-472: The lowercase variables represent vectors. Matrices W q {\displaystyle W_{q}} and U q {\displaystyle U_{q}} contain, respectively, the weights of the input and recurrent connections, where the subscript q {\displaystyle _{q}} can either be the input gate i {\displaystyle i} , output gate o {\displaystyle o} ,

2332-1213: The mathematical literature, namely the Hadamard root and Hadamard power (which are in effect the same thing because of fractional indices), defined for a matrix such that: For B = A ∘ 2 B i j = A i j 2 {\displaystyle {\begin{aligned}{B}&={A}^{\circ 2}\\B_{ij}&={A_{ij}}^{2}\end{aligned}}} and for B = A ∘ 1 2 B i j = A i j 1 2 {\displaystyle {\begin{aligned}{B}&={A}^{\circ {\frac {1}{2}}}\\B_{ij}&={A_{ij}}^{\frac {1}{2}}\end{aligned}}} The Hadamard inverse reads: B = A ∘ − 1 B i j = A i j − 1 {\displaystyle {\begin{aligned}{B}&={A}^{\circ -1}\\B_{ij}&={A_{ij}}^{-1}\end{aligned}}} A Hadamard division

2385-424: The matrix product is written using matmul , %*% , +.× , +/ .* and . , respectively. The R package matrixcalc introduces the function hadamard.prod() for Hadamard Product of numeric matrices or vectors. The Hadamard product appears in lossy compression algorithms such as JPEG . The decoding step involves an entry-for-entry product, in other words the Hadamard product. In image processing ,

2438-411: The network can learn grammatical dependencies. An LSTM might process the sentence " Dave , as a result of his controversial claims, is now a pariah" by remembering the (statistically likely) grammatical gender and number of the subject Dave , note that this information is pertinent for the pronoun his and note that this information is no longer important after the verb is . In the equations below,

2491-522: The official blog post, the new model cut transcription errors by 49%. 2016: Google started using an LSTM to suggest messages in the Allo conversation app. In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%. Apple announced in its Worldwide Developers Conference that it would start using

2544-401: The output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value. Many applications use stacks of LSTM RNNs and train them by connectionist temporal classification (CTC) to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given

2597-444: The picture may suggest). In other words, the gates i , o {\displaystyle i,o} and f {\displaystyle f} calculate their activations at time step t {\displaystyle t} (i.e., respectively, i t , o t {\displaystyle i_{t},o_{t}} and f t {\displaystyle f_{t}} ) also considering

2650-405: The same dimensions and returns a matrix of the multiplied corresponding elements. This operation can be thought as a "naive matrix multiplication" and is different from the matrix product . It is attributed to, and named after, either French mathematician Jacques Hadamard or German mathematician Issai Schur . The Hadamard product is associative and distributive . Unlike the matrix product, it

2703-923: The same dimensions of C {\displaystyle C} and B {\displaystyle B} with D {\displaystyle D} . ( A ∙ B ) ⊙ ( C ∙ D ) = ( A ⊙ C ) ∙ ( B ⊙ D ) , {\displaystyle (A\bullet B)\odot (C\bullet D)=(A\odot C)\bullet (B\odot D),} where ∙ {\displaystyle \bullet }  denotes face-splitting product . ( A ∙ B ) ( C ∗ D ) = ( A C ) ⊙ ( B D ) , {\displaystyle (A\bullet B)(C\ast D)=(AC)\odot (BD),} where ∗ {\displaystyle \ast }  is column-wise Khatri–Rao product . The Hadamard product of two positive-semidefinite matrices

SECTION 50

#1732904799655

2756-438: The time step. Letting the superscripts d {\displaystyle d} and h {\displaystyle h} refer to the number of input features and number of hidden units, respectively: The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM). Peephole connections allow the gates to access the constant error carousel (CEC), whose activation

2809-431: Was published in 1997 in the journal Neural Computation . By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem . The initial version of LSTM block included cells, input and output gates. ( Felix Gers , Jürgen Schmidhuber, and Fred Cummins, 1999) introduced the forget gate (also called "keep gate") into the LSTM architecture in 1999, enabling the LSTM to reset its own state. This

#654345