Unfolding Computational Graphs

Recurrent Neural Network

Some examples of important design patterns for recurrent neural networks include the following:

• Recurrent networks that produce an output at each time step and have recurrent connections between hidden units

• Recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step to the hidden units at the next time step

• Recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output

Teacher Forcing and Networks with Output Recurrence

Models that have recurrent connections from their outputs leading back into the model may be trained with teacher forcing. Teacher forcing is a procedure that emerges from the maximum likelihood criterion, in which during training the model receives the ground truth output y(t) as input at time t + 1.

Modeling Sequence Conditioned on Context with RNN

The prediction value (y(t)) stationary needs to be verified.

RNN can take a sequence of vector x as input, it can also take in a fix number of inputs as inputs:

When input x is a sequence: