LazyProgrammer - Deep Learning: Recurrent Neural Networks in Python: LSTM, GRU, and more RNN machine learning architectures in Python and Theano (Machine Learning in Python)

Recurrent Neural Networks are all about learning sequences. You may have already learned about Markov models for sequence modeling, which make use of the Markov assumption, p{ x(t) | x(t-1), ..., x(1) } = p{ x(t) | x(t-1) }. In other words, the current value depends only on the last value. While easy to train, one can imagine this may not be very realistic. Ex. the previous word in a sentence is and, whats the next word? Whereas Markov Models are limited by the Markov assumption, Recurrent Neural Networks are not.

As a result, they are more expressive, and more powerful than anything weve seen on tasks that we havent made progress on in decades. In the first section of the book we are going to add time to our neural networks. Ill introduce you to the Simple Recurrent Unit, also known as the Elman unit. The Simple Recurrent Unit will help you understand the basics of recurrent neural networks - the types of tasks they can be used for, how to construct the objective functions for these tasks, and backpropagation through time. We are going to revisit a classical neural network problem - the XOR problem, but were going to extend it so that it becomes the parity problem - youll see that regular feedforward neural networks will have trouble solving this problem but recurrent networks will work because the key is to treat the input as a sequence. In the next section of the course, we are going to revisit one of the most popular applications of recurrent neural networks - language modeling, which plays a large role in natural language processing or NLP.

Another popular application of neural networks for language is word vectors or word embeddings. The most common technique for this is called Word2Vec, but Ill show you how recurrent neural networks can also be used for creating word vectors. In the section after, well look at the very popular LSTM, or long short-term memory unit, and the more modern and efficient GRU, or gated recurrent unit, which has been proven to yield comparable performance. Well apply these to some more practical problems, such as learning a language model from Wikipedia data and visualizing the word embeddings we get as a result. One tip for getting through this book: Understand the mechanics first, worry about the meaning later. When we talk about LSTMs were going to discuss its ability to remember and forget things - keep in mind these are just convenient names by way of analogy.

Were not actually building something thats remembering or forgetting - they are just mathematical formulas. So worry about the math, and let the meaning come naturally to you. What you especially dont want to do is the opposite - try to understand the meaning without understanding the mechanics. When you do that, the result is usually a sensationalist media article or a pop-science book - this book is the opposite of that. We want to understand on a technical level whats happening - explaining things in layman terms or thinking of real-life analogies is icing on the cake only if you understand the technicalities. All the code for this class is hosted on Github, and you can get it from https://github.com/lazyprogrammer/machine_learning_examples in the folder rnn_class.

Git is a version control system that allows me to push updates and keep a history of all the changes. You should always do a git pull to make sure you have the latest version.