Timothy Masters - Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks

Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks — read online for free the complete book (whole text) full work

Below is the text of the book, divided by pages. System saving the place of the last page read, allows you to conveniently read the book "Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks" online for free, without having to search again every time where you left off. Put a bookmark, and you can go to the page where you finished reading at any time.

Light

Font size:

↓

↑

Georgia Tahoma Arial Verdana

Reset

Interval:

↓

↑

Bookmark:

Make

1234567...51

Timothy Masters 2018

Timothy Masters Deep Belief Nets in C++ and CUDA C: Volume 1

1. Introduction

Timothy Masters 1

(1)

Ithaca, New York, USA

This book is intended primarily for readers who already have at least a basic knowledge of neural networks but are interested in learning about, experimenting with, and perhaps even programming deep belief nets. The salient features of this book are the following:

• The book provides motivation for the deep belief net paradigm.
• It presents the most important equations for the most common deep belief net components and justifies them to a modest degree.
• The book provides training, execution, and analysis algorithms for common deep belief net paradigms in language-independent forms.
• This book serves as a detailed users manual for the DEEP program, which is available as a free download from the authors web site. I describe the internal operations of the program in depth.
• The book provides C++ code for many essential deep belief net algorithms. This includes versions for multiple-thread execution on Windows-based computers, as well as CUDA C implementations for using the supercomputer capabilities of NVIDIA CUDA-capable GPU cards.

It must be noted that several items are not included in this book.

• I largely avoid detailed mathematical theory. If we want to understand the quite advanced theory behind deep belief nets, numerous papers are available on the Internet. I will identify a few of the best later in this chapter.
• I present only those models I found to be of greatest practical, real-world value in my own work. This does not imply that the omitted models are inferior, only that I have not found them to be outstandingly useful in my particular applications.

In summary, I have attempted to fill gaps in the public domain material on deep belief nets. Rigorous theory is available in numerous papers, especially those of Dr. Geoffrey Hinton and other pioneers in the field. Reproducing these excellent discussions would be redundant. Also, general statements of basic algorithms are widely available on the Internet, though these are generally devoid of the practical nuances that make the difference between a useful algorithm and something suitable for only toy problems. What appears to be lacking in the public domain are the specific, practical bits of information needed by someone who wants to program deep belief nets and use them to solve real-world problems. This book focuses on such practicalities.

Review of Multiple-Layer Feedforward Networks

A multiple-layer feedforward network is generally illustrated as a stack of layers of neurons, similar to what is shown in Figures . The bottom layer is the input to the network, what would be referred to as the independent variables or predictors in traditional modeling literature. The layer above the input layer is the first hidden layer . Each neuron in this layer attains an activation that is computed by taking a weighted sum of the inputs and then applying a nonlinear function. Each hidden neuron in this layer will have a different set of input weights.

If there is a second hidden layer, the activations of each of its neurons are computed by taking a weighted sum of the activations of the first hidden layer and applying a nonlinear function. This process is repeated for as many hidden layers as desired.

The topmost layer is the output of the network. There are many ways of computing the activations of the output layer, and several of them will be discussed later. For now lets assume that the activation of each output neuron is just a weighted sum of the activations of the neurons in the prior layer, without use of a nonlinear function.

Figure 1-1

A shallow network

Figures show only a small subset of the connections. Actually, every neuron in every layer feeds into every neuron in the next layer above.

Figure 1-2

A deep network

To be more specific, Equation 1-1 shows the activation of a hidden neuron , expressed as a function of the activations of the prior layer. In this equation, x = { x 1, , x K } is the vector of prior-layer activations, w = { w 1, , w K } is the vector of associated weights, and b is a bias term.

(1-1)

Its often more convenient to consider the activation of an entire layer at once. In Equation 1-2, the weight matrix W has K columns, one for each neuron in the prior layer, and as many rows as there are neurons in the layer being computed. The bias and layer inputs are column vectors. The nonlinear activation function is applied element-wise to the vector.

(1-2)

There is one more way of expressing the computation of activations, which is most convenient in some situations. The bias vector b can be a nuisance, so it can be absorbed into the weight matrix W by appending it as one more column at the right side. We then augment the x vector by appending 1.0 to it: x = { x 1, , x K , 1}. The equation for the layers activations then simplifies to the activation function operating on a simple matrix/vector multiplication, as shown in Equation 1-3.

(1-3)

What about the activation function? Traditionally, the hyperbolic tangent function has been used because it has some properties that make training faster. However, for reasons that will become clear later, we will exclusively use the logistic function shown in Equation 1-4 and graphed in Figure .

(1-4)

There are numerous theorems that show the power of a neural network having even a single hidden layer. We will not pursue these here, but know that in a broad class of problems such a network is theoretically capable of solving the problem. Adding a second hidden layer for all practical purposes mops up the few remaining issues. So, its no surprise that multiple-layer feedforward networks are so popular.

Figure 1-3

The logistic activation function

What Are Deep Belief Nets, and Why Do We Like Them?

Prior to the development of neural networks, researchers generally relied on large doses of human intelligence when designing prediction and classification systems. One would measure variables of interest and then brainstorm ways of massaging these raw variables into new variables that (at least in the mind of the researcher) would make it easier for algorithms such as linear discriminant analysis to perform their job. For example, if the raw data were images expressed as arrays of gray-level pixels, one might apply edge detection algorithms or Fourier transforms to the raw image data and feed the results of these intermediate algorithms into a classifier.

Next page

1234567...51

Light

Font size:

↓

↑

Georgia Tahoma Arial Verdana

Reset

Interval:

↓

↑

Bookmark:

Make

Similar books «Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks»

Look at similar books to Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks. We have selected literature similar in name and meaning in the hope of providing readers with more options to find new, interesting, not yet read works.

Géron

Hands-On Machine Learning with Scikit-Learn and Tensorflow: Concepts, Tools, and Techniques to Build Intelligent Systems

Seneque Gareth

Hands-On Deep Learning with Go a practical guide to building and implementing neural network models using Go

Subramanian

Deep learning with PyTorch: a practical approach to building neural network models using PyTorch

Lanham

Hands-on deep learning for games: leverage the power of neural networks and reinforcement learning to build intelligent games

Gianmario Spacagna

Python Deep Learning: Next generation techniques to revolutionize computer vision, AI, speech and data analysis

Yuxi (Hayden) Liu

Hands-On Deep Learning Architectures with Python: Create deep neural networks to solve computational problems using TensorFlow and Keras

Valentino Zocca

Python Deep Learning

Jaegeun Han

Learn CUDA Programming: A beginners guide to GPU programming and parallel computing with CUDA 10.x and C/C++

Dr. PKS Prakash

R Deep Learning Cookbook: Solve complex neural net problems with TensorFlow, H2O and MXNet (English Edition)

Aurélien Géron

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

Gareth Seneque

Hands-On Deep Learning with Go

Shane Cook

CUDA Programming: A Developers Guide to Parallel Computing with GPUs

Discussion, reviews of the book Deep Belief Nets in C++ and CUDA C: Volume 1: Restricted Boltzmann Machines and Supervised Feedforward Networks and just readers' own opinions. Leave your comments, write what you think about the work, its meaning or the main characters. Specify what exactly you liked and what you didn't like, and why you think so.