Andre Ye - Modern Deep Learning Design and Application Development: Versatile Tools to Solve Deep Learning Problems

This Apress imprint is published by the registered company APress Media, LLC part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

Deep learning is almost universally regarded as falling under the field of computer science, with some intersections in statistics and mathematics broadly, like most computer science disciplines. Thus, upon first impression, it may be perceived to be a strict, exacting science a rigid process of adjusting architectures and methods in a rote fashion to squeeze out a few more percentage points of accuracy on some dataset. While this is arguably a necessary component and period of deep learnings trajectory of growth and development, if embraced too wholeheartedly, it becomes damaging. Such a perspective is lethal toward innovation in deep learning, for many and, possibly, most of the major advances in deep learning were and are not derived through a strict, rigorous scientific process of adjusting but rather through wild and free ideas.

Deep learning, as will be further explicated in this book, has always been somewhat of an experiment-first explanation-after field. That is, the success of a method is almost always explained in fuller detail after the method itself has been empirically observed to be successful. The method itself usually emerges from a hunch or intuition developed through a long series of experimentation and revision rather than a carefully constructed system in which a strong and rational explanation for a methods success precedes its empirical success. This is not, of course, to suggest that to make strides in deep learning one must be messy; it is only to propose that deep learning is a study in a stage guided primarily by the spirit of freedom in thought and experimentation.

Part of a different but seemingly more popular perspective of deep learning as rigid and more or less a continuation of the find the best model paradigm within machine learning, undoubtedly, arises from an immense focus on validation metrics that was spurred in the relatively early development of more advanced and powerful deep learning architectures and methods in the early to middle 2010s and the continuation of this spirit in popular deep learning competitions and competition platforms.

Beyond this phase, though, deep learning is often presented in a way that encourages this problematic perspective. Books, courses, online articles, and other mediums explain deep learning through a series of concrete examples, packaging their primary value as lying within the implementation of said examples. Concrete examples are important and come at little cost in a digital-based field, which is likely why they have become so prized in the computer science context.

The trouble here is that the focus is often too narrowly placed upon specific examples rather than broader concepts. It is very much reflective of the give a man a fish and you feed him for a day, teach him how to fish and you feed him for his lifetime aphorism. Rather than feeding single fish to readers, teaching motion and freedom the act of fishing for new ideas and applications must be a key component in the presentation of deep learning. Examples and implementations that are shown should be anchor points that guide broader conceptual understandings rather than being the sole occupants of the readers understanding.

Think of this in relation to the overfitting paradigm introduced early to machine learning practitioners. If a simple curve fitting model places too high a weight on passing through each point in its dataset, it models the data perfectly but fails to model the underlying phenomena the concept that the data represents. This is overfitting. The model is thus prone to difficulty generalizing the ideas that the data represents, failing to be useful in new applications and contexts. Many of the measures used in classical machine learning as well as deep learning seek to maximize the models performance with respect to modeling the concepts underlying the data rather than the data in and of itself. Likewise, the presentation of deep learning will inevitably be heavily tied to specific examples and developments, but should not consist just of these specific examples, for someone who has learned deep learning this way will likely have difficulty using deep learning in a context outside the narrow range that they learned those examples in. Moreover, they will find it challenging to innovate new ideas and approaches a skill incredibly valuable in such a quickly evolving field.

The goal of this book is to take a step toward this concept-prioritizing vision of deep learning communication , to communicate the ideas of deep learning design, rather than to develop the readers knowledge of deep learning in a manner consisting only of specific architectures and methods with no understanding of how to generalize and find innovative freedom within deep learning.

Make no mistake this goal does not mean that this book is lacking in examples or implementation. The first chapter, A Deep Dive into Keras, is dedicated entirely toward developing a deep understanding of Keras, the deep learning library we will use to realize concepts as implementations and applications. The remainder of the book is organized generally in an intuition/theory-first, implementation-second bipartite structure; the concept is introduced first with examples, analogy, and necessary instruments of theory and intuition, followed by a walk-through of code implementations and variations. Additionally, each of the six chapters following the first contains three case studies, each of which covers a deep learning paper from recent deep learning research in a perspective relevant to the chapters topic, some of which contain code walk-throughs. The objective of the case studies is multifold: to explore the diversity of topics explored in current deep learning research, to provide empirical evidence on the success of certain methods, and to provide examples of deep learning innovations that push the boundaries and correspondingly advance the fields state.