Jiawei Jiang - Distributed Machine Learning and Gradient Optimization

The big data paradigm presents a number of challenges for university curricula on big data or data science related topics. On the one hand, new research, tools and technologies are currently being developed to harness the increasingly large quantities of data being generated within our society. On the other, big data curricula at universities are still based on the computer science knowledge systems established in the 1960s and 70s. The gap between the theories and applications is becoming larger, as a result of which current education programs cannot meet the industry's demands for big data talents.

The scope of the series includes, but is not limited to, titles in the areas of database management, data mining, data analytics, search engines, data integration, NLP, knowledge graphs, information retrieval, social networks, etc. Other relevant topics will also be considered.

More information about this series at https://link.springer.com/bookseries/15869

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.

The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

In recent years, with the rapid development of technologies in many industrial applications, such as online shopping, social networks, intelligent healthcare, and IOTs, we have witnessed an explosive increase of generated data. To mine buried knowledge from raw data, machine learning technology is a widely used tool and has become the de facto technique for data analytics in both academia and industry, especially for unstructured data that is beyond the understanding of human beings. Training machine learning models using a single machine has been well-studied, either in a sequential way or in a multicore manner. However, in the data-intensive applications listed earlier, many real datasets can be up to hundreds of terabytes or even petabytes, which have far overwhelmed the computation and storage capability of a single physical machine. Traditional stand-alone machine learning training methods have encountered great challenges accordingly. In order to meet the trend of big data and solve the bottleneck of a stand-alone system, many researchers and practitioners resort to training machine learning models over a set of distributed machines, yielding a new research area in academiadistributed machine learning. Specifically, a class of supervised machine learning algorithms is often solved by a series of iterative gradient-based optimization algorithms, such as stochastic gradient descent (SGD), due to their convergence guarantee and ease of parallelization. These machine learning models include linear regression, logistic regression, support vector machine, gradient boosting decision tree, neural networks, and so on. For these machine learning models, the key to training them in a distributed way is to execute the gradient optimization algorithm in parallel rather than the original sequential execution. Although there is a rich literature on gradient optimization algorithms proposed for distributed machine learning, there is a lack of work that presents a comprehensive overview that elaborates the concepts, principles, basic building blocks, methodology, taxonomy, and recent progress.