Book contents
- Frontmatter
- Contents
- Contributors
- Preface
- 1 Scaling Up Machine Learning: Introduction
- Part One Frameworks for Scaling Up Machine Learning
- 2 MapReduce and Its Application to Massively Parallel Learning of Decision Tree Ensembles
- 3 Large-Scale Machine Learning Using DryadLINQ
- 4 IBM Parallel Machine Learning Toolbox
- 5 Uniformly Fine-Grained Data-Parallel Computing for Machine Learning Algorithms
- Part Two Supervised and Unsupervised Learning Algorithms
- Part Three Alternative Learning Settings
- Part Four Applications
- Subject Index
- References
3 - Large-Scale Machine Learning Using DryadLINQ
from Part One - Frameworks for Scaling Up Machine Learning
Published online by Cambridge University Press: 05 February 2012
- Frontmatter
- Contents
- Contributors
- Preface
- 1 Scaling Up Machine Learning: Introduction
- Part One Frameworks for Scaling Up Machine Learning
- 2 MapReduce and Its Application to Massively Parallel Learning of Decision Tree Ensembles
- 3 Large-Scale Machine Learning Using DryadLINQ
- 4 IBM Parallel Machine Learning Toolbox
- 5 Uniformly Fine-Grained Data-Parallel Computing for Machine Learning Algorithms
- Part Two Supervised and Unsupervised Learning Algorithms
- Part Three Alternative Learning Settings
- Part Four Applications
- Subject Index
- References
Summary
This chapter describes DryadLINQ, a general-purpose system for large-scale data-parallel computing, and illustrates its use on a number of machine learning problems.
The main motivation behind the development of DryadLINQ was to make it easier for nonspecialists to write general-purpose, scalable programs that can operate on very large input datasets. In order to appeal to nonspecialists, we designed the programming interface to use a high level of abstraction that insulates the programmer from most of the detail and complexity of parallel and distributed execution. In order to support general-purpose computing, we embedded these high-level abstractions in .NET, giving developers access to full-featured programming languages with rich type systems and proven mechanisms (such as classes and libraries) for managing complex, long-lived, and geographically distributed software projects. In order to support scalability over very large data and compute clusters, the DryadLINQ compiler generates code for the Dryad runtime, a well-tested and highly efficient distributed execution engine.
As machine learning moves into the industrial mainstream and operates over diverse data types including documents, images, and graphs, it is increasingly appealing to move away from domain-specific languages like MATLAB and toward general-purpose languages that support rich types and standardized libraries. The examples in this chapter demonstrate that a general-purpose language such as C# supports effective, concise implementations of standard machine learning algorithms and that DryadLINQ efficiently scales these implementations to operate over hundreds of computers and very large datasets primarily limited by disk capacity.
- Type
- Chapter
- Information
- Scaling up Machine LearningParallel and Distributed Approaches, pp. 49 - 68Publisher: Cambridge University PressPrint publication year: 2011
References
- 2
- Cited by