Extremely high performance. To achieve high performance, BigDL uses Intel® Math Kernel Library (Intel® MKL) and multithreaded programming in each Spark task. BigDL can efficiently scale out to perform data analytics at “big data scale” by using Spark as well as efficient implementations of synchronous stochastic gradient descent (SGD) and all-reduce communications in Spark. Modeled after Torch BigDL provides comprehensive support for deep learning, including numeric computing (via Tensor and high-level neural networks in addition, you can load pretrained Caffe* or Torch models into the Spark framework, and then use the BigDL library to run inference applications on their data. You can write deep learning applications as Scala or Python programs. Deep learning as a pipeline of feature extractors What Is BigDL?īigDL is a distributed deep learning library for Spark that can run directly on top of existing Spark or Apache Hadoop* clusters. These multiple phases in a deep learning pipeline increase the overall prediction accuracy of the resulting model through hierarchical feature extraction.įigure 1. Where traditional approaches tend to focus on feature extraction as a single phase for training a machine learning model, deep learning introduces depth by creating a pipeline of feature extractors in the model. These methods are based on artificial neural network topologies and can scale with larger data sets.įigure 1 explores the fundamental difference between traditional and deep learning approaches. What Is Deep Learning?ĭeep learning is a branch of machine learning that uses algorithms to model high-level abstractions in data. Labels on spark for mac how to#
This article introduces BigDL, shows you how to build the library on a variety of platforms, and provides examples of BigDL in action. Using BigDL, you can write deep learning applications as Scala or Python* programs and take advantage of the power of scalable Spark clusters.
BigDL is a distributed deep learning library for Apache Spark*.
0 kommentar(er)
