Apache Spark

Apache Spark
Original author(s) Matei Zaharia
Developer(s) Apache Software Foundation, UC Berkeley AMPLab, Databricks
Initial release May 30, 2014 (2014-05-30)
Stable release v1.6.1 / March 9, 2016 (2016-03-09)
Development status Active
Written in Scala, Java, Python, R
Operating system Microsoft Windows, OS X, Linux
Type Data analytics, machine learning algorithms
License Apache License 2.0
Website spark.apache.org

Apache Spark is an open source cluster computing framework. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation that has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance.

Overview

Spark provides programmers with an application programming interface centered on a data structure called the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.[1] It was developed in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.[2]

The availability of RDDs facilitates the implementation of both iterative algorithms, that visit their dataset multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications (compared to Apache Hadoop, a popular MapReduce implementation) may be reduced by several orders of magnitude.[1][3] Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Spark.[4]

Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster), Hadoop YARN, or Apache Mesos.[5] For distributed storage, Spark can interface with a wide variety, including Hadoop Distributed File System (HDFS),[6] MapR File System (MapR-FS),[7] Cassandra,[8] OpenStack Swift, Amazon S3, Kudu, or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

Spark Core

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities, exposed through an application programming interface (for Java, Python, Scala, and R) centered on the RDD abstraction. This interface mirrors a functional/higher-order model of programming: a "driver" program invokes parallel operations such as map, filter or reduce on an RDD by passing a function to Spark, which then schedules the function's execution in parallel on the cluster.[1] These operations, and additional ones such as joins, take RDDs as input and produce new RDDs. RDDs are immutable and their operations are lazy; fault-tolerance is achieved by keeping track of the "lineage" of each RDD, the sequence of operations produced it, so that it can be reconstructed in the case of data loss. RDDs can contain any type of Python, Java, or Scala objects.

Aside from the RDD-oriented functional style of programming, Spark provides two restricted forms of shared variables: broadcast variables reference read-only data that needs to be available on all nodes, while accumulators can be used to program reductions in an imperative style.[1]

A typical example of RDD-centric functional programming is the following Scala program that computes the frequencies of all words occurring in a set of text files and prints the most common ones. Each map, flatMap (a variant of map) and reduceByKey takes an anonymous function that performs a simple operation on a single data item (or a pair of items), and applies its argument to transform an RDD into a new RDD.

  val conf = new SparkConf().setAppName("wiki_test")       // create a spark config object
  val sc = new SparkContext(conf)                          // Create a spark context
  val data = sc.textFile("/path/to/somedir")               // Read files from "somedir" into an RDD of (filename, content) pairs.
  val tokens = data.map(_.split(" "))                      // Split each file into a list of tokens (words).
  val wordFreq = tokens.map((_, 1)).reduceByKey(_ + _)     // Add a count of one to each token, then sum the counts per word type.
  wordFreq.sortBy(s=>s._2).map(x => (x._2, x._1)).top(10)  // Get the top 10 words. Swap word and count to sort by count.

Spark SQL

Spark SQL is a component on top of Spark Core that introduces a new data abstraction called DataFrames,[lower-alpha 1] which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language to manipulate DataFrames in Scala, Java, or Python. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server.

    import org.apache.spark.sql.SQLContext

    val url = "jdbc:mysql://yourIP:yourPort/test?user=yourUsername;password=yourPassword"   // URL for your database server.
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)              // Create a sql context object

    val df = sqlContext
      .read
      .format("jdbc")
      .option("url", url)
      .option("dbtable", "people")
      .load()

    df.printSchema() // Looks the schema of this DataFrame.
    val countsByAge = df.groupBy("age").count() // Counts people by age

Spark Streaming

Spark Streaming leverages Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD transformations on those mini-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, on a single engine.

MLlib Machine Learning Library

Spark MLlib is a distributed machine learning framework on top of Spark Core that, due in large part of the distributed memory-based Spark architecture, is as much as nine times as fast as the disk-based implementation used by Apache Mahout (according to benchmarks done by the MLlib developers against the Alternating Least Squares (ALS) implementations, and before Mahout itself gained a Spark interface), and scales better than Vowpal Wabbit.[9] Many common machine learning and statistical algorithms have been implemented and are shipped with MLlib which simplifies large scale machine learning pipelines, including:

GraphX

GraphX is a distributed graph processing framework on top of Spark. It provides an API for expressing graph computation that can model the Pregel abstraction. It also provides an optimized runtime for this abstraction.

Like Spark, GraphX initially started as a research project at UC Berkeley's AMPLab and Databricks, and was later donated to the Apache Software Foundation and the Spark project.[11]

History

Spark was initially started by Matei Zaharia at UC Berkeley's AMPLab in 2009, and open sourced in 2010 under a BSD license.

In 2013, the project was donated to the Apache Software Foundation and switched its license to Apache 2.0. In February 2014, Spark became a Top-Level Apache Project.[12]

In November 2014, Spark founder M. Zaharia's company Databricks set a new world record in large scale sorting using Spark.[13]

Spark had in excess of 1000 contributors in 2015,[14] making it one of the most active projects in the Apache Software Foundation[15] and one of the most active open source big data projects.[16]

Version Original release date Latest version Release date
Old version, no longer supported: 0.5 2012-06-12 0.5.1 2012-10-07
Old version, no longer supported: 0.6 2012-10-14 0.6.1 2012-11-16
Old version, no longer supported: 0.7 2013-02-27 0.7.3 2013-07-16
Old version, no longer supported: 0.8 2013-09-25 0.8.1 2013-12-19
Old version, no longer supported: 0.9 2014-02-02 0.9.2 2014-07-23
Old version, no longer supported: 1.0 2014-05-30 1.0.2 2014-08-05
Old version, no longer supported: 1.1 2014-09-11 1.1.1 2014-11-26
Old version, no longer supported: 1.2 2014-12-18 1.2.2 2015-04-17
Current stable version: 1.3 2015-03-13 1.3.1 2015-04-17
Current stable version: 1.4 2015-06-11 1.4.1 2015-07-15
Current stable version: 1.5 2015-09-09 1.5.2 2015-11-09
Current stable version: 1.6 2016-01-04 1.6.1 2016-03-09
Future release: 2.0 2016 2.0.0 2016
Legend:
Old version
Older version, still supported
Latest version
Latest preview version
Future release

Notes

  1. ↑ Called SchemaRDDs before Spark 1.3.

References

  1. 1 2 3 4 Zaharia, Matei; Chowdhury, Mosharaf; Franklin, Michael J.; Shenker, Scott; Stoica, Ion. Spark: Cluster Computing with Working Sets (PDF). USENIX Workshop on Hot Topics in Cloud Computing (HotCloud).
  2. ↑ Zaharia, Matei; Chowdhury, Mosharaf; Das, Tathagata; Dave, Ankur; Ma,, Justin; McCauley, Murphy; J., Michael; Shenker, Scott; Stoica, Ion. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (PDF). USENIX Symp. Networked Systems Design and Implementation.
  3. ↑ Xin, Reynold; Rosen, Josh; Zaharia, Matei; Franklin, Michael; Shenker, Scott; Stoica, Ion (June 2013). "Shark: SQL and Rich Analytics at Scale" (PDF).
  4. ↑ Harris, Derrick (28 June 2014). "4 reasons why Spark could jolt Hadoop into hyperdrive". Gigaom.
  5. ↑ "Cluster Mode Overview - Spark 1.2.0 Documentation - Cluster Manager Types". apache.org. Apache Foundation. 2014-12-18. Retrieved 2015-01-18.
  6. ↑ Figure showing Spark in relation to other open-source Software projects including Hadoop
  7. ↑ MapR ecosystem support matrix
  8. ↑ Doan, DuyHai (2014-09-10). "Re: cassandra + spark / pyspark". Cassandra User (Mailing list). Retrieved 2014-11-21.
  9. ↑ Sparks, Evan; Talwalkar, Ameet (2013-08-06). "Spark Meetup: MLbase, Distributed Machine Learning with Spark". slideshare.net. Spark User Meetup, San Francisco, California. Retrieved 10 February 2014.
  10. ↑ "MLlib | Apache Spark". spark.apache.org. Retrieved 2016-01-18.
  11. ↑ Gonzalez, Joseph; Xin, Reynold; Dave, Ankur; Crankshaw, Daniel; Franklin, Michael; Stoica, Ion (Oct 2014). "GraphX: Graph Processing in a Distributed Dataflow Framework" (PDF).
  12. ↑ "The Apache Software Foundation Announces Apache&#8482 Spark&#8482 as a Top-Level Project". apache.org. Apache Software Foundation. 27 February 2014. Retrieved 4 March 2014.
  13. ↑ Spark officially sets a new record in large-scale sorting
  14. ↑ Open HUB Spark development activity
  15. ↑ "The Apache Software Foundation Announces Apache&#8482 Spark&#8482 as a Top-Level Project". apache.org. Apache Software Foundation. 27 February 2014. Retrieved 4 March 2014.
  16. ↑ Introduction to Apache Spark

External links

This article is issued from Wikipedia - version of the Friday, May 06, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.