Spark

Spark is a large-scale data computing engine, which is written by Scala.

Features of Apache Spark

1. Faster speed: In memory,theoretically, the speed of Spark is 100 times faster than Speed of Hadoop
2. In-memory computing engine: provides Cache mechanism to support continuous iterations or retrieve shared data, reduce cost of I/O of data.
3. DAG engine: reduce cost of HDFS Read/Write during computing
4. Use multi-threaded pool to reduce cost of task startup.
5. Easy to use:
   over 80 high-level operators.
   supports Java, Python, Scala, R
   the amount of code is 2 to 5 times less than MapReduce’s
6. Generalization: provides many libraries like Spark SQL, Spark MLlib, Spark GraphX, Spark Streaming.
7. Supports multiple resource schedulers: Hadoop Yarn, Apache Mesos, and Standalone cluster scheduler.

Resilient Distributed Datasets(RDD) : Key Point

1. Collections in cluster, which is read only and consists of Partitions.
2. Stores in disk or memory.
3. Operations is composed by transformation and action
4. Automatically reorganization if it fails.

Features of Transformation and Action

1. Different APIs: T: RDD[X]->RDD[Y]; A:RDD[X]->Z(Z is probably a basic data type like array)
2. Lazy actions:
   Transformation only record RDD transform relation
   Action triggers executions of program

Spark Streaming

Splits streaming data into small pieces and processes batch data. It will reduce latency.

Difference between MR and Spark

MR:

1. Only privides Map and Reduce operations
2. Low efficiency of processing: bad at iterating computing, interface processing, and log analysis
3. The middle result of Map needs to write into disk and Reduce needs to write into HDFS.
4. High cost of Task Scheduling and Startups.
5. Cannot fully use of memory
6. Map and Reduce need to sort data.

Implements of Spark

1. Log collecting and searching
2. User’s tap prediction
3. Common friends
4. ETL task of SparkSQL and DAG

Reference

Spark: Computing Engine