This work is licensed under a Creative Commons

This work is licensed under a Creative Commons Attribution-Share. Alike 4. 0 International License. This presentation is released under Creative Commons. A 6 ribute, on 4. 0 License. You are free to use, distribute and modify it , including for commercial purposes, provided you acknowledge the source.

Spark Source: https: //nptel. ac. in/courses/106104189/

100 times faster than traditional MR Systems

Word count Example

Define Lineage-its log of a coarse grained operations on RDDs. And very much useful in case of fault tolerance

Recovery of RDD In case of failure-Suppose the node that contains the result is failed. In this case Lineage will be used. Spark will retrace map() output and ask to recompute reduce() and generate the result.

Apache Spark can be used with programming languages such as Python, R and Scala. In order to run Spark, cloud-based applications are commonly used; such as Amazon Web Services, Microsoft Azure and Databricks (which provides a free community edition).

Source: https: //www. youtube. com/watch? v=tg. J 0 LSc 3 nc 8

• In Memory Computation: The result of map applied on RDDs are stored in RAM itself and directly provided to reducer. No Hard disk is required so its very fast in comparison to Map. Reduce • Lazy Computation: Spark will log every action that is happening but will not display any output until corresponding action is provoke. • Fault Tolerance: Any lost partition of RDD can be rolled back by applying simple transformation on the last part. • Immutability: The RDDs are immutable because the access provided by RDD is just Read Only. • Partitioning: Partitioning is required for parallel processing. RDDs can be partitioned into many different RDD to keep on different worker nodes. • Persistence (optional): You can save your RDD for future use. By default it is not persistent. • Coarse grained operations like map(), join(), filter(), flatmap() etc.

Any type of data means structured, unstructured or semi structured can be used with Spark SQL

Spark Architecture/Framework

Discussion • A user can submit a Spark job using Spark-submit. Once the Spark job is submitted, Sparkcontext driver program will be opened which will then go to Cluster Master (Manager) Node. • From Cluster Master Node, Containers will be opened in Worker Nodes. Followed by Containers, Executors will open up in Worker Nodes, and these Executors will start interacting with Sparkcontext Driver.

RDDs on Spark Cluster In RDD, Resilient Distributed Datasets, data is partitioned across Worker Nodes by Spark. You can also control the number of partitions created.