Spark Spark Streaming Graphx MLlib Machine learning Spark Slides: 31 Download presentation Spark生态系统 Spark Streaming Graphx MLlib Machine learning Spark Mesos YARN Tachyon HDFS MPI Map. Reduce Spark SQL 二、Spark平台的相关技术 (二)Spark程序运行模式-独立(Standlone)模式 zookeeper Master Slave Task Driver Task Your. Application Executor Slave Task Executor Driver Your Application Operator示例 RDD 1 Operation 1 2 3 partitions 4 5 6 7 RDD 2 Operation 2 3 4 MAP( + 1 ) 5 6 SAVEASTEXTFILE(“HD FS: //…”) 7 8 作用在RDD上的operation Storage System HDFS 一个完整的实例:wordcount import org. apache. spark. _ import Spark. Context. _ object Word. Count { def main(args: Array[String]) { if (args. length != 3 ){ println("usage is org. test. Word. Count <master> <input> <output>") return } Master地址 作业名称 val sc = new Spark. Context(args(0), "Word. Count", Spark安装 目录 System. getenv("SPARK_HOME"), Seq(System. getenv("SPARK_TEST_JAR"))) val text. File = sc. text. File(args(1)) val result = text. File. flat. Map(line => line. split("\s+")) 输入数据所在目录,比如: hdfs: //host: port/input/data. map(word => (word, 1)). reduce. By. Key(_ + _) result. save. As. Text. File(args(2)) } } 数据输出目录,比如: hdfs: //host: port/output/data 依赖的jar包 (三)应用 ✤技术架构和部署架构 Web Service Spark MVC 离 线 加 层 Logback ���理� ����理� Mahout Spark streaming Map/Reduce MLLib JDBC FTP HDFS 具 Sqlite Spark HDFS HBase 数据层访问层 数据层 What is tensor in machine learningApache spark mllibLogistic regression sparkSentiment analysis using pysparkSpark streaming graceful shutdownSpark sql: relational data processing in sparkConcept learning task in machine learningAnalytical learning in machine learningPac learning model in machine learningMachine learning t mitchellInductive and analytical learning in machine learningInductive learningInstance based learning in machine learningInductive learning machine learningFirst order rule learning in machine learningEager learning examplesDeep learning vs machine learningCuadro comparativo de e-learningSpark your learningSpark your learningSpark your learningBall setting and streamingHandling of time in the prime of miss jean brodieDynamic adaptive streaming over http implementationPaul merollaHadoop streaming pythonUnited streaming discoveryStreaming telemetry dataSse simdStreaming potentialSeverance streamingRealtime streaming protocol