频道栏目
首页 > 程序开发 > 综合编程 > 其他综合 > 正文
[完]Spark安装学习实践
2016-09-20 09:50:09         来源:无名的博客  
收藏   我要投稿

一、前提

安装Hadoop2.6.0以上; 安装JAVA JDK 1.7以上。

二、下载Spark

官方网站:http://spark.apache.org/downloads.html
1. 选择版本:Spark 1.6.2
2. 选择包类型:Pre-build with user-provided Hadoop [can use with most Hadoop distributions]
3. 选择下载类型:Select Apache Mirror
4. 下载Spark:点击接下来的链接,即可下载

三、安装Spark

假设Spark下载到当前用户的HOME目录下。

# 解压缩
sudo tar -zxf spark-1.6.2-bin-without-hadoop -C /usr/local/
cd /usr/local
sudo mv ./spark-1.6.2-bin-without-hadoop/ ./spark
# 修改权限
sudo chown -R hadoop:hadoop ./spark

配置Spark,修改配置文件spark-env.sh。

cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh

添加配置信息。

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)

配置完成,无需Hadoop那样运行启动命令,可直接使用。使用示例程序,验证Spark是否安装成功。

cd /usr/local/spark
bin/run-example SparkPi
# 2>&1,将所有信息都输出到stdout中
bin/run-example SparkPi 2>&1 | grep "Pi is"

示例程序结果:

hadoop@ubuntu:/usr/local/spark$ bin/run-example SparkPi 2>&1 | grep "Pi is"
Pi is roughly 3.14576

四、使用Spark Shell编写代码

启动Spark Shell,会自动创建为sc的spark context对象和名为sqlContext的sql context对象。
cd /usr/local/spark
bin/spark-shell

运行spark shell后结果:

......
16/09/14 05:18:32 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/09/14 05:18:33 INFO repl.SparkILoop: Created sql context..
SQL context available as sqlContext.
scala> 
加载text文件,spark创建sc,可加载本地文件和HDFS文件创建RDD。
scala> val textFile = sc.textFile("file:///usr/local/spark/README.md")
简单的RDD操作
# 获取RDD文件textFile的第一行内容
scala> textFile.first()
# 获取RDD文件textFile所有项的计数
scala> textFile.count()
# 抽取含有"Spark"的行,返回一个新的RDD
scala> val lineWithSpark = textFile.filter(line => line.contains("Spark"))
# 统计新的RDD的行数
scala> lineWithSpark.count()
# 通过组合RDD操作,实现简易MapReduce操作
scala> textFile.map(line => line.split(" ").size).reduce((a, b) => if (a>b) a else b)
退出Spark Shell,输入exit,或者Ctrl+C,即可退出Spark Shell Scala编写的程序需要使用sbt进行编译打包 Java程序使用Maven编译打包 Python程序则通过spark-submit直接提交

5-1 安装sbt

sbt是Spark用来对Scala程序进行打包的工具。

下载地址:https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.11/sbt-launch.jar 安装在/usr/local/sbt目录下:
sudo mkdir /usr/local/sbt
sudo chown -R hadoop:hadoop /usr/local/sbt
cd /usr/local/sbt
cp ~/sbt-launch.jar .
# 创建sbt脚本
vim ./sbt
脚本sbt中,添加下面内容:
#!/bin/bash
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"
为脚本添加可执行权限
chmod u+x ./sbt
检查sbt是否可用,确保电脑处于联网状态,首次运行会出现“Getting org.scala-sbt sbt 0.13.11 …”的下载信息。
./sbt sbt-version

  出现如下结果,表示安装成功

......
    [SUCCESSFUL ] org.fusesource.jansi#jansi;1.4!jansi.jar (6739ms)
:: retrieving :: org.scala-sbt#boot-scala
    confs: [default]
    5 artifacts copied, 0 already retrieved (24494kB/222ms)
[info] Set current project to sbt (in build file:/usr/local/sbt/)
[info] 0.13.11

5-2 Scala应用程序代码

创建一个文件夹 sparkapp 作为应用程序根目录,在目录下创建一个名为 SimpleApp.scala 的文件。
cd ~
mkdir sparkapp
mkdir -p ./sparkapp/src/main/scala      # 创建所需的文件夹结构
vim ./sparkapp/src/main/scala/SimpleApp.scala
在SimpleApp.scala文件中,编写Scala应用程序代码
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
        def main(args:Array[String]) {
                val logFile = "file:///usr/local/spark/README.md"
                val conf = new SparkConf().setAppName("Simple Application")
                val sc = new SparkContext(conf)
                val logData = sc.textFile(logFile, 2).cache()
                val numAs = logData.filter(line => line.contains("a")).count()
                val numBs = logData.filter(line => line.contains("b")).count()
                println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
        }
}

  该程序用于计算/usr/local/spark/README.md中含有“a”的行数和含有“b”的行数。程序依赖于Spark API,需要使用sbt进行编译打包。

~/sparkapp中新建文件simple.sbt(vim ./sparkapp/simple.sbt),添加下面内容,声明改程序的信息以及与Spark的依赖关系
name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2"

  在上面的配置信息中,scalaVersion用来指定scala的版本,sparkcore用来指定spark的版本,这两个版本信息都可以在之前的启动 Spark shell 的过程中,从如下的屏幕的显示信息中找到。

......
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.2
      /_/

Using Scala version 2.10.5 (OpenJDK Client VM, Java 1.7.0_111)
......

5-3 使用sbt打包Scala程序

检查应用程序的目录结构
cd ~/sparkapp
find .

  文件结构应如下所示:

.
./simple.sbt
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala
将应用程序打包成JAR(首次运行需要下载依赖包),生成的JAR包位置为~/sparkapp/target/scala-2.10/simple-project_2.10-1.0.jar
hadoop@ubuntu:~/sparkapp$ /usr/local/sbt/sbt package
......
[info] Packaging /home/hadoop/sparkapp/target/scala-2.10/simple-project_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 7 s, completed Sep 17, 2016 11:31:28 PM
通过spark-submit运行程序
# 显示完整信息
hadoop@ubuntu:~/sparkapp$ /usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp/target/scala-2.10/simple-project_2.10-1.0.jar 
16/09/17 23:50:00 INFO spark.SparkContext: Running Spark version 1.6.2
16/09/17 23:50:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
......
16/09/17 23:50:12 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
Lines with a: 58, Lines with b: 26
16/09/17 23:50:12 INFO spark.SparkContext: Invoking stop() from shutdown hook
......
16/09/17 23:50:13 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
# 显示所需要的信息
hadoop@ubuntu:~/sparkapp$ /usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp/target/scala-2.10/simple-project_2.10-1.0.jar 2>&1 | grep "Lines with a:"
Lines with a: 58, Lines with b: 26

六、Java独立应用编程

6-1 安装Maven

下载
去官网下载 https://maven.apache.org/download.cgi wget http://apache.fayea.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip 安装
sudo unzip apache-maven-3.3.9-bin.zip -d /usr/local
cd /usr/local
sudo mv apache-maven-3.3.9/ maven
/usr/local$ sudo chown -R hadoop:hadoop maven/

6-2 Java应用程序代码

进入HOME目录,创建相关目录,建立SimpleApp.java文件
cd ~
mkdir -p sparkapp2/src/main/java
vim sparkapp2/src/main/java/Simple.java
SimpleApp.java文件中添加如下代码:
import org.apache.spark.api.java.*;
import org.apache.spark.api.java.function.Function;

public class SimpleApp {
        public static void main(String[] args){
                String logFile = "file:///usr/local/spark/README.md";
                JavaSparkContext sc = new JavaSparkContext("local", "Simple App", "file:///usr/local/spark/", 
                    new String[]{"target/simple-project-1.0.jar"});
                JavaRDD logData = sc.textFile(logFile).cache();
                long numAs = logData.filter(new Function() {
                        public Boolean call(String s) {
                                return s.contains("a");
                        }
                }).count();

                long numBs = logData.filter(new Function() {
                        public Boolean call(String s) {
                                return s.contains("b");
                        }
                }).count();

                System.out.println("Lines with a: " + numAs + ", Lines with b: " + numBs);
        }
}
该程序依赖Spark Java API,需要通过Maven进行编译打包。在./sparkapp2中新建文件pom.xml(vim ~/sparkapp2/pom.xml),添加下面内容,声明该程序信息以及与Spark的依赖关系:

        edu.berkeley
        simple-project
        4.0.0
        Simple Project
        jar
        1.0
        
                
                        Akka repository
                        http://repo.akka.io/releases
                
        
        
                
                        org.apache.spark
                        spark-core_2.11
                        2.0.0-preview
                
        

6-3 使用maven打包java程序

检查应用程序文件结构
hadoop@ubuntu:~/sparkapp2$ find
.
./src
./src/main
./src/main/java
./src/main/java/Simple.java
./pom.xml
将应用程序打包成JAR文件(首次运行需要下载依赖包,需要联网,消耗一定的时间):
hadoop@ubuntu:~/sparkapp2$ /usr/local/maven/bin/mvn package
......
[INFO] Building jar: /home/hadoop/sparkapp2/target/simple-project-1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 32.926 s
[INFO] Finished at: 2016-09-18T18:59:14-07:00
[INFO] Final Memory: 26M/63M
[INFO] ------------------------------------------------------------------------
通过spark-submit运行程序
hadoop@ubuntu:~/sparkapp2$ /usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp2/target/simple-project-1.0.jar
......
hadoop@ubuntu:~/sparkapp2$ /usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp2/target/simple-project-1.0.jar 2>&1 | grep "Lines with a"
Lines with a: 58, Lines with b: 26
点击复制链接 与好友分享!回本站首页
上一篇:Hibernate查询、连接池、二级缓存
下一篇:第17 & 18 章 Jenkins –服务器维护 & 持续部署
相关文章
图文推荐
文章
推荐
点击排行

关于我们 | 联系我们 | 广告服务 | 投资合作 | 版权申明 | 在线帮助 | 网站地图 | 作品发布 | Vip技术培训 | 举报中心

版权所有: 红黑联盟--致力于做实用的IT技术学习网站