频道栏目
首页 > 资讯 > 云计算 > 正文

spark 环境搭建及几种模式测试

16-08-13        来源:[db:作者]  
收藏   我要投稿

spark安装部署

标签(空格分隔): spark


hadoop,spark,kafka交流群:459898801

1,spark环境的安装

创建四个目录

sudo mkdir /opt/modules
sudo mkdir /opt/softwares
sudo mkdir /opt/tools
sudo mkdir /opt/datas

sudo chmod 777 -R /opt/

1,安装jdk1.7

先卸载自带的jdk

rpm –qa | grep java

sudo rpm -e --nodeps (自带java包)

安装jdk1.7

export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

2,spark编译

安装mvn

export MAVEN_HOME=/usr/local/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

3,安装scala

export SCALA_HOME=/opt/modules/scala-2.10.4
export PATH=$PATH:$SCALA_HOME/bin

4,修改mvn镜像源

编译之前先配置镜像及域名服务器,来提高下载速度,进而提高编译速度,用nodepad++打开/opt/compileHadoop/apache-maven-3.0.5/conf/setting.xml。(nodepad已经通过sftp链接到了机器)

  
  nexus-spring
  cdh.repo
  spring
  http://repo.spring.io/repo/
  

  
  nexus-spring2
  cdh.releases.repo
  spring2
  http://repo.spring.io/repo/
  

5,配置域名解析服务器

sudo vi /etc/resolv.conf
添加内容:
    nameserver 8.8.8.8
    nameserver 8.8.4.4

6,编译spark

为了提高编译速度,修改如下内容

VERSION=1.3.0
SPARK_HADOOP_VERSION=2.6.0-cdh5.4.0
SPARK_HIVE=1
#VERSION=$("$MVN" help:evaluate -Dexpression=project.version 2>/dev/null | grep -v "INFO" | tail -n 1)
#SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | tail -n 1)
#SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
#    | grep -v "INFO"\
#    | fgrep --count "hive";\
#    # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
#    # because we use "set -o pipefail"
#    echo -n)

执行编译指令:

./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
去掉下面编译会很快,即使编译失败也不会每次都清除
-DskipTests clean package

spark编译成功后.png-52.8kB

4 安装hadoop2.6

1,添加java主目录位置

hadoop-env.sh
mapred-env.sh
yarn-env.sh
添加如下:
export JAVA_HOME=/opt/modules/jdk1.7.0_67

2,core-site.xml配置

    
        hadoop.tmp.dir
        /opt/modules/hadoop-2.5.0/data/tmp
    
    
        fs.defaultFS
        hdfs://spark.learn.com:8020
    

3,hdfs-site.xml配置

    
        dfs.replication
        1
    

4,mapred-site.xml配置

    
        mapreduce.framework.name
        yarn
    

5,yarn-site.xml配置

    
        yarn.resourcemanager.hostname
        miaodonghua.host
    

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    

6,slaves配置

miaodonghua.host//主机名:nodemanager和datanode地址

7,格式化namenode

bin/hdfs namenode -format

3,spark几种模式的安装部署

1 spark本地模式的安装

本地模式基础语法测试
1,直接运行

bin/spark-shell

运行成功.png-54.4kB
2,spark的webAPP

http://spark.learn.com:4040

webapp.png-73.9kB
3,读取readme.cm

val textFile = sc.textFile("README.md")

读取readmemd.png-25.6kB
4,count

testFile.count()

count.png-82.7kB
5,first

testFile.first()

first.png-64.1kB
6,filter

val linesWithSpark = testFile.filter(line => line.contains("Spark"))

lineFilter.png-5kB

testFile.filter(line => line.contains("Spark")).count()

lineFilter2.png-83.6kB
相对复杂计算 complex computations

testFile.map(line => line.split(" ").size).reduce((a, b) => if (a > b) a else b)

复杂1.png-81.8kB

import java.lang.Math
testFile.map(line => line.split(" ").size).reduce((a, b) => Math.max(a, b))

JAVAmax.png-87.1kB

val wordCounts = testFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)

wordcount.png-8kB

 wordCounts.collect()

显示.png-52.3kB

cache

scala> linesWithSpark.cache()

cache.png-49.7kB

scala> linesWithSpark.count()

cache1.png-14.6kB

2 spark standalone模式的安装

Cluster Mode.png-25.4kB
1,配置spark-env.sh

HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.4.0/etc/hadoop
JAVA_HOME=/opt/modules/jdk1.7.0_67
SCALA_HOME=/opt/modules/scala-2.10.4

SPARK_MASTER_IP=spark.learn.com
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081
SPARK_WORKER_INSTANCES=1

2,配置spark-defaults.conf

spark.master spark://hadoop-spark.com:7077

3,配置slaves

spark.learn.com

4,启动spark

sbin/start-master.sh
sbin/start-slaves.sh

启动成功.png-7kB

5,命令测试
读取本地的需要注释掉

HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.4.0/etc/hadoop
val textFile = sc.textFile("hdfs://spark.learn.com:8020/user/hadoop/spark")

读取hdfs成功.png-28.6kB

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).collect()

输出结果.png-27.8kB

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).sortByKey(true).collect()

输出有序结果.png-23kB

val wordcount = textFile.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey((a,b)=>a+b).map(x=>(x._2,x._1)).sortByKey(false).collect()

按value排序.png-17.6kB

sc.textFile("hdfs://spark.learn.com:8020/user/cyhp/spark/wc.input").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).collect

3 spark on yarn

查看spark-submit参数

[hadoop@spark spark-1.3.0-bin-2.6.0-cdh5.4.0]$ bin/spark-submit --help
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Usage: spark-submit [options]  [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]

Options:
  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.
  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                              on one of the worker machines inside the cluster ("cluster")
                              (Default: client).
  --class CLASS_NAME          Your application's main class (for Java / Scala apps).
  --name NAME                 A name of your application.
  --jars JARS                 Comma-separated list of local jars to include on the driver
                              and executor classpaths.
  --packages                  Comma-separated list of maven coordinates of jars to include
                              on the driver and executor classpaths. Will search the local
                              maven repo, then maven central and any additional remote
                              repositories given by --repositories. The format for the
                              coordinates should be groupId:artifactId:version.
  --repositories              Comma-separated list of additional remote repositories to
                              search for the maven coordinates given with --packages.
  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
                              on the PYTHONPATH for Python apps.
  --files FILES               Comma-separated list of files to be placed in the working
                              directory of each executor.

  --conf PROP=VALUE           Arbitrary Spark configuration property.
  --properties-file FILE      Path to a file from which to load extra properties. If not
                              specified, this will look for conf/spark-defaults.conf.

  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 512M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).

  --proxy-user NAME           User to impersonate when submitting the application.

  --help, -h                  Show this help message and exit
  --verbose, -v               Print additional debug output
  --version,                  Print the version of current Spark

 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).
  --supervise                 If given, restarts the driver on failure.
  --kill SUBMISSION_ID        If given, kills the driver specified.
  --status SUBMISSION_ID      If given, requests the status of the driver specified.

 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
  --executor-cores NUM        Number of cores per executor (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.

1,submit本地模式
ClientMode.png-13.1kB

bin/spark-submit \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10  

submit.png-15.3kB
2,submit运行于standalone模式
Cluster Mode.png-14.2kB

bin/spark-submit \
    --deploy-mode cluster \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10

submit的standalone模式.png-35.6kB

3,submit提交到yarn上
yarn Cluster mode
yarnCluster.png-42.4kB
yarn Client mode
yarn Client.png-107.6kB

bin/spark-submit \
    --master yarn \
    --class org.apache.spark.examples.SparkPi \
    lib/spark-examples-1.3.0-hadoop2.6.0-cdh5.4.0.jar \
    10  

submit到yarn上.png-89.6kB

4,spark监控界面

1,启动spark监控服务

执行下面指令启动spark historyserver

./sbin/start-history-server.sh

查看方式

http://:18080

2,spark监控的配置项介绍

1,整体配置项

我们可以对historyserver进行如下配置:

Environment Variable Meaning
SPARK_DAEMON_MEMORY Memory to allocate to the history server (default: 512m).
SPARK_DAEMON_JAVA_OPTS JVM options for the history server (default: none).
SPARK_PUBLIC_DNS The public address for the history server. If this is not set, links to application history may use the internal address of the server, resulting in broken links (default: none).
SPARK_HISTORY_OPTS spark.history.* configuration options for the history server (default: none).

2,SPARK_HISTORY_OPTS配置

Property Name Default Meaning
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider Name of the class implementing the application history backend. Currently there is only one implementation, provided by Spark, which looks for application logs stored in the file system.
spark.history.fs.logDirectory file:/tmp/spark-events Directory that contains application event logs to be loaded by the history server
spark.history.fs.updateInterval 10 The period, in seconds, at which information displayed by this history server is updated. Each update checks for any changes made to the event logs in persisted storage.
spark.history.retainedApplications 50 The number of application UIs to retain. If this cap is exceeded, then the oldest applications will be removed.
spark.history.ui.port   18080
spark.history.kerberos.enabled false Indicates whether the history server should use kerberos to login. This is useful if the history server is accessing HDFS files on a secure Hadoop cluster. If this is true, it uses the configs spark.history.kerberos.principal and spark.history.kerberos.keytab.
spark.history.kerberos.principal (none) Kerberos principal name for the History Server.
spark.history.kerberos.keytab (none) Location of the kerberos keytab file for the History Server.
spark.history.ui.acls.enable false Specifies whether acls should be checked to authorize users viewing the applications. If enabled, access control checks are made regardless of what the individual application had set for spark.ui.acls.enable when the application was run. The application owner will always have authorization to view their own application and any users specified via spark.ui.view.acls when the application was run will also have authorization to view that application. If disabled, no access control checks are made.

3,标记一个任务是否完成

要注意的是historyserver仅仅显示完成了的spark任务。标记任务完成的一种方式是直接调用sc.stop()。

4,spark-defaults.conf中的配置

Property Name Default Meaning
spark.eventLog.compress false
spark.eventLog.dir file:///tmp/spark-events
spark.eventLog.enabled false

2,具体配置

1,在spark-env.sh

SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://spark.learn.com:8020/user/hadoop/spark/history"

2,在spark-defaults.conf

spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://spark.learn.com:8020/user/hadoop/spark/history
spark.eventLog.compress         true

3,测试

启动相关服务

sbin/start-master.sh
sbin/start-slaves.sh 
sbin/start-history-server.sh
bin/spark-shell

执行spark应用

val textFile = sc.textFile("hdfs://spark.learn.com:8020/user/hadoop/spark/input/")
textFile.count
sc.stop

执行成功.png-25.9kB

 
 
相关TAG标签
上一篇:maven基础篇(6)-maven仓库
下一篇:决策树建模
相关文章
图文推荐

关于我们 | 联系我们 | 广告服务 | 投资合作 | 版权申明 | 在线帮助 | 网站地图 | 作品发布 | Vip技术培训 | 举报中心

版权所有: 红黑联盟--致力于做实用的IT技术学习网站