Sqoop 主要用于在Hadoop(Hive)与传统的数据库(如mysql)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle等)中的数据导进到Hadoop的HDFS中,也可以将HDFS的数据导进到关系型数据库中。
数据库的操作使用可视化工具会更方便,这里使用Navicat。
使用Navicat连接linux中的MySQL数据库出现错误:Access denied for user ‘root’@’localhost’ (using password:YES)
这里是因为MySQL的权限只能本地连接
mysql -uhadoop -p show databases;//第一步 mysql> use mysql//第二步 Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> select host,user from user;//第三步 +-----------+--------+ | host | user | +-----------+--------+ | % | root | | 127.0.0.1 | root | | ::1 | root | | localhost | | | localhost | hadoop | | master | | | master | hadoop | | master | root | +-----------+--------+ 8 rows in set (0.00 sec) mysql> update user set host = '%' where user = 'hadoop' and host='localhost';//第四步 Query OK, 0 rows affected (0.00 sec) Rows matched: 0 Changed: 0 Warnings: 0 mysql> flush privileges;//第五步 Query OK, 0 rows affected (0.00 sec)
Sqoop官网:https://sqoop.apache.org/
下载地址:https://mirrors.hust.edu.cn/apache/sqoop/
从官网看主要有两个版本,1.4.7和1.99.7两者并不兼容。而最新的版本还是16年发布的,两年没更新新版本。刚学大数据后面的知识还没看,猜测Sqoop可能以后用到的不多吧。不过还是得了解它。
这里安装1.99.7版本
下载gz文件上传到linux中并解压,为了方便这里吧解压后的目录改名为sqoop
在解压目录中创建文件夹extra来存放MySQL的驱动jar包
mkdir extra
把mysql的驱动jar文件复制到这个目录下,跟上一篇hive中驱动一样
配置环境变量
vim /etc/profile SQOOP_HOME=/home/chs/software/sqoop SQOOP_SERVER_EXTRA_LIB=/home/chs/software/extra
进入到解压目录中的 conf目录中更改sqoop.properties文件
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/chs/hadoop-2.7.3/etc/hadoop org.apache.sqoop.security.authentication.type=SIMPLE org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler org.apache.sqoop.security.authentication.anonymous=true
配置完成后验证
在安装目录下面
bin/sqoop2-tool verify
Setting conf dir: /home/chs/software/sqoop/bin/../conf Sqoop home directory: /home/chs/software/sqoop Sqoop tool executor: Version: 1.99.7 Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine Running tool: class org.apache.sqoop.tools.tool.VerifyTool 0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server. 14 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/chs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/chs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. Verification was successful. Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.
开启服务器
bin/sqoop2-server start
Setting conf dir: /home/chs/software/sqoop/bin/../conf Sqoop home directory: /home/chs/software/sqoop Starting the Sqoop2 server... 0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server. 6 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/chs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/chs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. Sqoop2 server started.
开启服务器后生成了两个目录
@BASEDIR@ 和 @LOGDIR@
查看运行日志
cd cd @LOGDIR@/ ll
-rw-r--r--. 1 root root 0 May 23 20:10 audit.log -rw-r--r--. 1 root root 669 May 23 20:11 derbyrepo.log -rw-r--r--. 1 root root 81227 May 23 20:11 sqoop.log
关闭服务器
bin/sqoop2-server stop
Setting conf dir: /home/chs/software/sqoop/bin/../conf Sqoop home directory: /home/chs/software/sqoop Stopping the Sqoop2 server... Sqoop2 server stopped.
开启客户端:
bin/sqoop2-shell
Setting conf dir: /home/chs/software/sqoop/bin/../conf Sqoop home directory: /home/chs/software/sqoop May 23, 2018 8:16:06 PM java.util.prefs.FileSystemPreferences$1 run INFO: Created user preferences directory. Sqoop Shell: Type 'help' or '\h' for help. sqoop:000>
退出客户端: :x