sqoop错误解决

./sqoop import --connect jdbc:mysql://localhost:3306/xxxx--username dba--password123456--direct --table ehm_hosts--target-dir /data/ehm_hosts -m1出现错误: java.net.ConnectExcept

黑肚皮的窝

32598人浏览 · 2013-06-25 16:51:21

黑肚皮的窝 · 2013-06-25 16:51:21 发布

 ./sqoop import --connect jdbc:mysql://localhost:3306/xxxx  --username dba  --password  123456  --direct --table ehm_hosts  --target-dir /data/ehm_hosts -m1

出现错误:

java.net.ConnectException
MESSAGE: Connection refused

STACKTRACE:

java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at java.net.Socket.<init>(Socket.java:375)
        at java.net.Socket.<init>(Socket.java:218)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:173)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:267)
        at com.mysql.jdbc.Connection.createNewIO(Connection.java:2739)
        at com.mysql.jdbc.Connection.<init>(Connection.java:1553)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:278)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)


** END NESTED EXCEPTION **



Last packet sent to the server was 22 ms ago.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:193)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
        ... 9 more
Caused by: com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: 

** BEGIN NESTED EXCEPTION ** 

java.net.ConnectException
MESSAGE: Connection refused

STACKTRACE:

java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at java.net.Socket.<init>(Socket.java:375)
        at java.net.Socket.<init>(Socket.java:218)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:173)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:267)
        at com.mysql.jdbc.Connection.createNewIO(Connection.java:2739)
        at com.mysql.jdbc.Connection.<init>(Connection.java:1553)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:278)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:162)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)


** END NESTED EXCEPTION **



Last packet sent to the server was 22 ms ago.
        at com.mysql.jdbc.Connection.createNewIO(Connection.java:2814)
        at com.mysql.jdbc.Connection.<init>(Connection.java:1553)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:278)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:187)
        ... 10 more

换成:

./sqoop import --connect jdbc:mysql://192.168.205.101:3306/xxxx  --username dba  --password  123456  --direct --table ehm_hosts  --target-dir /data/ehm_hosts -m1

问题解决!

出现问题:

3/06/26 00:37:18 INFO mapred.JobClient: Task Id : attempt_201306250027_0021_m_000000_0, Status : FAILED
java.io.IOException: Cannot run program "mysqldump": java.io.IOException: error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
        at java.lang.Runtime.exec(Runtime.java:593)
        at java.lang.Runtime.exec(Runtime.java:466)
        at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:403)
        at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:47)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
        ... 12 more

解决方案:

看mR日志.

50027_0021_m_000000_0	task_201306250027_0021_m_000000	slave1	FAILED	

java.io.IOException: Cannot run program "mysqldump": java.io.IOException: error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
	at java.lang.Runtime.exec(Runtime.java:593)
	at java.lang.Runtime.exec(Runtime.java:466)
	at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:403)
	at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:47)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
	at java.lang.ProcessImpl.start(ProcessImpl.java:65)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
	... 12 more

导出数据在slave1上，所以需要在slave1上安装mysqldump.安装后，执行成功。

Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: $HADOOP_HOME is deprecated.

13/06/26 00:51:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/06/26 00:51:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/06/26 00:51:23 INFO tool.CodeGenTool: Beginning code generation
13/06/26 00:51:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `ehm_hosts` AS t LIMIT 1
13/06/26 00:51:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `ehm_hosts` AS t LIMIT 1
13/06/26 00:51:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr
Note: /tmp/sqoop-hadoop/compile/a067fc87107ca67800cb30e3e4bd56f9/ehm_hosts.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/06/26 00:51:27 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/a067fc87107ca67800cb30e3e4bd56f9/ehm_hosts.jar
13/06/26 00:51:27 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/06/26 00:51:27 INFO mapreduce.ImportJobBase: Beginning import of ehm_hosts
13/06/26 00:51:30 INFO mapred.JobClient: Running job: job_201306250027_0023
13/06/26 00:51:31 INFO mapred.JobClient:  map 0% reduce 0%
13/06/26 00:51:51 INFO mapred.JobClient:  map 100% reduce 0%
13/06/26 00:51:56 INFO mapred.JobClient: Job complete: job_201306250027_0023
13/06/26 00:51:56 INFO mapred.JobClient: Counters: 18
13/06/26 00:51:56 INFO mapred.JobClient:   Job Counters 
13/06/26 00:51:56 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=19622
13/06/26 00:51:56 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/26 00:51:56 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/06/26 00:51:56 INFO mapred.JobClient:     Launched map tasks=1
13/06/26 00:51:56 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
13/06/26 00:51:56 INFO mapred.JobClient:   File Output Format Counters 
13/06/26 00:51:56 INFO mapred.JobClient:     Bytes Written=332
13/06/26 00:51:56 INFO mapred.JobClient:   FileSystemCounters
13/06/26 00:51:56 INFO mapred.JobClient:     HDFS_BYTES_READ=87
13/06/26 00:51:56 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=31929
13/06/26 00:51:56 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=332
13/06/26 00:51:56 INFO mapred.JobClient:   File Input Format Counters 
13/06/26 00:51:56 INFO mapred.JobClient:     Bytes Read=0
13/06/26 00:51:56 INFO mapred.JobClient:   Map-Reduce Framework
13/06/26 00:51:56 INFO mapred.JobClient:     Map input records=1
13/06/26 00:51:56 INFO mapred.JobClient:     Physical memory (bytes) snapshot=62537728
13/06/26 00:51:56 INFO mapred.JobClient:     Spilled Records=0
13/06/26 00:51:56 INFO mapred.JobClient:     CPU time spent (ms)=1360
13/06/26 00:51:56 INFO mapred.JobClient:     Total committed heap usage (bytes)=16252928
13/06/26 00:51:56 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=417800192
13/06/26 00:51:56 INFO mapred.JobClient:     Map output records=3
13/06/26 00:51:56 INFO mapred.JobClient:     SPLIT_RAW_BYTES=87
13/06/26 00:51:56 INFO mapreduce.ImportJobBase: Transferred 332 bytes in 28.6117 seconds (11.6037 bytes/sec)
13/06/26 00:51:56 INFO mapreduce.ImportJobBase: Retrieved 3 records.

问题:

Note: Recompile with -Xlint:deprecation for details.
13/06/26 00:49:52 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/2a8b237421375fb4414d761cf7c7c998/ehm_hosts.jar
13/06/26 00:49:52 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/06/26 00:49:52 INFO mapreduce.ImportJobBase: Beginning import of ehm_hosts
13/06/26 00:49:54 INFO mapred.JobClient: Cleaning up the staging area hdfs://master:9000/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201306250027_0022
13/06/26 00:49:54 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /data/ehm_hosts already exists
13/06/26 00:49:54 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /data/ehm_hosts already exists
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:887)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
        at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
        at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
        at org.apache.sqoop.manager.DirectMySQLManager.importTable(DirectMySQLManager.java:92)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

解决方案:

由于目录在hdfs上建立了，必须删除。

 hadoop fs -rmr  /data/ehm_hosts

欢迎使用若依开源

快速构建 Web 应用程序

更多推荐

深入 Composer autoload

这几天看到 phphub 上面有人开始进坑怒看 laravel 源代码，于是我也凑个热闹来看下这个故事。众所周知 composer 是现代PHP 项目的基石，与古老的 pear 不同， composer 并不是一款专注于系统级别php 管理的包管理系统，而是基于项目的一个库管理系统。这就好比 npminstall -g 和 npminstall 的区别。而且最主要

RuoYi 若依

实体类（VO，DO，DTO）的划分

经常会接触到VO，DO，DTO的概念，本文从领域建模中的实体划分和项目中的实际应用情况两个角度，对这几个概念进行简析。得出的主要结论是：在项目应用中，VO对应于页面上需要显示的数据（表单），DO对应于数据库中存储的数据（数据表），DTO对应于除二者之外需要进行传递的数据。一、实体类百度百科中对于实体类的定义如下：实体类的主要职责是存储和管理系统内部的信息，它也可以有行为，甚至很复杂的行为，但这些行

RuoYi 若依

计算机课如何断开学生端,极域课堂管理系统怎么连接老师学生端连接问题解决方法...

现在的大学课堂都紧跟着时代的步伐，选择用电子教室代替传统的教室，电子教室的好处就是可以让坐在角落的同学也能清楚的看到老师的每一个操作，除此之外还可以电子举手，提交作业获得老师的及时反馈，学习效率倍增。如果你发现自己的电脑出现连接不上电脑的情况的，来看看小编是怎么解决的把！类别：教育管理大小：16.38M 语言：简体中文评分：61、首先，正常的学生机器是开机就会自动的连接老师端的，如果...