Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 4. Exception = Connection refused
1) check oozie service running by typing
oozie admin -oozie http://OOZIE_IP:11000/oozie -status
if it doesnt show System mode: NORMAL, oozie is not started correctly
2) set in core-site.xml in hadoop config ,
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property>
Hope this works
Related
I'm trying to run a simple sqoop import using oozie from hue ( Cloudera VM ).Few seconds after submitting, the job gets hung with heart beat issue for ever, I did some search and found this thread https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-launcher-never-ends/td-p/13330, I added the XML properties that is mentioned in all the below yarn-site.xml files not knowing which specific file, but no use, I'm still facing the same issue can anyone give some insights on this?
/etc/hive/conf.cloudera.hive/yarn-site.xml
/etc/hadoop/conf.empty/yarn-site.xml
/etc/hadoop/conf.pseudo/yarn-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/yarn-site.xml
/etc/hive/conf.cloudera.hive/yarn-site.xml
job log
'12480 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of order_items
13225 [main] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
16314 [main] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation
18408 [main] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://quickstart.cloudera:8088/proxy/application_1484596399739_0002/
18409 [main] INFO org.apache.hadoop.mapreduce.Job - Running job: job_1484596399739_0002
25552 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1484596399739_0002 running in uber mode : false
25553 [main] INFO org.apache.hadoop.mapreduce.Job - map 0% reduce 0%
Heart beat
Heart beat
workflow XML
<workflow-app name="Oozie_Test1" xmlns="uri:oozie:workflow:0.5">
<start to="sqoop-e57e"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-e57e">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --m 1 --connect jdbc:mysql://quickstart.cloudera:3306/retail_db --username=retail_dba --password=cloudera --table order_items --hive-database sqoopimports --create-hive-table --hive-import --hive-table sqoop_hive_order_items</command>
<file>/user/oozie/share/lib/mysql-connector-java-5.1.34-bin.jar#mysql-connector-java-5.1.34-bin.jar</file>
</sqoop>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
Thanks
Mx
this thread helped me to resolve the issue
https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-sqoop-action-in-CDH-5-2-Heart-beat-issue/td-p/22181/page/2
after getting through this error I got stuck in "Failure to launch flume" issue and this thread helped me to fix the issue
oozie Sqoop action fails to import data to hive
I have installed oozie-4.2 on AWS as root and am running the workflow as root as well.I have hadoop-2.4 running on a cluster
I have added the following in oozie-site.xml
<property>
<name>oozie.service.ProxyUserService.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.root.groups</name>
<value>*</value>
</property>
In addition my hadoop core-site.xml also has the following:
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
After the config changes above, I stopped hdfs/yarn and restarted them. When I start oozied, I get these warnings
2016-02-03 07:12:22,987 WARN ConfigurationService:523 - SERVER[]
Invalid configuration defined,
[oozie.service.ProxyUserService.proxyuser.root.groups]
2016-02-03 07:12:22,987 WARN ConfigurationService:523 - SERVER[]
Invalid configuration defined,
[oozie.service.ProxyUserService.proxyuser.root.hosts]
When I start spark action based workflow, I get these errors :
Message [JA002: User: root is not allowed to impersonate root]
org.apache.oozie.action.ActionExecutorException: JA002: User: root is not allowed to impersonate root
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:462)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.comma
Can someone pl help with what I am doing wrong ?
On HDP 2.3.4, using Oozie 4.2.0 and Sqoop 1.4.2, I'm trying to create a coordinator app that will execute sqoop jobs on a daily basis. I need the sqoop action to execute jobs because these are incremental imports.
I've configured sqoop-site.xml and started the sqoop-metastore and I'm able to create, list, and delete jobs via the command line but the workflow encounters the error: Cannot restore job: streamsummary_incremental
stderr
Sqoop command arguments :
job
--exec
streamsummary_incremental
Fetching child yarn jobs
tag id : oozie-26fcd4dc0afd8f53316fc929ac38eae2
2016-02-03 09:46:47,193 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at <myHost>/<myIP>:8032
Child yarn jobs are found -
=================================================================
>>> Invoking Sqoop command line now >>>
2241 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2016-02-03 09:46:47,404 WARN [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2263 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6.2.3.4.0-3485
2016-02-03 09:46:47,426 INFO [main] sqoop.Sqoop (Sqoop.java:<init>(97)) - Running Sqoop version: 1.4.6.2.3.4.0-3485
2552 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - Cannot restore job: streamsummary_incremental
2016-02-03 09:46:47,715 ERROR [main] hsqldb.HsqldbJobStorage (HsqldbJobStorage.java:read(254)) - Cannot restore job: streamsummary_incremental
2552 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - (No such job)
2016-02-03 09:46:47,715 ERROR [main] hsqldb.HsqldbJobStorage (HsqldbJobStorage.java:read(255)) - (No such job)
2553 [main] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Cannot restore missing job streamsummary_incremental
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:256)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
sqoop-site.xml
<property>
<name>sqoop.metastore.client.enable.autoconnect</name>
<value>false</value>
<description>If true, Sqoop will connect to a local metastore
for job management when no other metastore arguments are
provided.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://<myhost>:12345</value>
<description>The connect string to use when connecting to a
job-management metastore. If unspecified, uses ~/.sqoop/.
You can specify a different path here.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>SA</value>
<description>The username to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
<description>The password to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.server.location</name>
<value>/tmp/sqoop-metastore/shared.db</value>
<description>Path to the shared metastore database files.
If this is not set, it will be placed in ~/.sqoop/.
</description>
</property>
<property>
<name>sqoop.metastore.server.port</name>
<value>12345</value>
<description>Port that this metastore should listen on.
</description>
</property>
workflow.xml
<action name="sqoop-import-job">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${outputDir}"/>
</prepare>
<arg>job</arg>
<arg>--exec</arg>
<arg>${jobId}</arg>
</sqoop>
<ok to="hive-load"/>
<error to="kill-sqoop"/>
</action>
Additional info:
We're only running a single-node cluster.
Only Sqoop Client is
installed.
I'm thinking maybe Oozie isn't able to connect to the metastore because we don't have sqoop server? Could anyone confirm this? If not that, could I have missed anything else?
Thanks!
I managed to resolve this issue with the help of #SamsonScharfrichter in the comments. I explicitly passed the metastore URL in the Oozie workflow and it worked:
<arg>job</arg>
<arg>--meta-connect</arg>
<arg>jdbc:hsqldb:hsql://<myhost>:12345/sqoop</arg>
<arg>--exec</arg>
<arg>myjob</arg>
It seems that Oozie tries to connect to a local metastore because it doesn't have a copy of sqoop-site.xml and so it doesn't know the metastore url (even though I'm running a single-node configuration).
rhive.connect(host="172.16.210.130",port=10000)
Warning:
+----------------------------------------------------------+
+ / hiveServer2 argument has not been provided correctly. +
+ / RHive will use a default value: hiveServer2=TRUE. +
+----------------------------------------------------------+
15/11/05 07:54:41 INFO jdbc.Utils: Supplied authorities: 172.16.210.130:10000
15/11/05 07:54:41 INFO jdbc.Utils: Resolved authority: 172.16.210.130:10000
15/11/05 07:54:41 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://172.16.210.130:10000/default
Error: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [root] does not have [CREATE] privilege on [/r]
How to solve this error?
I found the resolution of issue:
Set permissions 777 on hfs directory and sub-directories
Changes to following files:
hive-site.xml (in both conf & conf.server dirs)
set:
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
<property>
change:
<property>
<name>hive.security.metastore.authorization.manager</name>
<value> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider,org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly</value>
to
<property>
<name>hive.security.metastore.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
</property>
hiveserver2-site.xml (in both conf & conf.server directory)
set:
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
</property>
hdfs-site.xml
set:
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
SSH to sandbox and log into Hive as with hue user:
su hue
hive
Reference:
http://hortonworks11.rssing.com/browser.php?indx=7980721&item=4941
Login as a hdfs or hive user
su hdfs
or
su hive
and then try your steps on R console.
I tried to install oozie on my pc and looks like successfully installed
oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL
But when running an oozie job it is showing below error
Error: IO_ERROR : java.net.UnknownHostException: master
Can you please suggest what could be the reason?
The reason for this failure could be further examined in oozie.log, which usually located in /var/log/oozie folder for CDH distribution. Or the log location could be examined with command:
ps -ef | grep oozie
and look for "-Doozie.log.dir=..."
And the oozie host needs to be reachable from the host where the command line is invoked. Try telnet to oozie port to make sure connection is good. An example session is like this:
[cloudera#localhost hadoop-yarn]$ telnet master 11000
Trying 127.0.0.1...
Connected to master.
Escape character is '^]'.