rhive.connect(host="172.16.210.130",port=10000)
Warning:
+----------------------------------------------------------+
+ / hiveServer2 argument has not been provided correctly. +
+ / RHive will use a default value: hiveServer2=TRUE. +
+----------------------------------------------------------+
15/11/05 07:54:41 INFO jdbc.Utils: Supplied authorities: 172.16.210.130:10000
15/11/05 07:54:41 INFO jdbc.Utils: Resolved authority: 172.16.210.130:10000
15/11/05 07:54:41 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://172.16.210.130:10000/default
Error: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [root] does not have [CREATE] privilege on [/r]
How to solve this error?
I found the resolution of issue:
Set permissions 777 on hfs directory and sub-directories
Changes to following files:
hive-site.xml (in both conf & conf.server dirs)
set:
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
<property>
change:
<property>
<name>hive.security.metastore.authorization.manager</name>
<value> org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider,org.apache.hadoop.hive.ql.security.authorization.MetaStoreAuthzAPIAuthorizerEmbedOnly</value>
to
<property>
<name>hive.security.metastore.authorization.manager</name>
<value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
</property>
hiveserver2-site.xml (in both conf & conf.server directory)
set:
<property>
<name>hive.security.authorization.enabled</name>
<value>false</value>
</property>
hdfs-site.xml
set:
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
SSH to sandbox and log into Hive as with hue user:
su hue
hive
Reference:
http://hortonworks11.rssing.com/browser.php?indx=7980721&item=4941
Login as a hdfs or hive user
su hdfs
or
su hive
and then try your steps on R console.
Related
Problem
I am trying to login to our API store using an account from a secondary user store, every attempt results in the following error
Error! Login failed. Please recheck the username and password and try again.
Debugging
what seems to works
I can make an account on the primary store,
username: fred#text.com
password: password
This user can login without a problem.
The secondary user store is a JDBC connection pointing at our Oracle database, called storeTwo.
I can see the users, totaling 49,000, in the carbon management interface, along with the roles they have.
I have given a subset of these users every permission possible.
what doesn't work
When I take one of these accounts and try to login, I get the error mentioned above.
I cannot tell what the issues here are.
To me, it looks like wso2 isn't checking the secondary store when I try and login.
Other info
server OS: Centos 6
wso2 version: 2.1.0
wso2 info: standalone setup of wso2 API Manager.
config files
storeTwo.xml
<UserStoreManager class="org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager">
<Property name="url">jdbc:oracle:thin:#<IP>:<HOST></Property>
<Property name="userName"><user></Property>
<Property encrypted="true" name="password"><pasword></Property>
<Property name="driverName">oracle.jdbc.driver.OracleDriver</Property>
<Property name="Disabled">false</Property>
<Property name="ReadOnly">true</Property>
<Property name="ReadGroups">true</Property>
<Property name="WriteGroups">true</Property>
<Property name="UsernameJavaRegEx">^[\S]{5,30}$</Property>
<Property name="UsernameJavaScriptRegEx">^[\S]{5,30}$</Property>
<Property name="UsernameJavaRegExViolationErrorMsg">Username pattern policy violated.</Property>
<Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>
<Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>
<Property name="PasswordJavaRegExViolationErrorMsg">Password pattern policy violated.</Property>
<Property name="RolenameJavaRegEx">^[\S]{5,30}$</Property>
<Property name="RolenameJavaScriptRegEx">^[\S]{5,30}$</Property>
<Property name="CaseInsensitiveUsername">true</Property>
<Property name="SCIMEnabled">false</Property>
<Property name="IsBulkImportSupported">false</Property>
<Property name="PasswordDigest">MD5</Property>
<Property name="MultiAttributeSeparator">,</Property>
<Property name="StoreSaltedPassword">false</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="MaxRoleNameListLength">100</Property>
<Property name="UserRolesCacheEnabled">false</Property>
<Property name="UserNameUniqueAcrossTenants">false</Property>
<Property name="validationQuery"/>
<Property name="validationInterval"/>
carbon.xml
...
<LoginConfig>
<UserIdLogin primary="true">
<ClaimUri></ClaimUri>
</UserIdLogin>
<EmailLogin primary="true">
<ClaimUri>http://wso2.org/claims/emailaddress</ClaimUri>
</EmailLogin>
</LoginConfig>
...
wso2carbon.log
TID: [-1234] [] [2017-10-20 10:59:49,288] WARN {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - Failed Administrator login attempt 'temp#example.com[-1234]' at [2017-10-20 10:59:49,288+0100] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - contentType from the OMOutputFormat =application/soap+xml {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - contentType returned =application/soap+xml; charset=UTF-8; action="urn:loginResponse" {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - start writeTo() {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - preserve=false {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - isOptimized=false {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,288] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - isDoingSWA=false {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,289] DEBUG {org.apache.axis2.transport.TransportUtils} - createSOAPEnvelope using Builder (class org.apache.axis2.builder.SOAPBuilder) selected from type (application/soap+xml) {org.apache.axis2.transport.TransportUtils}
TID: [-1234] [] [2017-10-20 10:59:49,289] DEBUG {org.apache.axis2.transport.http.SOAPMessageFormatter} - end writeTo() {org.apache.axis2.transport.http.SOAPMessageFormatter}
TID: [-1234] [] [2017-10-20 10:59:49,290] ERROR {org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject} - Login failed. Please recheck the username and password and try again. {org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject}
By setting up a writable secondary user store, I've have found what looks to be the root problem. when you choose MD5 for the 'Password Hashing Algorithm' WSO2 seems to do the following.
MD5(password) -binary | base 64 = X03MO1qnZdYdgyfeuILPmQ==
whereas
MD5(password) = 5f4dcc3b5aa765d61d8327deb882cf99
this would then mean that when wso2 is checking the read-only secondary store mention above the passwords will never (very rarely) match.
The last post at 1 has an example (not for wso2) of this - below are the commands needed to test on the terminal in case link fails.
echo -n password | openssl dgst -md5 -binary | base64
echo -n password | openssl dgst -md5
Example md5 calls
I'm trying to run a simple sqoop import using oozie from hue ( Cloudera VM ).Few seconds after submitting, the job gets hung with heart beat issue for ever, I did some search and found this thread https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-launcher-never-ends/td-p/13330, I added the XML properties that is mentioned in all the below yarn-site.xml files not knowing which specific file, but no use, I'm still facing the same issue can anyone give some insights on this?
/etc/hive/conf.cloudera.hive/yarn-site.xml
/etc/hadoop/conf.empty/yarn-site.xml
/etc/hadoop/conf.pseudo/yarn-site.xml
/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/yarn-site.xml
/etc/hive/conf.cloudera.hive/yarn-site.xml
job log
'12480 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of order_items
13225 [main] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
16314 [main] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation
18408 [main] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://quickstart.cloudera:8088/proxy/application_1484596399739_0002/
18409 [main] INFO org.apache.hadoop.mapreduce.Job - Running job: job_1484596399739_0002
25552 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1484596399739_0002 running in uber mode : false
25553 [main] INFO org.apache.hadoop.mapreduce.Job - map 0% reduce 0%
Heart beat
Heart beat
workflow XML
<workflow-app name="Oozie_Test1" xmlns="uri:oozie:workflow:0.5">
<start to="sqoop-e57e"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-e57e">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --m 1 --connect jdbc:mysql://quickstart.cloudera:3306/retail_db --username=retail_dba --password=cloudera --table order_items --hive-database sqoopimports --create-hive-table --hive-import --hive-table sqoop_hive_order_items</command>
<file>/user/oozie/share/lib/mysql-connector-java-5.1.34-bin.jar#mysql-connector-java-5.1.34-bin.jar</file>
</sqoop>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
Thanks
Mx
this thread helped me to resolve the issue
https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-sqoop-action-in-CDH-5-2-Heart-beat-issue/td-p/22181/page/2
after getting through this error I got stuck in "Failure to launch flume" issue and this thread helped me to fix the issue
oozie Sqoop action fails to import data to hive
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 4. Exception = Connection refused
1) check oozie service running by typing
oozie admin -oozie http://OOZIE_IP:11000/oozie -status
if it doesnt show System mode: NORMAL, oozie is not started correctly
2) set in core-site.xml in hadoop config ,
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property>
Hope this works
I have installed oozie-4.2 on AWS as root and am running the workflow as root as well.I have hadoop-2.4 running on a cluster
I have added the following in oozie-site.xml
<property>
<name>oozie.service.ProxyUserService.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.root.groups</name>
<value>*</value>
</property>
In addition my hadoop core-site.xml also has the following:
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
After the config changes above, I stopped hdfs/yarn and restarted them. When I start oozied, I get these warnings
2016-02-03 07:12:22,987 WARN ConfigurationService:523 - SERVER[]
Invalid configuration defined,
[oozie.service.ProxyUserService.proxyuser.root.groups]
2016-02-03 07:12:22,987 WARN ConfigurationService:523 - SERVER[]
Invalid configuration defined,
[oozie.service.ProxyUserService.proxyuser.root.hosts]
When I start spark action based workflow, I get these errors :
Message [JA002: User: root is not allowed to impersonate root]
org.apache.oozie.action.ActionExecutorException: JA002: User: root is not allowed to impersonate root
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:462)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:436)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1132)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1286)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.comma
Can someone pl help with what I am doing wrong ?
On HDP 2.3.4, using Oozie 4.2.0 and Sqoop 1.4.2, I'm trying to create a coordinator app that will execute sqoop jobs on a daily basis. I need the sqoop action to execute jobs because these are incremental imports.
I've configured sqoop-site.xml and started the sqoop-metastore and I'm able to create, list, and delete jobs via the command line but the workflow encounters the error: Cannot restore job: streamsummary_incremental
stderr
Sqoop command arguments :
job
--exec
streamsummary_incremental
Fetching child yarn jobs
tag id : oozie-26fcd4dc0afd8f53316fc929ac38eae2
2016-02-03 09:46:47,193 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at <myHost>/<myIP>:8032
Child yarn jobs are found -
=================================================================
>>> Invoking Sqoop command line now >>>
2241 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2016-02-03 09:46:47,404 WARN [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2263 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6.2.3.4.0-3485
2016-02-03 09:46:47,426 INFO [main] sqoop.Sqoop (Sqoop.java:<init>(97)) - Running Sqoop version: 1.4.6.2.3.4.0-3485
2552 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - Cannot restore job: streamsummary_incremental
2016-02-03 09:46:47,715 ERROR [main] hsqldb.HsqldbJobStorage (HsqldbJobStorage.java:read(254)) - Cannot restore job: streamsummary_incremental
2552 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage - (No such job)
2016-02-03 09:46:47,715 ERROR [main] hsqldb.HsqldbJobStorage (HsqldbJobStorage.java:read(255)) - (No such job)
2553 [main] ERROR org.apache.sqoop.tool.JobTool - I/O error performing job operation: java.io.IOException: Cannot restore missing job streamsummary_incremental
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:256)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
sqoop-site.xml
<property>
<name>sqoop.metastore.client.enable.autoconnect</name>
<value>false</value>
<description>If true, Sqoop will connect to a local metastore
for job management when no other metastore arguments are
provided.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://<myhost>:12345</value>
<description>The connect string to use when connecting to a
job-management metastore. If unspecified, uses ~/.sqoop/.
You can specify a different path here.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>SA</value>
<description>The username to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
<description>The password to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.server.location</name>
<value>/tmp/sqoop-metastore/shared.db</value>
<description>Path to the shared metastore database files.
If this is not set, it will be placed in ~/.sqoop/.
</description>
</property>
<property>
<name>sqoop.metastore.server.port</name>
<value>12345</value>
<description>Port that this metastore should listen on.
</description>
</property>
workflow.xml
<action name="sqoop-import-job">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${outputDir}"/>
</prepare>
<arg>job</arg>
<arg>--exec</arg>
<arg>${jobId}</arg>
</sqoop>
<ok to="hive-load"/>
<error to="kill-sqoop"/>
</action>
Additional info:
We're only running a single-node cluster.
Only Sqoop Client is
installed.
I'm thinking maybe Oozie isn't able to connect to the metastore because we don't have sqoop server? Could anyone confirm this? If not that, could I have missed anything else?
Thanks!
I managed to resolve this issue with the help of #SamsonScharfrichter in the comments. I explicitly passed the metastore URL in the Oozie workflow and it worked:
<arg>job</arg>
<arg>--meta-connect</arg>
<arg>jdbc:hsqldb:hsql://<myhost>:12345/sqoop</arg>
<arg>--exec</arg>
<arg>myjob</arg>
It seems that Oozie tries to connect to a local metastore because it doesn't have a copy of sqoop-site.xml and so it doesn't know the metastore url (even though I'm running a single-node configuration).