I am using the below syntax to read from Teradata Aster database Table transaction and load into Hadoop/Hive Table
I have added the below jar files in /usr/iop/4.1.0.0/sqoop/lib folder
terajdbc4.jar
tdgssconfig.jar
noarch-aster-jdbc-driver.jar
Syntax:
sqoop import --connect jdbc:ncluster://hostname.gm.com:2406/Database=test --username abcde --password test33 --table aqa.transaction
Error:
Warning: /usr/iop/4.1.0.0/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/12/14 15:38:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6_IBM_20
16/12/14 15:38:49 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/12/14 15:38:49 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:ncluster://hostname.gm.com:2406/Database=test
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:256)
at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Add --connection-manager <class-name> in your sqoop command if available for your RDBMS in sqoop.
Otherwise, add --driver <driver-name> in your sqoop command to use Generic connection manager.
You can try with JDBC jar from Aster.
Here are some steps that I followed to create an external Hive table after importing an Aster table using Sqoop:
Download JDBC jar from https://aster-community.teradata.com/docs/DOC-2254
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$PWD/noarch-aster-jdbc-driver.jar
sqoop import -D mapreduce.job.name="Sqoop Hive Import for Aster table tableName" --connect "jdbc:ncluster://X.X.X.X/database" --driver com.asterdata.ncluster.Driver --username "user1" --password "password" --query "select * from schema.table where \$CONDITIONS limit 10" --split-by col1 --as-avrodatafile --target-dir /tmp/aster/tableName
Create an external Hive table on target directory or replace avrodatafile with hive table options.
Related
I'm attempting to write a bash script that will dump a database and then import it to a staging database. I would like the staging database to match the 'master' database.
I have the following code, however I recieve:
ERROR 1062 (23000) at line 23: Duplicate entry '1' for key 'PRIMARY'
# Dump production master database, excluding school_hosts table
mysqldump -h $MYSQL_HOST -u $MYSQL_USERNAME -p$MYSQL_PASSWORD --no-create-info --ignore-table=hcl_master.school_hosts hcl_master > hcl_master.sql
# Dump hcl staging database, for backup.
mysqldump -h $MYSQL_HOST -u $MYSQL_USERNAME -p$MYSQL_PASSWORD hclstaging_master > hclstaging_master_backup.sql
# Import dump file into staging master database
mysql -h $MYSQL_HOST -u $MYSQL_USERNAME -p$MYSQL_PASSWORD hclstaging_master < hcl_master.sql
After searching, I found that I could add --replace to the mysql command that is importing, however I recieve an error stating that:
mysql: unknown option '--replace'
Can anybody help with getting this script to work correctly? I'm unsure how I can drop the staging database before i import or how to get it to overwrite the primary key record?
Any help would be much appreciated. I am using MariaDB.
--replace is a mysqldump option that you specify when creating the dump, not something you can tell mysql when importing the dump.
How do I issue commands to MariaDB via the CLI without actually jumping into the interactive use mode?
I know I can type mysql which will then jump me into the interactive mode where I can write SQL commands like CREATE DATABASE dbname; and then exit to go back to the regular terminal.
However I'd like to skip that and do something like mysql 'CREATE DATABASE dbname;' all in one line.
mysql --help | grep "\-execute"
Output:
-e, --execute=name Execute command and quit
So to create a database with command line client, you just need to execute
mysql -uuser -p -e"CREATE DATABASE dbname"
You can also concatenate several SQL statements, e.g.
mysql -uuser -p -e"CREATE DATABASE dbname;SHOW DATABASES"
Put the commands that you want executed into a text file (optionally with a file extension of .sql) then, from the command line, do mysql -uuser -p < yourtextfile.sql to have all of the commands in the file executed.
I am trying to use Flyway to set up a DB2 test/demo environment in a Docker container. I have an image of DB2 running in a docker container and now am trying to get flyway to create the database environment. I can connect to the DB2 docker container and create DB2 objects and load them with data, but am looking for a way for non-technical users to do this (i.e. clone a GitHub repo and issue a single docker run command).
The Flyway Docker site (https://github.com/flyway/flyway-docker) indicates that it supports the following volumes:
| Volume | Description |
|-------------------|--------------------------------------------------------|
| `/flyway/conf` | Directory containing a flyway.conf |
| `/flyway/drivers` | Directory containing the JDBC driver for your database |
| `/flyway/sql` | The SQL files that you want Flyway to use |
I created the conf, drivers, and sql directories. In the conf directory, I placed the file flyway.conf that contained my flyway Url, user name, and password:
flyway.url=jdbc:db2://localhost:50000/apidemo
flyway.user=DB2INST1
flyway.passord=mY%tEst%pAsSwOrD
In the drivers directory, I added the DB2 JDBC Type 4 drivers (e.g. db2jcc4.jar, db2jcc_license_cisuz.jar),
And in the sql directory I put in a simple table creation statement (file name: V1__make_temp_table.sql):
CREATE TABLE EDS.REFT_TEMP_DIM (
TEMP_ID INTEGER NOT NULL )
, TEMP_CD CHAR (8)
, TEMP_NM VARCHAR (255)
)
DATA CAPTURE NONE
COMPRESS NO;
Attempting to perform the docker run with the flyway/flyway image as described in the GitHub Readme.md, it is not recognizing the flyway.conf file, since it does not know the url, user, and password.
docker run --rm -v sql:/flyway/sql -v conf:/flyway/conf -v drivers:/flyway/drivers flyway/flyway migrate
Flyway Community Edition 6.5.5 by Redgate
ERROR: Unable to connect to the database. Configure the url, user and password!
I then put the url, user, and password inline and It could not find the JDBC driver.
docker run --rm -v sql:/flyway/sql -v drivers:/flyway/drivers flyway/flyway -url=jdbc:db2://localhost:50000/apidemo -user=DB2INST1 -password=mY%tEst%pAsSwOrD migrate
ERROR: Unable to instantiate JDBC driver: com.ibm.db2.jcc.DB2Driver => Check whether the jar file is present
Caused by: Unable to instantiate class com.ibm.db2.jcc.DB2Driver : com.ibm.db2.jcc.DB2Driver
Caused by: java.lang.ClassNotFoundException: com.ibm.db2.jcc.DB2Driver
Therefore, I believe it is the way that I am setting up the local file system or associating to local files with the flyway volumes that is causing the issue. Does anyone have an idea of what I am doing wrong?
You need to supply absolute paths to your volumes for docker to mount them.
Changing the relative paths to absolute paths fixed the volume mount issue.
docker run --rm \
-v /Users/steve/github-ibm/flyway-db-migration/sql:/flyway/sql \
-v /Users/steve/github-ibm/flyway-db-migration/conf:/flyway/conf \
-v /Users/steve/github-ibm/flyway-db-migration/drivers:/flyway/drivers \
flyway/flyway migrate
I used the following command from shell to export the schema of the database:
mysqldump -u username -p -no-data database_name> gs://test.sql
I got the following error:
mysqldump: unknown option '-o'.
Then instead I used this:
mysqldump -u username -p -d database_name, ( -d instead of -no-data flag)
I got this error:
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
When trying to connect.
Any idea how to export only table definitions in the schema and not all the database.
You're missing the hostname.
mysqldump -h hostname -u username -p -d database_name
I am able to use unixodbc without any problem with my default user. But when I switch to another user, I get an error.
[centos# ~]$ odbcinst -q -s
[ODBC]
[Amazon Redshift DSN 32]
[centos# ~]$ su ruser
Password:
[ruser# centos]$ odbcinst -q -s
odbcinst: SQLGetPrivateProfileString failed with Unable to find component name.
Environment variables are set in both of the users:
AMAZONREDSHIFTODBCINI=/etc/amazon.redshiftodbc.ini
ODBCSYSINI=/usr/local/odbc
ODBCINI=/etc/odbc.ini
LD_LIBRARY_PATH=/usr/local/lib
LD_PRELOAD=/usr/local/lib/libodbcinst.so
Odbc configuration is as follows:
[ruser# centos]$ odbcinst -j
unixODBC 2.3.4
DRIVERS............: /usr/local/odbc /odbcinst.ini
SYSTEM DATA SOURCES: /usr/local/odbc /odbc.ini
FILE DATA SOURCES..: /usr/local/odbc /ODBCDataSources
USER DATA SOURCES..: /etc/odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
By the way, I don't understand why there are spaces in the above paths. I don't know if there is a way to change them. Any ideas to solve this issue? Overall odbc configuration seems the same in both of the users.
I found the exact same issue on Centos, where I could use the default centos user to connect, but not any other users.
I was able to resolve the issue by copying the working user (or system) odbc.ini and odbcinst.ini files across into the home directory of the other user, like so (where I have renamed them to .odbc.ini and .odbcinst.ini respectively):
~/.odbc.ini
[MSSQLTest]
Driver = ODBC Driver 17 for SQL Server
Server = tcp:<ip of server>
~/.odbcinst.ini
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.2.so.0.1
UsageCount=1
Finally, I just had to set the following environment variables inside my ~/.bashrc file, and I was able to connect.
export ODBCSYSINI="<full path to user folder, which would be the evaluated path of {echo ~}>"
export ODBCINSTINI=".odbcinst.ini"
export ODBCINI="<full path to user folder, which would be the evaluated path of {echo ~}>/.odbc.ini"
For some reason, I wasn't able to use ~ to reference the user folders, so I had to manually specify the full user path in the environment variables, and thus my complete .bashrc file is simply:
export ODBCSYSINI="/home/mitch"
export ODBCINSTINI=".odbcinst.ini"
export ODBCINI="/home/mitch/.odbc.ini"
With this setup, I could now run the following and connect successfully:
$ isql -v MSSQLTest <sql server username> <sql server password>
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL>