i have a csv file in hadoop and i have a Hive table ,now i want to laoad that csv file into this Hive table
i have used load LOAD DATA local 'path/to/csv/file' overwrite INTO TABLE tablename;
ended up with this error :
Error in .verify.JDBC.result(r, "Unable to retrieve JDBC result set for ", :
Unable to retrieve JDBC result set for LOAD DATA local
'path/to/csv/file' overwrite INTO TABLE tablename
(Error while processing statement: FAILED:
ParseException line 1:16 missing INPATH at ''path/tp csv/file'' near '<EOF>'
)
Note: i am trying this using RJDBC connection in r
I think the command to load CSV to Hive table is ( when CSV is in HDFS).
LOAD DATA INPATH '/user/test/my.csv' INTO TABLE my_test;
As your file is already present in the HDFS remove the keyword Local
LOAD DATA inpath 'path/to/csv/file' overwrite INTO TABLE tablename;
I have developed a tool to generate hive scripts from a csv file. Following are few examples on how files are generated.
Tool -- https://sourceforge.net/projects/csvtohive/?source=directory
Select a CSV file using Browse and set hadoop root directory ex: /user/bigdataproject/
Tool Generates Hadoop script with all csv files and following is a sample of
generated Hadoop script to insert csv into Hadoop
#!/bin/bash -v
hadoop fs -put ./AllstarFull.csv /user/bigdataproject/AllstarFull.csv
hive -f ./AllstarFull.hive
hadoop fs -put ./Appearances.csv /user/bigdataproject/Appearances.csv
hive -f ./Appearances.hive
hadoop fs -put ./AwardsManagers.csv /user/bigdataproject/AwardsManagers.csv
hive -f ./AwardsManagers.hive
Sample of generated Hive scripts
CREATE DATABASE IF NOT EXISTS lahman;
USE lahman;
CREATE TABLE AllstarFull (playerID string,yearID string,gameNum string,gameID string,teamID string,lgID string,GP string,startingPos string) row format delimited fields terminated by ',' stored as textfile;
LOAD DATA INPATH '/user/bigdataproject/AllstarFull.csv' OVERWRITE INTO TABLE AllstarFull;
SELECT * FROM AllstarFull;
Thanks
Vijay
Related
I am trying to create a sealed command for my build pipeline which inserts data and quits.
So far I have created my data files
things-to-import-001.sql and 002 etc, which contains all the INSERT statements I'd like to run, with a file per table.
I have created a command file to run them
-- import-all.sql
.read ./things-to-import-001.sql
.read ./things-to-import-002.sql
.quit
However when I run my command
sqlite3 -init ./import-all.sql ./database.sqlite
..the data is inserted, but the program remains running and shows the sqlite> prompt, despite the .quit command. I have also tried using .exit 0.
From the sqlite3 --help
-init FILENAME read/process named file
Docs: https://www.sqlite.org/cli.html#reading_sql_from_a_file
How can I tell sqlite to exit once my inserts have finished?
I have managed to find a dirty workaround for this issue.
I have updated my import file to include a bad command, and executed using -bail to quit on first error.
-- import-all.sql
.read ./things-to-import-001.sql
.read ./things-to-import-002.sql
.fakeErrorToQuitWithBail
Then you can execute with
sqlite3 -init import-all.sql -bail
and it should quit with
Error: unknown command or invalid arguments: "fakeErrorToQuitWithBail". Enter ".help" for help
Try using ".exit" at the place of ".quit". For some reason SQLite dont doccumented this commands.
https://www.tutorialspoint.com/sqlite/sqlite_commands.htm
I am trying to export data using the TDload utility of Teradata and I need the exported file to be in parquet format.
The command I have used is:
tdload --SourceTdpid xxx.xxx.xxx.xxx --SourceUserName dbc
--SourceUserPassword dbc --SourceTable DimAccount
--TargetFilename DimAccount.parquet
But this does not export the data in parquet.
How to achieve it?
Using a SQL style approach to Liquibase changesets (which is our codestyle, we don't use XML) I am trying to load an CSV file using the following SQL changeset
SQL
-- changeset user:insert-prices-data-temp-table
LOAD DATA LOCAL INFILE 'foo/src/main/resources/liquibase/changelogs/2021/prices.csv'
INTO TABLE prices_temp
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
When deploying the WAR file to Wildfly the following exception
java.io.FileNotFoundException:
foo/src/resources/liquibase/changelogs/2021/prices.csv (No such file
or directory) 2021-09-22T20:52:18.581085123Z at
java.io.FileInputStream.open0(Native Method)
2021-09-22T20:52:18.581087214Z at
java.io.FileInputStream.open(FileInputStream.java:195)
2021-09-22T20:52:18.581089768Z at
java.io.FileInputStream.(FileInputStream.java:138)
2021-09-22T20:52:18.581092029Z at
java.io.FileInputStream.(FileInputStream.java:93)
2021-09-22T20:52:18.581094150Z at
com.mysql.jdbc.MysqlIO.sendFileToServer(MysqlIO.java:3772)
The Master Change log file references the 2021 directory with the SQL and CSV file existing in the same directory.
<includeAll path="liquibase/changelogs/2021/" filter="xml" errorIfMissingOrEmpty="true" />
I have tried the following other paths but they all still yield a FileNotFoundException
prices.csv
liquibase/changelogs/2021/prices.csv
WEB-INF/classes/liquibase/changelogs/2021/CCC-220-marking-time-prices.csv
(Absolute path) /home/xxxx/Work/foo-service/foo/src/main/resources/liquibase/changelogs/2021/prices.csv
Liquibase Version: 3.5.1
Wildfly Jboss Version : 21
I have checked the CSV file is present in the WAR file
Any ideas how to fix this?
The problem is that you are trying to use some mysql function LOAD DATA LOCAL INFILE which doesn't know about your classpath. It's trying to look at your filesystem and such path doesn't exists. Even if you provide something like yourapp.jar!liquibase/changelogs/2021/prices.csv it won't be able to read that file. You will need to pull prices.csv out of your application to filesystem and point mysql function to that location.
Or you can use liquibase's loadData if that helps.
Brand new to SQL lite, running on a mac. I'm trying to import a csv file from the SQL lite tutorial:
http://www.sqlitetutorial.net/sqlite-import-csv/
The 'cities' data I'm trying to import for the tutorial is here:
http://www.sqlitetutorial.net/wp-content/uploads/2016/05/city.csv
I try and run the following code from Terminal to import the data into a database named 'data' and get the following error:
sqlite3
.mode csv
.import cities.csv data;
CREATE TABLE data;(...) failed: near ";": syntax error
A possible explanation may be the way I'm downloading the data - I copied the data from the webpage into TextWrangler and saved it as a .txt file. I then manually changed the extension to .csv. This doesn't seem very eloquent but that was the advice I found online for creating the .csv file: https://discussions.apple.com/thread/7857007
If this is the issue then how can I resolve it? If not then where am I going wrong?
Another potentially useful point - when I executed the code yesterday there was no problem, it created a database with the data. However, running the same code today produces the error.
sqlite3 dot commands such as .import are not SQL and don't need semicolon at end. Replace
.import cities.csv data;
with
.import cities.csv data
I have one doubt. I am trying to pull data from oracle and want to push it into HDFS using sqoop 1.4.6.The table which I want to migrate contains column named "COMMENT"(which is a reserved keyword in oracle), but when I tried to push table into HDFS using sqoop, the error occurred was:
15/09/30 14:52:49 ERROR db.DBRecordReader: Top level exception:
java.sql.SQLSyntaxErrorException: ORA-00936: missing expression
I have tried by putting \ and " for this column as:
"\"\"COMMENT\"\"" when I listed column names during query
so how to get this error fixed..
Please try to use --query option for sqoop import.
eg: sqoop import --query "select COMMENT from Table_Name ....."