I know how to run a SQL query in minitab using :
ODBC;
Connect "Driver={SQLite3 ODBC Driver};Database=...";
SQLString "SELECT * FROM Table".
However, this is very constraining since the length of the command line is limited in Minitab... Instead of manually shorten each line to right length, would it be possible to directly read the query from a text file?
You can have Minitab read text command files from an Exec file. You can create an Exec file using any text editor, and saving the file with the .mtb extension.
Related
I hope that what I am about to write makes some sense.
If you look at
How to deal with a 50GB large csv file in r language?
it is explained how to query à la SQL, a csv file from R.
In my case, I have vast amount of data stored as large (or larger than my RAM) flat files.
I would like to storer for instance one of these as an SQLite database without loading it in memory entirely.
Imagine if you could automatically read a limited chunk of that file which is suitable for your RAM, store it into a SQL, then free up some memory, process the next chunk and so on and so forth until all the file is in the database.
Is this doable in R? If the table could be stored as tibble, it would be even better, but it is not crucial.
Any suggestion is appreciated.
Thanks!
1) dbWriteTable dbWriteTable can read the file into a database without going through R. The database is created if it does not already exist.
library(RSQLite)
cat("a,b\n1,2\n", file = "myfile.csv") # create test file
con <- dbConnect(SQLite(), "mydb")
dbWriteTable(con, "mytable", "myfile.csv")
dbGetQuery(con, "select count(*) from mytable") # ensure it is there
dbDisconnect(con)
2) SQLite CLI We could alternately do it using the sqlite cli which can be downloaded from the sqlite download site.
https://www.sqlite.org/download.html
This does not involve R at all in creating the database. mydb will be created if it does not exist. This first line is entered at the shell or cmd prompt and that will provide its own prompt at which the remaining lines cana be entered.
sqlite3 mydb
.mode csv
.import myfile.csv mytable
.quit
3) Other database Another option is to use a database that has the ability to read csv files directly into it. H2 has csvread, MySQL has load data infile and PostgreSQL has copy.
Apparently there is already a function for that
https://raw.githubusercontent.com/inbo/inborutils/master/R/csv_to_sqlite.R
I am testing it. I do not see any progress bar even if the corresponding option is selected, but it appears to get the job done.
I need to get data from a PostgreSQL table into an SQLite table. I am exporting the data from PostgreSQL into a CSV file, and then trying to import it into SQLite from a command-line prompt. First I tried this:
c:> /sqlite3/sqlite3 sample.trd ".import source_file.csv trend_data"
For every line in the file, I got an error message saying SQLite expected three values and there was only one. Not really surprising because I never told SQLite that the file was a CSV file. So then I tried this:
c:> /sqlite3/sqlite3 sample.trd ".mode csv .import source_file.csv trend_data"
I got no errors, but I also got no data.
Thank you.
Each dot command must be alone in a single line.
You could pipe multiple commands into sqlite3, but in this case, it would be easier to set CSV mode with a parameter:
sqlite3 -csv sample.trd ".import source_file.csv trend_data"
I download the file using download.file function in R
Save it locally
Move it to HDFS
My hive table is mapped to this location of file in HDFS. Now, this file has , as the delimiter/separator which I cannot change using download.file function from R.
There is a field that has , in its content. How do I tell hive to drop this delimiter if found anywhere within the field contents?
I understand we can change the delimiter but is there really a way I can drop it like sqoop allows us to do?
My R script is as simple as
download.file()
hdfs.put
Is there a workaround?
I have been given a minitab worksheet file (.mtw), and I would like to open it in R.
Unfortunately, I only know of R's foreign package to open other filetypes, which only includes minitab's portable files (.mtp). Is there a way to open .mtw files, and if not how would I go about converting the file to .mtp?
If you want to convert your mtw worksheet to mtp extension, simply open the mtw file on minitab and from File menu select "Save Current Worksheet As" and save it to mtp extension. It will save it as Text not Encrypted
You would be able then to use foreign
Is it possible to transfer the date from the Teradata Table into .csv file directly.
Problem is - my table has more that 18 million rows.
If yes, please send tell me the process
For a table that size I would suggest using the FastExport utility. It does not natively support a CSV export but you can mimic the behavior.
Teradata SQL Assistant will export to a CSV but it would not be appropriate to use with a table of that size.
BTEQ is another alternative that may be acceptable for a one-time dump if the table.
Do you have access to any of these?
It's actually possible to change the delimiter of exported text files within Teradata SQL Assistant, without needing any separate applications:
Go to Tools > Options > Export/Import. From there, you can change the Use this delimiter between column option from {Tab} to ','.
You might also want to set the 'Enclose column data in' option to 'Double Quote', so that any commas in the data itself don't upset the file structure.
From there, you use the regular text export: File > Export Results, run the query, and select one of the Delimited Text types.
Then you can just use your operating system to manually change the file extension from .txt to .csv.
These instructions are from SQL Assistant version 16.20.0.7.
I use the following code to export data from the Teradata Table into .csv file directly.
CREATE EXTERNAL TABLE
database_name.table_name (to be created) SAMEAS database_name.table_name (already existing, whose data is to be exported)
USING (DATAOBJECT ('C:\Data\file_name.csv')
DELIMITER '|' REMOTESOURCE 'ODBC');
You can use FastExport utility from Teradata Studio for exporting the table in CSV format. You can define the delimiter as well.
Very simple.
Basic idea would be to export first table as a TXT file and then converting TXT t o CSV using R...read.table ()---> write.csv().....
Below are the steps of exporting TD table as txt file:
Select export option from file
Select all records from the table you want to export
Save it as a TXT file
Then use R to convert TXT file to CSV (set working directory to the location where you have saved your big TXT file):
my_table<-read.table("File_name.txt", fill = TRUE, header = TRUE)
write.csv(my_table,file = "File_name.csv")
This had worked for 15 million records table. Hope it helps.