I've requirement such that need to read an xlsx file from flyway locations through pl/sql procedure, end objective is to store the xlsx file as blob in the database, any inputs on how to achieve this.
Related
I have a question about autoload writestream.
I have below user case:
Days before I uploaded 2 csv files into databricks file system, then read and write it to table by autoloader.
Today, I found that the files uploaded days before has wrong data that faked. so I deleted these old csv file, and then uploaded 2 new correct csv file.
Then I read and write the new files by autoloader streaming.
I found that the streaming can read the data from new files successfully, but failed to write to table by writestream.
Then I tried to delete the checkpoint folder and all sub folders or files and re-create the checkpoint folder, and read and write by stream again, found that the data is write to table successfully.
Questions:
Since the autoloader has detect the new files, why it can't write to table succesfully until I delete the checkpoint folder and create a new one.
AutoLoader works best when new files are ingested into a directory. Overwriting files might give unexpected results. I haven't worked with the option cloudFiles.allowOverwrites set to True yet, but this might help you (see documentation below).
On the question about readStream detecting the overwritten files, but writeStream not: This is because of the checkpoint. The checkpoint is always linked to the writeStream operation. If you do
df = (spark.readStream.format("cloudFiles")
.option("cloudFiles.schemaLocation", "<path_to_checkpoint>")
.load("filepath"))
display(df)
then you will always view the data of all the files in the directory. If you use writeStream, you need to add .option("checkpointLocation", "<path_to_checkpoint>"). This checkpoint will remember that the files (that were overwritten) already have been processed. This is why the overwritten files will only be processed again after you deleted the checkpoint.
Here is some more documentation about the topic:
https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/faq#does-auto-loader-process-the-file-again-when-the-file-gets-appended-or-overwritten
cloudFiles.allowOverwrites https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-options
Is it possible to access a csv file directly with the csv storage engine without going through the trouble of loading it first?
We deploy a data warehouse where during load CSV files are read into a temp table and then insert the content in production fact tables. I wonder if we could just pass the load into and "directly go to jail insert"?
We have a CSV files being loaded automatically in Unix machine.
Requirement: We need to load the csv file from that remote server to my oracle DB. We do have an ODI as our ETL tool. Can someone advice on how to proceed further. What is the way to load the CSV from Unix server to Oracle DB.Please help us with some document if this case is possible.
Thanks ,
Gowtham Raja S
Oracle is providing some tutorials (Oracle By Example), one of them explains how to load a flat file to an Oracle table with ODI 12c : https://apexapps.oracle.com/pls/apex/f?p=44785:112:::::P112_CONTENT_ID:7947
You will just need to change the field delimiter to a comma instead of a tab.
The other tutorials can be found on the product page : http://www.oracle.com/technetwork/middleware/data-integrator/learnmore/index.html
if you know how to create a data store for file, in ODI we have a LKM called LKM file to Oracle (SQLLDR) OR LKM file to Oracle (external table), both can be used to load data quickly, or if you feel like this is bit difficult, however since you have the Sqlldr to load data manually from file to DB, what ever the command you are using to start sqlldr place it in ODI procedure with technology as OSCommand whihc loads data automatically.
let me know if any other suggestios are required.
I have a script that was reading data from a sqlite3 database and while this script was running I made a copy of the database cp mydatabase mydatabase.bak. Will this affect either the script that was reading from the db or the copy of the db? I had a look at the sqlite documentation here [0] but I didn't put a lock on the db as per the instructions.
[0] http://www.sqlite.org/backup.html
Copying the file should be analogous to another application reading the database, so it shouldn't be a problem. Multiple applications can safely read the database file at the same time (per the SQLite FAQ).
As another point, consider that you can read from a database even if the database and its directory both lack write permissions. Since in that scenario there's no way for the reading application to be modifying the database file or creating a temp file that needs to be incorporated into it, there's no way for any of a number of simultaneously reading applications to affect what any of the others see.
How to prepopulate an sqlite database at development time? I found a tutorial how to use a prepopulated sqlite with android but I want to populate the database with data from a text file. Is it possible to write a java program that will read the data from a text file insert into sqlite database and save the database?
Yes, what you described is what you need to do:
Read data from a text file
Generate "insert" sentences from it
Run them against your SQLite database
Done. Your DB is now pre-populated.
Make sure you flag this "pre population:" command so you don't run it twice.