We are using Teradata Fastexport connection in Informatica to export data from few tables by joining them in Source Qualifier query and write to a CSV file. We have around 180 columns to pull, recently we added 2 new columns to this flow and we found that data for few of the records looks junk. We figured out those records and tried to run the SQ query only for those records, to our surprise the columns which were junk earlier now throwing us expected data.
Is there any fast export limitation to export the columns ? or any properties should be increased at Informatica level.
We are clueless on this issue, please help.
Related
Context: We store historical data in Azure Data Lake as versioned parquet files from our existing Databricks pipeline where we write to different Delta tables. One particular log source is about 18 GB a day in parquet. I have read through the documentation and executed some queries using Kusto.Explorer on the external table I have defined for that log source. In the query summary window of Kusto.Explorer I see that I download the entire folder when I search it, even when using the project operator. The only exception to that seems to be when I use the take operator.
Question: Is it possible to prune columns to reduce the amount of data being fetched from external storage? Whether during external table creation or using an operator at query time.
Background: The reason I ask is that in Databricks it is possible to use the SELCECT statement to only fetch the columns I'm interested in. This reduces the query time significantly.
As David wrote above, the optimization does happen on Kusto side, but there's a bug with the "Downloaded Size" metric - it presents the total data size, regardless of the selected columns. We'll fix. Thanks for reporting.
I have a very large dataset to which I can only access by a pivot table in Excel.
I would like to access to the raw database to work with R.
I have tried several things:
To copy and paste in both, text and excel files, the pivot database and then import to R : It does not work. its shows only the filters that I have selected but not the totality of the database
to click on any cell of the pivot table to see the underlying data : It only shows the first 1000 entries. This is by far the best result that I have got, the only problem is that I get the first 1000 entries and I need all of them.
What I have been doing just until now is to select the variables that interest me, then, I copy them to a new sheet and finally I export to R. But I always forget a variable and its very expensive in time to redo all this procedure everyday.
Does anyone know how can I access to the totality of a database from a pivot table in Excel?
I hope I have been clear, do nto hesitate to ask for more information if needed.
Thank you in advanced
I'm trying to Export a very large amount of data as XML using plsql (around 80000000 characters). Unfortunately I'm currently unable to use FILE_UTL due to system privileges, and have been suggested to store it in a local table. Annoyingly, this table is cutting off the data at around 800 characters. While I'm aware plsql has a limit of 4000 characters, I'm still unsure how to proceed, and to keep my data stored so I can manually export it, or automatically do so.
TIA
I'm trying to read a .sql file into SQLite, but I'm getting syntax errors because the file was dumped from MySQL, which can add multiple entries at once, but I'm using SQLite v3.7.7, which can't read more than one entry to a table at a time with the VALUES command.
My understanding is that I either need to upgrade SQLite, or somehow modify the file to read in one entry at a time into the tables. Please note I'm dealing with tens of thousands of entries, so inserting the UNION SELECT command probably won't be very easy.
You need at least SQLite 3.7.11 to use the VALUES syntax you're interested in. But mysqldump has about 100 command-line options. And one of them, --skip-extended-insert, can disable extended inserts. (So you get one INSERT statement per row.) Read the mysqldump documentation, and run the dump again with options that better fit your target.
Or better yet, look at the list of SQLite converter tools.
I have lots of data to wrangle and I need some help.
I have been using an excel file that has two worksheets of interest to me. They each produce a OLAP pivot table with the data I need to work with. What I would like to do is move those (.odc) connections to access queries so I don't have to hand paste all of this info out and manipulate it and then go through the whole process several more times.
One table is Throughput (number of parts through an operation(s)) by Part Number and by Date. The other is Hours Logged at the operation(s) by Part Number and by Date. I also have a master list of all part numbers with some more data that I have to mix in.
Biggest problem: Each chart is producing its own subset of dates and part numbers so I have to take care to match up the data to run the calculations. I've tried:
By hand. Got tired with that real quick.
Using LOOKUP, VLOOKUP, MATCH with INDIRECT and all sorts of tricks.
It's a mess. But I'm confident that if I can put the original pivot tables into Access I can add a few joins and write up a couple queries and it will turn out beautifully.
Worst comes to worse I can copy/paste the pivot table data into access by hand, but what if I want to change or expand the data set? I'd rather work with the raw data.
EDIT:
The data is held on SQL Server and I cannot change that.
The excel pivot tables use a .ODC file for the connection. They gives the following connection string:
Provider=MSOLAP.3;Integrated Security=SSPI;Persist Security Info=True;Initial Catalog=[MyCatalog];Data Source=[MySource];MDX Compatibility=1;Safety Options=2;MDX Missing Member Mode=Error
Provider=MSOLAP.4;Integrated Security=SSPI;Persist Security Info=True;Initial Catalog=[MyCatalog];Data Source=[MySource];MDX Compatibility=1;Safety Options=2;MDX Missing Member Mode=Error
(I replaced the actual catalog and source)
Can I use the .odc file information to create a pass through query in Access?
Have you consider using a proper OLAP server?
Comparison of OLAP Servers
Once setup you'll be able to connect your Excel's Pivot Table to the server (as well as other reporting tools).
Talked to our IT dept. The guy who built the Cubes is working on querying the same info into MS Access for me.
Thanks everyone.