We need to use GenerateTableFetch to fetch huge data from teradata using nifi. But since Teradata is not present as a database type it generate Limit keyword. I am able to replace it with Sample keyword. But sample everytime give random value so how can i use Nifi with Teradata for huge table?
Maybe try the "MS SQL 2008" adapter, it generates a TOP statement as well as a ROW_NUMBER() OVER clause. If that doesn't work, we'd need a TeradataDatabaseAdapter implementation, please feel free to file an improvement Jira as needed.
Related
I am new to datastage, I created a simple job to get data from .ds file and load it in teradata using teradata connector, in the properties of teradata connector I set the
access_method=Bulk, max_session=2, min_session=1,load_type=load,
max_buffer_size=10000 and max_partition_sessions=1
but the job is continously in running state without displaying amount of rows transfered. Whereas when I choose the access_method=immediate then it starts to proceed, can any one suggest me the right way to do load in parallel.
Are you aware of the nodes you are using? Add APT_CONFIG_FLE environment variable and try to see from director how many nodes are used. N number of nodes means 2n number of processes. This might help you.
If this doesn't help try looking into the source database connector stage to see if increasing values for any options helps.
I've had difficult to set new limit for sql length in the sqlite (version 3.10.2). I read the sql documentation and I noticed there is a function that allows to change the limit: sqlite3_limit(db,SQLITE_LIMIT_SQL_LENGTH,size)
However, this function is for C interface.
I'd like to know if there is a function can i use in sql to change the sql length because I'll work with large sql statements.
Did anyone get this problem?
Thank you very much.
Carlos
There is no SQL function for that.
And the sqlite3_limit() function allows to reduce the connection's limits, but not to increase them.
To increase the maximum length of an SQL statement, you would have to recompile the SQLite library with different compilation options.
I have a Conceptual Data Model in powerdesigner and I want generate Sqlite DB from it, how to generate sqlite database from CDM Model in PowerDesigner?
I have started a DBMS for SQLite 3
When you generate Physical Data Model, you can select, in the dropbox button for database selection, ANSI Level 2. It works flawlessly that way. Confirm your choice when you generate the script.
Just make sure to remove or comment the drop statements at the beginning of the resulting script and you should not have any error when running the script in a database client.
Use the Tools->Generate Physical Data Model... command and select an appropriate database from the list (probably the ODBC or ANSI options since SQLLite isn't an out-of-the-box option.
OR, you could first create a database XEM for SQLLite, but that's a pretty advanced task. I'd stick with the generic if possible.
Ok. I have been pulling my hair out for the past couple of days trying to do the worlds simplest report...
So after figuring out the table associations, and pulling sample data, it has come to my attention that I need to change up how Im pulling the data.
Part of this is pulling Year-to-Date from the Invoice table.
However, qodbc is stupid (its probably me, but it makes me feel better to blame the driver)
SELECT * FROM Invoice WHERE TimeCreated > '2014-01-01 00:00:00.000' keeps giving me the error of Invalid operand for operator: >
Searching Google has provided no help for me.
Soooo... I need help with searching by a date field. Any one have ANY ideas or suggestions?
Also, bonus points but related... Anyone else have issues with the speed of the qodbc driver? Some tables can be searched just as fast as mysql, but some tables... Holy crap. 10 mins for a simple query. Ideas for improving the speed of those?
Date Format
SELECT *
from InvoiceLine
WHERE Txndate >= {d '2005-09-23'}
Time Stamps Format
SELECT *
FROM Customer
WHERE TimeCreated = {ts '1999-07-29 14:24:18.000'}
SELECT *
from InvoiceLine
WHERE TimeModified >= {ts '2005-09-23 00:00:00.000'}
Refer: http://support.flexquarters.com/esupport/index.php?/Default/Knowledgebase/Article/View/2203/50/how-are-dates-formatted-in-sql-queries-when-using-the-quickbooks-generated-time-stamps
How to make QODBC running faster
Keep in mind that QODBC is not a database tool, but rather a translation tool. QuickBooks is a non-normalized flat file system which has no indexes available and will not perform like SQL Server or dBase files. Every transaction you request must be translated and communicated to QuickBooks via large complicated XML transactions.
Try and keep your result set as small as possible to get a feel for the system, carefully design and test any multi-file joins, and keep the number of returned fields to a minimum for maximum performance.
Our main goal is to make it easier to access QuickBooks data in a standardised database-like fashion, but queries must be optimized to perform as fast as possible.
Also, try and use ranges of dates on TxnDate, TxnDateMacro and TimeModified as much as possible to narrow down the data to the smallest possible segment. For example, make something similar to this in the first part of your WHERE clause:
Invoice.TimeModified >= {ts'2003-09-01 17:01:09'} AND
Invoice.TimeModified <= {ts'2003-09-02 17:01:09'}
I would suggest to use Optimizer. sp_optimizefullsync All
See: How to setup QODBC Optimizer and where are the Optimizer options for all the details about optimizer. (http://support.flexquarters.com/esupport/index.php?/Default/Knowledgebase/Article/View/2358/48/how-to-setup-qodbc-optimizer-and-where-are-the-optimizer-options)
By default the optimizer will update new and changed entries in a table from QuickBooks first and then execute the query against the local optimized table. This is faster than reading everything out of QuickBooks every time, especially the more data you have.
Never used this driver but try this:
SELECT * FROM Invoice WHERE TimeCreated > {d'2014-01-01 00:00:00.000'}
Might need to muck with the format of the date string a little, just a guess.
As far as the speed of your selects, if the queries have WHERE clauses this can be impacted by not having an index on the table. Tables with indexes will return results faster than tables without.
I'm migrating/consolidating multiple FMP6 databases to a single C# application backed by SQL Server 2008. the problem I have is how to export the data to a real database (SQL Server) so I can work on data quality and normalisation. Which will be significant, there are a number of repeating fields that need to be normalised into child tables.
As I see it there are a few different options, most of which involve either connecting to to FMP over ODBC and using an intermediate to copy the data across (either custom code or MS Acess linked tables), or, exporting to flat file format (CSV with no header or xml) and either use excel to generate insert statements or write some custom code to load the file.
I'm leaning towards writing some custom code to do the migration (like this article does, but in C# instead of perl) over ODBC, but I'm concerned about the overhead of writing a migrator that will only be used once (as soon as the new system is up the existing DB's will be archived)...
a few little joyful caveats: in this version of FMP there's only one table per file, and a single column may have multi-value attributes, separated by hex 1D, which is the ASCII group separator, of course!
Does anyone have experience with similar migrations?
I have done this in the past, but using MySQL as the backend. The method I use is to export as csv or merge format and them use the LOAD DATA INFILE statement.
SQL Server may have something similar, maybe this link would help bulk insert