RecordSortedList and temporary table - axapta

I have a performance issue with multiple temporary tables that I'm trying to solve with RecordSortedList, but I'm getting strange results. I have a temporary table that has a couple hundred thousand records being inserted into it, and then used elsewhere for joins to other temporary tables. The problem is after trace parsing this solution the insert is taking too long for all the individual inserts and I was hoping to use a RecordSortedList to bulk insert into the staging table. However, I can't find a handle to the temporary table after the RecordSortedList.insertDatabase() call.
I've tried something like this:
RecordSortedList tmpTableSortedList;
MyTempTable myTempTable;
AssetTrans assetTrans;
int i = 1;
tmpTableSortedList = new RecordSortedList(tableNum(MyTempTable));
tmpTableSortedList.sortOrder(fieldNum(MyTempTable, LineNum));
//the real scenario has a much more complicated data gathering, but just for sample
while select * from AssetTrans
{
myTempTable.AssetGroup = assetTrans.AssetGroup
myTempTable.LineNum = i;
tmpTableSortedList.ins(myTempTable);
i++;
}
tmpTableSortedList.insertDatabase();
//strange things happen here
MyTempTable myTempTableCopy;
AnotherTmpTable anotherTmpTable;
tmpTableSortedList.first(myTempTableCopy); //returns a buffer, but not usable buffer in join.
//does not work, I imagine because the myTempTableCopy isn't actually pointing to the
//inserted records above; somehow the temp table is out of scope.
while select * from anotherTmpTable
join myTempTableCopy
where anotherTmpTable.id == myTempTableCopy.id
{
//logic
}
Is there a way to get a pointer to the temp table after the call to RecordSortedList.insertDatabase()? I've also tried linkPhysicalTable() and a few other things, but maybe RecordSortedList was not supposed to be used with tempDb tables?
Edit: Like Aliaksandr points out below this works with RecordInsertList instead of RecordSortedList

but maybe RecordSortedList was not supposed to be used with tempDb tables?
Error message when using TempDb tables:
RecordInsertList or RecordSortedList operations are not allowed with database temporary tables.
So it's not allowed, which might make sense because RecordSortedList is a memory-based object and TempDb tables are not. I would think you could though because I'm not sure there's a huge difference in a TempDb table and a Regular table when they're both stored on disk?
If you wanted to use an InMemory table, look at \Classes\CustVendSettle specifically the variable rslTmpOverUnderReverseTax, which uses an InMemory table.
IF TempDb tables were allowed, you would use getPhysicalTableName() to get the handle combined with useExistingTempDBTable().
Or did I misread your question?

does not work, I imagine because the myTempTableCopy isn't actually pointing to the inserted records above; somehow the temp table is out of scope.
Method new of RecordSortedList has additional Common parameter where you should pass your tempDB table buffer.
Error message when using TempDb tables:
RecordInsertList or RecordSortedList operations are not allowed with database temporary tables.
So it's not allowed, which might make sense because RecordSortedList is a memory-based object and TempDb tables are not.
Although the message says we can't use temporary tables for such operations, indeed we can. We just need to be careful because the code must be executed on the server.
RecordSortedList objects must be server-located before the insertDatabase method can be called. Otherwise, an exception is thrown.
I have a temporary table that has a couple hundred thousand records being inserted into it
There is no limit to the size of a RecordSortedList object, but they are completely memory-based, so there are potential memory consumption problems. So this may not be the best solution in your case.

Related

Accessing a TEMP TABLE in a TRIGGER on a VIEW

I need to parameterize a view, and I am doing so by creating a TEMP TABLE which has the parameters for the view.
CREATE TEMP TABLE parms (parm1 INTEGER, parm2 INTEGER);
CREATE VIEW tableview AS ...
The VIEW is rather complex, but it basically uses these two parameters to kick start a recursive CTE, and there isn't any other way that I have found to express the view without these parameters.
The parameters must be stored in a temporary table because each connection should be able to have its own view with different parameters.
In any case, this works fine for creating the view itself, so long as I create the same TEMP TABLE at the start of any queries that use the view, e.g.:
CREATE TEMP TABLE parms (parm1 INTEGER, parm2 INTEGER);
INSERT INTO parms (parm1,parm2) VALUES (5,66);
SELECT * FROM tableview;
I am able to do the same thing to create a trigger to allow inserts on the view:
CREATE TEMP TABLE parms (parm1 INTEGER, parm2 INTEGER);
CREATE TRIGGER tableinsert INSTEAD OF INSERT ON tableview ...
However, when I try to do an actual INSERT (re-creating the TEMP TABLE first as before) I get an error:
no such table: main.parms
If I create a non-temporary table, I do not get this error, but then I have the problem that different connections can't have their own separate views.
I have review the documentation for triggers, and it mentions caveats of using temporary triggers on a non-temporary table, but I don't see anything regarding the reverse.
I did find a reference elsewhere that indicated that "the table... must exist in the same database as the table or view to which the trigger is attached". I thought a temporary table was part of the current database, is this not true? Is there some way to make this true?
I also tried accessing the parms table as temp.parms in the TRIGGER, but got the error:
qualified table names are not allowed on INSERT, UPDATE, and DELETE
statements within triggers
If I can't use a temporary table, is there some way to work around it to accomplish the same thing?
Update: Ok, so it seems to be an SQLite limitation. After digging around a bit in the SQLite source code, it seems to be pretty trivial to allow SELECT access to a temporary table in a trigger. However, allowing UPDATE access appears to be a lot harder.
Temporary objects are created in a separate database named temp, so they are not accessible from triggers in other databases.
The remaining mechanism to get a connection-specific value into a trigger is to use a user-defined function.

Dynamic Sql VS Temporary Tables

I have a stored procedure, in which temporary tables "on delete preserve rows" are created dynamically and data was inserted, when i try to execute any other dynamic sql statement, data in the temporary tables are deleted. But I need a data for further process.
Can any one tell me why data is losing, and what is the solution for this.
Thank you.
Three possible reasons for this:
There is an explicit commit.
There is an implicit commit (a DDL statement, typically).
You are are closing the session and starting a new one.
If you cannot avoid these then you'll have to create a permanent table.

Teradata: Is there a way to generate DDL from a view or select statement?

I am using a global application user account to access database A. This user account does not have permissions to modify database A's schema (ie, create tables, modify tables, etc). This user also has access to database B, but only views. I need to run SQL to feed data from a view in database B into a table in database A.
In a perfect world, I would be able to use this SQL:
create database_a.mytable as (select * from database_b) with no data
However, the user can't create tables in database A. If I could get the DDL of the select statement then I could log in under my personal account (which doesn't have any access to database B) and run the DDL in database A to create the table.
The only other option is to manually write the SQL, but I don't want to do that, especially since this view I am wanting to copy has many columns of varying data types and sizes.
Edit: I may be getting closer. I just experimented with this:
show (select * from database_b.myview)
However, it generated the DLL of every single table that is used in the view itself, as well as the definition for the view. This doesn't really help me since I just want the schema of the select statement itself. In other words, I need what would be generated if I were to use the create table as statement mentioned above.
Edit for Rob: Perhaps "DDL" was the wrong term to use. Using show view db.myview just shows the definition of the view, not the schema it represents. In my above example of create table as, I show how you can create a table that mimics the schema of a result set returned in a select. It generates a DDL on the back end for creating a table and then executes that DDL to actually create the table. You can then say show table db.newtable and see the new table's DDL. I want to get that DDL directly from a select statement so that I can copy it, log out of the app account, into my personal account, and then execute the DDL to create the table.
This is only to save me the headache of having to type out the DDL manually by hand to save time and reduce typing errors, especially since the source view has so many columns. That said, I think hitting up the DBA or writing some snazzy stored procedure to do dynamic stuff would be a bit over the top for my needs. I think there has to be a way to get the DDL for creating a table schema directly from a select statement.
Generate DDL Statements for objects:
SHOW TABLE {DatabaseB}.{Table1};
SHOW VIEW {DatabaseB}.{View1};
Breakdown of columns in a view:
HELP VIEW {DatabaseB}.{View1};
However, without the ability to create the object in the target database DatabaseA your don't have much leverage. Obviously, if the object already existed INSERT INTO SELECT ... FROM DatabaseB.Table1 or MERGE INTO would be options that you already explored.
Alternative Solution
Would it be possible to have a stored procedure created that dynamically created the table based on the view name that is provided? The global application account would simply need privilege to execute the procedure. Generally the user creating the stored procedure would need the permissions to perform the actions contained within the stored procedure. (You have some additional flexibility with this in Teradata 13.10.)
There are some caveats with this approach. You are attempting to materialize views that could reference anywhere from hundreds to billions of records. These aren't simple 1:1 views that are put on top of the target tables. Trying to determine the required space in the target database to materialize the view will be difficult. Performance can and will vary depending on the complexity of the view and the data volumes. This will not be a fast-path or data block optimized operation.
As a DBA, I would be concerned with this approach being taken on by a global application account without fully understanding the intent. I trust you have an open line of communication with the DBA(s) involved for supporting this system. I'm sure there are reasons for your madness that can't be disclosed here.
Possible Solution - VOLATILE TABLE
Unless the implicit privilege for CREATE TABLE has been revoked from the global application account this solution should work.
Volatile tables do not require perm space. There table definitions persist for the duration of the session and any data inserted into them relies on the spool space of the user who instantiated it.
CREATE VOLATILE TABLE {Global Application UserID}.{TableA_Copy} AS
(
SELECT *
FROM {DatabaseB}.{TableA}
)
WITH NO DATA
NO PRIMARY INDEX
ON COMMIT PRESERVE ROWS;
SHOW TABLE {Global Application UserID}.{TableA_Copy};
I opted to use a Teradata 13.10 feature called NO PRIMARY INDEX. By default, CREATE TABLE AS will take the first column of the SELECT statement and make it the PRIMARY INDEX of the table. This could lead to skewing and perm space issues in your testing depending on the data demographics. You can specify an explicit PRIMARY INDEX on your own as you understand the underlying data. (See the DDL manuals for details on the syntax if you're uncertain.)
The use of ON COMMIT PRESERVE ROWS for the intent of this example is probably extraneous. But in reality if you popped any data into that table for testing this clause would be beneficial in Teradata mode as the data would otherwise be lost immediately after the CREATE TABLE or any other data manipulation was performed against the volatile table.

SQL Delete taking too long

We have a table(say T1) that is referenced by about 16 other tables with foreign keys in our SQL Server database. The data is accessed through an ASP.NET application with LINQToSQL. When the user tried to delete a record from T1 the statement would time out. So we decided to first delete the records from the tables that reference T1 and only then delete the record in T1. The problem is that deletion from T1 does not work as fast as expected.
My question is: is it normal that deletion from a table referenced by many other tables to be so time-consuming even if the record itself does not have any 'children' records?
EDIT: Apparently the cause for the timeout was not the delete itself but another query that retrieved data from the same DataContext. Thank you for your suggestions, I have marked as answer the suggestion to add indexes for all foreign keys because it improved our script's execution plan.
I suspect that you may need to look into the indexing on your child tables.
It sounds as if you FKs are set to Cascade Deletes, so I would suspect that some of your tables do not have an index that includes the key to the parent as the first in the index.
In this way your delete will be full scanning the child tables - even if you've already deleted the child records it will still check as you've still got the Cascade set.
When you define a relationship in DB, you can set the Delete rule as Cascade in SQL server. In this way, when you delete the record from the parent table, it will be automatically deleted from the child tables.
Please see the image below:
If it taking long time, you may have set other constraint that will slow
down the process of deletion.
Linq does not do bulk deletes if you're having it operate directly on the record set -- instead, it is probably deleting one record at a time.
To improve performance, use a stored procedure instead for any bulk insert, update or delete operations.

Why does this query timeout? V2

This question is a followup to This Question
The solution, clearing the execution plan cache seemed to work at the time, but i've been running into the same problem over and over again, and clearing the cache no longer seems to help. There must be a deeper problem here.
I've discovered that if I remove the .Distinct() from the query, it returns rows (with duplicates) in about 2 seconds. However, with the .Distinct() it takes upwards of 4 minutes to complete. There are a lot of rows in the tables, and some of the where clause fields do not have indexes. However, the number of records returned is fairly small (a few dozen at most).
The confusing part about it is that if I get the SQL generated by the Linq query, via Linqpad, then execute that code as SQL or in SQL Management Studio (including the DISTINCT) it executes in about 3 seconds.
What is the difference between the Linq query and the executed SQL?
I have a short term workaround, and that's to return the set without .Distinct() as a List, then using .Distinct on the list, this takes about 2 seconds. However, I don't like doing SQL Server work on the web server.
I want to understand WHY the Distinct is 2 orders of magnitude slower in Linq, but not SQL.
UPDATE:
When executing the code via Linq, the sql profiler shows this code, which is basically identical query.
sp_executesql N'SELECT DISTINCT [t5].[AccountGroupID], [t5].[AccountGroup]
AS [AccountGroup1]
FROM [dbo].[TransmittalDetail] AS [t0]
INNER JOIN [dbo].[TransmittalHeader] AS [t1] ON [t1].[TransmittalHeaderID] =
[t0].[TransmittalHeaderID]
INNER JOIN [dbo].[LineItem] AS [t2] ON [t2].[LineItemID] = [t0].[LineItemID]
LEFT OUTER JOIN [dbo].[AccountType] AS [t3] ON [t3].[AccountTypeID] =
[t2].[AccountTypeID]
LEFT OUTER JOIN [dbo].[AccountCategory] AS [t4] ON [t4].[AccountCategoryID] =
[t3].[AccountCategoryID]
LEFT OUTER JOIN [dbo].[AccountGroup] AS [t5] ON [t5].[AccountGroupID] =
[t4].[AccountGroupID]
LEFT OUTER JOIN [dbo].[AccountSummary] AS [t6] ON [t6].[AccountSummaryID] =
[t5].[AccountSummaryID]
WHERE ([t1].[TransmittalEntityID] = #p0) AND ([t1].[DateRangeBeginTimeID] = #p1) AND
([t1].[ScenarioID] = #p2) AND ([t6].[AccountSummaryID] = #p3)',N'#p0 int,#p1 int,
#p2 int,#p3 int',#p0=196,#p1=20100101,#p2=2,#p3=0
UPDATE:
The only difference between the queries is that Linq executes it with sp_executesql and SSMS does not, otherwise the query is identical.
UPDATE:
I have tried various Transaction Isolation levels to no avail. I've also set ARITHABORT to try to force a recompile when it executes, and no difference.
The bad plan is most likely the result of parameter sniffing: http://blogs.msdn.com/b/queryoptteam/archive/2006/03/31/565991.aspx
Unfortunately there is not really any good universal way (that I know of) to avoid that with L2S. context.ExecuteCommand("sp_recompile ...") would be an ugly but possible workaround if the query is not executed very frequently.
Changing the query around slightly to force a recompile might be another one.
Moving parts (or all) of the query into a view*, function*, or stored procedure* DB-side would be yet another workaround.
 * = where you can use local params (func/proc) or optimizer hints (all three) to force a 'good' plan
Btw, have you tried to update statistics for the tables involved? SQL Server's auto update statistics doesn't always do the job, so unless you have a scheduled job to do that it might be worth considering scripting and scheduling update statistics... ...tweaking up and down the sample size as needed can also help.
There may be ways to solve the issue by adding* (or dropping*) the right indexes on the tables involved, but without knowing the underlying db schema, table size, data distribution etc that is a bit difficult to give any more specific advice on...
 * = Missing and/or overlapping/redundant indexes can both lead to bad execution plans.
The SQL that Linqpad gives you may not be exactly what is being sent to the DB.
Here's what I would suggest:
Run SQL Profiler against the DB while you execute the query. Find the statement which corresponds to your query
Paste the whole statment into SSMS, and enable the "Show Actual Execution Plan" option.
Post the resulting plan here for people to dissect.
Key things to look for:
Table Scans, which usually imply that an index is missing
Wide arrows in the graphical plan, indicating lots of intermediary rows being processed.
If you're using SQL 2008, viewing the plan will often tell you if there are any indexes missing which should be added to speed up the query.
Also, are you executing against a DB which is under load from other users?
At first glance there's a lot of joins, but I can only see one thing to reduce the number right away w/out having the schema in front of me...it doesn't look like you need AccountSummary.
[t6].[AccountSummaryID] = #p3
could be
[t5].[AccountSummaryID] = #p3
Return values are from the [t5] table. [t6] is only used filter on that one parameter which looks like it is the Foreign Key from t5 to t6, so it is present in [t5]. Therefore, you can remove the join to [t6] altogether. Or am I missing something?
Are you sure you want to use LEFT OUTER JOIN here? This query looks like it should probably be using INNER JOINs, especially because you are taking the columns that are potentially NULL and then doing a distinct on it.
Check that you have the same Transaction Isolation level between your SSMS session and your application. That's the biggest culprit I've seen for large performance discrepancies between identical queries.
Also, there are different connection properties in use when you work through SSMS than when executing the query from your application or from LinqPad. Do some checks into the Connection properties of your SSMS connection and the connection from your application and you should see the differences. All other things being equal, that could be the difference. Keep in mind that you are executing the query through two different applications that can have two different configurations and could even be using two different database drivers. If the queries are the same then that would be only differences I can see.
On a side note if you are hand-crafting the SQL, you may try moving the conditions from the WHERE clause into the appropriate JOIN clauses. This actually changes how SQL Server executes the query and can produce a more efficient execution plan. I've seen cases where moving the filters from the WHERE clause into the JOINs caused SQL Server to filter the table earlier in the execution plan and significantly changed the execution time.

Resources