Linq and SQL query comparison - asp.net

I have this SQL query:
SELECT Sum(ABS([Minimum Installment])) AS SumOfMonthlyPayments FROM tblAccount
INNER JOIN tblAccountOwner ON tblAccount.[Creditor Registry ID] = tblAccountOwner.
[Creditor Registry ID] AND tblAccount.[Account No] = tblAccountOwner.[Account No]
WHERE (tblAccountOwner.[Account Owner Registry ID] = 731752693037116688)
AND (tblAccount.[Account Type] NOT IN
('CA00', 'CA01', 'CA03', 'CA04', 'CA02', 'PA00', 'PA01', 'PA02', 'PA03', 'PA04'))
AND (DATEDIFF(mm, tblAccount.[State Change Date], GETDATE()) <=
4 OR tblAccount.[State Change Date] IS NULL)
AND ((tblAccount.[Account Type] IN ('CL10','CL11','PL10','PL11')) OR
CONTAINS(tblAccount.[Account Type], 'Mortgage')) AND (tblAccount.[Account Status ID] <> 999)
I have created a Linq query:
var ownerRegistryId = 731752693037116688;
var excludeTypes = new[]
{
"CA00", "CA01", "CA03", "CA04", "CA02",
"PA00", "PA01", "PA02", "PA03", "PA04"
};
var maxStateChangeMonth = 4;
var excludeStatusId = 999;
var includeMortgage = new[] { "CL10", "CL11", "PL10", "PL11" };
var sum = (
from account in context.Accounts
from owner in account.AccountOwners
where owner.AccountOwnerRegistryId == ownerRegistryId
where !excludeTypes.Contains(account.AccountType)
where account.StateChangeDate == null ||
(account.StateChangeDate.Month - DateTime.Now.Month)
<= maxStateChangeMonth
where includeMortgage.Contains(account.AccountType) ||
account.AccountType.Contains("Mortgage")
where account.AccountStatusId != excludeStatusId
select account.MinimumInstallment).ToList()
.Sum(minimumInstallment =>
Math.Abs((decimal)(minimumInstallment)));
return sum;
Are they equal/same ? I dont have records in db so I cant confirm if they are equal. In SQL there are brackets() but in Linq I didnt use them so is it ok?
Please suggest.

It is not possible for us to say anything about this, because you didn't show us the DBML. The actual definition of the mapping between the model and the database is important to be able to see how this executes.
But before you add the DBML to your question: we are not here to do your work, so here are two tips to find out whether they are equal or not:
Insert data in your database and run the queries.
Use a SQL profiler and see what query is executed by your LINQ provider under the covers.
If you have anything more specific to ask, we will be very willing to help.

The brackets will be generated by LINQ provider, if necessary.
The simplest way to check if the LINQ query is equal to the initial SQL query is to log it like #Atanas Korchev suggested.
If you are using Entity Framework, however, there is no Log property, but you can try to convert your query to an ObjectQuery, and call the ToTraceString method then:
string sqlQuery = (sum as ObjectQuery).ToTraceString();
UPD. The ToTraceString method needs an ObjectQuery instance for tracing, and the ToList() call already performs materialization, so there is nothing to trace. Here is the updated code:
var sum = (
from account in context.Accounts
from owner in account.AccountOwners
where owner.AccountOwnerRegistryId == ownerRegistryId
where !excludeTypes.Contains(account.AccountType)
where account.StateChangeDate == null ||
(account.StateChangeDate.Month - DateTime.Now.Month)
<= maxStateChangeMonth
where includeMortgage.Contains(account.AccountType) ||
account.AccountType.Contains("Mortgage")
where account.AccountStatusId != excludeStatusId
select account.MinimumInstallment);
string sqlQuery = (sum as ObjectQuery).ToTraceString();
Please note that this code will not perform the actual query, it is usable for testing purposes only.
Check out this article if you are interested in ready-for-production logging implementation.

There can be a performance difference:
The SQL query returns a single number (SELECT Sum...) directly from the database server to the client which executes the query.
In your LINQ query you have a greedy operator (.ToList()) in between:
var sum = (...
...
select account.MinimumInstallment).ToList()
.Sum(minimumInstallment =>
Math.Abs((decimal)(minimumInstallment)));
That means that the query on the SQL server does not contain the .Sum operation. The query returns a (potentially long?) list of MinimumInstallments. Then the .Sum operation is performed in memory on the client.
So effectively you switch from LINQ to Entities to LINQ to Objects after .ToList().
BTW: Can you check the last proposal in your previous question here which would avoid .ToList() on this query (if the proposal should work) and would therefore be closer to the SQL statement.

Related

LINQ Entities Where clause not in correct place

Apparently I'm missing something with how LINQ to entities works. Hopefully one of you all can educate me.
Please try the below locally and let me know if you are seeing the same results. Something is really strange here...
Lets look at a very simple LINQ expression using navigation properties.
This was generated in LinqPad in a C# statement.
var result = (from ge in group_execution
where ge.automation_sequences.project.client_id == 1 && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
});
result.Dump();
OR, we can use joins...which will lead to the same bad results, but lets continue...
var result = (from ge in group_execution
join aseq in automation_sequences on ge.automation_sequence_id equals aseq.id
join p in project on aseq.project_id equals p.id
where p.client_id == 1 && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
p.client_id
});
result.Dump();
These very simple LINQ expressions generate the following SQL:
SELECT
[Filter1].[id1] AS [id],
[Filter1].[parent_group_exec_id] AS [parent_group_exec_id],
[Extent5].[client_id] AS [client_id]
FROM (SELECT [Extent1].[id] AS [id1], [Extent1].[automation_sequence_id] AS [automation_sequence_id], [Extent1].[parent_group_exec_id] AS [parent_group_exec_id]
FROM [dbo].[group_execution] AS [Extent1]
INNER JOIN [dbo].[automation_sequences] AS [Extent2] ON [Extent1].[automation_sequence_id] = [Extent2].[id]
INNER JOIN [dbo].[project] AS [Extent3] ON [Extent2].[project_id] = [Extent3].[id]
WHERE ([Extent1].[parent_group_exec_id] IS NOT NULL) AND (1 = [Extent3].[client_id]) ) AS [Filter1]
LEFT OUTER JOIN [dbo].[automation_sequences] AS [Extent4] ON [Filter1].[automation_sequence_id] = [Extent4].[id]
LEFT OUTER JOIN [dbo].[project] AS [Extent5] ON [Extent4].[project_id] = [Extent5].[id]
This baffles me. For the life of me I can't understand why LINQ is doing this. It's horrible, just look at the execution plan:
Now lets manually clean this up in SSMS and view the correct SQL and execution plan:
Much better, but how do we get LINQ to act this way?
Is anyone else seeing this? Has anyone else ever saw this and corrected it and if so how?
Thanks for looking into this.
UPDATE, attempting Chris Schaller fix:
var result = (from ge in group_execution
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
}).Where(x=>x.client_id == 1 && x.parent_group_exec_id != null);
result.Dump();
Just so you all know I'm monitoring the SQL through SQL Server Profiler. If anyone knows of any issues doing it this way let me know.
UPDATE, a fix for JOINS, but not nav properties, and a cause, but why?
Here's your solution:
var result = (from ge in group_execution.Where(x=>x.parent_group_exec_id != null)
join aseq in automation_sequences on ge.automation_sequence_id equals aseq.id
join p in project on aseq.project_id equals p.id
where p.client_id == 1// && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
p.client_id
});
result.Dump();
Null checks shouldn't cause the framework to mess up like this. Why should I have to write it this way? This just seems like a defect to me in the framework. It will make my dynamic expressions a little bit more difficult to write, but maybe I can find a way.
Navigation Properties still mess up...so I'm still really sad. Picture below:
var result = (from ge in group_execution.Where(x=>x.parent_group_exec_id != null)
where ge.automation_sequences.project.client_id == 1// && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
});
result.Dump();
move your where clause to after you have defined the structure of the select statement
var result = (from ge in group_execution
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
}).Where(x => x.client_id == 1 && x.parent_group_exec_id != null)
result.Dump();
Remember that Linq-to-entities flattens the results of queries to execute as SQL and then hydrates the object graph from those results.
When your query uses navigation properties or joins the query parser has to allow for zero results from those sub queries (Extents) to make sure that all columns that are required in the output and any interim processing are represented. By explicitly specifying a filter on a table for != null early in the query the parser knows that there is no further possibility that the field and any relationships linked by that field will be null, until then the parser prepares the query as if the joins will return null results
It is worth checking, but i wonder if UseDatabaseNullSemantics has anything to do with this?
Try:
dbContext.Configuration.UseDatabaseNullSemantics = false
In Linq, we can specify Where clauses as often as we like, improve the resulting SQL we should filter early in the query and granularly.
The parser engine is optimized to follow and implement your query sequentially, and generate good SQL at the end of it. Don't try to write linq-to-entities the same way that you structure your SQL, I know it's counter intuitive because the syntax is similar
A good technique is to assume that before each clause all the records from the previous statements have been loaded into memory, and that the next operation will affect all of those records. So you want to reduce the records before each additional operation by specifying a filter before moving on to the next clause
In general, if you have a filter condition based on the root table, apply this to the query before define all other joins and filters and even selects, you will get much cleaner sql.

How to bind select query as datasource in report?

I have a select statement that I want to bind as data source in a report.
I have not found a way to design an appropriate AOT query.
This is how it looks like in X++
public void insertData(date data = today())
{
BHNEmployeesOnDay ins;
EmplTable tbl;
CompanyInfo info;
BHNEmplAgreements Agreemnt;
BHNEmplAgreements Agreemnt2;
BHNEMPLHISTORYCOMPANY history;
BHNEMPLHISTORYCOMPANY history_test;
BHNDIVISIONTABLE division;
BHNPOSITIONTABLE position;
SysCompanyUserInfo sys1;
SysUserInfo sys2;
UserInfo usrInfo;
Date infinity = mkdate(1,1,1900);
;
delete_from ins;
while select * from tbl
join Info where info.dataAreaId == tbl.dataAreaId && info.BLX_companyForDW == 1
join sys1 where sys1.EmplId==tbl.EmplId && sys1.dataAreaId == tbl.dataAreaId
join sys2 where sys1.UserId==sys2.Id
join usrInfo where usrInfo.id==sys1.UserId
exists join history_test
where history_test.EmplId==tbl.EmplId && history_test.dataAreaId==tbl.dataAreaId
join Agreemnt where Agreemnt.HistoryId==history_test.HistoryId
&& (agreemnt.DateTo >= data || agreemnt.DateTo==infinity)
{
select firstonly *
from history order by history.DateFrom desc, Agreemnt2.DateFrom desc
where history.EmplId==tbl.EmplId && history.dataAreaId==tbl.dataAreaId
join Agreemnt2 where Agreemnt2.HistoryId==history.HistoryId
&& Agreemnt2.DateFrom<=data && (Agreemnt2.DateTo >= data || Agreemnt2.DateTo==infinity)
join division where division.DivisionId==agreemnt.DivisionId
join position where position.PositionId==agreemnt.PositionId;
ins.adddRecord(tbl.EmplId, tbl.Name_BHN, tbl.BirthDate, division.Name, position.FullName);
}
}
Currently I generate data into a table [during run() method of the report], then simply select from that table. So far only 1 person uses this report so it's not a problem, but if two people run the same report simultaneously, I'm gonna get dirty reads.
I know it's bad approach, but I'm out of ideas. I thought of making a View on T-SQL side and try to select from it - but I was told that it might not be detected or simply not transferred to other instances of our AX during export, so it has to be done on AX side.
How can I solve this?
Just in case this is query in T-SQL SQL query on pastebin
You could overwrite the report's fetch method and just use your X++ code as is to get the records and then use the report's send method to process them.
See here for an example.
The example uses a query object but you could easily swap that with your own X++ code - you just eventually have to call send for the records you want to be processed by the report.
Update:
For example you could just fetch any record of SalesTable and call send.
In this example a member variable salesTable is assumed so that you can access the current record in a display method in case you need it.
public boolean fetch()
{
boolean ret;
//ret = super();
;
select firstOnly salesTable;
this.send(salesTable);
return true;
}

Efficient joining the most recent record from another table in Entity Framework Core

I am comming to ASP .NET Core from PHP w/ MySQL.
The problem:
For the illustration, suppose the following two tables:
T: {ID, Description, FK} and States: {ID, ID_T, Time, State}. There is 1:n relationship between them (ID_T references T.ID).
I need all the records from T with some specific value of FK (lets say 1) along with the related newest record in States (if any).
In terms of SQL it can be written as:
SELECT T.ID, T.Description, COALESCE(s.State, 0) AS 'State' FROM T
LEFT JOIN (
SELECT ID_T, MAX(Time) AS 'Time'
FROM States
GROUP BY ID_T
) AS sub ON T.ID = sub.ID_T
LEFT JOIN States AS s ON T.ID = s.ID_T AND sub.Time = s.Time
WHERE FK = 1
I am struggling to write an efficient equivalent query in LINQ (or the fluent API). The best working solution I've got so far is:
from t in _context.T
where t.FK == 1
join s in _context.States on t.ID equals o.ID_T into _s
from s in _s.DefaultIfEmpty()
let x = new
{
id = t.ID,
time = s == null ? null : (DateTime?)s.Time,
state = s == null ? false : s.State
}
group x by x.id into x
select x.OrderByDescending(g => g.time).First();
When I check the resulting SQL query in the output window when executed it is just like:
SELECT [t].[ID], [t].[Description], [t].[FK], [s].[ID], [s].[ID_T], [s].[Time], [s].[State]
FROM [T] AS [t]
LEFT JOIN [States] AS [s] ON [T].[ID] = [s].[ID_T]
WHERE [t].[FK] = 1
ORDER BY [t].[ID]
Not only it selects more columns than I need (in the real scheme there are more of them). There is no grouping in the query so I suppose it selects everything from the DB (and States is going to be huge) and the grouping/filtering is happening outside the DB.
The questions:
What would you do?
Is there an efficient query in LINQ / Fluent API?
If not, what workarounds can be used?
Raw SQL ruins the concept of abstracting from a specific DB technology and its use is very clunky in current Entity Framework Core (but maybe its the best solution).
To me, this looks like a good example for using a database view - again, not really supported by Entity Framework Core (but maybe its the best solution).
What happens if you try to do a more straight forward translation to LINQ?
var latestState = from s in _context.States
group s by s.ID_T into sg
select new { ID_T = sg.Key, Time = sg.Time.Max() };
var ans = from t in _context.T
where t.FK == 1
join sub in latestState on t.ID equals sub.ID_T into subj
from sub in subj.DefaultIfEmpty()
join s in _context.States on new { t.ID, sub.Time } equals new { s.ID, s.Time } into sj
from s in sj.DefaultIfEmpty()
select new { t.ID, t.Description, State = (s == null ? 0 : s.State) };
Apparently the ?? operator will translate to COALESCE and may handle an empty table properly, so you could replace the select with:
select new { t.ID, t.Description, State = s.State ?? 0 };
OK. Reading this article (almost a year old now), Smit's comment to the original question and other sources, it seems that EF Core is not really production ready yet. It is not able to translate grouping to SQL and therefore it is performed on the client side, which may be (and in my case would be) a serious problem. It corresponds to the observed behavior (the generated SQL query does no grouping and selects everything in all groups). Trying the LINQ queries out in Linqpad it always translates to a single SQL query.
I have downgraded to EF6 following this article. It required some changes in my model's code and some queries. After changing .First() to .FirstOrDefault() in my original LINQ query it works fine and translates to a single SQL query selecting only the needed columns. The generated query is much more complex than it is needed, though.
Using a query from NetMage's answer (after small fixes), it results in a SQL query almost identical to my own original SQL query (there's only a more complex construct than COALESCE).
var latestState = from s in _context.States
group s by s.ID_T into sg
select new { ID = sg.Key, Time = sg.Time.Max() };
var ans = from t in _context.T
where t.FK == 1
join sub in latestState on t.ID equals sub.ID into subj
from sub in subj.DefaultIfEmpty()
join s in _context.States
on new { ID_T = t.ID, sub.Time } equals new { s.ID_T, s.Time }
into sj
from s in sj.DefaultIfEmpty()
select new { t.ID, t.Description, State = (s == null ? false : s.State) };
In LINQ it's not as elegant as my original SQL query but semantically it's the same and it does more or less the same thing on the DB side.
In EF6 it is also much more convenient to use arbitrary raw SQL queries and AFAIK also the database views.
The biggest downside of this approach is that full .NET framework has to be targeted, EF6 is not compatible with .NET Core.

Unable to create a constant value of type 'X.Models.Game'. Only primitive types or enumeration types are supported in this context

I've got a complex query written in SQL which works.
SELECT Instruments.Id as [Id], Instruments.ShareCode as [Share Code], Instruments.Name AS [Short Name], Instruments.Description as [Share Name],
InstrumentGames.Instrument_Id,
InstrumentGames.Game_Id, Games.Name AS [Game Name],
Entries.Name AS [Entry Name], AspNetUsers.UserName, AspNetUsers.Id as [User_Id],
Sectors.Name AS Sector_Id, Sectors.ShortName AS Sector
FROM AspNetUsers INNER JOIN
Entries ON AspNetUsers.Id = Entries.User_Id INNER JOIN
Games ON Entries.Game_Id = Games.Id INNER JOIN
InstrumentGames ON Games.Id = InstrumentGames.Game_Id INNER JOIN
Instruments ON InstrumentGames.Instrument_Id = Instruments.Id INNER JOIN
Sectors ON Instruments.Sector_Id = Sectors.Id
WHERE Instruments.Listed = 'true' and InstrumentGames.Game_Id = 2 and Entries.User_Id = 'd28d6552-7d98-476c-82cb-063e7ef45cb6'
I'm using Entity code first models and trying to convert what I have in SQL to a linq query.
I've come up with:
public static Models.Instrument GetShare(string shareSearchCriteria,
Models.Game selectedGame,
string userId)
{
var _db = new JSEChallenge.Models.ApplicationDbContext();
var records = (from instru in _db.Instruments
from e in _db.Entries
where (instru.ShareCode.Contains(shareSearchCriteria) ||
instru.Name.Contains(shareSearchCriteria) ||
instru.Description.Contains(shareSearchCriteria))
where (instru.Listed == true &&
instru.Games.Contains(selectedGame) &&
e.User_Id == userId)
select instru).ToList();
return records.FirstOrDefault();
}
But I keep getting this error:
Unable to create a constant value of type 'X.Models.Game'. Only primitive types or enumeration types are supported in this context.
I think the issue is the m2m table InstrumentGames. In my SQL query I can join it easily but in my C# I cannot. The way I usually find m2m records is syntax like instru.Games.Contains(selectedGame)
Unfortunately I still cannot get this to work.
How do I implement this kind of query in Linq?
Not sure if it will work, but try instru.Games.Any(q => q.Id == selectedGame.Id) instead of instru.Games.Contains(selectedGame).
Hope this helps!

LINQ - 'Could not translate expression' with previously used and proven query condition

I am fairly new to LINQ and can't get my head around some inconsistency in behaviour. Any knowledgeable input would be much appreciated. I see similar issues on SO and elsewhere but they don't seem to help.
I have a very simple setup - a company table and an addresses table. Each company can have 0 or more addresses, and if > 0 one must be specified as the main address. I'm trying to handle the cases where there are 0 addresses, using an outer join and altering the select statement accordingly.
Please note I'm currently binding the output straight to a GridView so I would like to keep all processing within the query.
The following DOES work
IQueryable query =
from comp in context.Companies
join addr in context.Addresses on comp.CompanyID equals addr.CompanyID into outer // outer join companies to addresses table to include companies with no address
from addr in outer.DefaultIfEmpty()
where (addr.IsMain == null ? true : addr.IsMain) == true // if a company has no address ensure it is not ruled out by the IsMain condition - default to true if null
select new {
comp.CompanyID,
comp.Name,
AddressID = (addr.AddressID == null ? -1 : addr.AddressID), // use -1 to represent a company that has no addresses
MainAddress = String.Format("{0}, {1}, {2} {3} ({4})", addr.Address1, addr.City, addr.Region, addr.PostalCode, addr.Country)
};
but this displays an empty address in the GridView as ", , ()"
So I updated the MainAddress field to be
MainAddress = (addr.AddressID == null ? "" : String.Format("{0}, {1}, {2} {3} ({4})", addr.Address1, addr.City, addr.Region, addr.PostalCode, addr.Country))
and now I'm getting the Could not translate expression error and a bunch of spewey auto-generated code in the error which means very little to me.
The condition I added to MainAddress is no different to the working condition on AddressID, so can anybody tell me what's going on here?
Any help is greatly appreciated.
The error you are getting is telling you that LinqToSql cannot translate your null check and then string.Format expression into SQL. If you look at the SQL your first query is generating (using either LinqPad or SQL Profiler), you'll see something like:
SELECT [t0].[CompanyID], [t0].[Name],
(CASE
WHEN [t1].[AddressID] IS NULL THEN #p0
ELSE [t1].[AddressID]
END) AS [AddressID],
[t1].[Address1] AS [value],
[t1].[City] AS [value2],
[t1].[Region] AS [value3],
[t1].[PostalCode] AS [value4],
[t1].[Country] AS [value5]
FROM [Company] AS [t0]
LEFT OUTER JOIN [Address] AS [t1] ON [t0].[CompanyID] = [t1].[CompanyID]
WHERE ([t1].[IsMain] IS NULL) OR ([t1].[IsMain] = 1)
For your AddressID field, you can see that it uses a CASE-WHEN to handle the condition when AddressID is null. When you add a CASE-WHEN for MainAddress, it's trying to do the same thing for that field, but there is no SQL equivalent to string.Format it can use for the ELSE clause, so it blows up.
An easy way around this problem is to use a method to format the string. By calling a private method, LinqToSql won't try to translate the string.Format to SQL, and will instead return all of the fields necessary to populate the Address object. The method can then take care of the formatting.
For example:
LINQ:
....
select new {
comp.CompanyID,
comp.Name,
AddressID = (addr.AddressID == null ? -1 : addr.AddressID),
MainAddress = FormatAddress(addr)
};
Method:
private static string FormatAddress(Address addr)
{
return (addr == null ? "" :
string.Format("{0}, {1}, {2} {3} ({4})",
addr.Address1, addr.City,
addr.Region, addr.PostalCode, addr.Country));
}

Resources