I am constructing a composite filter, It has OR and then AND filter operator when I set the sort order to this query it is not giving the correct result.
the query that I am trying to execute is { status == Initiated OR (Status == Failed AND failedCount <= 5 ) }. it actually executing the query without any error but the result is not exactly the correct. it retrieving only "status == failed AND failedCount <= 5 " not the status == initiated. and the order I need is "createdAt" field.
Filter initiatedFilter = new FilterPredicate("status", FilterOperator.EQUAL, "Initiated");
Filter failedFilter = new FilterPredicate("status", FilterOperator.EQUAL, "Failed");
Filter failedCountFilter = new FilterPredicate("failedCount", FilterOperator.LESS_THAN_OR_EQUAL, 5);
CompositeFilter failedCompositeFilter = CompositeFilterOperator.and(failedFilter, failedCountFilter);
CompositeFilter initaiedFailedFilter = CompositeFilterOperator.or(initiatedFilter, failedCompositeFilter);
Query query2 = new Query("JobInstance").setFilter(initaiedFailedFilter).addSort("failedCount", SortDirection.ASCENDING).addSort("createdAt", SortDirection.ASCENDING);
jobEntitesList = datastoreService.prepare(query2).asList(FetchOptions.Builder.withLimit(limit));
for (Entity jobEntity2 : jobEntitesList) {
/* doing some task */
}
What I have observed is If a remove the addSort() its working fine. but for me sort is important. I want the result should be in "createdAt" order. If I remove failedCount then it is throwing some error.
Or filters are implemented entirely in the client (like IN query filters: https://cloud.google.com/appengine/docs/standard/java/datastore/queries#filters). So your sort is happening in the client. It's likely that you are getting all of the results from the status == Failed AND FailedCount <= 5 query before getting any results from the status == Initiated.
From your comment, it's unclear what sort order you are looking for. As documented in the Restrictions on queries, the primary sort order must be on failedCount since that has an inequality filter.
Related
Consider these queries:
SELECT COUNT(1) AS failures
FROM c
WHERE c.time = 1623332779 AND c.status = 'FAILURE'
SELECT COUNT(1) AS successes
FROM c
WHERE c.time = 1623332779 AND c.status = 'SUCCESS'
How can I combine these two distinct queries into one query?
I tried repurposing the answers from How to get multiple counts with one SQL query?, but ran into a few problems:
COUNT(*) throws an error "Syntax error, incorrect syntax near '*'."
UNION throws "Syntax error, incorrect syntax near 'UNION'."
I also experimented with
SELECT
SUM(CASE WHEN c.time = 1623332779 THEN 1 else 0 end)
FROM c
but this leads to another syntax error. I noticed that
SELECT COUNT(1) AS mycounter, COUNT(1) AS mycounter2
FROM c
WHERE c.time = 1623332779
returns
[
{
"mycounter": 3,
"mycounter2": 3
}
]
but I was unable to link these distinct counters to distinct queries.
The following should work. The count operator skips values that are undefined which allows you to filter out rows from it:
SELECT
COUNT(c.status = 'SUCCESS' ? 1 : undefined) AS successes,
COUNT(c.status = 'FAILURE' ? 1 : undefined) AS failures
FROM c
WHERE c.time = 1623332779
It ruins performance though as it doesn't use indexing at all for the count. So you're better off using two seperate queries. If you really want to use a single request you could create a stored procedure that runs both queries and pastes the results together.
Instead of doing counts of the overall query, you can use GROUP BY to get counts in a single query. For example:
SELECT c.time, c.status, COUNT(c.status) AS statuscount
FROM c
WHERE c.time = "1623332779"
GROUP BY c.time, c.status
This won't give you explicit counts called "successes" and "failures" but it will return both counts, something like:
[
{
"time": "1623332779",
"status": "FAILURE",
"statuscount": 123
},
{
"time": "1623332779",
"status": "SUCCESS",
"statuscount": 456
}
]
Apparently I'm missing something with how LINQ to entities works. Hopefully one of you all can educate me.
Please try the below locally and let me know if you are seeing the same results. Something is really strange here...
Lets look at a very simple LINQ expression using navigation properties.
This was generated in LinqPad in a C# statement.
var result = (from ge in group_execution
where ge.automation_sequences.project.client_id == 1 && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
});
result.Dump();
OR, we can use joins...which will lead to the same bad results, but lets continue...
var result = (from ge in group_execution
join aseq in automation_sequences on ge.automation_sequence_id equals aseq.id
join p in project on aseq.project_id equals p.id
where p.client_id == 1 && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
p.client_id
});
result.Dump();
These very simple LINQ expressions generate the following SQL:
SELECT
[Filter1].[id1] AS [id],
[Filter1].[parent_group_exec_id] AS [parent_group_exec_id],
[Extent5].[client_id] AS [client_id]
FROM (SELECT [Extent1].[id] AS [id1], [Extent1].[automation_sequence_id] AS [automation_sequence_id], [Extent1].[parent_group_exec_id] AS [parent_group_exec_id]
FROM [dbo].[group_execution] AS [Extent1]
INNER JOIN [dbo].[automation_sequences] AS [Extent2] ON [Extent1].[automation_sequence_id] = [Extent2].[id]
INNER JOIN [dbo].[project] AS [Extent3] ON [Extent2].[project_id] = [Extent3].[id]
WHERE ([Extent1].[parent_group_exec_id] IS NOT NULL) AND (1 = [Extent3].[client_id]) ) AS [Filter1]
LEFT OUTER JOIN [dbo].[automation_sequences] AS [Extent4] ON [Filter1].[automation_sequence_id] = [Extent4].[id]
LEFT OUTER JOIN [dbo].[project] AS [Extent5] ON [Extent4].[project_id] = [Extent5].[id]
This baffles me. For the life of me I can't understand why LINQ is doing this. It's horrible, just look at the execution plan:
Now lets manually clean this up in SSMS and view the correct SQL and execution plan:
Much better, but how do we get LINQ to act this way?
Is anyone else seeing this? Has anyone else ever saw this and corrected it and if so how?
Thanks for looking into this.
UPDATE, attempting Chris Schaller fix:
var result = (from ge in group_execution
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
}).Where(x=>x.client_id == 1 && x.parent_group_exec_id != null);
result.Dump();
Just so you all know I'm monitoring the SQL through SQL Server Profiler. If anyone knows of any issues doing it this way let me know.
UPDATE, a fix for JOINS, but not nav properties, and a cause, but why?
Here's your solution:
var result = (from ge in group_execution.Where(x=>x.parent_group_exec_id != null)
join aseq in automation_sequences on ge.automation_sequence_id equals aseq.id
join p in project on aseq.project_id equals p.id
where p.client_id == 1// && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
p.client_id
});
result.Dump();
Null checks shouldn't cause the framework to mess up like this. Why should I have to write it this way? This just seems like a defect to me in the framework. It will make my dynamic expressions a little bit more difficult to write, but maybe I can find a way.
Navigation Properties still mess up...so I'm still really sad. Picture below:
var result = (from ge in group_execution.Where(x=>x.parent_group_exec_id != null)
where ge.automation_sequences.project.client_id == 1// && ge.parent_group_exec_id != null
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
});
result.Dump();
move your where clause to after you have defined the structure of the select statement
var result = (from ge in group_execution
select new
{
ge.id,
ge.parent_group_exec_id,
ge.automation_sequences.project.client_id
}).Where(x => x.client_id == 1 && x.parent_group_exec_id != null)
result.Dump();
Remember that Linq-to-entities flattens the results of queries to execute as SQL and then hydrates the object graph from those results.
When your query uses navigation properties or joins the query parser has to allow for zero results from those sub queries (Extents) to make sure that all columns that are required in the output and any interim processing are represented. By explicitly specifying a filter on a table for != null early in the query the parser knows that there is no further possibility that the field and any relationships linked by that field will be null, until then the parser prepares the query as if the joins will return null results
It is worth checking, but i wonder if UseDatabaseNullSemantics has anything to do with this?
Try:
dbContext.Configuration.UseDatabaseNullSemantics = false
In Linq, we can specify Where clauses as often as we like, improve the resulting SQL we should filter early in the query and granularly.
The parser engine is optimized to follow and implement your query sequentially, and generate good SQL at the end of it. Don't try to write linq-to-entities the same way that you structure your SQL, I know it's counter intuitive because the syntax is similar
A good technique is to assume that before each clause all the records from the previous statements have been loaded into memory, and that the next operation will affect all of those records. So you want to reduce the records before each additional operation by specifying a filter before moving on to the next clause
In general, if you have a filter condition based on the root table, apply this to the query before define all other joins and filters and even selects, you will get much cleaner sql.
I have an enquiry form which works from a view with a custom query. The form has filters, which I use in the executeQuery method of the view on the form to add ranges on various fields.
A new requirement is to filter based on two fields in the query.
Example: The PurchLine table is one of the tables in the query.
A new range is needed :
if PurchLine.ItemId != “” then
filter by PurchLine.PurchStatus == None
but, if the Item has a SPECIFIC value,
then filter by PurchStatus == Received.
(Ok, this is just an example!).
I am unable to modify my query to add a range on the PurchStatus based on the Item field.
I know exactly how the string value of the query must look, but how can I modify the query string?
The current query string looks like this (if I breakpoint on super in executeQuery):
SELECT FIRSTFAST * FROM OpenPOLinesView(OpenPOLinesView) WHERE ((CreatedDateTime<='2016-11-30T23:59:59')) AND ((VendAccount = N'S000001048'))
I want to add this at the end:
AND (((ItemId = N'') AND (PurchStatus = 0)) OR ((ItemId = N'XXX123') AND (PurchStatus = 2)))
How can I modify the query string in code?
You can use query expression for this, e.g.
queryBuildRange.value(strFmt('((%1 == %2) || ((%1 == %3) && (%4 == "%5")))',
fieldStr(InventTable, ItemType),
any2int(ItemType::Service),
any2int(ItemType::Item),
fieldStr(InventTable, ProjCategoryId),
queryValue("Spares")));
Please refer to this link Using Expressions in Query Ranges [AX 2012] for details.
I am fairly new to LINQ and can't get my head around some inconsistency in behaviour. Any knowledgeable input would be much appreciated. I see similar issues on SO and elsewhere but they don't seem to help.
I have a very simple setup - a company table and an addresses table. Each company can have 0 or more addresses, and if > 0 one must be specified as the main address. I'm trying to handle the cases where there are 0 addresses, using an outer join and altering the select statement accordingly.
Please note I'm currently binding the output straight to a GridView so I would like to keep all processing within the query.
The following DOES work
IQueryable query =
from comp in context.Companies
join addr in context.Addresses on comp.CompanyID equals addr.CompanyID into outer // outer join companies to addresses table to include companies with no address
from addr in outer.DefaultIfEmpty()
where (addr.IsMain == null ? true : addr.IsMain) == true // if a company has no address ensure it is not ruled out by the IsMain condition - default to true if null
select new {
comp.CompanyID,
comp.Name,
AddressID = (addr.AddressID == null ? -1 : addr.AddressID), // use -1 to represent a company that has no addresses
MainAddress = String.Format("{0}, {1}, {2} {3} ({4})", addr.Address1, addr.City, addr.Region, addr.PostalCode, addr.Country)
};
but this displays an empty address in the GridView as ", , ()"
So I updated the MainAddress field to be
MainAddress = (addr.AddressID == null ? "" : String.Format("{0}, {1}, {2} {3} ({4})", addr.Address1, addr.City, addr.Region, addr.PostalCode, addr.Country))
and now I'm getting the Could not translate expression error and a bunch of spewey auto-generated code in the error which means very little to me.
The condition I added to MainAddress is no different to the working condition on AddressID, so can anybody tell me what's going on here?
Any help is greatly appreciated.
The error you are getting is telling you that LinqToSql cannot translate your null check and then string.Format expression into SQL. If you look at the SQL your first query is generating (using either LinqPad or SQL Profiler), you'll see something like:
SELECT [t0].[CompanyID], [t0].[Name],
(CASE
WHEN [t1].[AddressID] IS NULL THEN #p0
ELSE [t1].[AddressID]
END) AS [AddressID],
[t1].[Address1] AS [value],
[t1].[City] AS [value2],
[t1].[Region] AS [value3],
[t1].[PostalCode] AS [value4],
[t1].[Country] AS [value5]
FROM [Company] AS [t0]
LEFT OUTER JOIN [Address] AS [t1] ON [t0].[CompanyID] = [t1].[CompanyID]
WHERE ([t1].[IsMain] IS NULL) OR ([t1].[IsMain] = 1)
For your AddressID field, you can see that it uses a CASE-WHEN to handle the condition when AddressID is null. When you add a CASE-WHEN for MainAddress, it's trying to do the same thing for that field, but there is no SQL equivalent to string.Format it can use for the ELSE clause, so it blows up.
An easy way around this problem is to use a method to format the string. By calling a private method, LinqToSql won't try to translate the string.Format to SQL, and will instead return all of the fields necessary to populate the Address object. The method can then take care of the formatting.
For example:
LINQ:
....
select new {
comp.CompanyID,
comp.Name,
AddressID = (addr.AddressID == null ? -1 : addr.AddressID),
MainAddress = FormatAddress(addr)
};
Method:
private static string FormatAddress(Address addr)
{
return (addr == null ? "" :
string.Format("{0}, {1}, {2} {3} ({4})",
addr.Address1, addr.City,
addr.Region, addr.PostalCode, addr.Country));
}
I have this SQL query:
SELECT Sum(ABS([Minimum Installment])) AS SumOfMonthlyPayments FROM tblAccount
INNER JOIN tblAccountOwner ON tblAccount.[Creditor Registry ID] = tblAccountOwner.
[Creditor Registry ID] AND tblAccount.[Account No] = tblAccountOwner.[Account No]
WHERE (tblAccountOwner.[Account Owner Registry ID] = 731752693037116688)
AND (tblAccount.[Account Type] NOT IN
('CA00', 'CA01', 'CA03', 'CA04', 'CA02', 'PA00', 'PA01', 'PA02', 'PA03', 'PA04'))
AND (DATEDIFF(mm, tblAccount.[State Change Date], GETDATE()) <=
4 OR tblAccount.[State Change Date] IS NULL)
AND ((tblAccount.[Account Type] IN ('CL10','CL11','PL10','PL11')) OR
CONTAINS(tblAccount.[Account Type], 'Mortgage')) AND (tblAccount.[Account Status ID] <> 999)
I have created a Linq query:
var ownerRegistryId = 731752693037116688;
var excludeTypes = new[]
{
"CA00", "CA01", "CA03", "CA04", "CA02",
"PA00", "PA01", "PA02", "PA03", "PA04"
};
var maxStateChangeMonth = 4;
var excludeStatusId = 999;
var includeMortgage = new[] { "CL10", "CL11", "PL10", "PL11" };
var sum = (
from account in context.Accounts
from owner in account.AccountOwners
where owner.AccountOwnerRegistryId == ownerRegistryId
where !excludeTypes.Contains(account.AccountType)
where account.StateChangeDate == null ||
(account.StateChangeDate.Month - DateTime.Now.Month)
<= maxStateChangeMonth
where includeMortgage.Contains(account.AccountType) ||
account.AccountType.Contains("Mortgage")
where account.AccountStatusId != excludeStatusId
select account.MinimumInstallment).ToList()
.Sum(minimumInstallment =>
Math.Abs((decimal)(minimumInstallment)));
return sum;
Are they equal/same ? I dont have records in db so I cant confirm if they are equal. In SQL there are brackets() but in Linq I didnt use them so is it ok?
Please suggest.
It is not possible for us to say anything about this, because you didn't show us the DBML. The actual definition of the mapping between the model and the database is important to be able to see how this executes.
But before you add the DBML to your question: we are not here to do your work, so here are two tips to find out whether they are equal or not:
Insert data in your database and run the queries.
Use a SQL profiler and see what query is executed by your LINQ provider under the covers.
If you have anything more specific to ask, we will be very willing to help.
The brackets will be generated by LINQ provider, if necessary.
The simplest way to check if the LINQ query is equal to the initial SQL query is to log it like #Atanas Korchev suggested.
If you are using Entity Framework, however, there is no Log property, but you can try to convert your query to an ObjectQuery, and call the ToTraceString method then:
string sqlQuery = (sum as ObjectQuery).ToTraceString();
UPD. The ToTraceString method needs an ObjectQuery instance for tracing, and the ToList() call already performs materialization, so there is nothing to trace. Here is the updated code:
var sum = (
from account in context.Accounts
from owner in account.AccountOwners
where owner.AccountOwnerRegistryId == ownerRegistryId
where !excludeTypes.Contains(account.AccountType)
where account.StateChangeDate == null ||
(account.StateChangeDate.Month - DateTime.Now.Month)
<= maxStateChangeMonth
where includeMortgage.Contains(account.AccountType) ||
account.AccountType.Contains("Mortgage")
where account.AccountStatusId != excludeStatusId
select account.MinimumInstallment);
string sqlQuery = (sum as ObjectQuery).ToTraceString();
Please note that this code will not perform the actual query, it is usable for testing purposes only.
Check out this article if you are interested in ready-for-production logging implementation.
There can be a performance difference:
The SQL query returns a single number (SELECT Sum...) directly from the database server to the client which executes the query.
In your LINQ query you have a greedy operator (.ToList()) in between:
var sum = (...
...
select account.MinimumInstallment).ToList()
.Sum(minimumInstallment =>
Math.Abs((decimal)(minimumInstallment)));
That means that the query on the SQL server does not contain the .Sum operation. The query returns a (potentially long?) list of MinimumInstallments. Then the .Sum operation is performed in memory on the client.
So effectively you switch from LINQ to Entities to LINQ to Objects after .ToList().
BTW: Can you check the last proposal in your previous question here which would avoid .ToList() on this query (if the proposal should work) and would therefore be closer to the SQL statement.