Why EF core executes n query for n+1 query - asp.net-core-2.2

I am trying to get Departments with last DepartmentLog, but ef core executes n+1 query. And works so slow. I have 10 000 rows on Department table and 12 000 rows on DepartmentLogs. I am not sure it must work 1.5s - 2s for executing like this query.
var query2 = _unitOfWork.Departments.GetDbSet()
.Include(p => p.CreatedUser)
.ThenInclude(p => p.Employee)
.OrderByDescending(p => p.CreatedAt)
.Select(p => new
{
Department = p,
DepartmentLog = _unitOfWork.DepartmentLogs.GetDbSet()
.Include(m => m.CreatedUser)
.ThenInclude(m => m.Employee)
.OrderByDescending(m => m.CreatedAt)
.FirstOrDefault(m => m.DepartmentId == p.Id)
})
.Take(10).ToList();

The lambda you're passing to Select is executed for each item in the result set. Since you're making a query in that lambda, you're going to have N queries issued, one for each item in the result set.
The reason it only generates one extra query when you remove FirstOrDefault, is that you're returning the all the department logs, which then allows EF to add them to the object cache, where it can pull them out from later, instead of issuing a new query. With FirstOrDefault, you're only returning one, so EF caches that, but on the next run of the lambda, you're selecting a different one that it doesn't have cached.
To use FirstOrDefault and still have just one extra query, you should run this query first:
var departmentLogs = await _unitOfWork.DepartmentLogs.GetDbSet()
.Include(m => m.CreatedUser)
.ThenInclude(m => m.Employee)
.OrderByDescending(m => m.CreatedAt)
.ToListAsync();
Then, in your Select:
.Select(p => new
{
Department = p,
DepartmentLog = departmentLogs.FirstOrDefault(m => m.DepartmentId == p.Id)
})
Then, it will run the FirstOrDefault on the result set already in memory. However, this requires returning all the logs, which could be more problematic performance-wise, if there's a ton of them. You'll have to profile each way and see which ends up being better in your particular circumstance.
Alternatively, a probably preferably, you can simply create this query as a stored procedure, and then just call the stored procedure to get your results, instead of using LINQ to build a query. There's only so much EF can do to optimize the query, and this is a highly inefficient query, no matter what you do.

Related

Deleting records from look up table not working in .NET Core 6

The bounty expires in 1 hour. Answers to this question are eligible for a +100 reputation bounty.
eia92 wants to draw more attention to this question.
I recently upgraded my project to .NET Core 6 and now removing records from my look up tables is not working. I have a Risk object that has a collection of Users. Removing users from the risk object no longer works. Any ideas what I'm doing wrong?
My lookup table is called RiskItemUser, and it has two columns, RiskItemId and UserId.
Code:
var postSavedRisk = _riskService.Queryable().Include(c => c.AssignedTo).Where(w => w.Id == riskitem.Id).FirstOrDefault();
List<User> usersToRemove = postSavedRisk.AssignedTo.Where(c => userNamesToRemove.Contains(c.UserName)).ToList();
using (var db = new ApplicationDbContext())
{
var postSavedAssginedTo = db.RiskItemUser
.Where(w => w.RiskItemId == riskitem.Id)
.ToList();
foreach (var userToRemove in usersToRemove)
{
foreach (var riskAssignedTo in postSavedAssginedTo)
{
if(userToRemove.Id == riskAssignedTo.UserId)
db.RiskItemUser.Remove(riskAssignedTo);
await db.SaveChangesAsync().ConfigureAwait(false);
}
}
}
The code, as you show it, looks like it should work, although some parts are hidden. Therefore, it's hard to tell how to make it work. But there's room for simplification, which should result in working code.
You want to remove users whose names are specified by userNamesToRemove from a risk that's specified by riskitem.Id. Assuming that there's a navigation property RiskItemUser.User, removing these data could be done by essentially one line of code:
db.RiskItemUser.RemoveRange(
db.RiskItemUser.Where(ru => ru.RiskItemId == riskitem.Id
&& userNamesToRemove.Contains(ru.User.Name)));
await db.SaveChangesAsync().ConfigureAwait(false);
You tagged EFC 6, but as of EFC 7.0, there's support for bulk delete (and update) functions, allowing for single-statement deletion of multiple database records:
db.RiskItemUser
.Where(db.RiskItemUser.Where(ru => ru.RiskItemId == riskitem.Id
&& userNamesToRemove.Contains(ru.User.Name)))
.ExecuteDelete();
This will execute one delete statement, whereas the previous method will execute one statement per row.
Note that this bulk method is like executing raw SQL. There's no communication with EF's change tracker and EF can't coordinate the correct order of statements. I think the general advice should be to not mix these bulk methods with regular SaveChanges calls.

SelectMany doesn't work on cosmosDb for child properties?

When I try to use selectMany on queryable that I build against cosmosdb I always get exception:
The LINQ expression 'DbSet
.Where(t => t.Id == __timelineId_0)
.SelectMany(
source: t => EF.Property>(t, "GraduationEvents")
.AsQueryable(),
collectionSelector: (t, c) => new TransparentIdentifier(
Outer = t,
Inner = c
))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
The query doesn't matter, always when I use selectMany I get this error.
Example query:
await _dbContext.Timelines.Where(x => x.Id == timelineId).Select(x => x.GraduationEvents).SelectMany(x => x).ToListAsync();
My entity configuration:
public void Configure(EntityTypeBuilder<Timeline> builder)
{
builder.HasKey(x => x.Id);
builder.HasAlternateKey(x => x.Id);
builder.OwnsMany(x => x.GraduationEvents, x => x.OwnsMany(graduationEvent => graduationEvent.Subjects));
}
I also tried to use native cosmosClient but when I query the base with plain sql I get empty objects (all nulls). Any thoughts what am I doing wrong?
Sajid - I tried your soulution but the exception remains the same
Try calling directly .SelectMany() over the List property (GraduationEvents).
I usually then call AsDocumentQuery() to generate the query to CosmosDB and then execute that query to retrieve the results.
Some pseudo c# to clarify this a bit:
var query = this.documentClient.CreateDocumentQuery(uri, options)
.SelectMany(x => x.GraduationEvents).AsDocumentQuery();
List<T> results = new List<T>();
while (query.HasMoreResults)
{
results.AddRange(await query.ExecuteNextAsync());
}
Edit: This approach uses the native Azure DocumentClient library.

Multiple request instead of one in EF Core

I have this select which I expected to be executed by sending one command to SQL Server but I can see 10s requests instead. How can I fix it?
We have Companies which have Customers and Orders. For some reasons Orders are under Company entity.
var q = _dbContext.Companies
.Include(x => x.Customers)
.Include(c => c.Orders)
.Where(a => a.CompanyId == 123);
var total = await q.CountAsync();
q = q.OrderByDescending(x => x.CompanyCode)
.Skip((pageIndex - 1) * pageSize).Take(pageSize);
var res = await q.Select(x => new ResultDto()
{
CompanyCode = x.CompanyCode,
Customers = x.Customers
.Where(c => c.IsActive)
.Select(c => new CustomerDto()
{
FirstName = c.FirstName,
Surname = c.Surname,
Orders = x.Orders
.Where(o => o.IsOpen)
.Select(o => new OrderDto()
{
DateCreated = o.DateCreated
}).ToList()
}).FirstOrDefault(),
}).ToListAsync();
This is EF.NetCore optimization.
You actually cant achieve one query when your navigation properties are collections.
I can't find any links right now, but this is by design.
Once you have a collection in your navigations inside select or inside includes it will produce a separate query for each root entity. The reason I believe is the redundant data amount produced by such a query.
I suggest leave it as is if you have not a lot of data < 1000 rows in a result. You will see a lot of queries but they will be really fast.
As I can see you have pagination here so it shouldn't be a problem.
Otherwise, select your collections separately and join them in memory carefully.
Unfortunately, there is no other way for EF Core
Also, I recommend to turn on EF core logs to what is going on early. I bet FirstOrDefault will produce some warning.

Linq Query Distinct Function do not work with DropdownList

I want to assign Linq Query result to dropdownlist which contain a
Distinct function
My Code:-
var area = de.City_Area_View
.Select(m => new { m.Area_Id, m.Area_Name})
.Distinct()
.ToList();
drpfilter.DataTextField = "Area_Name";
drpfilter.DataValueField = "Area_Id";
drpfilter.DataSource = area;
drpfilter.DataBind();
Problem :- When I write this code then I get Below Error
Error:- The method 'Distinct' is not supported.
I get System.NotSupportedException.
I want to assign a Distinct name of area to the DropDownList
So please help me for this problem.
If your set is small enough (so you don't mind fetching all the values from the database), the simplest thing would be to force the distinct part to be performed locally:
var area = de.City_Area_View
.Select(m => new { m.Area_Id, m.Area_Name})
.AsEnumerable()
.Distinct()
.ToList();
AsEnumerable simply "changes" the expression type to IEnumerable<T> instead of IQueryable<T>, so that the compiler calls Enumerable.Distinct instead of Queryable.Distinct - and Enumerable.Distict will definitely work.

Autocomplete extender 'extended' across entire data table

I have a request to create an auto complete that will search an data table. Is this achieveable quickly and simply or is it a case of writing a reasonable amount of code?
Originally, I have been using a webservice and linq to point at a single column's worth of data (IDDesc) and pull back the list of products:
Product.FinalProductsDataContext dbac = new Product.FinalProductsDataContext();
return dbac.tblProduct
.Where(r => r.Account== HttpContext.Current.Session["AccountKey"].ToString() && r.IDDesc.Contains(prefixText))
.Distinct()
.OrderBy(r => r.IDDesc)
.Select(r => r.IDDesc)
.Take(count)
.ToArray();
However, if I wish the autocomplete to look at all the columns, is it a case of repeating similar LINQ statements for each of the columns contained within the datatable or is there a 'quick fix'?
I personally don't think this is an ideal scenario but it is a request I must work towards.
Any help or advice, greatly appreciated.
Rather than trying to solve this entirely with LINQ (and repeating all those statements for each column in the table, as well as hitting the database repeatedly), I think I'd look to put something in the database to do the heavy lifting here.
You could create a view that takes the fields from the table and amalgamates them into one column e.g.
CREATE VIEW [dbo].[ProductView]
AS
SELECT CAST(ProductName AS NVARCHAR(50)) AS 'ProductColumn'
FROM dbo.Products
UNION
SELECT CAST(SupplierName AS NVARCHAR(50))
FROM dbo.Products
UNION
...
which if you added the view to your context would then allow you to modify your existing LINQ query and point it at that view e.g.:
Product.FinalProductsDataContext dbac = new Product.FinalProductsDataContext();
return dbac.ProductView
.Where(r => r.Account== HttpContext.Current.Session["AccountKey"].ToString() && r.ProductColumn.Contains(prefixText))
.Distinct()
.OrderBy(r => r.ProductColumn)
.Select(r => r.ProductColumn)
.Take(count)
.ToArray();

Resources