I have this select which I expected to be executed by sending one command to SQL Server but I can see 10s requests instead. How can I fix it?
We have Companies which have Customers and Orders. For some reasons Orders are under Company entity.
var q = _dbContext.Companies
.Include(x => x.Customers)
.Include(c => c.Orders)
.Where(a => a.CompanyId == 123);
var total = await q.CountAsync();
q = q.OrderByDescending(x => x.CompanyCode)
.Skip((pageIndex - 1) * pageSize).Take(pageSize);
var res = await q.Select(x => new ResultDto()
{
CompanyCode = x.CompanyCode,
Customers = x.Customers
.Where(c => c.IsActive)
.Select(c => new CustomerDto()
{
FirstName = c.FirstName,
Surname = c.Surname,
Orders = x.Orders
.Where(o => o.IsOpen)
.Select(o => new OrderDto()
{
DateCreated = o.DateCreated
}).ToList()
}).FirstOrDefault(),
}).ToListAsync();
This is EF.NetCore optimization.
You actually cant achieve one query when your navigation properties are collections.
I can't find any links right now, but this is by design.
Once you have a collection in your navigations inside select or inside includes it will produce a separate query for each root entity. The reason I believe is the redundant data amount produced by such a query.
I suggest leave it as is if you have not a lot of data < 1000 rows in a result. You will see a lot of queries but they will be really fast.
As I can see you have pagination here so it shouldn't be a problem.
Otherwise, select your collections separately and join them in memory carefully.
Unfortunately, there is no other way for EF Core
Also, I recommend to turn on EF core logs to what is going on early. I bet FirstOrDefault will produce some warning.
Related
Why does this EF Core 3.1 typical query result in:
System.InvalidOperationException: Lambda expression used inside
Include is not valid.
(Note: IgnoreQueryFilters() is not the issue here. Even without it, it results in the same error)
var tenant = await context.Tenants.IgnoreQueryFilters()
.Include(i => i.Locations).ThenInclude(x => x.CurrentResidents.DefaultIfEmpty()).IgnoreQueryFilters()
.Include(i => i.Locations).ThenInclude(x => x.DefaultResidents.DefaultIfEmpty()).IgnoreQueryFilters()
.Where(l => l.Id == tenantId)
.FirstOrDefaultAsync()
.ConfigureAwait(false);
Here's my sql interpretation of what I'm trying to achieve:
select t.* from tenants
left join locations l on l.TenantId = t.Id
left join residents c on c.CurrentLocationId = l.Id
left join residents d on d.DefaultLocationId = l.Id
where t.Id = #tenantId
Please help. :pray:
Your problem is including Locations twice. Not sure from your SQL what you are trying to retrieve. The residents for whom both CurrentLocationId AND DefaultLocationId matches LocationId? then use AND in the LINQ statement.
If something else, then what is the type of CurrentResidents and DefaultResidents? Same Resident type? Then you probably want to get two sets...
Please provide more information...
As Felix pointed out the problem is that you include Locations twice which is not correct in ef core. Here is how you can achieve this:
var tenant = await context.Tenants
.IgnoreQueryFilters()
.Include(i => i.Locations.CurrentResidents.DefaultIfEmpty())
.Include(i => i.Locations.DefaultResidents.DefaultIfEmpty())
.FirstOrDefaultAsync(l => l.Id == tenantId, cancellationToken)
.ConfigureAwait(false);
Calling IgnoreQueryFilters once is sufficient, you don't need to call it 3 times.
Calling where and the first or default is unnecessary. You can just use the lambda in the first or default.
This was what I was looking for:
var tenant = await context.Tenants
.IgnoreQueryFilters()
.Include(i => i.Locations).ThenInclude(x => x.CurrentResidents)
.Include(i => i.Locations).ThenInclude(x => x.CurrentResidents)
.FirstOrDefaultAsync(l => l.Id == tenantId)
.ConfigureAwait(false);
Sql generated by ef core is a little weird but yields the result I'm interested in so all good.
#bencel: thank you for pointing out those issues. I was focused on getting the query running so overlooked those.
Thanks everyone!!
When I try to use selectMany on queryable that I build against cosmosdb I always get exception:
The LINQ expression 'DbSet
.Where(t => t.Id == __timelineId_0)
.SelectMany(
source: t => EF.Property>(t, "GraduationEvents")
.AsQueryable(),
collectionSelector: (t, c) => new TransparentIdentifier(
Outer = t,
Inner = c
))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
The query doesn't matter, always when I use selectMany I get this error.
Example query:
await _dbContext.Timelines.Where(x => x.Id == timelineId).Select(x => x.GraduationEvents).SelectMany(x => x).ToListAsync();
My entity configuration:
public void Configure(EntityTypeBuilder<Timeline> builder)
{
builder.HasKey(x => x.Id);
builder.HasAlternateKey(x => x.Id);
builder.OwnsMany(x => x.GraduationEvents, x => x.OwnsMany(graduationEvent => graduationEvent.Subjects));
}
I also tried to use native cosmosClient but when I query the base with plain sql I get empty objects (all nulls). Any thoughts what am I doing wrong?
Sajid - I tried your soulution but the exception remains the same
Try calling directly .SelectMany() over the List property (GraduationEvents).
I usually then call AsDocumentQuery() to generate the query to CosmosDB and then execute that query to retrieve the results.
Some pseudo c# to clarify this a bit:
var query = this.documentClient.CreateDocumentQuery(uri, options)
.SelectMany(x => x.GraduationEvents).AsDocumentQuery();
List<T> results = new List<T>();
while (query.HasMoreResults)
{
results.AddRange(await query.ExecuteNextAsync());
}
Edit: This approach uses the native Azure DocumentClient library.
I am trying to get Departments with last DepartmentLog, but ef core executes n+1 query. And works so slow. I have 10 000 rows on Department table and 12 000 rows on DepartmentLogs. I am not sure it must work 1.5s - 2s for executing like this query.
var query2 = _unitOfWork.Departments.GetDbSet()
.Include(p => p.CreatedUser)
.ThenInclude(p => p.Employee)
.OrderByDescending(p => p.CreatedAt)
.Select(p => new
{
Department = p,
DepartmentLog = _unitOfWork.DepartmentLogs.GetDbSet()
.Include(m => m.CreatedUser)
.ThenInclude(m => m.Employee)
.OrderByDescending(m => m.CreatedAt)
.FirstOrDefault(m => m.DepartmentId == p.Id)
})
.Take(10).ToList();
The lambda you're passing to Select is executed for each item in the result set. Since you're making a query in that lambda, you're going to have N queries issued, one for each item in the result set.
The reason it only generates one extra query when you remove FirstOrDefault, is that you're returning the all the department logs, which then allows EF to add them to the object cache, where it can pull them out from later, instead of issuing a new query. With FirstOrDefault, you're only returning one, so EF caches that, but on the next run of the lambda, you're selecting a different one that it doesn't have cached.
To use FirstOrDefault and still have just one extra query, you should run this query first:
var departmentLogs = await _unitOfWork.DepartmentLogs.GetDbSet()
.Include(m => m.CreatedUser)
.ThenInclude(m => m.Employee)
.OrderByDescending(m => m.CreatedAt)
.ToListAsync();
Then, in your Select:
.Select(p => new
{
Department = p,
DepartmentLog = departmentLogs.FirstOrDefault(m => m.DepartmentId == p.Id)
})
Then, it will run the FirstOrDefault on the result set already in memory. However, this requires returning all the logs, which could be more problematic performance-wise, if there's a ton of them. You'll have to profile each way and see which ends up being better in your particular circumstance.
Alternatively, a probably preferably, you can simply create this query as a stored procedure, and then just call the stored procedure to get your results, instead of using LINQ to build a query. There's only so much EF can do to optimize the query, and this is a highly inefficient query, no matter what you do.
I created a table with relation with ApplicationUser and when I want to get with eager-loading, I could not get all columns from that table my search base on user authentication.
var UserSites = await _SqldbContext.Users
.Where(x => x.UserName == User.Identity.Name)
.Include(x => x.sites)
.ToListAsync() ;
return Json(UserSites);
But in return I only get one row of that table with two columns
[{"sites":[{"id":1,"userId":"c0e8be95-535c-449c-9aa1-06702cd4c983"
but I have more rows with this userId and also here I get only two columns but I have more than two columns, I am not sure what is wrong here please help me.
I think ToListAsync() works properly and If you make break point you may see all data from your site but here I think Json does not show it properly, instead of start from User table in your example I started with your proposed table name sites (to get all data of this table)
var UserSites = await _SqldbContext.sites.Include(x => x.[Name Of User in relation table]).Where(y=>y.[Name Of User in relation table].UserName== User.Identity.Name)
.ToListAsync();
List<sites> siteObject=new List<sites>();
UserSites.ForEach(x =>
{
sites site = new sites() {
// fill property of your class
};
siteObject.Add(site);
});
return Json(siteObject);
I hope this code works for you.
Normally I use stored procedures / work in SQL so apologies if I get the terminology slightly off here..
I have a database, with 3 seperate tables, and I need to search multiple fields in each of the 3 tables.
Im sure that I am not doing this the mose effective way, initially I am trying to do it in simple seteps to understand it.
I have the following;
var foo1 = entities.table1.Where(a => a.bodyText.Contains(searchString) || a.pageTitle.Contains(searchString));
var foo2 = entities.table2.Where(b => b.newsArticle.Contains(searchString) || b.newsArticle.Contains(searchString));
var foo3 = entities.table3.Where(c => c.ImageDescriptionContains(searchString));
I need to combine all these results into a single repeater for display.
At this point all 3 sets of data are in seperate, unique collections of anonymous data. So whats the best way of converting these into a single coherent bindable source?
I was thinking of itereating through each list in turn, pulling out the fields I need to display and putting them in a new class, then binding a lsit of these classes to the repeater.
But it all seems a bit clunky to me.
Is there a way of doing the search across all 3 tables in one go, and returning just the fields I need from each table, with a common name (i.e. in SQL I could write
select b.newsArticle as myText,
or
select newsArticle, ''
to return the news article and an empty string).
This would combine:
var foos = foo1.ToList();
foos.AddRange(foo2);
foos.AddRange(foo3);
To get just what you want:
var myExtractedValues = foos.Select(x => new {
Article = !string.IsNullOrEmpty(x.newsArticle))
? x.newsArticle
: string.Empty});
I have used an anonymous type here but you could swap the new {} with a type of your own.
I reverse the operator on the IsNullOrEmpty but that is just a personal preference (I prefer how is reads.)
To get all the results in one go you'll need to define a common class that will be used by all three queries to store the result. This class may be as well anonymous but I'll name it just for clarity.
class Data
{
public string Text{ get; set;}
}
Now, in your code you'll fetch instances of Data from database and you can use Union:
using( var entities = new YourDataContext)
{
var foo1 = entities.table1
.Where(a => a.bodyText.Contains(searchString) ||
a.pageTitle.Contains(searchString))
.Select(a => new Data{ Text = a.bodyText});
var foo2 = entities.table2
.Where(b => b.newsArticle.Contains(searchString) ||
b.newsArticle.Contains(searchString))
.Select(b => new Data{ Text = b.newsArticle});
var foo3 = entities.table3
.Where(c => c.ImageDescription.Contains(searchString))
.Select(c => new Data{ Text = c.ImageDescription});
return foo1.Union(foo2).Union(foo3);
}