We are using asp.net mvc entity framework
We have a number of queries that represent large static data. Im wondering what is the best and simplest way to avoid querying the data each time.
We currently use a custom cache but it seems to have issues. Im wondering how others achieve this.
For example I have a locations data set that has over 10k records
Its used throughout the site and can be queried when the app starts doesn't need to get fetched again.
Just use the System.Runtime.Caching.MemoryCache
Maybe this helps:
void SomeAction()
{
var test1 = LoadIds();
var test2 = LoadIds();
// test1 & test2 should be the same
}
int[] LoadIds() {
var random = new Random();
return MemoryCache.Default.GetCached("LoadIds", 60, () => new int[10].Select(x => random.Next()).ToArray() );
}
public static class CacheExtensions {
public static T GetCached<T>(this ObjectCache cache, string key, int cacheTime, Func<T> acquire)
{
if (cache.Contains(key))
return (T) cache[key];
var result = acquire();
if (cacheTime > 0)
cache.Add(new CacheItem(key, result), new CacheItemPolicy {AbsoluteExpiration = DateTime.Now + TimeSpan.FromMinutes(cacheTime)});
return result;
}
}
Now instead of using this random initialization...
MemoryCache.Default.GetCached("LoadIds", 60, () => new int[10].Select(x => random.Next()).ToArray());
... you can use your query. Just make sure to add a .ToList() or .ToArray() at the end, since you want to cache the data and not the just the query. e.g.
MemoryCache.Default.GetCached("LoadIds", 60, () => dbContext.Persons.Select(x => x.Id).ToArray());
Or in your case maybe
MemoryCache.Default.GetCached("LoadLocations", 60, () => dbContext.Locations.ToArray());
This extension method above is not perfect though. In case that two requests access the same content at the same time it might double load that data...
I'm using this method in large scale web apps, while abstracting out the MemoryCache in order to replace it with other types of caches like Redis, azure storage or even the HttpContext.Current.Items for caches per request.
Related
I am working on a basic Support Ticket System. I get the Tickets from Firebase (Either as a Stream or Future).
I want to allow some Filtering Options (e.g. sort by Status and Category).
For this, I thought about using A Future Provider to get the List and a StateNotiferProvider to update the List depending on which filter is being used.
This is the code I have so far:
final ticketListStreamProvider =
RP.FutureProvider((_) => FirestoreService.getTicketList());
class TicketListNotifier extends RP.StateNotifier<List<Ticket>> {
TicketListNotifier() : super([]);
void addTicket(Ticket ticket) {
state = List.from(state)..add(ticket);
}
void removeTicket(Ticket ticket) {
state = List.from(state)..remove(ticket);
}
}
final ticketsController =
RP.StateNotifierProvider<TicketListNotifier, List<Ticket>>(
(ref) => TicketListNotifier(),
);
There are multiple issues I have with that. Firstly it doesn't work.
The StateNotifier accepts a List and not a Future<List>. I need to convert it somehow or rewrite the StateNotifier to accept the Future.
I was trying to stay close to one of the official examples.
(https://github.com/rrousselGit/riverpod/tree/master/examples/todos)
Unfortunately, they don't use data from an outside source like firebase to do it.
What's the best approach to get resolve this issue with which combination of providers?
Thanks
You can fetch your ticketlist in your TicketListNotifier and set its state with your ticket list.
class TicketListNotifier extends RP.StateNotifier<List<Ticket>> {
TicketListNotifier() : super([]);
Future<void> fetchTicketList() async {
FirestoreService.getTicketList().when(
//success
// state = fetched_data_from_firestore
// error
// error handle
)
}
}
final ticketsController =
RP.StateNotifierProvider<TicketListNotifier, List<Ticket>>(
(ref) => TicketListNotifier(),
);
Call this method where you want to fetch it /*maybe in your widget's initState method
ref.read(ticketsController.notifier).fetchTicketList();
Now ref.read(ticketsController); will return your ticket list
Since you have the ticket list in your TicketListNotifier's state you can use your add/remove method like this:
ref.read(ticketsController.notifier).addTicket(someTicket);
With Flutter and Firestore, I am trying to get more than 10 documents into a Stream<List>. I can do this with a .where clause on a collection mapping the QuerySnapshot. However, the 10 limit is a killer.
I'm using the provider package in my app. So, in building a stream in Flutter with a StreamProvider, I can return a
Stream<List<Map from the entire collection. too expensive. 200 plus docs on these collections and too many users. Need to get more efficient.
Stream<List<Map using a .where from a Collection that returns a Stream List 10 max on the list...doesn't cut the mustard.
Stream<Map from a Document, that returns 1 stream of 1 document.
I need something in between 1 and 2.
I have a Collection with up to 500 Documents, and the user will choose any possible combination of those 500 to view. The user assembles class rosters to view their lists of users.
So I'm looking for a way to get individual streams of, say 30 documents, and then compile them into a List: But I need this List<Stream<Map to be a Stream itself so each individual doc is live, and I can also filter and sort this list of Streams. I'm using the Provider Package, and if possible would like to stay consistent with that. Here's where I am currently stuck:
So, my current effort:
Future<Stream<List<AttendeeData>>> getStreams() async {
List<Stream<AttendeeData>> getStreamsOutput = [];
for (var i = 0; i < teacherRosterList.length; i++) {
Stream thisStream = await returnTeacherRosterListStream(facility, teacherRosterList[i]);
getStreamsOutput.add(thisStream);
}
return StreamZip(getStreamsOutput).asBroadcastStream();
}
Feels like I'm cheating below: I get an await error if I put the snapshot directly in Stream thisStream above as Stream is not a future if I await, and if I don't await, it moves too fast and gets a null error.
Future<Stream<AttendeeData>> returnTeacherRosterListStream(String thisFacility, String thisID) async {
return facilityList.doc(thisFacility).collection('attendance').doc(thisID).snapshots().map(_teacherRosterListFromSnapshot);
}
}
Example of how I'm mapping in _teacherRosterListFromSnapshot (not having any problem here):
AttendeeData _teacherRosterListFromSnapshot(DocumentSnapshot doc) {
// return snapshot.docs.map((doc) {
return AttendeeData(
id: doc.data()['id'] ?? '',
authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
);
}
My StreamProvider Logic and the error:
return MultiProvider(
providers: [
StreamProvider<List<AttendeeData>>.value(
value: DatabaseService(
teacherRosterList: programList,
facility: user.claimsFacility,
).getStreams()),
]
Error: The argument type 'Future<Stream<List>>' can't be assigned to the parameter type 'Stream<List>'.
AttendeeData is my Map Class name.
So, the summary of questions:
Can I even do this? I'm basically Streaming a List of Streams of Maps....is this a thing?
If I can, how do I do it?
a. I can't get this into the StreamProvider because getStreams is a Future...how can I overcome this?
I can get the data in using another method from StreamProvider, but it's not behaving like a Stream and the state isn't updating. i'm hoping to just get this into Provider, as I'm comfortable there, and I can manage state very easily that way. However, beggars can't be choosers.
Solved this myself, and since there is a dearth of good start to finish answers, I submit my example for the poor souls who come after me trying to learn these things on their own. I'm a beginner, so this was a slog:
Objective:
You have any number of docs in a collection and you want to submit a list of any number of docs by their doc number and return a single stream of a list of those mapped documents. You want more than 10 (firestore limit on .where query), less than all the docs...so somewhere between a QuerySnapshot and a DocumentSnapshot.
Solution: We're going to get a list of QuerySnapshots, we're going to combine them and map them and spit them out as a single stream. So we're getting 10each in chunks (the max) and then some odd number left over. I plug mine into a Provider so I can get it whenever and wherever I want.
So from my provider I call this as the Stream value:
Stream<List<AttendeeData>> filteredRosterList() {
var chunks = [];
for (var i = 0; i < teacherRosterList.length; i += 10) {
chunks.add(teacherRosterList.sublist(i, i + 10 > teacherRosterList.length ? teacherRosterList.length : i + 10));
} //break a list of whatever size into chunks of 10.
List<Stream<QuerySnapshot>> combineList = [];
for (var i = 0; i < chunks.length; i++) {
combineList.add(*[point to your collection]*.where('id', whereIn: chunks[i]).snapshots());
} //get a list of the streams, which will have 10 each.
CombineLatestStream<QuerySnapshot, List<QuerySnapshot>> mergedQuerySnapshot = CombineLatestStream.list(combineList);
//now we combine all the streams....but it'll be a list of QuerySnapshots.
//and you'll want to look closely at the map, as it iterates, consolidates and returns as a single stream of List<AttendeeData>
return mergedQuerySnapshot.map(rosterListFromTeacherListDocumentSnapshot);
}
Here's a look at how I mapped it for your reference (took out all the fields for brevity):
List<AttendeeData> rosterListFromTeacherListDocumentSnapshot(List<QuerySnapshot> snapshot) {
List<AttendeeData> listToReturn = [];
snapshot.forEach((element) {
listToReturn.addAll(element.docs.map((doc) {
return AttendeeData(
id: doc.data()['id'] ?? '',
authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
);
}).toList());
});
return listToReturn;
}
I've a JSON model that contains strings instead of dates (the model is generated via T4TS, so I cannot change that).
The code is currently using an expanded model extending the original json, where the dates are recalculated on new fields.
I was wondering if it would be possible to apply the filters on the fields being string without adding that additional step of extending the model.
private makeNumeric(label: string, property: string) {
return {
label: label,
key: property,
prepareDimension: (crossfilter) => (CrossfilterUtils.makeNumeric(crossfilter, property)),
prepareGroup: (dimension) => {
if (!this.values[property]) {
var group = CrossfilterUtils.makeNumericGroup(dimension);
this.values[property] = group;
}
return this.values[property];
},
valuesAreOrdinal: false
};
}
I haven't used the crossfilter library much before and by looking at the documentation I can't seem to reconcile it with the code (heritage code, to put it that way).
The incoming date format looks like this: "2020-10-22T07:26:00Z"
The typescript model I'm working with is like this:
interface MyModel {
...
CreatedDate?: string;
}
Any idea?
The usual pattern in JavaScript is to loop through the data and do any conversions you need:
data.forEach(function(d) {
d.date = new Date(d.date);
d.number = +d.number;
});
const cf = crossfilter(data);
However, if this is not allowed due to TS, you can also make the conversions when creating your dimensions and groups:
const cf = crossfilter(data);
const dateDim = cf.dimension(d => new Date(d.date));
const monthGroup = dateDim.group(date => d3.timeMonth(date))
.reduceSum(d => +d.number);
I find this a little less robust because you have to remember to do this everywhere. It's a little harder to reason about the efficiency since you have to trust that crossfilter uses the accessors sparingly, but I don't recall seeing this be a problem in practice.
I have one row in database to count total user logins
I have tried to increase number by getting the row and adding +1 to it
And i'm not sure about concurrency after I have tried this, counter was increased by 1 and not by 2 as it "should" (if many users will login at the same time)
using(var db = new Database()) {
db.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
using(var db2 = new Database()) {
db2.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db2.SaveChanges();
}
db.SaveChanges();
}
Why not make a single table for storing the number of people who have logged in increment the field when someone logs in successfully and decrease when the user logs out. For example for login:
_Users = context.Users.First(aa => aa.UserName.ToUpper() == _UserName.ToUpper() && aa.MDesktop == true);
if (_Users != null)
{
context.LogEntry.FirstOrDefault().Counter++;
context.SaveChanges();
}
This is old but it is still a relevant discussion for new EF developers and it deserves an explanation.
OP's example uses two different DBContext's, effectively OP has defined two different units of work, and importantly, neither of these is aware that the other exists at all.
Lets assume that the current value of the "Logins" setting is 5
For the purposes of this walkthrough lets save the two instances that are requested from Settings into variables outside of the scope of the DB contexts in question:
Setting setting1 = null;
Setting setting2 = null;
using(var db = new Database()) {
// DB: 5, Setting1: null, Setting2: null
// Load the value of setting1 from the database
setting1 = db.Settings.FirstOrDefault(x => x.Name == "Logins");
// DB: 5, Setting1: 5, Setting2: null
// Increment the value of setting1
setting1.Counter++;
// at this point, no changes have been saved yet, the DB still holds the original value for "Logins"
// DB: 5, Setting1: 6, Setting2: null
// Create a new context called DB2
using(var db2 = new Database()) {
// load setting2 from the DB
setting2 = db2.Settings.FirstOrDefault(x => x.Name == "Logins");
// right now setting2 still has a value of 5, the previous change was not yet committed
// DB: 5, Setting1: 6, Setting2: 5
setting2.Counter++;
// DB: 5, Setting1: 6, Setting2: 6
// Save the value of Setting2 back to the database
db2.SaveChanges();
// DB: 6, Setting1: 6, Setting2: 6
// At this point setting1, setting2, and the DB all agree the value is 6.
}
// The context is only aware that we previously set the value of setting1 to 6
// so it issues an update to the DB
db.SaveChanges();
// ultimately this update would not actually change anything.
}
Entity Framework, Unit of Work and Repository data access patterns all exhibit this behaviour, when you create a new DbContext IRepository or IUnitOfWork it is done so in isolation of any others that might exist at the same point in time, there is no difference between instantiating a new context in the same method, or a different thread or even executing on entirely different servers. If you need to implement counters or incremental values there is always a degree of uncertainty when we first cache the value of the field, then increment the value and later write that value back to the database.
To minimise the potential conflict, read the record and save it immediately after, then as a rule always re-query the value of this setting before you use it.
You can call .SaveChanges() multiple times in your logic, in this example simply saving before instantiating the second context, or at least before the second context loaded the record from the DB would have been enough to see the value incremented twice:
using(var db = new Database()) {
db.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db.SaveChanges(); // save it back as soon as we've made the change
using(var db2 = new Database()) {
db2.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db2.SaveChanges();
}
db.SaveChanges();
}
Where possible, you will find the code simpler if you can avoid a schema where an incrementing or counter fields is required, instead you could turn the count logic into a query based solution.
Counters are of course a special case, you could always make direct SQL calls to the database, both for read or increment to ensure that that we bypass any potential caching that might occur with the records through EF.
You could do this as a one liner to increment the value:
dbContext.Database.ExecuteSqlCommand("UPDATE Setting SET[Counter] = IsNull([Counter],0) + 1 WHERE[Name] = 'Logins'");
Or if you want to inspect the new value:
int newCount = dbContext.Database.SqlQuery<int>(#"
UPDATE Setting SET[Counter] = IsNull([Counter],0) + 1
OUTPUT inserted.[Counter]
WHERE [Name] = 'Logins'").First();
If you need to ge tthe current value, and know that it is the most up-to-date then you can simply query it from any context in the same way:
int logins = dbContext.Database.SqlQuery<int>(#"
SELECT [Counter] FROM Setting
WHERE [Name] = 'Logins'").First();
I hope this sheds some light on why your code only incremented the value once, its not a fault in EF, just something that we need to be aware of, once EF has read values form the DB, they are potentially already stale or out of date. If optimistic concurrency is not appropriate for your use case, then you will need to think outside of the box a little bit ;)
the easy approach?
then I'd suggest using a manual transaction in EF Core
ef core transaction docs
Be sure to add an unique constraint of some sort eb. (settings id + logins counter)
using(var transaction = _context.Database.BeginTransaction())
{
try
{
var totalLoginsCounter = _context.Settings.FirstOrDefault(x => x.Name == "Logins").Counter;
totalLoginsCounter += 1;
await _context.SaveChanges();
transaction.Commit();
}
catch
{
commit.RollBack();
}
}
should concurrency happen the request will fail. Because it would try to put duplicate keys which is not possible. then HIGHLY recommend you'd then implement a retry pattern to avoid people not being able to actually login because a number in your database didn't get updated.
I'm currently building a ASP.NET web app search result page. I've implemented different types of sorting for users but some of them are giving unexpected results.
These two line of code are executed regardless of the sort type or any other factors:
var resources = ctx.Resource.Where(p => p.CityAlias == city.ToLower() && p.Company.Hidden != true && p.State == 1);
FilterResources(ref resources, resourceTypeId, serviceId);
FilterResources:
private void FilterResources(ref IQueryable<Resource> allRes, int resourceType, int selectedService)
{
allRes = allRes.Where(p => p.ResourceType_Id == resourceType && p.ResourceService.Any(x => x.Service_Id == selectedService));
}
Resource.ResourceService is of type ICollection that is mapped to a database table of the same name with a foreign key pointing to Resource.Id
So far so good. First a working sort example:
private void SortResourcesByName(ref IQueryable<Resource> resources)
{
resources = resources.OrderBy(p => p.Name);
}
Then we handle paging:
int count = resources.Count();
var resourceList = resources.Skip((page - 1) * 10).Take(10).ToList();
This set from entities seems to be correct and in the right order, but here's when things get weird:
If we use a different sort, some of the pages have duplicate results and in some cases if I go through the pages, I can see duplicate (or more) Resources scattered across the list.
For example:
private void SortResourcesByPrice(ref IQueryable<Resource> resources, int serviceId)
{
resources = resources.OrderByDescending(p => p.ResourceService.FirstOrDefault(s => s.Service_Id == serviceId).Price.Value)
.ThenByDescending(p => p.Company.CompanyService.FirstOrDefault(c => c.ServiceId == serviceId).Price.Value);
}
If I use this sort, the first page of the view is correct, but when I start navigating to other pages, I start to see duplicate results.
The query returns just over 200 results, which are divided among 21 pages. Pages 3 and 4 are exactly the same for some reason, pages 5 and 6 also yield same results. In these cases, ResourceService.Price = null.
Bonus weirdness:
using OrderBy() instead of OrderByDescending does not give duplicate results
private void SortResourcesByPrice(ref IQueryable<Resource> resources, int serviceId)
{
resources = resources.OrderBy(p => p.ResourceService.FirstOrDefault(s => s.Service_Id == serviceId).Price.Value)
.ThenBy(p => p.Company.CompanyService.FirstOrDefault(c => c.ServiceId == serviceId).Price.Value);
}
I'm at a total loss here, so I'd really appreciate help. If you can't make sense of my example code or can't understand the question in some other way, I'd happily try to provide more information.
Edit:
I'd like to add that the amount of results/Resources is always the same, regardless of what sort I'm using (just as it's supposed to).
Edit 2:
Fixed some numbers & typos
You can use https://expressprofiler.codeplex.com/ to view the SQL generate by EF copy and paste the sql query directly in SQL Server Management studio and look where is the duplicate origine, you can change query to remove duplicate entry and after search Linq responsible.
You can put the SQL here too