Fetching data from cache if available, fetch from database if not - asp.net

I have a page that need to run a query against a large dataset very often. To ease the burden on the database, I've set up a cache that will refresh itself every 5 minutes.
The logic is:
When a call is made, check if there is data in cache, if it is, run the queryu on the cache. If not, start a task of fetching from all rows from database while running a query on my repository to get out just the data needed for that call. When all rows is fetched, put it in the cache so it can be accessed on the next call. The problem is that I sometimes get a: "Message = "There is already an open DataReader associated with this Command which must be closed first." I guess this is because it runs two queries to the same repository at the same time (one for all rows and one for the query). I've got MARS enabled in my connections string.
My code
public IQueryable<TrackDto> TrackDtos([FromUri] int[] Ids)
{
if (HttpContext.Current.Cache["Tracks"] != null && ((IQueryable<TrackDto>)HttpContext.Current.Cache["Tracks"]).Any())
{
var trackDtos = Ids.Length > 0
? ((IQueryable<TrackDto>)HttpContext.Current.Cache["Tracks"]).Where(trackDto => Ids.Contains(trackDto.Id).AsQueryable()
: ((IQueryable<TrackDto>)HttpContext.Current.Cache["Tracks"]).AsQueryable();
return trackDtos;
}
else
{
UpdateTrackDtoCache(DateTime.Today);
var trackDtos = Ids.Length > 0
? WebRepository.TrackDtos.Where(trackDto => trackDto.Date == DateTime.Today && Ids.Contains(trackDto.Id)).AsQueryable()
: WebRepository.TrackDtos.Where(trackDto => trackDto.Date == DateTime.Today).AsQueryable().AsQueryable();
return trackDtos;
}
}
private IQueryable<TrackDto> MapTrackDtosFromDb(DateTime date)
{
return WebRepository.TrackDtos.Where(tdto => tdto.Date == date.Date);
}
private void UpdateTrackDtoCache(DateTime date)
{
if (CacheIsUpdating)
return;
CacheIsUpdating = true;
var task = Task.Factory.StartNew(
state =>
{
var context = (HttpContext)state;
context.Cache.Insert("Tracks", MapTrackDtosFromDb(date), null, Cache.NoAbsoluteExpiration,
new TimeSpan(0, 5, 0));
CacheIsUpdating = false;
},
HttpContext.Current);
}

I believe you are running DML or DDL sql queries using the same active connection. And MARS does not allow that. You can execute multiple select statements or bulk insert but if you run multiple update, delete statements or your sql execution will throw this kind of errors. Even if you run an update sql query while running a select statement on the same command you will get this error. For more info read this
http://msdn.microsoft.com/en-us/library/h32h3abf(v=vs.110).aspx

Related

Ingest from storage with persistDetails = true not save ingest status result

I'm now implement a program to migrate large amount of data to ADX base on Ingest from Storage feature of ADX and I'm need to check that status of each ingestion request each time the request finish but I'm facing an issue
Base on MS document in here
If I set the persistDetails = true for example with the command below it must save the ingestion status but currently this setting seem not work (with or without it)
.ingest async into table MigrateTable
(
h'correct blob url link'
)
with (
jsonMappingReference = 'table_mapping',
format = 'json',
persistDetails = true
)
Above command will return an OperationId and when I using it to check export status when the ingest task finish I always get this error message :
Error An admin command cannot be executed due to an invalid state: State='Operation 'DataIngestPull' does not persist its operation results' clientRequestId: KustoWebV2;
Can someone clarify for me what is the root cause relate to this? With me it seem like a bug relate to ADX
Ingesting data directly against the Data Engine, by running .ingest commands, is usually not recommended, compared to using Queued Ingestion (motivation included in the link). Using Kusto's ingestion client library allows you to track the ingestion status.
Some tools/services already do that for you, and you can consider using them directly. e.g. LightIngest, Azure Data Factory
If you don't follow option 1, you can still look for the state/status of your command using the operation ID you get when using the async keyword, by using .show operations
You can also use the client request ID to filter the result set of .show commands to view the state/status of your command.
If you're interested in looking specifically at failures, .show ingestion failures is also available for you.
The persistDetails option you specified in your .ingest command actually has no effect - as mentioned in the docs:
Not all control commands persist their results, and those that do usually do so by default on asynchronous executions only (using the async keyword). Please search the documentation for the specific command and check if it does (see, for example data export).
============ Update sample code follow suggestion from Yoni ========
Turn out, other member in my team mess up with access right with adx, after fixing it everything work fine
I just have one concern relate to PartiallySucceeded that need clarify from #yoni or someone have better knowledge relate to that
try
{
var ingestProps = new KustoQueuedIngestionProperties(model.DatabaseName, model.IngestTableName)
{
ReportLevel = IngestionReportLevel.FailuresAndSuccesses,
ReportMethod = IngestionReportMethod.Table,
FlushImmediately = true,
JSONMappingReference = model.IngestMappingName,
AdditionalProperties = new Dictionary<string, string>
{
{"jsonMappingReference",$"{model.IngestMappingName}" },
{ "format","json"}
}
};
var sourceId = Guid.NewGuid();
var clientResult = await IngestClient.IngestFromStorageAsync(model.FileBlobUrl, ingestProps, new StorageSourceOptions
{
DeleteSourceOnSuccess = true,
SourceId = sourceId
});
var ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
while (ingestionStatus.Status == Status.Pending)
{
await Task.Delay(WaitingInterval);
ingestionStatus = clientResult.GetIngestionStatusBySourceId(sourceId);
}
if (ingestionStatus.Status == Status.Succeeded)
{
return true;
}
LogUtils.TraceError(_logger, $"Error when ingest blob file events, error: {ingestionStatus.ErrorCode.FastGetDescription()}");
return false;
}
catch (Exception e)
{
return false;
}

xamarin forms azure mobile apps slow sync

I'm using Azure Mobile App with Xamarin.Forms to create an offline capable mobile app.
My solution is based on https://adrianhall.github.io/develop-mobile-apps-with-csharp-and-azure/chapter3/client/
Here is the code that I use for offline sync :
public class AzureDataSource
{
private async Task InitializeAsync()
{
// Short circuit - local database is already initialized
if (client.SyncContext.IsInitialized)
{
return;
}
// Define the database schema
store.DefineTable<ArrayElement>();
store.DefineTable<InputAnswer>();
//Same thing with 16 others table
...
// Actually create the store and update the schema
await client.SyncContext.InitializeAsync(store, new MobileServiceSyncHandler());
}
public async Task SyncOfflineCacheAsync()
{
await InitializeAsync();
//Check if authenticated
if (client.CurrentUser != null)
{
// Push the Operations Queue to the mobile backend
await client.SyncContext.PushAsync();
// Pull each sync table
var arrayTable = await GetTableAsync<ArrayElement>();
await arrayTable.PullAsync();
var inputAnswerInstanceTable = await GetTableAsync<InputAnswer>();
await inputAnswerInstanceTable.PullAsync();
//Same thing with 16 others table
...
}
}
public async Task<IGenericTable<T>> GetTableAsync<T>() where T : TableData
{
await InitializeAsync();
return new AzureCloudTable<T>(client);
}
}
public class AzureCloudTable<T>
{
public AzureCloudTable(MobileServiceClient client)
{
this.client = client;
this.table = client.GetSyncTable<T>();
}
public async Task PullAsync()
{
//Query name used for incremental pull
string queryName = $"incsync_{typeof(T).Name}";
await table.PullAsync(queryName, table.CreateQuery());
}
}
The problem is that the syncing takes a lot of time even when there isn't anything to pull (8-9 seconds on Android devices and more than 25 seconds to pull the whole database).
I looked at Fiddler to find how much time takes the Mobile Apps BackEnd to respond and it is about 50 milliseconds per request so the problem doesn't seem to come from here.
Does anyone have the same trouble ? Is there something that I'm doing wrong or tips to improve my sync performance ?
Our particular issue was linked to our database migration. Every row in the database had the same updatedAt value. We ran an SQL script to modify these so that they were all unique.
This fix was actually for some other issue we had, where not all rows were being returned for some unknown reason, but we also saw a substantial speed improvement.
Also, another weird fix that improved loading times was the following.
After we had pulled all of the data the first time (which, understandably takes some time) - we did an UpdateAsync() on one of the rows that were returned, and we did not push it afterwards.
We've come to understand that the way offline sync works, is that it will pull anything that has a date newer than the most recent updated at. There was a small speed improvement associated with this.
Finally, the last thing we did to improve speed was to not fetch the data again, if it already had cached a copy in the view. This may not work for your use case though.
public List<Foo> fooList = new List<Foo>
public void DisplayAllFoo()
{
if(fooList.Count == 0)
fooList = await SyncClass.GetAllFoo();
foreach(var foo in fooList)
{
Console.WriteLine(foo.bar);
}
}
Edit 20th March 2019:
With these improvements in place, we are still seeing very slow sync operations, used in the same way as mentioned in the OP, also including the improvements listed in my answer here.
I encourage all to share their solutions or ideas on how this speed can be improved.
One of the reasons for the slow Pull() is when more than (10) rows get the same UpdatedAt value. This happens when you update the rows at once, for example running an SQL command.
One way to overcome this is to modify the default trigger on the tables. To ensure every row gets a unique UpdateAt, we did something like this:
ALTER TRIGGER [dbo].[TR_dbo_Items_InsertUpdateDelete] ON [dbo].[TableName]
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
DECLARE #InsertedAndDeleted TABLE
(
Id NVARCHAR(128)
);
DECLARE #Count INT,
#Id NVARCHAR(128);
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM inserted;
INSERT INTO #InsertedAndDeleted
SELECT Id
FROM deleted
WHERE Id NOT IN
(
SELECT Id
FROM #InsertedAndDeleted
);
--select * from #InsertedAndDeleted;
SELECT #Count = Count(*)
FROM #InsertedAndDeleted;
-- ************************ UpdatedAt ************************
-- while loop
WHILE #Count > 0
BEGIN
-- selecting
SELECT TOP (1) #Id = Id
FROM #InsertedAndDeleted;
-- updating
UPDATE [dbo].[TableName]
SET UpdatedAt = Convert(DATETIMEOFFSET, DateAdd(MILLISECOND, #Count, SysUtcDateTime()))
WHERE Id = #Id;
-- deleting
DELETE FROM #InsertedAndDeleted
WHERE id = #Id;
-- counter
SET #Count = #Count - 1;
END;
END;

DocumentDB select document at specific index

Is it possible to select a document at a specific index?
I have a document import process, I get a page of items from my data source (250 items at once) I then import these into DocumentDB in concurrently. Assuming I get an error inserting these items into DocumentDB I wont be sure what individual item or items failed. (I could work it out but don't want to). It would be easier to just Upsert all the items from the page again.
The items i'm inserting have an ascending id. So if i query DocumentDB (ordered by id) and select the id at position (count of all Id's - page size) I can start importing from that point forward again.
I know SKIP is not implemented, I want to check if there is another option?
You could try a bulk import stored procedure. The sproc creation code below is from Azure's github repo. This sproc will report back the number of docs created in the batch and continue trying to create docs in multiple batches if the sproc times out.
Since the sproc is ACID, you will have to retry from the beginning (or the last successful batch) if there are any exceptions thrown.
You could also change the createDocument function to upsertDocument if you just want to retry the entire batch process if any exception is thrown.
{
id: "bulkImportSproc",
body: function bulkImport(docs) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
// The count of imported docs, also used as current doc index.
var count = 0;
// Validate input.
if (!docs) throw new Error("The array is undefined or null.");
var docsLength = docs.length;
if (docsLength == 0) {
getContext().getResponse().setBody(0);
return;
}
// Call the CRUD API to create a document.
tryCreate(docs[count], callback);
// Note that there are 2 exit conditions:
// 1) The createDocument request was not accepted.
// In this case the callback will not be called, we just call setBody and we are done.
// 2) The callback was called docs.length times.
// In this case all documents were created and we don't need to call tryCreate anymore. Just call setBody and we are done.
function tryCreate(doc, callback) {
var isAccepted = collection.createDocument(collectionLink, doc, callback);
// If the request was accepted, callback will be called.
// Otherwise report current count back to the client,
// which will call the script again with remaining set of docs.
// This condition will happen when this stored procedure has been running too long
// and is about to get cancelled by the server. This will allow the calling client
// to resume this batch from the point we got to before isAccepted was set to false
if (!isAccepted) getContext().getResponse().setBody(count);
}
// This is called when collection.createDocument is done and the document has been persisted.
function callback(err, doc, options) {
if (err) throw err;
// One more document has been inserted, increment the count.
count++;
if (count >= docsLength) {
// If we have created all documents, we are done. Just set the response.
getContext().getResponse().setBody(count);
} else {
// Create next document.
tryCreate(docs[count], callback);
}
}
}
}

Entity framework - increase number by 1

I have one row in database to count total user logins
I have tried to increase number by getting the row and adding +1 to it
And i'm not sure about concurrency after I have tried this, counter was increased by 1 and not by 2 as it "should" (if many users will login at the same time)
using(var db = new Database()) {
db.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
using(var db2 = new Database()) {
db2.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db2.SaveChanges();
}
db.SaveChanges();
}
Why not make a single table for storing the number of people who have logged in increment the field when someone logs in successfully and decrease when the user logs out. For example for login:
_Users = context.Users.First(aa => aa.UserName.ToUpper() == _UserName.ToUpper() && aa.MDesktop == true);
if (_Users != null)
{
context.LogEntry.FirstOrDefault().Counter++;
context.SaveChanges();
}
This is old but it is still a relevant discussion for new EF developers and it deserves an explanation.
OP's example uses two different DBContext's, effectively OP has defined two different units of work, and importantly, neither of these is aware that the other exists at all.
Lets assume that the current value of the "Logins" setting is 5
For the purposes of this walkthrough lets save the two instances that are requested from Settings into variables outside of the scope of the DB contexts in question:
Setting setting1 = null;
Setting setting2 = null;
using(var db = new Database()) {
// DB: 5, Setting1: null, Setting2: null
// Load the value of setting1 from the database
setting1 = db.Settings.FirstOrDefault(x => x.Name == "Logins");
// DB: 5, Setting1: 5, Setting2: null
// Increment the value of setting1
setting1.Counter++;
// at this point, no changes have been saved yet, the DB still holds the original value for "Logins"
// DB: 5, Setting1: 6, Setting2: null
// Create a new context called DB2
using(var db2 = new Database()) {
// load setting2 from the DB
setting2 = db2.Settings.FirstOrDefault(x => x.Name == "Logins");
// right now setting2 still has a value of 5, the previous change was not yet committed
// DB: 5, Setting1: 6, Setting2: 5
setting2.Counter++;
// DB: 5, Setting1: 6, Setting2: 6
// Save the value of Setting2 back to the database
db2.SaveChanges();
// DB: 6, Setting1: 6, Setting2: 6
// At this point setting1, setting2, and the DB all agree the value is 6.
}
// The context is only aware that we previously set the value of setting1 to 6
// so it issues an update to the DB
db.SaveChanges();
// ultimately this update would not actually change anything.
}
Entity Framework, Unit of Work and Repository data access patterns all exhibit this behaviour, when you create a new DbContext IRepository or IUnitOfWork it is done so in isolation of any others that might exist at the same point in time, there is no difference between instantiating a new context in the same method, or a different thread or even executing on entirely different servers. If you need to implement counters or incremental values there is always a degree of uncertainty when we first cache the value of the field, then increment the value and later write that value back to the database.
To minimise the potential conflict, read the record and save it immediately after, then as a rule always re-query the value of this setting before you use it.
You can call .SaveChanges() multiple times in your logic, in this example simply saving before instantiating the second context, or at least before the second context loaded the record from the DB would have been enough to see the value incremented twice:
using(var db = new Database()) {
db.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db.SaveChanges(); // save it back as soon as we've made the change
using(var db2 = new Database()) {
db2.Settings.FirstOrDefault(x => x.Name == "Logins").Counter++;
db2.SaveChanges();
}
db.SaveChanges();
}
Where possible, you will find the code simpler if you can avoid a schema where an incrementing or counter fields is required, instead you could turn the count logic into a query based solution.
Counters are of course a special case, you could always make direct SQL calls to the database, both for read or increment to ensure that that we bypass any potential caching that might occur with the records through EF.
You could do this as a one liner to increment the value:
dbContext.Database.ExecuteSqlCommand("UPDATE Setting SET[Counter] = IsNull([Counter],0) + 1 WHERE[Name] = 'Logins'");
Or if you want to inspect the new value:
int newCount = dbContext.Database.SqlQuery<int>(#"
UPDATE Setting SET[Counter] = IsNull([Counter],0) + 1
OUTPUT inserted.[Counter]
WHERE [Name] = 'Logins'").First();
If you need to ge tthe current value, and know that it is the most up-to-date then you can simply query it from any context in the same way:
int logins = dbContext.Database.SqlQuery<int>(#"
SELECT [Counter] FROM Setting
WHERE [Name] = 'Logins'").First();
I hope this sheds some light on why your code only incremented the value once, its not a fault in EF, just something that we need to be aware of, once EF has read values form the DB, they are potentially already stale or out of date. If optimistic concurrency is not appropriate for your use case, then you will need to think outside of the box a little bit ;)
the easy approach?
then I'd suggest using a manual transaction in EF Core
ef core transaction docs
Be sure to add an unique constraint of some sort eb. (settings id + logins counter)
using(var transaction = _context.Database.BeginTransaction())
{
try
{
var totalLoginsCounter = _context.Settings.FirstOrDefault(x => x.Name == "Logins").Counter;
totalLoginsCounter += 1;
await _context.SaveChanges();
transaction.Commit();
}
catch
{
commit.RollBack();
}
}
should concurrency happen the request will fail. Because it would try to put duplicate keys which is not possible. then HIGHLY recommend you'd then implement a retry pattern to avoid people not being able to actually login because a number in your database didn't get updated.

Collection was modified; enumeration operation may not execute

I have getting error
Collection was modified; enumeration operation may not execute.
at System.Collections.Queue.QueueEnumerator.MoveNext()
Queue ReqQ = (Application["ReqQ"] != null) ? ((Queue)Application["ReqQ"]) :
new Queue(50);
if (ReqQ != null)
{
foreach (object OReq in ReqQ)
{
string mId = (string)OReq;
if (mId.Split('~')[1].Equals(reqUid.Split('~')[1]) && (DateTime.Parse(mId.Split('~')[0]).AddMinutes(1 * int.Parse(string.IsNullOrEmpty(delay) ? "0" : delay)) > DateTime.Now))
{
isSuccess = false;
break;
}
}
}
else
{
ReqQ = new Queue(10);
isSuccess = true;
}
if (isSuccess)
{
if (ReqQ.Count >= 10) //only keep last 10 messages in application cache
ReqQ.Dequeue();
ReqQ.Enqueue(reqUid);
Application["ReqQ"] = ReqQ;
}
It looks like you've got a single collection which you're reading and modifying from multiple threads (for different requests). To start with that's not safe using Queue - and it's particularly not true if you're iterating through the collection while you modify it in another. (EDIT: I've just noticed you're not even using a generic collection. If you're using .NET 4, there's no reason to use the non-generic collections...)
It's unclear what you're trying to achieve - you may be able to get away with just changing to use ConcurrentQueue<T> instead, but you need to be aware that by the time you've iterated over the collection, the values you read may already have been dequeued in another thread.

Resources