It is possible to make a basic CRUD on JSON file? - asp.net

I have a news site in ASP.NET MVC, where all the data comes from the homepage of the database. And this is generating a large traffic to my hosting server.
I have sessions like: Politics, Sports, World, among others, and each session is a different request made to the database. But each request only a small number of data is returned from the database, maximum 8 results for query. In total there are 21 requests made to the database.
I thought about making a data.json file and persist the session information in this file and show them in the frontend.
But as the file data.json and not a database, I have to do all the work to relocate the information when an action to delete or edit is performed.
Is there a better way to get to the point I want, considering that the site is already published?

I would put a caching layer in between the site and the db so when you do one of these calls to the db it first checks the cache to see if the objects are there, if they are then it returns them from memory (super quick), if not it gets from db and populates the cache for next time. You must remember to clear the cache when the data in the DB gets changed though.
Cache Interface:
public interface ICacheProvider
{
void Add(string key, object value);
void Add(string key, object value, TimeSpan timeout);
object Get(string key);
object this[string key] { get; set; }
bool Remove(string key);
bool Contains(string key);
void ClearAll();
}
Cache Implementation:
public class HttpCacheProvider : ICacheProvider
{
/// <summary>
/// The http cache
/// </summary>
private readonly Cache _cache;
/// <summary>
/// How often to expire the cache
/// </summary>
public int CacheExpiryMinutes { get; set; }
public HttpCacheProvider()
: this(HttpContext.Current.Cache, 60)
{
}
public HttpCacheProvider(int cacheExpiryMinutes)
: this(HttpContext.Current.Cache, cacheExpiryMinutes)
{
}
public HttpCacheProvider(Cache cache, int cacheExpiryMinutes)
{
_cache = cache;
CacheExpiryMinutes = cacheExpiryMinutes;
}
public void Add(string key, object value)
{
if (_cache[key] == null)
_cache.Insert(key, value, null, DateTime.Now.AddMinutes(CacheExpiryMinutes), Cache.NoSlidingExpiration);
}
public void Add(string key, object value, TimeSpan timeout)
{
if (_cache[key] == null)
_cache.Insert(key, value, null, DateTime.Now.Add(timeout), Cache.NoSlidingExpiration);
}
public object Get(string key)
{
return _cache[key];
}
public bool Contains(string key)
{
return (this[key] != null);
}
public object this[string key]
{
get { return _cache[key]; }
set { _cache[key] = value; }
}
public bool Remove(string key)
{
return _cache.Remove(key) != null;
}
public void ClearAll()
{
var enumerator = _cache.GetEnumerator();
while (enumerator.MoveNext())
{
_cache.Remove(enumerator.Key.ToString());
}
}
}
Usage:
private ICacheProvider _cacheProvider;
public IEnumerable<Something> GetData(int param1, int param2)
{
var cacheKey = string.Format("{0}-{1}", param1, param2);
if (!_cacheProvider.Contains(cacheKey))
{
try
{
// get data from db
// add that data to cahce
_cacheProvider.Add(cacheKey, nodes);
}
catch (Exception ex)
{
// blah, logging
}
}
return _cacheProvider[cacheKey] as IEnumerable<Something>;
}

Related

Masstransit: How to build a cache filter in the Mediator

We used the Masstransit Mediator to write request/response "Consumers" called from API controllers. Before the consumer is taken action, some ConsumeFilters take place: Logging, Validation and DBTransaction. Next I would like to implement a Cache Filter using simple Microsoft In-Memory Cache. The filter should check if the request object is already in cache, if not the consumer pipe is called and the cache object is added, else the cached object should be returned immediatly.
I could not figure out how write such a filter. Do I need two filters? If I call RespondAsync from ConsumeContext how can a use a generic response type?
Has someone done it before, or should I do I directly in consumer?
Seems like something that should be in the consumer itself. The cache itself could be a dependency of the consumer, which is a single instance and injected into the consumer via the constructor. That way, it would be able to check if the results are in the cache before calling the backing service with the request detail.
Hiding that in a filter seems a little specific to the message type, so within the consumer will likely be easier for developers to understand later.
I figured out a solution to integrate in-memory caching as mass transit scope filter. Currently it is only used in a mediator. Some prequists:
All messages are records (with value bases equal methods)
The query request object inherits from ApplicationQueryRequest (a record)
The query response objects inherits from ApplicationResponse (a record)
The request object has an attribute named Cached Attribute:
[AttributeUsage(AttributeTargets.Class)]
public sealed class CacheAttribute : Attribute
{
public CacheAttribute(int slidingExpireSecs = 30, int absoluteExpireSecs = 100)
{
SlidingExpireSecs = slidingExpireSecs;
AbsoluteExpireSecs = absoluteExpireSecs;
}
public int SlidingExpireSecs { get; }
public int AbsoluteExpireSecs { get; }
}
Therefore each request object can have a cache attribute like:
[Cache]
public record FooRequest
{
}
Target is that the filter automatically fetches data from the cache and stores data in it.
First initialize the mediator with all consumer and send fiters, in our case only one scope filter exists but must be added for send and consume:
services.AddMediator(
configurator =>
{
(context, cfg) =>
{
cfg.UseSendFilter(typeof(CacheScopeFilter<>), context);
cfg.UseConsumeFilter(typeof(CacheScopeFilter<>), context);
Additionally the ICacheScope must be of
services.AddScoped...
THe scope filter looks like this:
public class CacheScopeFilter<T> :
IFilter<SendContext<T>>,
IFilter<ConsumeContext<T>> where T : class
{
private readonly ILogger<T> logger;
private readonly IMemoryCache memoryCache;
private readonly ICacheScope cacheScope;
private CacheOptions cacheOptions;
public CacheScopeFilter(ILogger<T> logger, IOptionsMonitor<CacheOptions> options, IMemoryCache memoryCache, ICacheScope cacheScope)
{
this.logger = logger;
cacheOptions = options.CurrentValue;
options.OnChange(
opts =>
{
logger.LogInformation($"Set Memory Cache enabled: {opts.EnableMemoryCache}");
cacheOptions = opts;
});
this.memoryCache = memoryCache;
this.cacheScope = cacheScope;
}
public async Task Send(ConsumeContext<T> context, IPipe<ConsumeContext<T>> next)
{
var requestName = typeof(T).Name;
logger.LogInformation($"----- Start check cache application query request {requestName} {context.Message}");
cacheScope.RequestKey = null;
if (context.TryGetMessage<ApplicationQueryRequest>(out var requestContext))
{
if(!cacheOptions.EnableMemoryCache)
{
logger.LogInformation("Cache is disabled");
await next.Send(context);
return;
}
var cacheAttribute = (CacheAttribute)Attribute.GetCustomAttribute(
requestContext.Message.GetType(),
typeof(CacheAttribute));
if (cacheAttribute == null)
{
await next.Send(context);
return;
}
cacheScope.RequestKey = typeof(T).FullName + ";" + JsonConvert.SerializeObject(context.Message);
cacheScope.SlidingExpireSecs = cacheAttribute.SlidingExpireSecs;
cacheScope.AbsoluteExpireSecs = cacheAttribute.AbsoluteExpireSecs;
if (memoryCache.TryGetValue(cacheScope.RequestKey, out ApplicationResponse cacheResponse))
{
logger.LogInformation($"Take data from cache {requestName} {context.Message}, CacheKey: {cacheScope.RequestKey}");
await context.RespondAsync(cacheResponse);
return;
}
logger.LogInformation($"Data not in cache, fetching data {requestName} {context.Message}");
}
await next.Send(context);
logger.LogInformation($"----- Finish check cache application query request {requestName} {context.Message}");
}
public async Task Send(SendContext<T> context, IPipe<SendContext<T>> next)
{
var requestName = typeof(T).Name;
logger.LogInformation($"----- Start handling cache application query response {requestName} {context.Message}");
var isCachedSet = context.TryGetPayload<CacheDoneMarker>(out _);
if (context.Message is ApplicationResponse && (cacheScope.RequestKey != null) && !isCachedSet)
{
logger.LogInformation($"Cache data {requestName} {context.Message}, CacheKey: {cacheScope.RequestKey}");
var cacheEntryOptions = new MemoryCacheEntryOptions().
SetSlidingExpiration(TimeSpan.FromSeconds(cacheScope.SlidingExpireSecs)).
SetAbsoluteExpiration(TimeSpan.FromSeconds(cacheScope.AbsoluteExpireSecs));
memoryCache.Set(cacheScope.RequestKey, context.Message, cacheEntryOptions);
context.GetOrAddPayload(() => new CacheDoneMarker());
}
await next.Send(context);
logger.LogInformation($"----- Finish handling cache application query response {requestName} {context.Message}");
}
public void Probe(ProbeContext context)
{
context.CreateFilterScope("cache");
}
}
public class CacheScope : ICacheScope
{
public string RequestKey { get; set; }
public int SlidingExpireSecs { get; set; }
public int AbsoluteExpireSecs { get; set; }
}
// Scope injected !
public interface ICacheScope
{
public string RequestKey { get; set; }
public int SlidingExpireSecs { get; set; }
public int AbsoluteExpireSecs { get; set; }
}
This should also work for distributed cache, but not added yet.

Aspnetboilerplate application gives runtime error while applying sort on a query for ISoftDelete entity

I want to write CRUD services for an entity named Location . Location has a composite primary key, so I couldn't use AsyncCrudAppService and copied the methods that I wanted from AsyncCrudAppService.
When I use GetAll Service without ApplySorting method, it works fine. But when I add sorting, I get this runtime error:
[35] ERROR BookingSystem.middlewares.HttpGlobalExceptionFilter [(null)] - The LINQ expression 'DbSet
.Where(l => __ef_filter__p_0 || !(((ISoftDelete)l).IsDeleted))
.OrderByDescending(l => l.Id)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). S
ee https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
public class Location : IEntity<int>, IPassivable, IFullAudited<User> {
public int Id { get; set; }
public int IdLocation { get; set; } //primary key 0
public LocationType TypeId { get; set; } //primary key 1
public string Name { get; set; }
}
public interface ILocationService : IApplicationService
{
Task<PagedResultDto<LocationDto>> GetAllAsync(PagedLocationResultRequestDto input);
}
public class LocationService : AbpServiceBase, ILocationService, IPerWebRequestDependency
{
private readonly IRepository<Location, int> _repository;
private IAsyncQueryableExecuter AsyncQueryableExecuter { get; set; }
public LocationService(IRepository<Location> repository)
{
_repository = repository;
AsyncQueryableExecuter = NullAsyncQueryableExecuter.Instance;
}
public async Task<PagedResultDto<LocationDto>> GetAllAsync(PagedLocationResultRequestDto input)
{
if (input.MaxResultCount > _appSettings.Value.MaxResultCount)
{
throw new BookingSystemUserFriendlyException(BookingSystemExceptionCode.InputNotValid,
nameof(input.MaxResultCount));
}
var query = CreateFilteredQuery(input);
var totalCount = await AsyncQueryableExecuter.CountAsync(query);
query = ApplySorting(query, input);
query = ApplyPaging(query, input);
var entities = await AsyncQueryableExecuter.ToListAsync(query);
return new PagedResultDto<LocationDto>(
totalCount,
entities.Select(MapToEntityDto).ToList()
);
}
protected virtual IQueryable<Location> ApplySorting(
IQueryable<Location> query, PagedLocationResultRequestDto input)
{
if (input is ISortedResultRequest sortInput &&
!sortInput.Sorting.IsNullOrWhiteSpace())
{
return query.OrderBy(sortInput.Sorting);
}
return query.OrderByDescending(e => e.Id);
}
protected virtual IQueryable<Location> ApplyPaging(
IQueryable<Location> query, PagedLocationResultRequestDto input)
{
if (input is IPagedResultRequest pagedInput)
{
return query.PageBy(pagedInput);
}
return query;
}
private IQueryable<Location> CreateFilteredQuery(PagedLocationResultRequestDto input)
{
return _repository.GetAll()
.WhereIf(!string.IsNullOrWhiteSpace(input.Name),
location => location.Name.ToLower().Contains(input.Name.Trim().ToLower()));
}
private LocationDto MapToEntityDto(Location entity)
{
return ObjectMapper.Map<LocationDto>(entity);
}
}
Abp package version: 5.1.0
Base framework: .Net Core
Well, I asked the same question in the project's GitHub and got the answer.
In ApplySorting, the default sorting is based on Id, which doesn't exist in my database table.
If you are using composite PK, then you probably don't have the Id field in the database, right? Then you should not sort by Id.
https://github.com/aspnetboilerplate/aspnetboilerplate/issues/5274#issuecomment-583946065

A durable entity does not deserialize

I am trying to use a durable entity in my Azure Function to cache some data. However, when I try to retrieve the entity (state) for the first time, I get an exception indicating an issue during the entity deserialization.
Here is my entity class and related code
[JsonObject(MemberSerialization.OptIn)]
public class ActionTargetIdCache : IActionTargetIdCache
{
[JsonProperty("cache")]
public Dictionary<string, ActionTargetIdsCacheItemInfo> Cache { get; set; } = new Dictionary<string, ActionTargetIdsCacheItemInfo>();
public void CacheCleanup(DateTime currentUtcTime)
{
foreach (string officeHolderId in Cache.Keys)
{
TimeSpan cacheItemAge = currentUtcTime - Cache[officeHolderId].lastUpdatedTimeStamp;
if (cacheItemAge > TimeSpan.FromMinutes(2))
{
Cache.Remove(officeHolderId);
}
}
}
public void DeleteActionTargetIds(string officeHolderId)
{
if (this.Cache.ContainsKey(officeHolderId))
{
this.Cache.Remove(officeHolderId);
}
}
public void DeleteState()
{
Entity.Current.DeleteState();
}
public void SetActionTargetIds(ActionTargetIdsCacheEntry entry)
{
this.Cache[entry.Key] = entry.Value;
}
public Task<ActionTargetIdsCacheItemInfo> GetActionTargetIdsAsync(string officeHolderId)
{
if (this.Cache.ContainsKey(officeHolderId))
{
return Task.FromResult(Cache[officeHolderId]);
}
else
{
return Task.FromResult(new ActionTargetIdsCacheItemInfo());
}
}
// public void Reset() => this.CurrentValue = 0;
// public int Get() => this.CurrentValue;
[FunctionName(nameof(ActionTargetIdCache))]
public static Task Run([EntityTrigger] IDurableEntityContext ctx)
=> ctx.DispatchAsync<ActionTargetIdCache>();
}
public class ActionTargetIdsCacheEntry
{
// officeHolderId
public string Key { get; set; } = string.Empty;
public ActionTargetIdsCacheItemInfo Value { get; set; } = new ActionTargetIdsCacheItemInfo();
}
[JsonObject(MemberSerialization.OptIn)]
public class ActionTargetIdsCacheItemInfo : ISerializable
{
public ActionTargetIdsCacheItemInfo()
{
lastUpdatedTimeStamp = DateTime.UtcNow;
actionTargetIds = new List<string>();
}
public ActionTargetIdsCacheItemInfo(SerializationInfo info, StreamingContext context)
{
lastUpdatedTimeStamp = info.GetDateTime("lastUpdated");
actionTargetIds = (List<string>)info.GetValue("actionTargetIds", typeof(List<string>));
}
[JsonProperty]
public DateTimeOffset lastUpdatedTimeStamp { get; set; } = DateTimeOffset.UtcNow;
[JsonProperty]
public List<string> actionTargetIds { get; set; } = new List<string>();
public void GetObjectData(SerializationInfo info, StreamingContext context)
{
info.AddValue("lastUpdated", lastUpdatedTimeStamp);
info.AddValue("actionTargetIds", actionTargetIds);
}
}
public interface IActionTargetIdCache
{
void CacheCleanup(DateTime currentUtcTime);
void DeleteActionTargetIds(string officeHolderId);
void DeleteState();
void SetActionTargetIds(ActionTargetIdsCacheEntry item);
// Task Reset();
Task<ActionTargetIdsCacheItemInfo> GetActionTargetIdsAsync(string officeHolderId);
// void Delete();
}
Here is the exception I get during the first attempt to access the state from an orchestration using the GetActionTargetIdsAsync method:
Exception has occurred: CLR/Microsoft.Azure.WebJobs.Extensions.DurableTask.EntitySchedulerException
Exception thrown: 'Microsoft.Azure.WebJobs.Extensions.DurableTask.EntitySchedulerException' in System.Private.CoreLib.dll: 'Failed to populate entity state from JSON: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'PolTrack.CdbGetFunctionApp.ActionTargetIdsCacheItemInfo' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
Path 'cache.officeHolderId1', line 1, position 29.'
Inner exceptions found, see $exception in variables window for more details.
Innermost exception Newtonsoft.Json.JsonSerializationException : Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'PolTrack.CdbGetFunctionApp.ActionTargetIdsCacheItemInfo' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
Path 'cache.officeHolderId1', line 1, position 29.
Could someone with the sufficient SO privileges please add the tag azure-durable-entities.
I did manage to get around this by following #silent suggestion. I re-designed the entity class to only use CLR types. In my case, that meant replacing Dictionary<string, ActionTargetIdsCacheItemInfo> with two dictionaries Dictionary<string, List<string>> and Dictionary<string, DateTimeOffset>.

HttpContext.Current and ConcurrentDictionary

Currently we have a very strange issue on our production server. For a specific param in query string, we get the data for query string in other request. I'm trying to figure out if this behavior can be caused, by the way I use ConcurrentDictionary in IHttpHandler:
Below is pseudo code example:
public class MyHandler : IHttpHandler
{
private static ConcurrentDictionary<string, DataObject> _dataCache = new ConcurrentDictionary<string, DataObject>();
public virtual bool IsReusable
{
get { return true; }
}
public virtual void ProcessRequest(HttpContext context)
{
Func<DataObject> getDataMethod = () =>
{
return DataFactory.GetData(context.Request.QueryString["dataid"].ToLower());
}
string cacheKey = HttpUtility.UrlDecode(context.Request.QueryString["dataid"].ToLower());
DataObject infoItem = _dataCache .GetOrAdd(cacheKey, (key) => { return getDataMethod(); })
//Other processing code
}
}
So it happens that for "dataid=1" i get the data for "dataid=2"...
When getDataMethod is executed, can I be sure that it will access the relevant context?

Cache an object in a .ashx handler

I'm not sure of the best way to approach this, but here is my scenario.
Let's say I have an ashx handler that feeds out some data to an ajax request from the client. Something simple like:
public class Handler : IHttpHandler
{
private List<MyObject> data;
public bool IsReusable
{
get { return true; }
}
public void ProcessRequest(HttpContext context)
{
data = GetData();
string result;
switch (context.Request.QueryString["requestType"])
{
case "case":
result = Foo.Bar(data).GetJSON();
break;
// .. several more data conversitions / comparisions
}
context.Response.Write(result);
}
public class MyObject
{
public int Id { get; set; }
public string Data { get; set; }
}
}
How would I cache the list of MyObject's so it is not rebuilt every time a request is made to the service? Let's say the list of MyObject gets thousands of results and I want to keep it cached for ~1 minute. Could I make use of context.Cache?
Basically; I do not want to cache the handlers output. Just an object with data that resides in it.
Edit:
I am looking for something along the lines of:
data = (List<MyObject>) context.Cache["data"];
if (data == null || !data.Any())
{
data = GetData();
context.Cache.Insert("data", data, null, DateTime.Now.AddMinutes(1), System.Web.Caching.Cache.NoSlidingExpiration);
}
You can add result to HttpContext.Current.Cache, something like this :
var requestType = context.Request.QueryString["requestType"];
HttpContext.Current.Cache[requestType] = result;

Resources