Get the most recent document in a Cosmos DB container using IQueryable - azure-cosmosdb

I need to get the most recent document in a Cosmos DB container using an IQueryable query.
The SQL query corresponding to what I want to do is:
SELECT top 1 * FROM c order by c._ts
What I have available is an IQueryable provided by a generic repository. The repository creates the IQueryable like this:
var queryable = container.GetItemLinqQueryable<TEntity>(false, null, requestOption);
And the Entity (in IQueryable) does NOT contain a field that maps to "_ts":
public class Entity {
public string id { get; set; }
public Name { get; set; }
}
Here is the code that I need to write, but there is a missing timestamp field:
var queryable = await _repo.GetAsync(partitionKey);
var query = queryable.OrderByDescending(e => e.{The Missing _ts field}).Take(1);
Is there a way to write such a LINQ query?
Note: I am using .NET Cosmos DB SDK (v3).

Add the ts attribute on your Entity class something like below. Now you can query on Timestamp.
[Newtonsoft.Json.JsonConverter(typeof(Microsoft.Azure.Documents.UnixDateTimeConverter))]
[Newtonsoft.Json.JsonProperty(PropertyName="_ts")]
private virtual DateTime Timestamp { get; }

Adding _ts attribute plainly throws error when ingesting the data without setting the value for TimeStamp field(Default value is "01/01/0001" for DateTime). It works fine when fetching data, but when ingesting we will run into issues. Solution is to use a nullable type which will work both for insertion, retrieval.
[Newtonsoft.Json.JsonConverter(typeof(Microsoft.Azure.Documents.UnixDateTimeConverter))]
[Newtonsoft.Json.JsonProperty(NullValueHandling=Newtonsoft.Json.NullValueHandling.Ignore, PropertyName="_ts")]
public DateTime? LastModified { get; }
The key here is NullValueHandling.Ignore and nullable type (DataTime?). If we have these values, Cosmos will internally update the _ts field when we insert or merge. So, no need to set the value in our code during ingestion. Querying for documents will fetch the value as usual.
The above will cause an issue if you try to fetch an element and update some properties of that element. Now if you try to push this element to cosmos, it will not update the latest modified time because we already have the value set in _ts.
To overcome this, add shouldserialize method.
private bool ShouldSerializeLastModified()
{
// never update last modified timestamp when updating entries in cosmos. Let cosmos do that for us.
return false;
}
Reference: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.databaseproperties.lastmodified?view=azure-dotnet
https://learn.microsoft.com/en-us/dotnet/desktop/winforms/controls/defining-default-values-with-the-shouldserialize-and-reset-methods?view=netframeworkdesktop-4.8

Related

Get Method does not find document (MongoDb) even if it exists

I have the following get-method:
public async Task<TModel> Get(string id)
{
var filter = Builders<TModel>.Filter.Eq("_id",id);
var result = await _collection.FindAsync(filter);
return result.FirstOrDefault();
}
My Model is defined like this:
public class Entity
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string Id { get; set; }
...
}
If I test this method I get 404 (Not found) back.
What I already checked:
The id parameter has the right id.
Used id exists in database.
_collection is the correct IMongoCollection.
MongoDB server is running and collection exits.
Connection string, credentials and permissions to connect to the MongoDB server is correct (other crud methods are working).
What other troubleshooting steps can I do? What could be the error?
Thank you for your help!
In the Entity class the _id field is defined to be ObjectId, but in the Get function it is string. MongoDB matches are type-sensitive, so you'll need to convert the string to an ObjectId first.

OrmLite - GUIDs as primary keys in Oracle

I'm using OrmLite with both SqlServer, Oracle and PostgreSQL dialects.
I want to use GUIDs as primary keys and have a simple object, using the AutoId attribute:
public class MyObject
{
[AutoId]
public Guid Id { get; set; }
[Required]
public string Name { get; set; }
...
All goes well with SqlServer and PostgreSQL dialetcs, but with Oracle I get an initial GUID with all zeros in the db, and a subsequent INSERT violates the unique key constraint of my primary key. How can this be accomplished db agnostic so it also works with Oracle?
Based on the source code I'm looking at, it doesn't appear to properly generate GUIDs for anything that's not SQL Server or PostgreSQL, regardless of what the documentation actually says on the README. Relevant code links below:
SQL Server
PostgreSQL
Base Dialect Provider
The best alternative I can provide here is to override the OracleOrmLiteDialectProvider. Specifically, I would override the GetAutoIdDefaultValue method to return "SYS_GUID()" if the field type is a GUID. Sample code below...
public OracleNewGuidOrmLiteDialectProvider : OracleOrmLiteDialectProvider
{
public static OracleNewGuidOrmLiteDialectProvider Instance = new OracleNewGuidOrmLiteDialectProvider();
public string AutoIdGuidFunction { get; set; } = "SYS_GUID()";
public override string GetAutoIdDefaultValue(FieldDefinition fieldDef)
{
return fieldDef.FieldType == typeof(Guid)
? AutoIdGuidFunction
: null;
}
}
To match the rest of the provider implementations, I would recommend creating a OracleNewGuidDialect class, like below...
public static class OracleNewGuidDialect
{
public static IOrmLiteDialectProvider Provider => OracleNewGuidOrmLiteDialectProvider.Instance;
}
Then you would set the provider when you instantiate your OrmLiteConnectionFactory to OracleNewGuidOrmLiteDialectProvider.Instance, similar to below...
var dbFactory = new OrmLiteConnectionFactory(oracleConnectionString, OracleNewGuidDialect.Provider);
This isn't the best solution, but the pluggable nature of ServiceStack ORMLite allows you to control everything to the extent that you need. Hope this helps. Also, quick caveat--I didn't fully bake this solution, so you may need to tweak it, but based on the other implementations of the providers, it seems straightforward.

Audit.net data models example

Does any one have a working example of how to added audit models to an existing project, for Audit.Net.
It is one fantastic component to use, and up until now, my team and I have gotten by with the standard JSON files, however, we'd like to migrate our current solution to our Xamarin application, and would like to store the auditing in the local SQLite database on the device.
However, the documentation for this project is somewhat lacking and there is no concise examples of how to get custom auditing working with Entity Framework.
We have worked through the MD files on the github repo, but we still cannot get auditing to work.
Another question, similar to this has been asked HERE, but there is no definitive example of what the Audit_{entity} table should look like, what fields it MUST contain, and how to set up relationships for it.
We tried to reverse engineer the JSON files into a relational structure, but at the time of asking this question, we have not gotten any auditing to write to the SQLite database.
Sorry about the documentation not helping too much, hope I (or anybody) can provide better documentation in the future.
I am assuming you are using EntityFramework to map your entities
to a SQLite database, and you want to use the EF data
provider
to store the audits events in the same database, in Audit_{entity} tables.
There is no constraint on the schema you want to use for your Audit_{entity} tables, as long as you have a one-to-one relation between your {entity} table and its Audit_{entity} table. Then the mapping can be configured on several ways.
The recommendation for the Audit_{entity} tables is to have the same columns as the audited {entity} table, with any common additional column needed, like a User and a Date defined on an Interface.
So, if all your Audit_{entity} tables has the same columns/properties as its {entity}, and you added some common columns (defined on an interface), the configuration can be set like this:
public class User
{
public int Id { get; set; }
public string Name { get; set; }
}
public class Audit_User : IAudit
{
public int Id { get; set; }
public string Name { get; set; }
// IAudit members:
public string AuditUser { get; set; }
public datetime AuditDate { get; set; }
public string Action { get; set } // "Insert", "Update" or "Delete"
}
Audit.Core.Configuration.Setup()
.UseEntityFramework(x => x
.AuditTypeNameMapper(typeName => "Audit_" + typeName)
.AuditEntityAction<IAudit>((ev, ent, auditEntity) =>
{
auditEntity.AuditDate = DateTime.UtcNow;
auditEntity.AuditUser = evt.Environment.UserName;
auditEntity.AuditAction = ent.Action;
});
Note the interface is not mandatory, but using it makes the configuration cleaner. Also note you can make your Audit_{entity} inherit from your {entity} if you wanted to.
Update
Maybe my assumption at the beginning is incorrect and you are not auditing EF entities, but any other type of audit. If that's the case, what you are looking for is a Data Provider that stores the audit events into your SQLite database.
At the time being, there is no built-in data provider that stores to SQLite, and if there was one, it would store just the JSON representation of the event in one column (like the SQL/MySql providers). But it looks like you want to have a custom schema, so you will need to implement your own data provider.
Check the documentation here.
Here is a sample skeleton of a data provider:
public class SQLiteDataProvider : AuditDataProvider
{
public override object InsertEvent(AuditEvent auditEvent)
{
// Insert the event into SQLite and return its ID
}
public override void ReplaceEvent(object eventId, AuditEvent auditEvent)
{
// Replace the event given its ID (only used for CreationPolicies InsertOnStartReplaceOnEnd and Manual)
}
// async implementation:
public override async Task<object> InsertEventAsync(AuditEvent auditEvent)
{
// Asynchronously insert the event into SQLite and return its ID
}
public override async Task ReplaceEventAsync(object eventId, AuditEvent auditEvent)
{
// Asynchronously replace the event given its ID
}
}
Then you just set it up with:
Audit.Core.Configuration.Setup()
.UseCustomProvider(new SQLiteDataProvider());

How can I issue a SQL Statement against my database and have it create a collection?

I am using EF6 but I don't know if there is any difference between that and earlier versions.
I have the following context:
public IdentityContext()
: base("name=Identity")
{
Database.SetInitializer<IdentityContext>(null);
Configuration.ProxyCreationEnabled = false;
Configuration.LazyLoadingEnabled = false;
Configuration.ValidateOnSaveEnabled = false;
Configuration.AutoDetectChangesEnabled = true;
Database.Log = logInfo => Debug.WriteLine(logInfo);
}
public DbSet<AspNetRole> AspNetRoles { get; set; }
public DbSet<AspNetUserClaim> AspNetUserClaims { get; set; }
public DbSet<AspNetUserLogin> AspNetUserLogins { get; set; }
public DbSet<AspNetUser> AspNetUsers { get; set; }
What I would like to do is to issue a SQL query against this:
I saw one example (not related to my context) which is:
using (var context = new BloggingContext())
{
var blogs = context.Blogs.SqlQuery("SELECT * FROM dbo.Blogs").ToList();
}
But this does not make any sense to me as I cannot see why there is .Blogs after the context.
For my context above can someone explain how I can issue a simple sql statement to get the Id from the AspNetUser table and place these in a collection.
Note that I am using EF6 but in this case I am not sure that is relevant to the question if I query the context. My real needs are for a complex SQL but first I need some suggestion to get me started. Thanks.
If you're using .SqlQuery, you're kind of bypassing the beauty of using LINQ-style lambda expressions to query your database.
You might need to fiddle with the syntax a bit, but something like this should work for your initial requirement of getting user IDs:
var allMyIds = context.AspNetUserLogins.Select(l => l.Id).ToList();
or, if you prefer...
var allMyIds = (from l in context.AspNetUserLogins
select l.Id).ToList();
However, if you really need to execute SQL directly, you can do it on one of the DBSet classes as in your question (in which case the EF makes a valiant attempt at change tracking), or if it's a general query you just want to execute that returns primitive types and isn't specific to any one table, do it against the Database class using ExecuteSqlCommand:
var allMyIds = context.Database.ExecuteSqlCommand("select id from AspNetUserLogins");
Hope that helps!

return linq result via web service

I'm trying to use Linq in a webservice that returns a data from a query. As
the data return from Linq to Sql is IEnumerable and it's not possible to
easily get a Dataset.what is the best format to return data?
DataClassesDataContext Dac = new DataClassesDataContext();
Dac.Connection.ConnectionString = ConfigurationManager.ConnectionStrings["ConnectionCs"].ConnectionString;
var query = from record in Dac.RetrieveWorkshops(WorkshopCode, Name) select record;
Web-service proxies instantiate collections as arrays, so I've gotten into the habit of defining 'data transfer objects' and returning IEnumerable's of those DTO's. In your case, You could define a WorkshopDTO object which contained only the public properties you wanted to return and have your LINQ query construct them based on each record it reads. Then your client-proxy will see an array of WorkshopDTO's to manipulate.
You can populate your data into an object and return a list of that object in Web Service.
Example:
public class WorkshopEntity
{
string Name { get; set; }
string Location { get; set; }
}
List<WorkshopEntity> workshopList = (from record in Dac.RetrieveWorkshops(WorkshopCode, Name) select new WorkshopEntity { Name=record.name, Location=record.Location }).ToList();
return query.ToArray();
but remember to add [Serializable] Attribiute at top of your WorkshopEntity class.
if your have complex type properties in WorkshopEntity inform me because you should do more.

Resources