Error with GetItemQueryIterator when using a custom CosmosSerializer - azure-cosmosdb

I'm trying to make use of System.Text.Json serialization for a project I'm working on. When I make use of a custom CosmosSerializer and call GetItemQueryIterator, the ToStream call is being sent a Microsoft.Azure.Cosmos.SqlQuerySpec that cannot be serialized. Here is a sample that should easily reproduce the problem. Any and all help is appreciated!
using Microsoft.Azure.Cosmos;
using System;
using System.IO;
using System.Text.Json.Serialization;
using System.Threading.Tasks;
namespace CosmosTestApp
{
class Program
{
const string endpoint = "<REPLACE>";
const string key = "<REPLACE>";
const string dbName = "test";
const string containerName = "items";
public static async Task Main(string[] args)
{
var options = new CosmosClientOptions()
{
Serializer = new CosmosJsonSerializer()
};
var client = new CosmosClient(endpoint, key, options);
var dbResponse = await client.CreateDatabaseIfNotExistsAsync(dbName);
var db = dbResponse.Database;
var containerDef = new ContainerProperties(containerName, "/id");
var containerResposne = await db.CreateContainerIfNotExistsAsync(containerDef);
var testContainer = containerResposne.Container;
var testDoc = new TestDoc();
var docResponse = await testContainer.CreateItemAsync(testDoc, new PartitionKey(testDoc.Id));
Console.WriteLine($"Created document {docResponse.Resource.Id}");
var query = testContainer.GetItemQueryIterator<TestDoc>("SELECT * FROM c");
while (query.HasMoreResults)
{
var doc = await query.ReadNextAsync();
foreach(var x in doc.Resource)
{
Console.WriteLine($"Retrieved document {x.Id}");
}
}
}
}
internal class CosmosJsonSerializer : CosmosSerializer
{
public override T FromStream<T>(Stream stream)
{
using (var memoryStream = new MemoryStream())
{
stream.CopyTo(memoryStream);
stream.Close();
var item = JsonSerializer.Parse<T>(memoryStream.ToArray());
return item;
}
}
//This errors on the SqlQuerySpec
public override Stream ToStream<T>(T input)
=> new MemoryStream(JsonSerializer.ToUtf8Bytes(input));
}
internal class TestDoc
{
[JsonPropertyName("id")]
public string Id { get; set; } = "1";
public string TestString { get; set; } = "testing CosmosJsonSerializer";
}
}
EDIT: Bug has been filed and confirmed here: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/575

Related

Query Cosmos DB to get a list of different derived types using the .Net SDK Microsoft.Azure.Cosmos

We have an interface and a base class with multiple derived types.
public interface IEvent
{
[JsonProperty("id")]
public string Id { get; set; }
string Type { get; }
}
public abstract class EventBase: IEvent
{
public string Id { get; set; }
public abstract string Type { get; }
}
public class UserCreated : EventBase
{
public override string Type { get; } = typeof(UserCreated).AssemblyQualifiedName;
}
public class UserUpdated : EventBase
{
public override string Type { get; } = typeof(UserUpdated).AssemblyQualifiedName;
}
We are storing these events of different derived types in the same container in Cosmos DB using v3 of .Net SDK Microsoft.Azure.Cosmos. We then want to read all the events and have them deserialized to the correct type.
public class CosmosDbTests
{
[Fact]
public async Task TestFetchingDerivedTypes()
{
var endpoint = "";
var authKey = "";
var databaseId ="";
var containerId="";
var client = new CosmosClient(endpoint, authKey);
var container = client.GetContainer(databaseId, containerId);
await container.CreateItemAsync(new UserCreated{ Id = Guid.NewGuid().ToString() });
await container.CreateItemAsync(new UserUpdated{ Id = Guid.NewGuid().ToString() });
var queryable = container.GetItemLinqQueryable<IEvent>();
var query = queryable.ToFeedIterator();
var list = new List<IEvent>();
while (query.HasMoreResults)
{
list.AddRange(await query.ReadNextAsync());
}
Assert.NotEmpty(list);
}
}
Doesn't seem to be any option to tell GetItemLinqQueryable how to handle types. Is there any other method or approach to support multiple derived types in one query?
It's ok to put the events in some kind of wrapper entity if that would help, but they aren't allowed to be stored as an serialized sting inside a property.
The comment from Stephen Clearly pointed me in the right direction and with the help of this blog https://thomaslevesque.com/2019/10/15/handling-type-hierarchies-in-cosmos-db-part-2/ I ended up with a solution similar to the following example were we have a custom CosmosSerializer that uses a custom JsonConverter that reads the Type property.
public interface IEvent
{
[JsonProperty("id")]
public string Id { get; set; }
[JsonProperty("$type")]
string Type { get; }
}
public abstract class EventBase: IEvent
{
public string Id { get; set; }
public string Type => GetType().AssemblyQualifiedName;
}
public class UserCreated : EventBase
{
}
public class UserUpdated : EventBase
{
}
EventJsonConverter reads the Type property.
public class EventJsonConverter : JsonConverter
{
// This converter handles only deserialization, not serialization.
public override bool CanRead => true;
public override bool CanWrite => false;
public override bool CanConvert(Type objectType)
{
// Only if the target type is the abstract base class
return objectType == typeof(IEvent);
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
// First, just read the JSON as a JObject
var obj = JObject.Load(reader);
// Then look at the $type property:
var typeName = obj["$type"]?.Value<string>();
return typeName == null ? null : obj.ToObject(Type.GetType(typeName), serializer);
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotSupportedException("This converter handles only deserialization, not serialization.");
}
}
The NewtonsoftJsonCosmosSerializer takes a JsonSerializerSettings that it uses for serialization.
public class NewtonsoftJsonCosmosSerializer : CosmosSerializer
{
private static readonly Encoding DefaultEncoding = new UTF8Encoding(false, true);
private readonly JsonSerializer _serializer;
public NewtonsoftJsonCosmosSerializer(JsonSerializerSettings settings)
{
_serializer = JsonSerializer.Create(settings);
}
public override T FromStream<T>(Stream stream)
{
if (typeof(Stream).IsAssignableFrom(typeof(T)))
{
return (T)(object)stream;
}
using var sr = new StreamReader(stream);
using var jsonTextReader = new JsonTextReader(sr);
return _serializer.Deserialize<T>(jsonTextReader);
}
public override Stream ToStream<T>(T input)
{
var streamPayload = new MemoryStream();
using var streamWriter = new StreamWriter(streamPayload, encoding: DefaultEncoding, bufferSize: 1024, leaveOpen: true);
using JsonWriter writer = new JsonTextWriter(streamWriter);
writer.Formatting = _serializer.Formatting;
_serializer.Serialize(writer, input);
writer.Flush();
streamWriter.Flush();
streamPayload.Position = 0;
return streamPayload;
}
}
The CosmosClient is now created with our own NewtonsoftJsonCosmosSerializer and EventJsonConverter.
public class CosmosDbTests
{
[Fact]
public async Task TestFetchingDerivedTypes()
{
var endpoint = "";
var authKey = "";
var databaseId ="";
var containerId="";
var client = new CosmosClient(endpoint, authKey, new CosmosClientOptions
{
Serializer = new NewtonsoftJsonCosmosSerializer(new JsonSerializerSettings
{
Converters = { new EventJsonConverter() }
})
});
var container = client.GetContainer(databaseId, containerId);
await container.CreateItemAsync(new UserCreated{ Id = Guid.NewGuid().ToString() });
await container.CreateItemAsync(new UserUpdated{ Id = Guid.NewGuid().ToString() });
var queryable = container.GetItemLinqQueryable<IEvent>();
var query = queryable.ToFeedIterator();
var list = new List<IEvent>();
while (query.HasMoreResults)
{
list.AddRange(await query.ReadNextAsync());
}
Assert.NotEmpty(list);
}
}

beautify by serializing class to generate token in C#

I need to prepare some token make request over rest API. I already did this and works fine but as you guess its kinda look like very dirty way to handle this.
var dict = new Dictionary<string, string>();
dict.Add("Parameter1", tokenData.par1);
dict.Add("Parameter2", tokenData.par2);
using (HttpClient cliesssnt = new HttpClient())
{
cliesssnt.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var token =new FormUrlEncodedContent(dict);
HttpResponseMessage respossnse =
cliesssnt.PostAsync("https://someurl", token ).Result;
var result= respossnse.Content.ReadAsStringAsync().Result;
}
class:
public class ERaporAuthVM
{
[JsonProperty("Parameter1")]
public int par1 { get; set; }
[JsonProperty("Parameter2")]
public string par2 { get; set; }
}
I wish I could do something like this to preapre this token;
var token = JsonSerializer<ERaporAuthVM>(tokenData);
but how can I make it work this way?
public FormUrlEncodedContent JsonSerializer<T>(T input) where T : class
{
var result = new Dictionary<string, string>();
foreach(var prop in input.GetType().GetProperties())
{
var value = prop.GetValue(input);
//to-do: get name from JsonPropertyAttribute if exists
result.Add(prop.Name, value == null ? null : value.ToString());
}
return new FormUrlEncodedContent(result);
}
Usage:
var token = JsonSerializer(tokenData);

Use asyn method for synchronous Sqlite on createMethod

I'm using Xamarin, also my SQLite tables contain a large amount of data.
Because I want to avoid UIThread problems in OnCreate(), I need to perform database actions asynchronously.
I'm looking for guidance if I am handling this properly.
First method, which I found on the net:
protected override void OnCreate(Bundle savedInstanceState)
{
base.OnCreate(savedInstanceState);
SetContentView(Resource.Layout.InventoryPreviewMain);
Thread thread = new Thread(() =>
{
SQLiteConnection db = new SQLiteConnection(dpPath);
var table = db.Query<InventoryPreviewClass>("select * from InventoryPreviewClass where CategoryID =" + Connection.CategoryID + "");
mItems = new List<InventoryPreviewClass>();
foreach (var item in table)
{
mItems.Add(new InventoryPreviewClass() { InventoryItemID = item.InventoryItemID, InventoryItemName = item.InventoryItemName, InventoryItemPrice = item.InventoryItemPrice });
}
MyListViewAdapterInventory adapter = new MyListViewAdapterInventory(this, Resource.Layout.InventoryPreview, mItems);
mlistview.Adapter = adapter;
});
thread.Start();
Second Method, using async
public async void StartTimer()
{
SQLiteConnection db = new SQLiteConnection(dpPath);
var table = db.Query<InventoryPreviewClass>("select * from InventoryPreviewClass where CategoryID =" + Connection.CategoryID + "");
mItems = new List<InventoryPreviewClass>();
foreach (var item in table)
{
mItems.Add(new InventoryPreviewClass() { InventoryItemID = item.InventoryItemID, InventoryItemName = item.InventoryItemName, InventoryItemPrice = item.InventoryItemPrice });
}
MyListViewAdapterInventory adapter = new MyListViewAdapterInventory(this, Resource.Layout.InventoryPreview, mItems);
mlistview.Adapter = adapter;
await Task.Delay(500);
}
Which of two examples are more safe for keeping alive UIthread? Is there any other solution for making this?What is more reccomended to do?
Answer
Use the Cross-platform SQLite Library made by #FrankKruger to create/access SQLite databases for Xamarin mobile apps.
This library has a built-in asynchronous connection, so you'll never need to worry about accessing the database from the UI Thread again!
Xamarin.Android Example
"Second Method"
public async Task StartTimer()
{
mItems = await InventoryPreviewClassDatabase.GetAllInventoryPreviewClassAsync();
MyListViewAdapterInventory adapter = new MyListViewAdapterInventory(this, Resource.Layout.InventoryPreview, mItems);
mlistview.Adapter = adapter;
}
BaseDatabase Class
using System;
using System.Threading.Tasks;
using SQLite;
namespace SampleApp
{
public abstract class BaseDatabase
{
#region Constant Fields
static readonly Lazy<SQLiteAsyncConnection> _databaseConnectionHolder = new Lazy<SQLiteAsyncConnection>(() => GetDatabaseConnection());
#endregion
#region Fields
static bool _isInitialized;
#endregion
#region Properties
static SQLiteAsyncConnection DatabaseConnection => _databaseConnectionHolder.Value;
#endregion
#region Methods
protected static async Task<SQLiteAsyncConnection> GetDatabaseConnectionAsync()
{
if (!_isInitialized)
await Initialize().ConfigureAwait(false);
return DatabaseConnection;
}
static async Task Initialize()
{
await DatabaseConnection.CreateTableAsync<InventoryPreviewClass>().ConfigureAwait(false);
_isInitialized = true;
}
SQLiteAsyncConnection GetDatabaseConnection()
{
var sqliteFilename = "YourDatabaseName.db3";
string documentsPath = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal); // Documents folder
var path = Path.Combine(documentsPath, sqliteFilename);
var conn = new SQLiteAsyncConnection(path, SQLiteOpenFlags.ReadWrite | SQLiteOpenFlags.Create | SQLiteOpenFlags.SharedCache);
return conn;
}
#endregion
}
}
Parent Database Class
using System.Linq;
using System.Threading.Tasks;
using System.Collections.Generic;
namespace SampleApp
{
public abstract class InventoryPreviewClassDatabase : BaseDatabase
{
#region Methods
public static async Task<IList<InventoryPreviewClass>> GetAllInventoryPreviewClassAsync()
{
var databaseConnection = await GetDatabaseConnectionAsync().ConfigureAwait(false);
return await databaseConnection.Table<InventoryPreviewClass>().ToListAsync().ConfigureAwait(false);
}
public static async Task<InventoryPreviewClass> GetInventoryPreviewClassByIDAsync(int id)
{
var databaseConnection = await GetDatabaseConnectionAsync().ConfigureAwait(false);
return await databaseConnection.Table<InventoryPreviewClass>().Where(x => x.ID.Equals(id)).FirstOrDefaultAsync().ConfigureAwait(false);
}
public static async Task<int> SaveInventoryPreviewClassAsync(InventoryPreviewClass inventoryPreview)
{
var databaseConnection = await GetDatabaseConnectionAsync().ConfigureAwait(false);
var isObjectInDatabase = await GetInventoryPreviewClassByIDAsync(inventoryPreview.ID).ConfigureAwait(false) != null;
if (isObjectInDatabase)
return await databaseConnection.UpdateAsync(inventoryPreview).ConfigureAwait(false);
return await databaseConnection.InsertAsync(inventoryPreview).ConfigureAwait(false);
}
public static async Task<int> DeleteItemAsync(OpportunityModel opportunity)
{
var databaseConnection = await GetDatabaseConnectionAsync().ConfigureAwait(false);
return await databaseConnection.DeleteAsync(opportunity).ConfigureAwait(false);
}
public static async Task<int> GetNumberOfRowsAsync()
{
var databaseConnection = await GetDatabaseConnectionAsync().ConfigureAwait(false);
return await databaseConnection.Table<InventoryPreviewClass>().CountAsync().ConfigureAwait(false);
}
#endregion
}
}
This code was inspired from this Xamarin.Forms sample app

ServiceStack RSS serialisation issue

I'm trying to create an RSS feed for a ServiceStack Service. I've followed various examples as closely as I can. My problem is that I get no output and I am not sure how to troubleshoot the issue. I suspect I have done something wrong on the serialisation. Here is (a simplified version of) what I have
My DTO's are
using System.Collections.Generic;
using ServiceStack;
using Library;
[Route("/MyCollection/Tomorrow/{ID}", "GET, POST")]
[Api("MyCollections Delivery")]
public class MyCollectionTomorrow
: IReturn<MyCollectionTomorrowResponse>
{
public long ID { get; set; }
}
public class MyCollectionTomorrowResponse : IHasResponseStatus
{
public long ID { get; set; }
public List<MyCollection> Result { get; set; }
public ResponseStatus ResponseStatus { get; set; }
}
public class MyCollection
{
public string Description { get; set; }
public string MyCollectionDayOfWeek { get; set; }
public DateTime MyCollectionDate { get; set; }
public bool Assisted { get; set; }
public string RoundType { get; set; }
public string Description { get; set; }
}
My service is
using System;
using Library;
using ServiceStack;
using ServiceStack.Configuration;
using System;
using Library;
using ServiceStack;
using ServiceStack.Configuration;
using MyCollection.Tomorrow;
using MyCollections.Tomorrow;
public class MyCollectionTomorrowService : Service
{
public object Any(WasteCollectionTomorrow request)
{
int id;
var param = new CollectionTomorrow();
param.ID = ID;
var response = client.Get<CollectionTomorrowResponse>(param);
return response;
}
catch (Exception ex)
{
var response = new CollectionTomorrowResponse();
response.Result = null
var status = new ResponseStatus { Message = ex.Message, StackTrace = ex.StackTrace };
response.ResponseStatus = status;
return response;
}
}
}
and my media type is
namespace DataFeedServices
{
using System;
using System.IO;
using System.ServiceModel.Syndication;
using System.Text;
using System.Xml;
using ServiceStack;
using ServiceStack.Data;
using ServiceStack.Web;
using MyCollections.Tomorrow;
public class RssFormat
{
private const string RssContentType = "application/rss+xml";
public static void Register(IAppHost appHost)
{
appHost.ContentTypes.Register(RssContentType, SerializeToStream, DeserializeFromStream);
}
public static void SerializeToStream(IRequest req, object response, Stream stream)
{
StreamWriter sw = null;
try
{
var syndicationFeedResponse = response as MyCollectionResponse;
sw = new StreamWriter(stream);
if (response != null)
{
WriteRssCollectionFeed(sw, syndicationFeedResponse);
}
}
finally
{
if (sw != null)
{
sw.Dispose();
}
}
}
public static void WriteRssCollectionFeed(StreamWriter sw, MyCollectionResponse Mycollections)
{
const string Baseuri = "example.com";
try
{
var uri = new Uri(Baseuri);
var syndicationFeed = new SyndicationFeed(
"MyCollection Service",
"Mycollections " ,
uri);
syndicationFeed.Authors.Add(new SyndicationPerson("email#mysite.com"));
if (Mycollections.Result != null)
{
foreach (var cats in Mycollections.Result)
{
syndicationFeed.Categories.Add(new SyndicationCategory(cats.RoundID));
}
}
syndicationFeed.Generator = "MyApp";
syndicationFeed.Copyright = new TextSyndicationContent("Copyright 2015");
syndicationFeed.LastUpdatedTime = DateTime.Now;
if (Mycollections.Result != null)
{
// set items
foreach (var coll in Mycollections.Result)
{
var item = new SyndicationItem { Title = new TextSyndicationContent(coll.CollectionDate) };
item.Links.Add(new SyndicationLink(uri));
item.Authors.Add(new SyndicationPerson("email#mysite.com"));
var itemContent = new StringBuilder();
itemContent.Append("My Item content");
item.Content = new TextSyndicationContent(
itemContent.ToString(),
TextSyndicationContentKind.Plaintext);
}
}
Rss20FeedFormatter rssFeed = syndicationFeed.GetRss20Formatter();
var xwriter = XmlWriter.Create(sw);
rssFeed.WriteTo(xwriter);
}
catch (Exception)
{
throw new Exception("Something bad happened");
}
}
public static object DeserializeFromStream(Type type, Stream stream)
{
throw new NotImplementedException();
}
}
}
Since your ContentType is not reusable and coupled to a specific MyCollectionResponse, it's easier to just return a raw string with the RSS XML:
[AddHeader(ContentType = "application/rss+xml")]
public object Any(WasteCollectionTomorrow request)
{
//..
return rssXml;
}
You can also write it directly to the Response Output Stream with something like:
public object Any(WasteCollectionTomorrow request)
{
//..
base.Response.ContentType = "application/rss+xml";
RssFormat.SerializeToStream(response, Response.OutputStream);
base.Response.EndRequest();
return null;
}

How do you mock ServiceStack ISession using Moq and StructureMap?

I'm using ServiceStack / StructureMap / Moq. The service makes a call to Session, which is type ServiceStack.CacheAccess.ISession. For unit tests, I created a Mock object using Moq, and added it to the StructureMap configuration:
protected Mock<ISession> sessionMock = new Mock<ISession>();
ObjectFactory.Configure(
cfg =>
{
cfg.For<ISession>().Use(sessionMock.Object);
However, I was not surprised when the Session object was null -- I'm pretty sure I'm leaving out a step. What else do I need to do to fill my Session property with a mock object?
[EDIT] Here's a simple test scenario
Code to test. Simple request / service
[Route("getKey/{key}")]
public class MyRequest:IReturn<string>
{
public string Key { get; set; }
}
public class MyService:Service
{
public string Get(MyRequest request)
{
return (string) Session[request.Key];
}
}
The base test class and MockSession classes
// test base class
public abstract class MyTestBase : TestBase
{
protected IRestClient Client { get; set; }
protected override void Configure(Container container)
{
// this code is never reached under any of my scenarios below
container.Adapter = new StructureMapContainerAdapter();
ObjectFactory.Initialize(
cfg =>
{
cfg.For<ISession>().Singleton().Use<MockSession>();
});
}
}
public class MockSession : ISession
{
private Dictionary<string, object> m_SessionStorage = new Dictionary<string, object>();
public void Set<T>(string key, T value)
{
m_SessionStorage[key] = value;
}
public T Get<T>(string key)
{
return (T)m_SessionStorage[key];
}
public object this[string key]
{
get { return m_SessionStorage[key]; }
set { m_SessionStorage[key] = value; }
}
}
And tests. See comments for where I'm seeing the failure. I didn't really expect versions 1 & 2 to work, but hoped version 3 would.
[TestFixture]
public class When_getting_a_session_value:MyTestBase
{
[Test]
public void Test_version_1()
{
var session = ObjectFactory.GetInstance<MockSession>();
session["key1"] = "Test";
var request = new MyRequest {Key = "key1"};
var client = new MyService(); // generally works fine, except for things like Session
var result = client.Get(request); // throws NRE inside MyService
result.ShouldEqual("Test");
}
[Test]
public void Test_version_2()
{
var session = ObjectFactory.GetInstance<MockSession>();
session["key1"] = "Test";
var request = new MyRequest {Key = "key1"};
var client = ObjectFactory.GetInstance<MyService>();
var result = client.Get(request); // throws NRE inside MyService
result.ShouldEqual("Test");
}
[Test]
public void Test_version_3()
{
var session = ObjectFactory.GetInstance<MockSession>();
session["key1"] = "Test";
var request = new MyRequest {Key = "key1"};
var client = CreateNewRestClient();
var result = client.Get(request); // throws NotImplementedException here
result.ShouldEqual("Test");
}
}
It looks like you're trying to create unit tests, but you're using an AppHost like you wound an Integration test. See this previous answer for differences between the two and docs on Testing.
You can mock the Session by registering an instance in Request.Items[Keywords.Session], e.g:
[Test]
public void Can_mock_IntegrationTest_Session_with_Request()
{
using var appHost = new BasicAppHost(typeof(MyService).Assembly).Init();
var req = new MockHttpRequest();
req.Items[Keywords.Session] = new AuthUserSession {
UserName = "Mocked"
};
using var service = HostContext.ResolveService<MyService>(req);
Assert.That(service.GetSession().UserName, Is.EqualTo("Mocked"));
}
Otherwise if you set AppHost.TestMode=true ServiceStack will return the IAuthSession that's registered in your IOC, e.g:
[Test]
public void Can_mock_UnitTest_Session_with_IOC()
{
using var appHost = new BasicAppHost
{
TestMode = true,
ConfigureContainer = container =>
{
container.Register<IAuthSession>(c => new AuthUserSession {
UserName = "Mocked",
});
}
}.Init();
var service = new MyService {
Request = new MockHttpRequest()
};
Assert.That(service.GetSession().UserName, Is.EqualTo("Mocked"));
}

Resources