Read File and Return Synchronously (Metro App) - asynchronous

I am writing a Metro App.
I am trying to read a file and return a float[] from the data. But no matter what I do, the function seems to return null. I have tried the solutions to similar questions to no luck.
For example if I use:
float[] floatArray = new ModelReader("filename.txt").ReadModel()
The result will be a null array.
However if I use:
new ModelReader("filename.txt")
The correct array will be printed to the console because "Test" also prints the array before returning it. This seems very weird to me.
Please give me some guidance, I have no idea what is wrong.
public class ModelReader
{
float[] array;
public ModelReader(String name)
{
ReadModelAsync(name);
}
public float[] ReadModel()
{
return array;
}
private async Task ReadModelAsync(String name)
{
await readFile(name);
}
async Task readFile(String name)
{
// settings
var path = #"Assets\models\" + name;
var folder = Windows.ApplicationModel.Package.Current.InstalledLocation;
// acquire file
var file = await folder.GetFileAsync(path);
// read content
var read = await Windows.Storage.FileIO.ReadTextAsync(file);
using (StringReader sr = new StringReader(read))
{
Test test = new Test(getFloatArray(sr));
this.array = test.printArray();
}
}
private float[] getFloatArray(StringReader sr) { ... }
public class Test
{
public float[] floatArray;
public Test(float[] floatArray)
{
this.floatArray = floatArray;
}
public float[] printArray()
{
for (int i = 0; i < floatArray.Length; i++)
{
Debug.WriteLine(floatArray[i]);
}
return floatArray;
}
}

You're trying to get the result of an asynchronous operation before it has completed. I recommend you read my intro to async / await and follow-up with the async / await FAQ.
In particular, your constructor:
public ModelReader(String name)
{
ReadModelAsync(name);
}
is returning before ReadModelAsync is complete. Since constructors cannot be asynchronous, I recommend you use an asynchronous factory or asynchronous lazy initialization as described on my blog (also available in my AsyncEx library).
Here's a simple example using an asynchronous factory approach:
public class ModelReader
{
float[] array;
private ModelReader()
{
}
public static async Task<ModelReader> Create(string name)
{
var ret = new ModelReader();
await ret.ReadModelAsync(name);
return ret;
}
...
}

Related

How to implement asynchronous data streaming in .Net Core Service Bus triggered Azure Function processing huge data not to get OutOfMemoryException?

I have a service bus triggered Azure Function which listens for messages containing just blob URL strings of JSON data which each one of them is at least 10MB.
Message queue is near real-time(If I use the correct term) so producers keep putting messaging to the queue with a frequency so there is always data in the queue to be processed.
I have designed a solution but it gets OutOfMemoryException most of the time. The steps involved in the current solution sequentially are:
Consume a message
Download the file from the URL within the consumed message to a temporary folder
Read the whole file as a string
Deserialize it to an object
Partition into the chunks to supply Mongo bulk upsert limit
Bulk upsert to Mongo
I have tried to solve OutOfMemoryException and I thought that it's because my function/consumer don't have the same pace with the producer, so I think that at the time t1 when it gets the first message and process it and then while it's upserting to the mongo the function keeps getting the messages and they accumulate in the memory and waiting to be upserted.
Is my reasoning right?
Thus I think that If I could implement a streaming solution starting from #3, reading from file by chunking and putting it to a stream then I would prevent the memory keep growing and reduce time also. I have mostly Java background and I somehow know that with custom iterator/spliterator/iterable it is possible to do streaming and asynchronous processing.
How can I do asynchronous data streaming with .Net Core in an Azure Function?
Are there other approaches to solve this problem?
namespace x.y.Z
{
public class MyFunction
{
//...
[FunctionName("my-func")]
public async Task Run([ServiceBusTrigger("my-topic", "my-subscription", Connection = "AzureServiceBus")] string message, ILogger log, ExecutionContext context)
{
var data = new PredictionMessage();
try
{
data = myPredictionService.genericDeserialize(message);
await myPredictionService.ValidateAsync(data);
await myPredictionService.AddAsync(data);
}
catch (Exception ex)
{
//...
}
}
}
}
public class PredictionMessage
{
public string BlobURL { get; set; }
}
namespace x.y.z.Prediction
{
public abstract class BasePredictionService<T> : IBasePredictionService<T> where T : PredictionMessage, new()
{
protected readonly ILogger log;
private static JsonSerializer serializer;
public BasePredictionService(ILogger<BasePredictionService<T>> log)
{
this.log = log;
serializer = new JsonSerializer();
}
public async Task ValidateAsync(T message)
{
//...
}
public T genericDeserialize(string message)
{
return JsonConvert.DeserializeObject<T>(message);
}
public virtual Task AddAsync(T message)
{
throw new System.NotImplementedException();
}
public async Task<string> SerializePredictionResult(T message)
{
var result = string.Empty;
using (WebClient client = new WebClient())
{
var tempPath = Path.Combine(Path.GetTempPath(), DateTime.Now.Ticks + ".json");
Uri srcPath = new Uri(message.BlobURL);
await client.DownloadFileTaskAsync(srcPath, tempPath);
using (FileStream fs = File.Open(tempPath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
result = sr.ReadToEnd();
}
}
Task.Run(() =>
{
File.Delete(tempPath);
});
return result;
}
}
protected TType StreamDataDeserialize<TType>(string streamResult)
{
var body = default(TType);
using (MemoryStream stream = new MemoryStream(Encoding.Default.GetBytes(streamResult)))
{
using (StreamReader streamReader = new StreamReader(stream))
{
body = (TType)serializer.Deserialize(streamReader, typeof(TType));
}
}
return body;
}
protected List<List<TType>> Split<TType>(List<TType> list, int chunkSize = 1000)
{
List<List<TType>> retVal = new List<List<TType>>();
while (list.Count > 0)
{
int count = list.Count > chunkSize ? chunkSize : list.Count;
retVal.Add(list.GetRange(0, count));
list.RemoveRange(0, count);
}
return retVal;
}
}
}
namespace x.y.z.Prediction
{
public class MyPredictionService : BasePredictionService<PredictionMessage>, IMyPredictionService
{
private readonly IMongoDBRepository<MyPrediction> repository;
public MyPredictionService(IMongoDBRepoFactory mongoDBRepoFactory, ILogger<MyPredictionService> log) : base(log)
{
repository = mongoDBRepoFactory.GetRepo<MyPrediction>();
}
public override async Task AddAsync(PredictionMessage message)
{
string streamResult = await base.SerializePredictionResult(message);
var body = base.StreamDataDeserialize<List<MyPrediction>>(streamResult);
if (body != null && body.Count > 0)
{
var chunkList = base.Split(body);
await BulkUpsertProcess(chunkList);
}
}
private async Task BulkUpsertProcess(List<List<MyPrediction>> chunkList)
{
foreach (var perChunk in chunkList)
{
var filterContainers = new List<IDictionary<string, object>>();
var updateContainer = new List<IDictionary<string, object>>();
foreach (var item in perChunk)
{
var filter = new Dictionary<string, object>();
var update = new Dictionary<string, object>();
filter.Add(/*...*/);
filterContainers.Add(filter);
update.Add(/*...*/);
updateContainer.Add(update);
}
await Task.Run(async () =>
{
await repository.BulkUpsertAsync(filterContainers, updateContainer);
});
}
}
}
}

Async Vala Example

In the Book "Introduction to Vala" by Dr Michael Lauer, he has mentioned that the lib Soup async api is broken. I'm struggling to write a simple example using session.queue_message that query radio stations using the service from radio-browser. Here is my code. I would appreciate any help form experienced Programmers like "Al Thomas". Thank you.
public class Station : Object {
// A globally unique identifier for the change of the station information
public string changeuuid { get; set; default = ""; }
// A globally unique identifier for the station
public string stationuuid { get; set; default = ""; }
// The name of the station
public string name { get; set; default = ""; }
// The stream URL provided by the user
public string url { get; set; default = ""; }
// and so on ... many properties
public string to_string () {
var builder = new StringBuilder ();
builder.append_printf ("\nchangeuuid = %s\n", changeuuid);
builder.append_printf ("stationuuid = %s\n", stationuuid);
builder.append_printf ("name = %s\n", name);
builder.append_printf ("url = %s\n", url);
return (owned) builder.str;
}
}
public class RadioBrowser : Object {
private static Soup.Session session;
// private static MainLoop main_loop;
public const string API_URL = "https://de1.api.radio-browser.info/json/stations";
public const string USER_AGENT = "github.aeldemery.radiolibrary";
public RadioBrowser (string user_agent = USER_AGENT, uint timeout = 50)
requires (timeout > 0)
{
Intl.setlocale ();
session = new Soup.Session ();
session.timeout = timeout;
session.user_agent = user_agent;
session.use_thread_context = true;
// main_loop = new MainLoop ();
}
private void check_response_status (Soup.Message msg) {
if (msg.status_code != 200) {
var str = "Error: Status message error %s.".printf (msg.reason_phrase);
error (str);
}
}
public Gee.ArrayList<Station> listStations () {
var stations = new Gee.ArrayList<Station> ();
var data_list = Datalist<string> ();
data_list.set_data ("limit", "100");
var parser = new Json.Parser ();
parser.array_element.connect ((pars, array, index) => {
var station = Json.gobject_deserialize (typeof (Station), array.get_element (index)) as Station;
assert_nonnull (station);
stations.add (station);
});
var msg = Soup.Form.request_new_from_datalist (
"POST",
API_URL,
data_list
);
// send_message works but not queue_message
// session.send_message (msg);
session.queue_message (msg, (sess, mess) => {
check_response_status (msg);
try {
parser.load_from_data ((string) msg.response_body.flatten ().data);
} catch (Error e) {
error ("Failed to parse data, error:" + e.message);
}
});
return stations;
}
}
int main (string[] args) {
var radio_browser = new RadioBrowser ();
var stations = radio_browser.listStations ();
assert_nonnull (stations);
foreach (var station in stations) {
print (station.to_string ());
}
return 0;
}
While I'm not Al Thomas, I still might be able to help. ;)
For async calls to work in there needs to be a main loop running, and typically from the main program thread. Thus you want to create and execute the main loop from your main() function, rather than in your application code:
int main (string[] args) {
var loop = new GLib.MainLoop ();
var radio_browser = new RadioBrowser ();
// set up async calls here
// then set the main loop running as the last thing
loop.run();
}
Also, if you want to wait for an async call to complete, you typically need to make the call using the yield keyword from another async function E.g:
public async Gee.ArrayList<Station> listStations () {
…
// When this call is made, execution of listStations() will be
// suspended until the soup response is received
yield session.send_async(msg);
// Execution then resumes normally
check_response_status (msg);
parser.load_from_data ((string) msg.response_body.flatten ().data);
…
return stations;
}
You can then call this from the (non-async) main function using the listStations.begin(…) notation:
int main (string[] args) {
var loop = new GLib.MainLoop ();
var radio_browser = new RadioBrowser ();
radio_browser.listStations.begin((obj, res) => {
var stations = radio_browser.listStations.end(res);
…
loop.quit();
});
loop.run();
}
As further reading, I would recommend the async section of the Vala Tutorial, and the asyc examples on the wiki as well.

Net Framework Xunit With Moq Unit Testing Keep Calling The Original Function

I have a problem with mocking cause it keep calling the original function. This is my demo code
First file is interface that contains the function that I want to mock.
public interface IDemoReplace
{
int FunctionToBeReplaced();
}
Second file is a class that actually has the implementation for the function
public class DemoReplace : IDemoReplace
{
public int FunctionToBeReplaced()
{
//this function contains sql query in my real project
return 1;
}
}
Third file is a class that I want to test
public class ClassToBeTested
{
public int TestThisFunction()
{
IDemoReplace replace = new DemoReplace();
var temp = replace.FunctionToBeReplaced();
return temp;
}
}
Last file is the test class
public class TestClass
{
[Fact]
public void TryTest()
{
using (var mock = AutoMock.GetLoose()) {
//Arrange
mock.Mock<IDemoReplace>()
.Setup(x => x.FunctionToBeReplaced())
.Returns(returnTwo());
var classToBeTested = mock.Create<ClassToBeTested>();
var expected = 2;
//Act
var actual = classToBeTested.TestThisFunction();
//Assert
Assert.Equal(expected, actual);
}
}
public int returnTwo() {
return 2;
}
}
This test will be failed with expected is 2 and actual is 1. When I tried to debug it doesn't call returnTwo but call the original function instead.
I am new to unit testing so what did I miss? Please be considered that the code above is only a demo of what is happened in my actual project. FunctionToBeReplaced is actually a function that execute and return record from database so I want to mock that function.
Thanks :)
This is a design issue. The subject under test is tight coupled to implementation concerns that make it difficult to isolation the subject so that it can be unit tested.
It (subject) is manually creating its dependency
IDemoReplace replace = new DemoReplace();
Ideally you want to explicitly inject dependencies. Those dependencies should also be abstractions and not concretions.
public class ClassToBeTested {
private readonly IDemoReplace dependency;
public ClassToBeTested(IDemoReplace dependency) {
this.dependency = dependency;
}
public int TestThisFunction() { ;
var temp = dependency.FunctionToBeReplaced();
return temp;
}
}
At run time, the implementation (or mock) can be injected, either purely, or via a container.
The test in the original example shown should now behave as expected.
public class TestClass {
[Fact]
public void TryTest() {
using (var mock = AutoMock.GetLoose()) {
//Arrange
var expected = returnTwo();
mock.Mock<IDemoReplace>()
.Setup(x => x.FunctionToBeReplaced())
.Returns(expected);
var classToBeTested = mock.Create<ClassToBeTested>();
//Act
var actual = classToBeTested.TestThisFunction();
//Assert
Assert.Equal(expected, actual);
}
}
public int returnTwo() {
return 2;
}
}

'Server side events' send with the ASP Web Api do not arrive?

I created a test source which should send a message to the client every x time. This is the ApiController:
public class TestSourceController : ApiController
{
private static readonly ConcurrentQueue<StreamWriter> ConnectedClients = new ConcurrentQueue<StreamWriter>();
[AllowAnonymous]
[Route("api/sources/test")]
public HttpResponseMessage Get()
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent((Action<Stream, HttpContent, TransportContext>) OnStreamAvailable,
"text/event-stream");
return response;
}
private static void OnStreamAvailable(Stream stream, HttpContent headers, TransportContext context)
{
var clientStream = new StreamWriter(stream);
ConnectedClients.Enqueue(clientStream);
}
private static void DoThings()
{
const string outboundMessage = "Test";
foreach (var clientStream in ConnectedClients)
{
clientStream.WriteLine("data:" + JsonConvert.SerializeObject(outboundMessage));
clientStream.Flush();
}
}
}
The clientStream.Flush(); is called like expected and without exceptions.
I handle it in AngularJS like this:
$scope.handleServerCallback = function (data) {
console.log(data);
$scope.$apply(function() {
$scope.serverData = data;
});
};
$scope.listen = function () {
$scope.eventSource = new window.EventSource("http://localhost:18270/api/sources/test");
$scope.eventSource.onmessage = $scope.handleServerCallback;
$scope.eventSource.onopen = function() { console.log("Opened source"); };
$scope.eventSource.onerror = function (e) { console.error(e); };
};
$scope.listen();
My guess is it's a problem with the server since I can see the "EventStream" from the test call is empty in the chrome debugger.
Does anyone know how to make sure the messages arrive at the client?
The solution was quite easy, according to the spec every line has to end with "\n" and the very last line with "\n\n".
So:
clientStream.WriteLine("data:" + JsonConvert.SerializeObject(outboundMessage) + "\n\n");
Solves it.

How can i use engine object in my console application

"How can i use engine in my console application"
I shouldn't use the ITemplate-interface and Transform-Method.
I am using Tridion 2011
Could anyone please suggest me.
You can't. The Engine class is part of the TOM.NET and that API is explicitly reserved for use in:
Template Building Blocks
Event Handlers
For all other cases (such as console applications) you should use the Core Service.
There are many good questions (and articles on other web sites) already:
https://stackoverflow.com/search?q=%5Btridion%5D+core+service
http://www.google.com/#q=tridion+core+service
If you get stuck along the way, show us the relevant code+configuration you have and what error message your get (or at what step you are stuck) and we'll try to help from there.
From a console application you should use the Core Service. I wrote a small example using the Core Service to search for items in the content manager.
Console.WriteLine("FullTextQuery:");
var fullTextQuery = Console.ReadLine();
if (String.IsNullOrWhiteSpace(fullTextQuery) || fullTextQuery.Equals(":q", StringComparison.OrdinalIgnoreCase))
{
break;
}
Console.WriteLine("SearchIn IdRef:");
var searchInIdRef = Console.ReadLine();
var queryData = new SearchQueryData
{
FullTextQuery = fullTextQuery,
SearchIn = new LinkToIdentifiableObjectData
{
IdRef = searchInIdRef
}
};
var results = coreServiceClient.GetSearchResults(queryData);
results.ToList().ForEach(result => Console.WriteLine("{0} ({1})", result.Title, result.Id));
Add a reference to Tridion.ContentManager.CoreService.Client to your Visual Studio Project.
Code of the Core Service Client Provider:
public interface ICoreServiceProvider
{
CoreServiceClient GetCoreServiceClient();
}
public class CoreServiceDefaultProvider : ICoreServiceProvider
{
private CoreServiceClient _client;
public CoreServiceClient GetCoreServiceClient()
{
return _client ?? (_client = new CoreServiceClient());
}
}
And the client itself:
public class CoreServiceClient : IDisposable
{
public SessionAwareCoreServiceClient ProxyClient;
private const string DefaultEndpointName = "netTcp_2011";
public CoreServiceClient(string endPointName)
{
if(string.IsNullOrWhiteSpace(endPointName))
{
throw new ArgumentNullException("endPointName", "EndPointName is not specified.");
}
ProxyClient = new SessionAwareCoreServiceClient(endPointName);
}
public CoreServiceClient() : this(DefaultEndpointName) { }
public string GetApiVersionNumber()
{
return ProxyClient.GetApiVersion();
}
public IdentifiableObjectData[] GetSearchResults(SearchQueryData filter)
{
return ProxyClient.GetSearchResults(filter);
}
public IdentifiableObjectData Read(string id)
{
return ProxyClient.Read(id, new ReadOptions());
}
public ApplicationData ReadApplicationData(string subjectId, string applicationId)
{
return ProxyClient.ReadApplicationData(subjectId, applicationId);
}
public void Dispose()
{
if (ProxyClient.State == CommunicationState.Faulted)
{
ProxyClient.Abort();
}
else
{
ProxyClient.Close();
}
}
}
When you want to perform CRUD actions through the core service you can implement the following methods in the client:
public IdentifiableObjectData CreateItem(IdentifiableObjectData data)
{
data = ProxyClient.Create(data, new ReadOptions());
return data;
}
public IdentifiableObjectData UpdateItem(IdentifiableObjectData data)
{
data = ProxyClient.Update(data, new ReadOptions());
return data;
}
public IdentifiableObjectData ReadItem(string id)
{
return ProxyClient.Read(id, new ReadOptions());
}
To construct a data object of e.g. a Component you can implement a Component Builder class that implements a create method that does this for you:
public ComponentData Create(string folderUri, string title, string content)
{
var data = new ComponentData()
{
Id = "tcm:0-0-0",
Title = title,
Content = content,
LocationInfo = new LocationInfo()
};
data.LocationInfo.OrganizationalItem = new LinkToOrganizationalItemData
{
IdRef = folderUri
};
using (CoreServiceClient client = provider.GetCoreServiceClient())
{
data = (ComponentData)client.CreateItem(data);
}
return data;
}
Hope this gets you started.

Resources