We are trying to load and resume workflows which have a delay. I have seen the Microsoft sample of Absolute Delay for this using store.WaitForEvents and LoadRunnableInstance to load the workflow. However here the workflow is already known.
In our case we want to have an event waiting for the store.WaitForEvents after every say 5 seconds to check if there is a runnable instance and if so only load and run that /those particular instances. Is there a way I could know which workflow instance is ready.
We are maintaing the workflow id and the xaml associated to it in our database, so if we could know the workflow instance id we could get the xaml mapped to it, create the workflow and then do a LOadRunnableInstance on it.
Any help would be greatly appreciated.
Microsoft sample (Absolute Delay)
public void Run(){
wfHostTypeName = XName.Get("Version" + Guid.NewGuid().ToString(),
typeof(WorkflowWithDelay).FullName);
this.instanceStore = SetupSqlpersistenceStore();
this.instanceHandle =
CreateInstanceStoreOwnerHandle(instanceStore, wfHostTypeName);
WorkflowApplication wfApp = CreateWorkflowApp();
wfApp.Run();
while (true)
{
this.waitHandler.WaitOne();
if (completed)
{
break;
}
WaitForRunnableInstance(this.instanceHandle);
wfApp = CreateWorkflowApp();
try
{
wfApp.LoadRunnableInstance();
waitHandler.Reset();
wfApp.Run();
}
catch (InstanceNotReadyException)
{
Console.WriteLine("Handled expected InstanceNotReadyException, retrying...");
}
}
Console.WriteLine("workflow completed.");
}
public void WaitForRunnableInstance(InstanceHandle handle)
{
var events=instanceStore.WaitForEvents(handle, TimeSpan.MaxValue);
bool foundRunnable = false;
foreach (var persistenceEvent in events)
{
if (persistenceEvent.Equals(HasRunnableWorkflowEvent.Value))
{
foundRunnable = true;
break;
}
}
if (!foundRunnable) {
Console.WriteLine("no runnable instance");
}
}
Thanks
Anamika
I had a similar problem with durable delay activities and WorkflowApplicationHost. Ended up creating my own 'Delay' activity that worked essentially the same way as the one out of the box, (takes an arg that describes when to resume the workflow, and then bookmarks itself). Instead of saving delay info in the SqlInstanceStore though, my Delay Activity created a record in a seperate db. (similar to the one you are using to track the Workflow Ids and Xaml). I then wrote a simple service that polled that DB for expired delays and initiated a resume of the necessary workflow.
Oh, and the Delay activity deleted it's record from that DB on bookmark resume.
HTH
I'd suggest having a separate SqlPersistenceStore for each workflow definition you're hosting.
Related
I am trying to consume a max of 1000 messages from kafka at a time. (I am doing this because i need to batch insert into MSSQL.) I was under the impression that kafka keeps an internal queue which fetches messages from the brokers and when i use the consumer.consume() method it just checks if there are any messages in the internal queue and returns if it finds something. otherwise it just blocks until the internal queue is updated or until timeout.
I tried to use the solution suggested here: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1164#issuecomment-610308425
but when i specify TimeSpan.Zero (or any other timespan up to 1000ms) the consumer never consumes any messages. but if i remove the timeout it does consume messages but then i am unable to exit the loop if there are no more messages left to be read.
I also saw an other question on stackoverflow which suggested to read the offset of the last message sent to kafka and then read messages until i reach that offset and then break from the loop. but currently i only have one consumer and 6 partitions for a topic. I haven't tried it yet but i think managing offsets for each of the partition might make the code messy.
Can someone please tell me what to do?
static List<RealTime> getBatch()
{
var config = new ConsumerConfig
{
BootstrapServers = ConfigurationManager.AppSettings["BootstrapServers"],
GroupId = ConfigurationManager.AppSettings["ConsumerGroupID"],
AutoOffsetReset = AutoOffsetReset.Earliest,
};
List<RealTime> results = new List<RealTime>();
List<string> malformedJson = new List<string>();
using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
consumer.Subscribe("RealTimeTopic");
int count = 0;
while (count < batchSize)
{
var consumerResult = consumer.Consume(1000);
if (consumerResult?.Message is null)
{
break;
}
Console.WriteLine("read");
try
{
RealTime item = JsonSerializer.Deserialize<RealTime>(consumerResult.Message.Value);
results.Add(item);
count += 1;
}
catch(Exception e)
{
Console.WriteLine("malformed");
malformedJson.Add(consumerResult.Message.Value);
}
}
consumer.Close();
};
Console.WriteLine(malformedJson.Count);
return results;
}
I found a workaround.
For some reason the consumer first needs to be called without a timeout. That means it will wait for a message until it gets at least one. after that using consume with timeout zero fetches all the rest of the messages one by one from the internal queue. this seems to work out for the best.
I had a similar problem, updating the Confluent.Kafka and lidrdkafka libraries from version 1.8.2 to 2.0.2 helped
Is it possible to execute a query asynchronously in hive server?
For eg, How can I /Is it possible to do something like this from the client-
QueryHandle handle = executeAsyncQuery(hiveQuery);
Status status = handle.checkStatus();
if(status.isCompleted()) {
QueryResult result = handle.fetchResult();
}
I also had a look at How do I make an async call to Hive in Java?. But did not help. The answers were mostly around the thrift clients taking a callback argument.
Any help would be appreciated. Thanks!
[EDIT 1]
I went through the HiveConnection.java in hive-jdbc. hive-jdbc by default uses the async thrift APIs. Hence it submits a query and polls for result sets (look at HiveStatement.java). Now i am able to write a piece of code which is purely non blocking. But the problem is as soon as the client disconnect the foot print about the query is lost.
Client 1
final TCLIService.Client client = new TCLIService.Client(createBinaryTransport(host, port, loginTimeout, sessConf, false)); // from HiveConnection.java
TSessionHandle sessionHandle = openSession(client) // from HiveConnection.java
TExecuteStatementReq execReq = new TExecuteStatementReq(sessionHandle, sql);
execReq.setRunAsync(true);
execReq.setConfOverlay(sessConf);
final TGetOperationStatusReq handle = client.ExecuteStatement(execReq)
writeHandleToFile("~/handle", handle)
Client 2
final TGetOperationStatusReq handle = readHandleFromFile("~/handle")
final TCLIService.Client client = new TCLIService.Client(createBinaryTransport(host, port, loginTimeout, sessConf, false));
while (true) {
System.out.println(client.GetOperationStatus(handle).getOperationState());
Thread.sleep(1000);
}
Client 2 keeps printing FINISHED_STATE as long as Client 1 is alive. But if client 1 process completes or gets killed, client 2 starts printing null which means hiveserver2 is cleaning up the resources as soon as a client disconnects.
Is it possible to configure hiveserver2 to configure this clean up process based on time or something?
Thanks!
Did some research and figured out that this happens only with binary transport (tcp)
#Override
public void deleteContext(ServerContext serverContext,
TProtocol input, TProtocol output) {
Metrics metrics = MetricsFactory.getInstance();
if (metrics != null) {
try {
metrics.decrementCounter(MetricsConstant.OPEN_CONNECTIONS);
} catch (Exception e) {
LOG.warn("Error Reporting JDO operation to Metrics system", e);
}
}
ThriftCLIServerContext context = (ThriftCLIServerContext) serverContext;
SessionHandle sessionHandle = context.getSessionHandle();
if (sessionHandle != null) {
LOG.info("Session disconnected without closing properly, close it now");
try {
cliService.closeSession(sessionHandle);
} catch (HiveSQLException e) {
LOG.warn("Failed to close session: " + e, e);
}
}
}
The above stub (from ThriftBinaryCLIService) gets executed through this piece of code from TThreadPoolServer which is used by ThriftBinaryCLIService.
eventHandler.deleteContext(connectionContext, inputProtocol,
outputProtocol);
Apparently http transport (ThriftHttpCLIService) has a different strategy of cleaning up operation handles (not greedy like tcp)
Will check with hive community on this to understand a bit more and see if there is an issue addressing this already.
I am trying to understand how to write a WWW call in Unity. According to the description here http://docs.unity3d.com/ScriptReference/WWW.html I can inspect the isDone property, but in the example on the same page, it makes no attempt to inspect isDone.
The question I have is, if I make a WWW call and it takes several seconds to reply, doesn't the game freeze?
I would like to think the right code is this, but is it?
StartCoroutine(WaitForResponse(myWWWObject));
private IEnumerator WaitForResponse(WWW aRequest ) {
while ( ! aRequest.isDone )
yield return aRequest;
}
Does the game freeze until aRequest is done? Or is it truly asynchronous?
You need to understand Coroutines - a fundamental feature of Unity that allows you to write long-duration code functions (eg: longer then a frame) that will not freeze your game.
http://docs.unity3d.com/Manual/Coroutines.html
In C# you can spot a coroutine function because it has a return type of IEnumerator. And you can spot the locations in the code where the function will suspend and continue from again by the C# keyword yield.
Each MonoBehaviour class can manage coroutines, and if you tell it to start one with StartCoroutine(), then MonoBehaviour will call the coroutine every frame (sometimes more then once) until the Coroutine reaches the end.
For WWW class (which supports co-routines), all you need is to call this:
WWW www = new WWW(url);
yield return www;
You create a WWW class with the URL to retrievve, and the yield will basically be called each frame automatically by the MonoDevelop coroutine manager until www object says it has completed (successfully or failure).
Your game will not freeze at all during this time.
In the linked documentation page for WWW, the following method is a coroutine method:
IEnumerator Start() {
WWW www = new WWW(url);
yield return www;
Renderer renderer = GetComponent<Renderer>();
renderer.material.mainTexture = www.texture;
}
The yield statement is used to pause execution and return control to Unity and then to resume execution in the next game frame. For example yield return null; will cause execution to resume in the next frame. Additionally, you can extend that behaviour by using one of the classes that derive from YieldInstruction class, for example WaitForSeconds.
Consider yield return new WaitForSeconds(1f); - this will resume execution after 1 second of game time has passed. The WWW class works in similar way. You can use an instance of that class to yield execution, returning only after the download has been completed, no need to manually poll the isDone property.
Keep in mind that you can only yield in Coroutines, I advise you read up on them. If you want to do a WWW request not in a Coroutine then you have to poll the isDone manually and proceed when completed.
To answer last question: Will the game freeze when creating an instance of the WWW class? - No. The class works asynchronously. You can use it in normal Update/Start etc. functions and inside Coroutines with yield.
if I make a WWW call and it takes several seconds to reply, doesn't the game freeze?
Not, Game Will not be freez try this code and you will see that log "after corou" will show immediately even the picture is downloading.
public class WWWDemo : MonoBehaviour {
bool b = true;
// Use this for initialization
void Start () {
}
// Update is called once per frame
void Update()
{
if (b) {
StartCoroutine(ExampleCoroutine());
Debug.LogWarning("after corou");
}
}
IEnumerator ExampleCoroutine()
{
b = false;
Debug.Log("Time : " + DateTime.Now);
string url = "http://images.earthcam.com/ec_metros/ourcams/fridays.jpg";
WWW www = new WWW(url);
yield return www;
Debug.Log("Time : " + DateTime.Now);
Renderer render = gameObject.GetComponent<Renderer>();
render.material.mainTexture = www.texture;
Debug.Log("Time : " + DateTime.Now);
}
}
but the execution of next line will definitely stop after yield return www statement it will wait unitl the picture download while the rest events of Unity will execute async.
Consider below example as well separately. Start Event executes ones it will wait but update will run continuously.
IEnumerator Start() {
Debug.Log("Time : " + DateTime.Now);
string url = "http://images.earthcam.com/ec_metros/ourcams/fridays.jpg";
WWW www = new WWW(url);
yield return www;
Debug.Log("Time : " + DateTime.Now);
Renderer render = gameObject.GetComponent<Renderer>();
render.material.mainTexture = www.texture;
Debug.Log("Time : " + DateTime.Now);
}
void Update()
{
Debug.Log("update is running while start is waiting for downloading");
}
I'm using the Booksleeve library in a C#/ASP.NET 4 application. Currently the RedisConnection object is a static object across my MonoLink class. Should I be keeping this connection open, or should I be open/closing it after each query/transaction (as I'm doing now)? Just slightly confused. Here's how I'm using it, as of now:
public static MonoLink CreateMonolink(string URL)
{
redis.Open();
var transaction = redis.CreateTransaction();
string Key = null;
try
{
var IncrementTask = transaction.Strings.Increment(0, "nextmonolink");
if (!IncrementTask.Wait(5000))
{
transaction.Discard();
throw new System.TimeoutException("Monolink index increment timed out.");
}
// Increment complete
Key = string.Format("monolink:{0}", IncrementTask.Result);
var AddLinkTask = transaction.Strings.Set(0, Key, URL);
if (!AddLinkTask.Wait(5000))
{
transaction.Discard();
throw new System.TimeoutException("Add monolink creation timed out.");
}
// Run the transaction
var ExecTransaction = transaction.Execute();
if (!ExecTransaction.Wait(5000))
{
throw new System.TimeoutException("Add monolink transaction timed out.");
}
}
catch (Exception ex)
{
transaction.Discard();
throw ex;
}
finally
{
redis.Close(false);
}
// Link has been added to redis
MonoLink ml = new MonoLink();
ml.Key = Key;
ml.URL = URL;
return ml;
}
Thanks, in advance, for any responses/insight. Also, is there any sort of official documentation for this library? Thank you S.O. ^_^.
According to the author of Booksleeve,
The connection is thread safe and intended to be massively shared;
don't do a connection per operation.
Should I be keeping this connection open, or should I be open/closing
it after each query/transaction (as I'm doing now)?
There is probably a little overhead if you will open a new connection each time you want to make a query/transaction and although redis is designed for high level of concurrently connected clients, there might be performance problems if their number is around tens of thousands. As far as I know connection pooling should be done by the client libraries (because redis itself doesn't have this functionality), so you should check if booksleeve supports this stuff. Otherwise you should open the connection when your application starts and keep it open for it's lifetime (in case you don't need parallel clients connected to redis for some reason).
Also, is there any sort of official documentation for this library?
The only documentation I was able to find regarding how to use it was tests folder in it's source codes.
For reference (continuing #bzlm's answer), I created a Singleton that always provides the same Redis connection using BookSleeve (if it's closed, it's being created. Else, the existing connection is being served).
Look at this: https://stackoverflow.com/a/8777999/290343
You consume it like that:
RedisConnection connection = Redis.RedisConnectionGateway.Current.GetConnection();
I'm building a Flex app which requires me to download files.
I have the following code:
public function execute(event:CairngormEvent) : void
{
var evt:StemDownloadEvent = event as StemDownloadEvent;
var req:URLRequest = new URLRequest(evt.data.file_path);
var localRef:FileReference = new FileReference();
localRef.addEventListener(Event.OPEN, _open);
localRef.addEventListener(ProgressEvent.PROGRESS, _progress);
localRef.addEventListener(Event.COMPLETE, _complete);
localRef.addEventListener(Event.CANCEL, _cancel);
localRef.addEventListener(Event.SELECT, _select);
localRef.addEventListener(SecurityErrorEvent.SECURITY_ERROR, _securityError);
localRef.addEventListener(IOErrorEvent.IO_ERROR, _ioError);
try {
localRef.download(req);
} catch (e:Error) {
SoundRoom.logger.log(e);
}
}
As you can see, I hooked up every possible event listener as well.
When this executes, I get the browse window, and am able to select a location, and click save. After that, nothing happens.
I have each event handler hooked up to my logger, and not a single one is being called! Is there something missing here?
The problem seems to be with my command being destroyed before this could finish.
For a proof of concept, I set my localRef variable to be static instead of an instance variable, and everything went through successfully! I guess Cairngorm commands kill themselves asap!