We have multiple jobs that serves as threads for file loading. But we want to trigger jobs only when file is received. So we created a file watcher job in control-M. We want to trigger thread Job for each file. So one file will be processed by a single thread job.
For example: If only one file is received only one thread job should be triggered say Thread1 job is triggered. Now after 1 min a new file is received then since Thread1 job is already running so Thread 2 job should be initiated.
I think, if we could post condition pro-grammatically in Control-M my purpose will be solved.
Please help and comment if any more information is required.
You could have the filewatcher post a generic out-condition then configure a dummy job at the start of each thread that would require exclusive control over a control resource and on completion delete it's in-condition and kickoff the rest of the thread.
3 file arrive.
Filewatcher completes and posts out-condition.
Only one thread header can start, it then removes the out-condition and continues the thread.
Filewatcher runs again, completes and posts out-condition.
Only one thread header can start, it then removes the out-condition and continues the thread.
etc..
Its not clear where you are trying to use the conditions, but its possible to add conditions programatically using the ctmcontb utility.
ex: ctmcontb -ADD Condition_Name ODAT
Related
I am trying to see if Airflow is the right tool for some functionality I need for my project. We are trying to use it as a scheduler for running a sequence of jobs
that start at a particular time (or possibly on demand).
The first "task" is to query the database for the list of job id's to sequence through.
For each job in the sequence send a REST request to start the job
Wait until job completes or fails (via REST call or DB query)
Go to next job in sequence.
I am looking for recommendations on how to break down the functionality discussed above into an airflow DAG. So far my approach would :
create a Hook for the database and another for the REST server.
create a custom operator that handles the start and monitoring of the "job" (steps 2 and 3)
use a sensor to poll handle waiting for job to complete
Thanks
I have two batch jobs in differents flows. The first, do an Upsert in Salesforce and when it finish, it call to the second flow that has another batch job.
This image represents the flows:
But when I see the log on the console, sometimes the log of the second batch is mixed with the log of the first.
I get the feeling that the batch processes are asynchronous and the second batch is called even though the first batch is being processed.
Am I wrong? Should I pay attention to the order of the logs?
If I wanted it to be totally synchronous, what would be the best way?
Mule Batch is asynchronous, it is like fire and forget. If you want to call the second batch after first batch is completed, then invoke the second batch at 'On Complete' phase of first batch as shown in below picture.
If you want to do some function before invoking the second batch, then you need to use request-reply scope to make batch component synchronous.
Yes the batch job is asynchronous. As soon as the batch execute is triggered the flow will move on to the next event processor.
If batch job 2 needs to run after batch job 1 only, then you can use the on-complete phase of the first batch job to trigger some event to indicate the first has finished so that can be used to trigger the second batch job.
Alternatively If the batch jobs are related that closely you might be able to combine them into one using multiple batch steps
Would you know how to run a background task on Symfony 4, based on the setup of a form ? This would avoid that the user has to remain on the form until the task is finished.
The idea would be that when the form is validated, it starts an independant background task. Then the user can continue its navigation and come back once the task is finished to get the results.
Thanks for your help,
You need to use pattern Message Bus. Symfony has own implementation of this pattern since version 4.1 introducing Messenger Component.
You can see documentation here: https://symfony.com/doc/current/components/messenger.html
To get it work you need some external program that will implement AMQP protocol. Most popular in PHP world IMHO RabbitMQ.
A very simple solution for this could be the following procedure:
Form is valid.
A temporary file is created.
Cronjob gets executed every five minutes and starts a symfony command.
The command checks if the file exists and if it's empty.
If so, the command works of the background task. But before this, the command write it's process id in the file to prevent from beeing excuted a second time.
Remove the file when the command has finished.
As long as the file exists you can show a hint for the user that the task is running.
When i changed the start time of a coordinator job in job.properties in oozie, the job is not taking the changed time, instead its running in the old scheduled time.
Old job.properties:
startMinute=08
startTime=${startDate}T${startHour}:${startMinute}Z
New job.properties:
startMinute=07
startTime=${startDate}T${startHour}:${startMinute}Z
The job is not running at the changed time:07th minute,its running at 08th minute in every hour.
Please can you let me know the solution, how i can make the job pickup the updated properties(changed timing) without restarting or killing the job.
You can't really change the timing of the co-ordinator via any methods given by Oozie(v3.3.2) . When you submit a job the contents properties are stored in the database whereas the actual workflow is in the HDFS.
Everytime you execute the co-ordinator it is necessary to have the workflow in the path specified in properties during job submission but the properties file is not needed. What I mean to imply is the properties file does not come into the picture after submitting the job.
One hack is to update the time directly in the database using SQL query.But I am not sure about the implications of it.The property might become inconsistent across the database.
You have to kill the job and resubmit a new one.
Note: oozie provides a way to change the concurrency,endtime and pausetime as specified in the official docs.
I am working on a project which will batching some 834 records in file.
I setup the batching trigger as when the record count reaches a number, a batch file will release. But I also want release a batch even the record count is not reached (for example, every night, release all queueing record as a final file).
I know it can be done by click the override button in Batch Configuration window, but it need be done automatically.
So, basically, my question is, what did BizTalk do when I clicked the override button? Does BizTalk prove anyway to let me do that in a program?
I must say I did not try to send a controlmessage to a batch setting as release per record count, if you know this works, please let me know.
You're almost there and to complete the process isn't that difficult.
Leave the Batch configuration at the records count as it is.
Then, setup a process where an External Release trigger is sent at the appropriate time. A Windows Scheduled Task is a viable option, it can copy a file to a File Receive Location.
This article describes how to create the trigger message: http://msdn.microsoft.com/en-us/library/bb246108.aspx