Recently got this question from one of my friend but i am not sure how it works in oozie.
Question:
What happens if one action fails in workflow(assume multiple actions are there) and when we restart again whether it starts from failed action or start from previously completed action.
Please clarify this doubt.
Thanks
Saravan
This property comes into the consideration, oozie.wf.rerun.failnodes if true: It will start from the failed node, false: it will start from the start.
Related
I have a subdag that uses a sensor operator which contains a soft_fail=true , in order to skip instead of failing the task.
It works well, except that the status of the whole subdag is shown as "succeeded" instead of "skipped" which could be misleading when monitoring the flow, as I wouldn't know if the file has been found, or simply skipped. Any thoughts on how to make the subdag status inherit the subtasks' status?
A "skipped" status isn't a failure though, you requested not to execute a task and it did just that. Also think about what the opposite would be, a user being surprised that their run had failed just because Airflow did as they asked and skipped all the tasks.
This issue regarding the skipped status has been covered before. For example, it was reported in 1.8.0 and fixed in 1.8.1, but in later versions this fix was not propagated.
You could open an issue and request the change by selecting Reference in new issue in the three dots of this link.
I am running a Oozie workflow. I have a Fork action in that workflow. Sometimes i am observing that thought an action in Fork Status is failed, but the the flow is going to Joining and completing the flow till that end.
Can someone please let me know in what cases the above scenario occurs.
Thanks.
I use OOZIE to run a workflow. But a simple official example shell-wf (echo hello oozie) stuck in RUNNING state and never end. The workflow can be submitted but stuck at RUNNING state. There is not any error in job log in OOZIE UI.
When submitting a shell with spark-submit inside, the job will be never submitted and can not be seen in Spark UI. I suspect the shell didn't run at all.
What's the possible problem?
A Quick Checklist
For those who have the same problem, there is a checklist to check your system. Hope it helps!
Check jobTracker in your Oozie configuration. Note: If a job has been successfully run, it probably not the problem of jobTracker. Related discussion can be found here
Check your disk usage. If ## Heading ##disk usage is greater than 90%, remove some files to make sure disk usage is less than 90%. (That's my case!)
Check Console URL of the stuck action. It can be found in Job - Job Info tab - Actions - Action - Action Info tab. Job state here may help you to find the problem.
Check Oozie log. It's typically in /usr/local/oozie/logs. Check oozie.log* to find if there are exceptions.
Details
Disk usage
If your action state is
YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM.
That may be the disk problem. Relative discussion can be found in MapReduce job hangs, waiting for AM container to be allocated. Solutions can be found in Why does Hadoop report "Unhealthy Node local-dirs and log-dirs are bad"?.
Would you know how to run a background task on Symfony 4, based on the setup of a form ? This would avoid that the user has to remain on the form until the task is finished.
The idea would be that when the form is validated, it starts an independant background task. Then the user can continue its navigation and come back once the task is finished to get the results.
Thanks for your help,
You need to use pattern Message Bus. Symfony has own implementation of this pattern since version 4.1 introducing Messenger Component.
You can see documentation here: https://symfony.com/doc/current/components/messenger.html
To get it work you need some external program that will implement AMQP protocol. Most popular in PHP world IMHO RabbitMQ.
A very simple solution for this could be the following procedure:
Form is valid.
A temporary file is created.
Cronjob gets executed every five minutes and starts a symfony command.
The command checks if the file exists and if it's empty.
If so, the command works of the background task. But before this, the command write it's process id in the file to prevent from beeing excuted a second time.
Remove the file when the command has finished.
As long as the file exists you can show a hint for the user that the task is running.
The method WorkflowInstanceProxy.EndResumeBookmark returns an enum BookmarkResumptionResult. When will this return NotReady? What I really want to know is how I can create a scenario so that this occurs, so that I can write tests to verify that my implementation of an extension is correct.
MSDN says(http://msdn.microsoft.com/en-us/library/system.activities.bookmarkresumptionresult.aspx):
NotReady: The bookmark resumption was not scheduled because the runtime has not yet created the bookmark. The host should wait until another idle point to try resumption again.
I saw it when you want to resume the bookmark and the workflow is not unloaded yet.
I hope this helps.