I just got a task colored in Blue which doesn't appear in the status legend. I'm curious if this is a bug or an undocumented status.
As you can see the color blue doesn't show up in the list of potential statuses on the right. I had just finished clearing all past, future, and upstream attempts fyi.
That's a known TaskInstance State; it's just not on the UI -- it stands for shutdown:
https://github.com/apache/incubator-airflow/blob/master/airflow/utils/state.py#L70
Other statuses that don't appear on the upper right hand side include:
UPSTREAM_FAILED: 'orange'
REMOVED: 'lightgrey'
SCHEDULED: 'tan'
Clearing a running task will often result in this color for a terminated/killed process. You just need to clear the task one additional time to reset it.
Related
I have a sensor that waits for a file to appear in an external file system
The sensor uses mode="reschedule"
I would like to trigger a specific behavior after X failed attempts.
Is there any straightforward way to know how many times the sensor has already attempted to run the poke method?
My quick fix so far has been to push an XCom with the attempt number, and increase it every time the poke method returns False. Is there any built-in mechanism for this?
Thank you
I had a similar problem when sensor mode = "reschedule", trying to poke a different path to a file based on the current time without directly referencing pendulum.now or datetime.now
I used task_reschedules (as done in the base sensor operator to get try_number for reschedule mode https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/sensors/base.html#BaseSensorOperator.execute)
def execute(self, context):
task_reschedules = TaskReschedule.find_for_task_instance(context['ti'])
self.poke_number = (len(task_reschedules) + 1)
super().execute(context)
then self.poke_number can be used within poke(), and current time is approximately execution_date + (poke_number * poke_interval).
Apparently, the XCom thing isn't working, because pushed XComs don't seem to be available between pokes; they always return undefined.
try_number inside task_instance doesn't help either, as pokes don't count as a new try number
I ended up computing the attempt number by hand:
attempt_no = math.ceil((pendulum.now(tz='utc') - kwargs['ti'].start_date).seconds / kwargs['task'].poke_interval)
The code will work fine as long as individual executions of the poke method don't last longer than the poke interval (which they shouldn't)
Best
I have a dag similar to this:
with DAG(..., concurrency=3) as dag:
important = [ op1, op2, op3, ... ]
less_important=[ opA, opB, opC, ... ]
important >> DummyOperator('Important', priority_weight=1000)
less_important >> DummyOperator('LessImportant', priority_weight=1)
The idea is to use the default weight rule being WeightRule.DOWNSTREAM to ensure that the important tasks are scheduled before the less_important ones.
Inspecting the task instance details I see that the priorities for the important and less_important tasks are correctly computed (1001 and 2, resp.).
However, the last run scheduled one of the less_important tasks right away, next to two of the important ones.
Am I overlooking something?
Oh, I'm using Airflow v. 1.8.0. I checked the changelog and I don't see any ticket regarding weight and priority, so I don't suppose it's something that was fixed, but I might of course be wrong.
I have a DAG without a schedule (it is run manually as needed). It has many tasks. Sometimes I want to 'skip' some initial tasks by changing the task state to SUCCESS manually. Changing task state of a manually executed DAG fails, seemingly because of a bug in parsing the execution_date.
Is there another way to individually setting task states for a manually executed DAG?
Example run below. The execution date of the Task is 01-13T17:27:13.130427, and I believe the milliseconds are not being parsed correctly.
Traceback
Traceback (most recent call last):
File "/opt/conda/envs/jumpman_prod/lib/python3.6/site-packages/airflow/www/views.py", line 2372, in set_task_instance_state
execution_date = datetime.strptime(execution_date, '%Y-%m-%d %H:%M:%S')
File "/opt/conda/envs/jumpman_prod/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "/opt/conda/envs/jumpman_prod/lib/python3.6/_strptime.py", line 365, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: ..130427
It's not working from Task Instances page, but you can do it in another page:
- open DAG graph view
- select needed Run (screen 1) and click go
- select needed task
- in a popup window click Mark success (screen 2)
- then confirm.
PS it relates to airflow 1.9 version
Screen 1
Screen 2
What you may want to do to accomplish this is using branching, which, as the name suggests, allows you to follow different execution paths according to some conditions, just like an if in any programming language.
You can use the BranchPythonOperator (documented here) to attain this goal: the idea is that this operator is configured by a python_callable, a function that outputs the task_id to execute next (which should, of course, be a task which is directly downstream from the BranchPythonOperator itself).
Using branching will set the skipped tasks to the proper state automatically, as mentioned in the documentation:
All other “branches” or directly downstream tasks are marked with a state of skipped so that these paths can’t move forward. The skipped states are propagated downstream to allow for the DAG state to fill up and the DAG run’s state to be inferred.
The resulting DAG would look something like the following:
(source: apache.org)
Branching is documented here, on the official Apache Airflow documentation.
From the man page (http://manpages.ubuntu.com/manpages/precise/en/man1/tmux.1.html):
refresh-client [-S] [-t target-client]
(alias: refresh)
Refresh the current client if bound to a key, or a single client
if one is given with -t. If -S is specified, only update the
client's status bar.
What does it mean for a client to be bound to a key? I'm trying to think of when I may actually use this.
This is by default bound to "r" in tmux, and I'm thinking of overriding it. Could someone explain an example use case for wanting to refresh the client? Thanks.
tmux only updates the screen when there is some new content to display. If you put something like date into your status line, the date will only update when the content of the pane changes, when you change between panes, or when run refresh-client. So in that case, you could use Ctrl-b r to refresh the screen.
Alternatively, you can also set set status-interval 1 to redraw every second, but that will cause CPU usage and drain your battery.
If you don't have anything dynamic in your status line, you can safely remap the key. And if you ever need to execute refresh-client, you can still run it with tmux refresh-client.
I'm looking for some kind of documentation that specifies how much time each browser (IE6/IE7/FF2/FF3, etc) will wait on a request before it just gives up and times out.
I haven't had any luck trying to get this.
Any pointers?
I managed to find network.http.connect.timeout for much older versions of Mozilla:
This preference was one of several
added to allow low-level tweaking of
the HTTP networking code. After a
portion of the same code was
significantly rewritten in 2001, the
preference ceased to have any effect
(as noted in all.js as early as
September 2001).
Currently, the timeout is determined
by the system-level connection
establishment timeout. Adding a way to
configure this value is considered
low-priority.
It would seem that network.http.connect.timeout hasn't done anything for some time.
I also saw references to network.http.request.timeout, so I did a Google search. The results include lots of links to people recommending that others include it in about:config in what appears to be a mistaken belief that it actually does something, since the same search turns up this about:config entries article:
Pref removed (unused).
Previously: HTTP-specific network
timeout. Default value is 120.
The same page includes additional information about network.http.connect.timeout:
Pref removed (unused).
Previously: determines how long to
wait for a response until registering
a timeout. Default value is 30.
Disclaimer: The information on the MozillaZine Knowledge Base may be incorrect, incomplete or out-of-date.
After the last Firefox update we had the same session timeout issue and the following setting helped to resolve it.
We can control it with network.http.response.timeout parameter.
Open Firefox and type in ‘about:config’ in the address bar and press Enter.
Click on the "I'll be careful, I promise!" button.
Type ‘timeout’ in the search box and network.http.response.timeout parameter will be displayed.
Double-click on the network.http.response.timeout parameter and enter the time value (it is in seconds) that you don't want your session not to timeout, in the box.
firstly I don't think there is just one solution to your problem....
As you know each browser is vastly differant.
But lets see if we can get any closer to the answer you need....
I think IE Might be easy...
Check this link
http://support.microsoft.com/kb/181050
For Firefox try this:
Open Firefox, and in the address bar, type "about:config" (without quotes). From there, scroll down to the Network.http.keep-alive and make sure that is set to "true". If it is not, double click it, and it will go from false to true. Now, go one below that to network.http.keep-alive.timeout -- and change that number by double clicking it. if you put in, say, 500 there, you should be good. let us know if this helps at all
For Google Chrome (Tested on ver. 62)
I was trying to keep a socket connection alive from the google chrome's fetch API to a remote express server and found the request headers have to match Node.JS's native <net.socket> connection settings.
I set the headers object on my client-side script with the following options:
/* ----- */
head = new headers();
head.append("Connnection", "keep-alive")
head.append("Keep-Alive", `timeout=${1*60*5}`) //in seconds, not milliseconds
/* apply more definitions to the header */
fetch(url, {
method: 'OPTIONS',
credentials: "include",
body: JSON.stringify(data),
cors: 'cors',
headers: head, //could be object literal too
cache: 'default'
})
.then(response=>{
....
}).catch(err=>{...});
And on my express server I setup my router as follows:
router.head('absolute or regex', (request, response, next)=>{
req.setTimeout(1000*60*5, ()=>{
console.info("socket timed out");
});
console.info("Proceeding down the middleware chain link...\n\n");
next();
});
/*Keep the socket alive by enabling it on the server, with an optional
delay on the last packet sent
*/
server.on('connection', (socket)=>socket.setKeepAlive(true, 10))
WARNING
Please use common sense and make sure the users you're keeping the socket connection open to is validated and serialized. It works for Firefox as well, but it's really vulnerable if you keep the TCP connection open for longer than 5 minutes.
I'm not sure how some of the lesser known browsers operate, but I'll append to this answer with the Microsoft browser details as well.