oozie job -info $coordinator
the command gives you the details of workflows belong to the coordinator, print their ID, status, created time and nominal time.
I'm trying to print the workflows of the oozie coordinator which are executed after a specific date.
As per their documentation,
-filter <arg> <key><comparator><value>[;<key><comparator><value>]*
(All Coordinator actions satisfying the filters will be retrieved).
key: status or nominal time
comparator: =, !=, <, <=, >, >=. = is used as OR and others as AND
status: values are valid status like SUCCEEDED, KILLED etc. Only = and != apply for status.
nominaltime: time of format yyyy-MM-dd'T'HH:mm'Z'
From this, it's understandable that status key supports only "=" or "!=" whereas nominal time key supports all comparators.
But when I try to use it, I'm getting below error.
[hadoop#xx ~]$ oozie job -info $coord -filter status nominalTime>2018-09-01'T'08:00'Z'
Error: E0421 : E0421: Invalid job filter [nominalTime], filter should be of format <key><comparator><value> pairs
The same command works if I put "=" or "!=" but throws an error if you use other comparators. (>,<,>=,<=)
Kindly suggest how to fix this or any other alternatives for this use case.
-filter 'nominaltime>2019-11-01T01:00Z'
Related
I have use case to create 2 tasks of BigqueryOperator that have same destination table but I need one to run daily, and the second one to be run manually just when I need.
Below are the illustration of Tree View
| task_3rd_adhoc
| task_3rd
|---- task_2nd
|---- task_1st_a
|---- task_1st_b
From example above, DAG are run daily. And I aim to the task will be:
task_1st_a and task_1st_b run first. Target table are:
project.dataset.table_1st_a with _PARTITIONTIME = execution date, and
project.dataset.table_1st_b with _PARTITIONTIME = execution date.
then task_2nd_a will run after task_1st_a and task_1st_b finish. BigQueryOperator use TriggerRule.ALL_SUCCESS. Target table is:
project.dataset.table_2nd with _PARTITIONTIME = execution date.
then task_3rd will run after task_2nd success. BigQueryOperator use TriggerRule.ALL_SUCCESS. Target table is:
project.dataset.table_3rd with PARTITIONTIME = D-2 from execution date.
task_3rd_adhoc will not run in daily job. I need this when I want to backfill table project.dataset.table_3rd. With target table:
project.dataset.table_3rd with _PARTITIONTIME = execution_date
But I still can't find what is the correct TriggerRule for step #4 above. I tried TriggerRule.DUMMY because I thought it can be used to set no Trigger, but task_3rd_adhoc also run in daily job when I tried create DAG above.
(based on this doc dependencies are just for show, trigger at will)
First of all, you've misunderstood TriggerRule.DUMMY.
Usually, when you wire tasks together task_a >> task_b, B would run only after A is complete (success / failed, based on B's trigger_rule).
TriggerRule.DUMMY means that even after wiring tasks A & B together as before, B would run independently of A (run at will). It doesn't mean run at your will, rather it runs at Airflow's will (it will trigger it whenever it feels like). So clearly tasks having dummy trigger rule will pretty much ALWAYS run, albeit, at an unpredictable time
What you need here (to have a particular task in DAG always but run it only when manually specified) is a combination of
AirflowSkipException
Variable
Here's roughly how you can do
A Variable should hold the command for this task (whether or not it should run). This Variable, of course, you can edit anytime from UI (thereby controlling whether or not that task runs in next DagRun)
In the Operator's code (execute() method for custom-operator or just python_callable in case of PythonOperator), you'll check value of Variable (whether or not the task is supposed to run)
Based on the Variable value, if the task is NOT supposed to run, you must throw an AirflowSkipException, so that the task will be marked at skipped. Or else, it will just run as usual
How do I find out if a variable has been defined in my Robot Framework script? I am doing API testing, not UI testing. I have a complex set up and tear-down sequence and, since I am interacting with multiple computers through the script, it is important to know the current state if a fatal error has occurred. I could track what I have done with some complex set of meta variables or a variable tracking list, but I would prefer to query if a particular variable has been defined and if so take the appropriate tear-down steps.
A simplified version is something like:
*** Test Cases ***
Check monitor
${monitored}= Connect to Monitor ${Monitor IP Address} ${User name} ${password}
${peer connected}= Connect to Monitor ${Peer IP Address} ${User name} ${password}
Get Information from Monitor ${IP Address}
Send Info to Peer ${buffer1}
Report back to Monitor ${Monitor IP Address}
We are assuming that the tear-down closes the connections. I want to close any connections that are open, but if I failed to open the peer connection I will close the monitor connection and fail on closing the monitor connection.
I am trying to determine if ${peer connected} is defined. Can I look into Robot Framework's variable storage to see if it is there (in that dictionary?)?
You can call Get Variables to get a dictionary of all variables, then check whether the variable you're interested in is in the dictionary.
*** Test cases ***
Example
${foo}= set variable hello, world
${variables}= Get variables
Should be true "\${foo}" in $variables
Should not be true "\${bar}" in $variables
There a pretty straightforward approach - the built-in keyword Get Variable Value returns python's None (by default) if there is no such variable defined:
${the var}= Get Variable Value ${peer connected}
${is set}= Set Variable If """${the var}""" != 'None' ${True} ${False}
I am fine with this approach. In case the variable is not defined, the test case does not fail....
${variables} Get variables
${status} Run Keyword And Return Status Evaluate $new_table in $variables
${new_table} Set variable if ${status}==${FALSE} new_tbl ${new_table}
Also possible is:
${variables} Get Variables
IF "\${dataPluginVersion}" in "${variables}"
No Operation
ELSE
${dataPluginVersion} Set Variable 0
END
Or:
${variables} Get Variables
IF not "\${dataPluginVersion}" in "${variables}"
${dataPluginVersion} Set Variable 0
END
A shorter way:
OEM-T01-99-Test-variables
[Tags] TEST
Variable Should Not Exist \${TESTDEVICE_SSH_CONNECTION}
Variable Should Exist \${TEST_NAME}
This method is more readable and less verbose than using "Get Variables" keyword, IMHO
Reference: Robotframework built-in keywords
I would like to mark the last time entry for a user as the current time entry. In other words, I would like to clear the end field of a given time entry, to indicate that it it not finished yet, it's still running.
Is it possible to do that?
I tried using the Update time entry on workspace endpoint, like so:
curl -H 'Content-Type':'application/json' \
-H 'X-Api-Key':'API_KEY' \
-X PUT https://api.clockify.me/api/workspaces/WORKSPACE_ID/timeEntries/TIME_ENTRY_ID \
-d '{"end":null}'
But the response code is 400 with the following response:
{"message":"text","code":3002}
What does this error code mean?
So the documentation for the UpdateTimeEntryRequest (https://clockify.github.io/clockify_api_docs/#/definitions/UpdateTimeEntryRequest) is a bit wrong or incomplete. I figured it out through some testing, here's what you need to do:
First, you need to make sure you have the current data of the time entry, specifically the start date and billable status (as these are the only two "required" fields in the API call). You are getting the 400 bad request because you are missing these two fields - the error message is obviously no help here, but a 400 response code means "the server was unable to process the request sent by the client due to invalid syntax.", which makes sense, since your input is invalid.
Assuming you have the time entry information already (which is the only way I could see how you would know the ID), you could recreate the whole UpdateTimeEntryRequest data but then null out the end field or leave it out completely, like this:
{
"start": START_FROM_THE_TIME_ENTRY,
"billable": BILLABLE_FROM_THE_TIME_ENTRY,
"description": DESC_FROM_THE_TIME_ENTRY,
"projectId": PROJECT_ID,
"taskId": TASK_ID,
"end": null,
"tagIds": TAGS_ARRAY
}
From my testing, this restarts the timer. If you leave out any field, those fields are reset, for example:
{
"start": START_FROM_THE_TIME_ENTRY,
"billable": BILLABLE_FROM_THE_TIME_ENTRY
}
would restart the timer, but it would clear the description, project/task, and tags, which is probably not preferable.
Trying to implement a listener interface for robot framework in order to collect information about keyword executions like time taken for execution, Pass/Fail status, failure message in case if status is fail. Sample code is given below
import os.path
import tempfile
class PythonListener:
ROBOT_LISTENER_API_VERSION = 2
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
def __init__(self, filename='listen.txt'):
outpath = os.path.join(tempfile.gettempdir(), filename)
self.outfile = open(outpath, 'w')
def end_keyword(self, name, attrs):
self.outfile.write(name + "\n")
self.outfile.write(str(attrs) + "\n")
def close(self):
self.outfile.close()
All the information apart from keyword failure message is available in the attributes which is passed to end_test method from robot framework.
Documentation can be found here. https://github.com/robotframework/robotframework/blob/master/doc/userguide/src/ExtendingRobotFramework/ListenerInterface.rst#id36
The failure message is available in the attributes for end_test() method. But this will not have information if a keyword is run using RunKeywordAndIgnoreError.
I could see that there is a special variable ${KEYWORD MESSAGE} in robot framework, which contains the possible error message of the current keyword Is it possible to access this variable in the listener class.?
https://github.com/robotframework/robotframework/blob/master/doc/userguide/src/CreatingTestData/Variables.rst#automatic-variables
Are there any other ways to collect the failure message information at the end of every keyword?
That's an interesting approach, indeed, end_test will ensure an attributes.message field containing the failure. (so it goes for end_suite if it fails during the suite setup/teardown)
With end_keyword you don't have such message, but at least you can filter for the FAIL status and detect which one failed. Then the message returned by Run Keyword And Ignore Error has to be logged explicitly by you so you can capture such triggering logs with the log_message hook. Otherwise nobody is aware of the message of the exception handled by the wrapper keyword which returns a tuple of (status, message).
There's also the message hook but couldn't manage to get it called from a normal breaking robot:
Called when the framework itself writes a syslog message.
message is a dictionary with the same contents as with log_message method.
Side note: To not expose these hooks as keywords, you can precede the method names with _. Examples:
def _end_test(self, name, attributes): ...
def _log_message(self, message): ...
I am writing a custom library for robot framework in python. I don't want to use builtin library for some reasons.
My python code :
import os
import re
output = "IP address is 1.1.1.1"
def find_ip():
cmd = 'ipconfig'
output = os.popen(cmd).read()
match1 = re.findall('.* (1.1.1.1).*',output)
mat1 = ['1.1.1.1']
if match1 == mat1:
print "PASS"
In the above program I have written python function to :
Execute a windows command "ipconfig"
Written regular expression to match 1.1.1.1
create a list variable, mat1 = ['1.1.1.1']
Now I want to put condition like, if "match1" and "mat1" are equal my TEST should PASS. else it should fail in Robot framework.
Any one please give idea on how to write python function for this purpose?
Please note I dont want to use "Should Match Regexp" keyword in Robot Framework. Because I know it will do the same whatever I am asking.
To make a keyword pass, you don't need to do anything except return normally to the caller. To fail, you need to raise an exception:
def find_ip():
...
if match1 != mat1:
raise Exception('expected the matches to be similar; they are not")
This is documented in the robot user guide in the section Returning Keyword Status:
Reporting keyword status is done simply using exceptions. If an
executed method raises an exception, the keyword status is FAIL, and
if it returns normally, the status is PASS.
The error message shown in logs, reports and the console is created
from the exception type and its message. With generic exceptions (for
example, AssertionError, Exception, and RuntimeError), only the
exception message is used, and with others, the message is created in
the format ExceptionType: Actual message.