I encountered a freezing test but actually it was stuck in a retry loop.
Wait Until Keyword Succeeds | 60 sec | 12 sec
The first parameter is the timeout for an attempt and the second is retry interval. However, if the task is something like "Xpath Should Match X Times" on a large data set it might never succeed in given time and next attempt starts from scratch again.
Is there way to limit the maximum number of retry attempts in this kind of situation?
Your situation is a bit strange because theoretically there is no such thing as "infinite retry loop" with Wait Until Keyword Succeeds because the first parameter is not "timeout for an attempt" but rather "global timeout for all the attempts". So in your case, it would try every 12 seconds and stop after 60 secs elapsed. Maybe you should reconsider the value you use with this information.
To answer your question, there is no way (AFAIK) to give a max number of attempts. I had such a need for some keywords and I came up with my own custom keyword:
Run_keyword_n_times_and_stop_if_success
[Arguments] ${keyword} ${number_tries}
:FOR ${index} IN RANGE ${number_tries}
\ ${result} ${error_message} = Run Keyword And Ignore Error ${keyword}
\ Pass Execution If '${result}' == 'PASS' keyword execution was successful
Fail ${error_message}
This works only with keywords having no arguments, but maybe that could be helpful for you (either because you don't have args, or because you could elaborate on that first version)
Thanks for the Retry keyword. To use it with a keyword with arguments.
Run Keyword N Times And Stop If Success
[Arguments] ${number_tries} ${keyword} #{args}
: FOR ${index} IN RANGE ${number_tries}
\ ${result} ${error_message} = Run Keyword And Ignore Error ${keyword} #{args}
\ Pass Execution If '${result}' == 'PASS' keyword execution was successful
Fail ${errorLog_message}
Sample usage:
#{args} Create List #{User} ${Url} ${Browser}
Run Keyword N Times And Stop If Success 5 Login #{args}
Since April 2015, this has been implemented as an argument for the keyword.
An example of usage could be seen as:
Wait Until Keyword Succeeds | 5x | 30 sec | Flaky Keyword
More information on the keyword is available in the BuiltIn documentation.
Related
I want to make a VDP scheduler job in Denodo 8 wait for a certain amount of time. The wait function in the job creation process is not working as expected so I figured I'd write it into the VQL. However when i try the suggested function from the documentation (https://community.denodo.com/docs/html/browse/8.0/en/vdp/vql/stored_procedures/predefined_stored_procedures/wait) the Denodo 8 VQL shell doesn't recognize the function.
--Not working
SELECT WAIT('10000');
Returns the following error:
Function 'wait' with arity 1 not found
--Not working
WAIT('10000');
Returns the following error:
Error parsing command 'WAIT('10000')'
Any suggestions would be much appreciated.
There are two ways of invoking WAIT:
Option #1
-- Wait for one minute
CALL WAIT(60000);
Option #2:
-- Wait for ten seconds
SELECT timeinmillis
FROM WAIT()
WHERE timeinmillis = 10000;
(1) The test suite with installation & running instructions:
https://github.com/TeddyTeddy/robot-fw-rest-instance-library-tests-v2
(2) The test suite is testing a locally running JSON RESTFUL API server:
https://github.com/typicode/json-server
The DB file for the server (i.e. db.json) is at the root of (1). The server reads the file and creates the API endpoints based on it.
An example run of the server:
(base) ~/Python/Robot/robot-fw-rest-instance-library-tests-v2$ json-server --watch db.json
\{^_^}/ hi!
Loading db.json
Done
Resources
http://localhost:3000/posts
http://localhost:3000/comments
http://localhost:3000/albums
http://localhost:3000/photos
http://localhost:3000/users
http://localhost:3000/todos
(3) With the given db.json, you can make the following request to the server:
GET /posts?_start=<start_index>&_end=<end_index>
where _start is inclusive and _end is exclusive. Note that start_index starts from 0 just like in Array.Slice method.
(4) To be able to comprehensively test (3), i wrote the following Robot Test Case, which is provided in (1):
Slicing Posts With All Possible Start And End Combinations
[Documentation] Referring to the API documentation:
... GET /posts?_start=20&_end=30
... where _start is inclusive and _end is exclusive
... This test case make the above API call with all possible combinations of _start and _end values.
... For each call, the test case fetches expected_posts from database for the same _start and _end.
... It then compares the expected_posts with observed_posts. It also calculates the expected length
... of observed_posts and compares that with the observed length of observed_posts
[Tags] read-tested slicing run-me-only
FOR ${start_index} IN RANGE ${0} ${NUMBER_OF_POSTS+10}
FOR ${end_index} IN RANGE ${start_index+1} ${NUMBER_OF_POSTS + 10 +1}
Log To Console start_index:${start_index}
Log To Console end_index:${end_index}
# note that start_index starts from zero when posts are fetched from database
${expected_posts} = Fetch Posts From Database ${start_index} ${end_index}
# note that start_index starts from 0 too when posts are fetched via API call
# test call
${observed_posts} = Get Sliced Posts ${start_index} ${end_index}
Should Be Equal ${expected_posts} ${observed_posts}
# note that start_index is between [0, NUMBER_OF_POSTS-1]
# and end_index is between [start_index+1, start_index+NUMBER_OF_POSTS]
# we expect observed_posts to be a non-empty list at least containing 1 item
${observed_length} = Get Length ${observed_posts}
# calculate expected_length of the observed_posts list
IF ${end_index} < ${NUMBER_OF_POSTS}
${expected_length} = Evaluate $end_index-$start_index
ELSE IF ${end_index} >= ${NUMBER_OF_POSTS} and ${start_index} < ${NUMBER_OF_POSTS}
${expected_length} = Evaluate $NUMBER_OF_POSTS-$start_index
ELSE
${expected_length} = Set Variable ${0}
END
Should Be Equal ${expected_length} ${observed_length}
Free Memory ${expected_posts} # (*)
Free Memory ${observed_posts} # (*)
END
Reload Library REST # (**)
END
Note that when you follow the instruction to run the test suite via ./run, you will only execute this test case (because of --include run-me-only tag in run command).
The problem
As the test case runs, the amount of memory Robot & RESTInstance use grows to gigabyte levels in a few minutes.
The Question
How can I prevent this from happening?
How can I free the memory used in inner loop's iteration?
My failed attempts to fix the problem
I added the codes marked with (*) into the test case with the following custom keyword:
#keyword
def free_memory(reference):
del reference
Note also that I use RESTInstance library to make the GET call:
Get Sliced Posts
[Documentation] start_index starts from 1 as we fetch from the API now
[Arguments] ${start_index} ${end_index}
GET /posts?_start=${start_index}&_end=${end_index}
${posts} = Output response body
[Return] ${posts}
AFAIK, RESTInstance library keeps a list of instance objects:
https://asyrjasalo.github.io/RESTinstance/#Rest%20Instances
So, this list is growing by adding an instance object per API call. So, I tried:
Reload Library REST # (**)
in the test case, once the iteration with the outermost FOR loop ended. I thought the list would be destroyed & re-created as we reload the library, but the memory consumption kept rising still.
When I run the code below I get an error message
Keyword 'BuiltIn.Log' expected 1 to 5 arguments, got 0.
for the second log to console (within the if-clause). Why? Why doesn't it "see" the variable I try to send as an argument?
Googling doesn't return anything useful. I have tried several combinations of spaces and apostrophes but nothing has worked.
Function key above F12
[Arguments] ${fkey}
${ValidFKeys}= Create List F13 F14 F15 F16 F17
log to console ${fkey}
Run Keyword If $fkey in $ValidFKeys run keywords
... log to console ${fkey}
Check the Run Keywords documentation:
By default all arguments are expected to be keywords to be executed.
What happened is Run Keywords treated Log To Console as the first keyword to be ran, then the value of ${fkey} as the second - it did not pass it as argument to the log. To overcome this, add an "AND" - so now it knows ${fkey} is an argument; if yo udon't have any other keyword to be ran, either drop Run Keywords, or use No Operation:
Run Keyword If $fkey in $ValidFKeys run keywords
... log to console ${fkey} AND No Operation
Here is my situation: I need to run number of test cases repeatedly over long period of time (Stress test + Longevity test). In the test case, there are number of events that should be passed all the time. However, I would like to catch any expected failure.
Is there a way to set robot test suite to keep executing over period of time or until it encounter a failure?
While you certainly can do an endurance test in Robot Framework, you may find that too much memory is consumed and may cause the interpreter to exit prematurely with a MemoryError. If you implement your test as a keyword, you can run your "test" many times with a for loop. In the below example, Scenario is where your test code would go. This is just a simulation that will fail after 200 runs.
*** Variables ***
${ITERATION} ${1}
*** Test Cases ***
Endurance Test
[Timeout] 4 hours
:FOR ${i} IN RANGE 1000
\ Scenario
*** Keywords ***
Scenario
Set Suite Variable ${ITERATION} ${ITERATION+1}
Run Keyword If ${ITERATION} > 200 Fail You wore me out
How to run a particular test case multiple times and display the pass and fail count under Test Statistics?
Below is the current code I have to run a test case multiple times. (The test case is implemented in a keyword and called)
*** Test Cases ***
Testcase
repeat keyword 5 Run Keyword And Continue On Failure Execute
*** Keywords ***
Execute
log Hello world!
The code is run from cmd using "pybot testcase.robot"
This code runs the test multiple times but I'm not getting the final pass/fail count in the logs.
I need to manually count the pass and fail test case repetitions.
So what modifications should I do to get the data automatically and should be seen in Test Statistics of the log also.
Instead of using "Repeat Keyword", use For loop.
Use "Run Keyword And Return Status" instead of "Run Keyword And Continue On Failure ".
*** Test Cases ***
Test Me
${fail}= Set Variable 0
:FOR ${index} IN RANGE 5
\ ${passed}= Run Keyword and Return Status Execute
\ Continue For Loop If ${passed}
\ ${fail}= ${fail} + 1
${success}= Set Variable 5 - ${fail}
Log Many Success: ${success}
Log Many fail: ${fail}