I'm trying to update large complexe json files and exit with detailled error message when detecting incoherent data (with jq 1.6).
I started to use functions and try/catch to produce a kind of java stacktrace containing input data from each level => easy, thank you JQ
But when I started to update array elements (using |=), I didn't find the solution
Here is a very simple example :
echo '{"array": [{"foo":"bar"}]}' | jq -c '.array[] |= try . catch (.)'
output : {"array":[{"__jq":0}]}
Do I made a mistake ? Is it a normal behaviour ?
Thanks for your help
The try-catch isn't really an expression, it yields no meaningful value, it merely executes some expression:
try-catch
Errors can be caught by using try EXP catch EXP. The first expression is executed, and if it fails then the second is executed with the error message. The output of the handler, if any, is output as if it had been the output of the expression to try.
emphasis mine.
So it would be wrong to use the value, you should perform the assignment within the try expression.
$ echo '{"array": [{"foo":"bar"}]}' | jq -c 'try (.array[] |= .) catch (.)'
You have stumbled upon a bug in jq 1.6. Using jq 1.5, one obtains the correct output:
{"array":[{"foo":"bar"}]}
However, the expression .array[] |= try . catch (.) is not one would ever use in practice, because if .array is a JSON array or JSON object, it just says: do nothing.
To understand try ... catch ..., it might help to consider this example:
$ jq -n 'try error("abc") catch ("The error message was " + .)'
"The error message was abc"
Related
We have a number of builtin format/escape functions: #csv, #sh, etc
… | #sh
#sh "\( … )"
It is possible to define a custom format/escape function, say #sql?
Um. Technically, yes. I don't think I could recommend it, but it does appear to be possible.
First of all, you need to understand that #csv, #sh etc aren't separate functions from jq's point of view. They're all implemented by the format/1 function. This function takes a single string argument that is the name of the format to use, e.g. "csv" or "sh". You can redefine this function, and lo and behold jq will use it for formatting!
jq -n 'def format(fstring): "XXX";[1,2,3]|#json'
"XXX"
Okay, that's not very useful. How about this?
jq -n '
def format(fstring):
if fstring=="sparkle" then
".:!" + (tostring) + "!:."
else
oldformat(fstring)
end
;
[1,2,3]|#sparkle
'
".:![1,2,3]!:."
It worked! But don't get too excited...
jq -n '
def format(fstring):
if fstring=="sparkle" then
".:!" + (tostring) + "!:."
else
oldformat(fstring)
end
;
[1,2,3]|#json
'
Uh oh, that just hangs. Although we were hoping to delegate to the original format when we don't know what to do with the format, we actually called our new format function recursively. It just keeps calling itself forever. It looks like we might be out of luck on our extremely cursed quest. However, if we read the jq manual carefully, there is a glimmer of hope:
Multiple definitions using the same function name are allowed. Each re-definition replaces the previous one for the same number of function arguments, but only for references from functions (or main program) subsequent to the re-definition. See also the section below on scoping.
Great! We can use this to save the old format function:
jq -n '
def oldformat(fstring):
format(fstring);
def format(fstring):
if fstring=="sparkle" then
".:!" + (tostring) + "!:."
else
oldformat(fstring)
end
;
[1,2,3]|(#sparkle,#json,#sh)
'
".:![1,2,3]!:."
"[1,2,3]"
"1 2 3"
I really don't recommend this: it's awkward to do, not terribly useful, and I have no idea if it's going to break something else. The format function doesn't appear to be documented so it's likely an unsupported implementation detail. But it was fascinating to find out that this is technically possible.
val SOME i = Int.fromString e
I have a line like this on my code and smlnj shows me this warning
vm.sml:84.7-84.32 Warning: binding not exhaustive
SOME i = ...
Is this bad practice? Should I use a function to handle the option or am I missing something?
If you're just working on a small script you'll run once, it's not necessarily bad practice: If Int.fromString e fails (and returns NONE instead of SOME _), then the value binding will fail and an exception will be raised to the appropriate handler (or the program will exit, if ther is no handler). To disable this warning, you can run the top-level statement (for SML-NJ 110.96): Control.MC.bindNonExhaustiveWarn := false;.
As an alternative approach, you could throw a custom exception:
val i =
case Int.fromString e
of SOME i => i
| NONE => raise Fail ("Expected string value to be parseable as an int; got: " ^ e)
The exception message should be written in a way that's appropriate to the provenance of the e value. (If e comes from command-line input, the program should tell the user that a number was expected there; if e comes from a file, the program should tell the user which file is formatted incorrectly and where the formatting error was found.)
As yet another alternative: If your program is meant to be long-running and builds up a lot of state, it wouldn't be very user-friendly if the program crashed as soon as the user entered an ill-formed string on the command line. (The user would be quite sad in this case, as all the state they built up in the program would have been lost.) In this case, you could repeatedly read from stdin until the user types in input that can be parsed as an int. This is incidentally more-or-less what the SML/NJ REPL does: instead of something like val SOME parsedProgram = SMLofNJ.parse (getUserInput ()), it would want to do something like:
fun getNextParsedProgram () =
case SMLofNJ.parse (getUserInput ())
of NONE => (print "ERROR: failed to parse\n"; getNextParsedProgram ())
| SOME parsedProgram => parsedProgram
In summary,
For a short-lived script or a script you don't intend on running often, turning off the warning is a fine option.
For a program where it's unexpected that e would be an unparseable string, you could raise a custom exception that explains what went wrong and how the user can fix it.
For longer-lived programs where better error handling is desired, you should respect the NONE case by pattern-matching on the result of fromString, which forces you to come up with some sort of error-handling behavior.
So, I have some code which is supposed to check whether a pre-check sequence has completed before I begin patching a server. I'm doing this by using an until loop and as the condition, I have a flag set to true. When the flag is set to false that means that the pre-checks are done and I can now proceed out of the until loop.
The problem is I get stuck in an infinite loop and I get an error message saying [: true: integer expression expected
I continually call a URL to conduct my checks, and it returns a file which I save. An example extract from that file would be:
inProgress":true,"status":"IN_PROGRESS","preCheckMessages":[],"precheckResultItems":[]}
The part I need to extract from this is the value from the inProgress field (which could be 'true' or 'false'). I then use this value to compare to my flag. So I extract the value from the inProgress field.
The section of code causing me an issue is as follows:
inProgress="true"
until [ "$inProgress" -eq "false" ]
do
long_URL_goes_here > jobIdCheck.txt
inProgress=$(grep -o 'inProgress.*$' jobIdCheck.txt)
word="inProgress\":"
tempVar=${inProgress##${word}}
inProgress=${tempVar%%,*}
echo inProgress value = ${inProgress}
done
echo "Pre-checks complete. Ready to apply patch"
This is where I hit the infinite loop with the error message saying [: true: integer expression expected. (I never see the "Pre-checks complete message")
But I don't understand why it would be expecting an integer? Surely the string comparison should suffice?
Is the fact that the value true in the inProgress field ISN'T encapsulated in double quotes important?
Any help will be greatly received.
first time posting here. I've a feeling that this is a really dumb question, but for some reason my code keeps failing and I just can't put my finger on what's wrong.
Here's is what I have:
*** Settings ***
Library Selenium2Library
Library OperatingSystem
Library String
Library Collections
*** Test Cases ***
Test Robot Framework Logging
#{ALLOWED}= Create List /page1 /page2 /page3
${ControllersList}= Get File ${EXEC_DIR}/Resources/controllers.txt
#{PAGES}= Split to lines ${ControllersList}
:FOR ${PAGE} IN #{PAGES}
\ Run Keyword If '${PAGE} IN #{ALLOWED}' Log Testing WARN
[Teardown] Close Browser
This is the output:
Evaluating expression ''/page1 IN [u'/page1', u'/page2', u'/page3']'' failed: SyntaxError: invalid syntax (<string>, line 1)
If I change the condition to something like this it works:
'${PAGE} == /page1'
I checked the documentation and it seems that the IN condition could be used. I'm totally lost here. Any hints? Thanks!
This is the proper way to do the expression:
Run Keyword If $PAGE in $ALLOWED Log Testing WARN
By removing the quotes and the curly braces, robot is able to treat PAGE and ALLOWED as python variables when evaluating the expression.
From the section Evaluating Expressions in the documentation for the BuiltIn library:
Starting from Robot Framework 2.9, variables themselves are automatically available in the evaluation namespace. They can be accessed using special variable syntax without the curly braces like $variable. These variables should never be quoted, and in fact they are not even replaced inside strings.
Also, when using keywords like Run Keyword If, the expression must be a valid python expression after variable substitution. Therefore, you must use the operator in rather than IN. The latter is used only by the robot :FOR statement.
Example
*** Variables ***
#{ALLOWED} /page1 /page2 /page3
*** Test Cases ***
Example that passes
${PAGE}= set variable /page1
Run Keyword If $PAGE in $ALLOWED
... pass execution ${PAGE} is allowed
fail ${PAGE} is not allowed
Example that fails
${PAGE}= set variable /page99
Run Keyword If $PAGE in $ALLOWED
... pass execution ${PAGE} is allowed
fail ${PAGE} is not allowed
The check was almost right, but you've effectively turned the Boolean expression into a String by surrounding it with the quotes. Here's the syntax that'll do it:
\ Run Keyword If $PAGE in $ALLOWED Log Testing WARN
Mind there are no curly brackets {} around the variable names - thus the check is (almost) the straight python's variable in another_variable
The documentation Evaluating Expressions does indeed specify that in construction used in the evaluation itself. An alternative approach is to use the Collections library keyword Get Match Count. This will return 0 when no results are found, and not generate a test failure.
*** Settings ***
Library Collections
*** Test Cases ***
Test Robot Framework Logging
#{ALLOWED}= Create List /page1 /page2 /page3
#{PAGES}= Create List /page1 /page3 /page5
:FOR ${PAGE} IN #{PAGES}
\ ${InList}= Get Match Count ${ALLOWED} ${PAGE}
\ Run Keyword If ('${InList}' == '0') Log To Console Page Not Allowed
\ ... ELSE IF ('${InList}' == '1') Log To Console 1 Page Found
\ ... ELSE Log To Console More pages found.
Is there a way to "die" in execution flow in an xquery file and output a nicely formatted printout of a sequence variable?
I'm trying something like:
return { fn:error(xs:QName("ERROR"), $xml) }
but that doesn't quite seem to work.
Thanks!
Based on your comment (you need it for debugging) I guess you are looking for the fn:trace function, described here http://www.xqueryfunctions.com/xq/fn_trace.html
If you want to abort the execution flow and output an error in your application you should in fact use the XQuery exception handling.
Try something like this, omitting the return if this isn't part of a FLWOR expression.
...
return fn:error((), "DEBUG", $xml)
There's no need for curly braces unless you're enclosing an expression, for example <x>{ current-time() }</x>. The return expression is not enclosed.
With MarkLogic it's best to leave the first parameter of fn:error empty. That way you don't have to worry about a QName, and anyway some folks believe that it's reserved for predefined errors. MarkLogic uses the second parameter to fill in error:code, and the third parameter for data.
For more on fn:error, see http://docs.marklogic.com/fn:error and https://github.com/robwhitby/xray/pull/11