How can I get a reliable measure of the execution time of an XQuery in eXist-db?
It seems like eXide takes into account even the render of the results in the browser, am I wrong?
eXide measures only the time required to execute the query, not to render the results in the browser or serialize the results. (To confirm, see the eXide source where queries are executed and the duration is measured: https://github.com/wolfgangmm/eXide/blob/develop/controller.xql#L155-L193. The first timestamp taken on line 159 and the 2nd on 189-90.)
You can measure the duration of your own queries using this same technique:
xquery version "3.1";
let $start-time := util:system-time()
let $query-needing-measurement := (: insert query or function call here :)
let $end-time := util:system-time()
let $duration := $end-time - $start-time
let $seconds := $duration div xs:dayTimeDuration("PT1S")
return
"Query completed in " || $seconds || "s."
Another common approach is to log this message or send it to the Monex app's console. For this, use util:log() (built-in) or console:log() (requires Monex, which if not already installed can be installed from the Dashboard > Package Manager).
Also, see the XQuery Wikibook's entry, https://en.wikibooks.org/wiki/XQuery/Timing_a_Query.
Note: Updated with suggestion by Gunther from comments below.
Related
I would like to compare a datetime value (from a database) with the current time and check whether the current time is before or after the value in the database. This is done in a Symfony project. I tried to follow the instructions on the Symfony website.
So I wrote the following Doctrine query in a Repository Class which checks for a user lockout time and checks if it still lies in the future:
$user_id = 1; // Just giving $user_id a value for this example
$qb = $this->createQueryBuilder('user')
->andWhere('user.lockout_time > :time')
->setParameter('time', date("Y-m-d H:i:s"))
->andWhere('user.user_id = :user_id')
->setParameter('user_id', $user_id);
$query = $qb->getQuery();
echo $query->getSQL();
die;
When running this, both Where clauses contain "?" in the comparative value (e.g. WHERE user.lockout_time > ?). Obviously I want the actual values to be used in the query.
Initially I thought the date() function might be the issue, but even if I just use the :user_id I get the above error.
If I write ->andWhere('user.user_id = 1') I get the desired result.
If I replace :time with some date in the format 'Y-m-d H:i:s', I get the message "Error: Expected end of string..." (with the ... being the value for the Hour).
So both my setParameter() lines are not passing the values set in them. What am I overlooking?
Edit:
The suggested question here is not a duplicate. That just helps me see the query that is sent off. It was helpful in preparation of this question.
So here my own answer after some (= way too much) time of digging around.
The "?" are escaped values and do not represent what is actually in the query (which is another reason the above link doesn't help). To resolve this I resorted to monitoring the MySQL general log.
Here how to get to it, if someone has the same question. This log shows the actual SQL query.
I am developing a quizing game in Delphi and I would like to have a timer so that players don´t have unlimited time to answer the questions. I am using the function "Time" to get the current time but I don´t know how to convert it to something like an integer so that when let´s say 10 seconds have passed the player loses it´s chance. It would look like something like this:
Var
CurrentTime,Aux:TDateTime;
Begin
CurrentTime:=Time; //Current Time is assigned to a global variable.
Aux:=CurrentTime;
While (Aux-10000<=CurrentTime) do
Begin
if (Answer:=True) then //If the answer is already given by the player we break the while loop
Break;
Aux:=Time; //We refresh the auxilary variable
if (Aux-10000>=CurrentTime) then //We check if 10 seconds have passed
Begin
showmessage('You have taken too much time, your turn is lost');
exit; //We leave the script
end;
end;
end;
The problem is I can´t do arithmetic operations in DateTimes, as far as I know, So I need a different method for comparing the 2 different time instances. Any help would be appreciated, thanks!
TDate, TTime, and TDateTime are implemented using floating-point numbers, so you can perform arithmetic operations on them.
But you really shouldn't, in this case. The DateUtils unit has many functions for working with date/time values, eg:
uses
..., DateUtils;
var
StartTime, Aux: TDateTime;
begin
StartTime := Time();
Aux := StartTime;
...
while (not Answer) and (MillisecondsBetween(Aux, StartTime) < 10000) do
begin
Sleep(0);
Aux := Time();
end;
if (not Answer) then
begin
ShowMessage('You have taken too much time, your turn is lost');
Exit; //We leave the script
end;
...
end;
Note that this is not really a good use for TDateTime. Your calculations are relying on the system clock in local time being accurate and unchanging, but it can be changed dynamically while your code is running (user updates, network updates, daylight saving time change, etc), throwing off the results.
Consider using TStopWatch instead. It is intended for exactly this kind of use-case (determining elapsed time between actions), eg:
uses
..., System.Diagnostics;
var
SW: TStopWatch;
begin
SW := TStopWatch.StartNew;
...
while (not Answer) and (SW.ElapsedMilliseconds < 10000) do
Sleep(0);
if (not Answer) then
begin
ShowMessage('You have taken too much time, your turn is lost');
Exit; //We leave the script
end;
...
end;
Or, you could use TEvent instead, and have the answer signal the event when ready, eg:
uses
..., SyncObjs;
var
AnsweredEvent: TEvent;
...
// when the answer is submitted:
AnsweredEvent.SetEvent;
...
begin
AnsweredEvent.ResetEvent;
...
if AnsweredEvent.WaitFor(10000) <> wrSignaled then
begin
ShowMessage('You have taken too much time, your turn is lost');
Exit; //We leave the script
end;
end;
initialization
AnsweredEvent := TEvent.Create;
finalization
AnsweredEvent.Free;
I have written a few applications like this, using a TTimer. The timer's interval is set to 1000, which is equivalent to 1 second (you can use a different value); every time the timer's OnTimer event executes, a global variable is incremented which is then checked against the time limit (10 seconds?); if the variable equals this limit then first the timer is stopped, then the code performs whatever is necessary to transfer to the next person, or next question.
There should be similar code that executes when the person enters an answer as this code too needs to first save the answer then transfers to the next person. This portion should also stop the timer.
The 'show next question' part should restart the timer and reset the global variable only after the next question has been displayed, as it might take some time for it to be fetched.
I was wondering if there is a way for me use a result from search function in MarkLogic and use it in multiple transformation queries that I have.
For eg.
let $uris := cts:uris(("/example/"),(),cts:element-query(xs:QName("cd:documentTitle"),cts:element-value-query(xs:QName("cd:id"),"abc")))
return (fn:count($uris), $uris)[1 to 20]
The above query say returns me URI for 20 documents. How can I save this result or re-use this result for multiple transformations that I have. All of them working on the same result set but performing different tasks.
Any help would be greatly appreciated.
I'm going to guess you're using CORB to process data based on that return.
You could instead run your query in QConsole or similar and write the following in your URIs module:
let $uris = ("/1.xml", "/2.xml", ...)
return (fn:count($uris), $uris)
I have some long running (> 1 minute) tasks that use to be run fine through an xQuery in REST. However, we have now placed these servers behind an Amazon load balancer and because of the way Amazon load balancers work, no single query can have a duration exceeding 29 seconds. Amazon will just timeout the query.
NOTE: There is no control over this
So, the solution we came up with is for the xQuery to just trigger a scheduled task to run which works fine. I had thought this would work and it does with one exception --- using something like this:
declare function jobs:create-job ($xquery-resource as xs:string, $period as xs:integer, $job-name as xs:string, $job-parameters as element()?, $delay as xs:integer, $repeat as xs:integer) as xs:boolean {
let $jobstatus := scheduler:schedule-xquery-periodic-job($xquery-resource, $period, $job-name, $job-parameters, $delay, $repeat)
return $jobstatus
And setting the $repeat to 0, it runs once but the job named "$job-title" is still in the list of scheduled jobs as "COMPLETE". Trying to run the code again will error. The error is apparently that another job with the same name cannot be created. If I change the job name it will run, so I know it is the name that is causing the error. The schedule log shows:
<scheduler:job name="Create Vault">
<scheduler:trigger name="Create Vault Trigger">
<expression>30000</expression>
<state>COMPLETE</state>
<start>2019-08-22T20:07:14.775Z</start>
<end/>
<previous>2019-08-22T20:07:14.775Z</previous>
<next/>
<final/>
</scheduler:trigger>
</scheduler:job>
Now, is there a different way that we could execute an xQuery triggered only once so that we do not have this job name issue? Or a way to tell the scheduled task to self-destruct and remove itself? Otherwise we would need to write some complicated code to create another task to delete the job after it is run (or maybe the create task code should delete any $job-name jobs) or leave them behind and use some GUID on the name.
Update I
The cleanest way we found is this (essentially if the job exists when we go to create it, delete it and then create another one):
declare function jobs:create-job ($xquery-resource as xs:string, $period as xs:integer, $job-name as xs:string, $job-parameters as element()?, $delay as xs:integer, $repeat as xs:integer) as xs:boolean {
let $cleanjob:= if(count(scheduler:get-scheduled-jobs()//scheduler:job[#name=$job-name]) > 0) then scheduler:delete-scheduled-job($job-name) else true()
let $jobstatus := if($cleanjob) then scheduler:schedule-xquery-periodic-job($xquery-resource, $period, $job-name, $job-parameters, $delay, $repeat) else false()
return $jobstatus
};
I would say we should also check the status and not delete if it is running ... however these tasks are built to run on-demand but the on-demand is likely once a day at most. The longest task is formatting about 3000 document to PDFSs which takes maybe 20 mins. It is not likely anyone would clash with another running task but we could add that to be sure.
Or should the answer be examining util:eval-async() but that is confusing as it does not really say that it just "shells" out the execution. If it threads it out and waits for the thread then that will not work either.
I use airflow python operators to execute sql queries against a redshift/postgres database. In order to debug, I'd like the DAG to return the results of the sql execution, similar to what you would see if executing locally in a console:
I'm using psycop2 to create a connection/cursor and execute the sql. Having this logged would be extremely helpful to confirm the parsed parameterized sql, and confirm that data was actually inserted (I have painfully experiences issues where differences in environments caused unexpected behavior)
I do not have deep knowledge of airflow or the low level workings of the python DBAPI, but the pscyopg2 documentation does seem to refer to some methods and connection configurations that may allow this.
I find it very perplexing that this is difficult to do, as I'd imagine it would be a primary use case of running ETLs on this platform. I've heard suggestions to simply create additional tasks that query the table before and after, but this seems clunky and ineffective.
Could anyone please explain how this may be possible, and if not, explain why? Alternate methods of achieving similar results welcome. Thanks!
So far I have tried the connection.status_message() method, but it only seems to return the first line of the sql and not the results. I have also attempted to create a logging cursor, which produces the sql, but not the console results
import logging
import psycopg2 as pg
from psycopg2.extras import LoggingConnection
conn = pg.connect(
connection_factory=LoggingConnection,
...
)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur = conn.cursor()
sql = """
INSERT INTO mytable (
SELECT *
FROM other_table
);
"""
cur.execute(sql)
I'd like the logger to return something like:
sql> INSERT INTO mytable (
SELECT ...
[2019-07-25 23:00:54] 912 rows affected in 4 s 442 ms
Let's assume you are writing an operator that uses postgres hook to do something in sql.
Anything printed inside an operator is logged.
So, if you want to log the statement, just print the statement in your operator.
print(sql)
If you want to log the result, fetch the result and print the result.
E.g.
result = cur.fetchall()
for row in result:
print(row)
Alternatively you can use self.log.info in place of print, where self refers to the operator instance.
Ok, so after some trial and error I've found a method that works for my setup and objective. To recap, my goal is to run ETL's via python scripts, orchestrated in Airflow. Referring to the documentation for statusmessage:
Read-only attribute containing the message returned by the last command:
The key is to manage logging in context with transactions executed on the server. In order for me to do this, I had to specifically set con.autocommit = False, and wrap SQL blocks with BEGIN TRANSACTION; and END TRANSACTION;. If you insert cur.statusmessage directly following a statement that deletes or inserts, you will get a response such as 'INSERT 0 92380'.
This still isn't as verbose as I would prefer, but it is a much better than nothing, and is very useful for troubleshooting ETL issues within Airflow logs.
Side notes:
- When autocommit is set to False, you must explicitly commit transactions.
- It may not be necessary to state transaction begin/end in your SQL. It may depend on your DB version.
con = psy.connect(...)
con.autocommit = False
cur = con.cursor()
try:
cur.execute([some_sql])
logging.info(f"Cursor statusmessage: {cur.statusmessage})
except:
con.rollback()
finally:
con.close()
There is some buried functionality within psycopg2 that I'm sure can be utilized, but the documentation is pretty thin and there are no clear examples. If anyone has suggestions on how to utilize things such as logobjects, or returning join PID to somehow retrieve additional information.