Creating and running a "scheduled task" once - xquery

I have some long running (> 1 minute) tasks that use to be run fine through an xQuery in REST. However, we have now placed these servers behind an Amazon load balancer and because of the way Amazon load balancers work, no single query can have a duration exceeding 29 seconds. Amazon will just timeout the query.
NOTE: There is no control over this
So, the solution we came up with is for the xQuery to just trigger a scheduled task to run which works fine. I had thought this would work and it does with one exception --- using something like this:
declare function jobs:create-job ($xquery-resource as xs:string, $period as xs:integer, $job-name as xs:string, $job-parameters as element()?, $delay as xs:integer, $repeat as xs:integer) as xs:boolean {
let $jobstatus := scheduler:schedule-xquery-periodic-job($xquery-resource, $period, $job-name, $job-parameters, $delay, $repeat)
return $jobstatus
And setting the $repeat to 0, it runs once but the job named "$job-title" is still in the list of scheduled jobs as "COMPLETE". Trying to run the code again will error. The error is apparently that another job with the same name cannot be created. If I change the job name it will run, so I know it is the name that is causing the error. The schedule log shows:
<scheduler:job name="Create Vault">
<scheduler:trigger name="Create Vault Trigger">
<expression>30000</expression>
<state>COMPLETE</state>
<start>2019-08-22T20:07:14.775Z</start>
<end/>
<previous>2019-08-22T20:07:14.775Z</previous>
<next/>
<final/>
</scheduler:trigger>
</scheduler:job>
Now, is there a different way that we could execute an xQuery triggered only once so that we do not have this job name issue? Or a way to tell the scheduled task to self-destruct and remove itself? Otherwise we would need to write some complicated code to create another task to delete the job after it is run (or maybe the create task code should delete any $job-name jobs) or leave them behind and use some GUID on the name.
Update I
The cleanest way we found is this (essentially if the job exists when we go to create it, delete it and then create another one):
declare function jobs:create-job ($xquery-resource as xs:string, $period as xs:integer, $job-name as xs:string, $job-parameters as element()?, $delay as xs:integer, $repeat as xs:integer) as xs:boolean {
let $cleanjob:= if(count(scheduler:get-scheduled-jobs()//scheduler:job[#name=$job-name]) > 0) then scheduler:delete-scheduled-job($job-name) else true()
let $jobstatus := if($cleanjob) then scheduler:schedule-xquery-periodic-job($xquery-resource, $period, $job-name, $job-parameters, $delay, $repeat) else false()
return $jobstatus
};
I would say we should also check the status and not delete if it is running ... however these tasks are built to run on-demand but the on-demand is likely once a day at most. The longest task is formatting about 3000 document to PDFSs which takes maybe 20 mins. It is not likely anyone would clash with another running task but we could add that to be sure.
Or should the answer be examining util:eval-async() but that is confusing as it does not really say that it just "shells" out the execution. If it threads it out and waits for the thread then that will not work either.

Related

Why does Hangfire wait for 15s every few seconds when polling sql server for jobs?

I’ve inherited a system that uses Hangfire with sql server job storage. Usually when a job is scheduled to be run immediately we notice it takes a few seconds before it’s triggered.
Looking at SQL Profiler when running in my dev environment, the SQL run against Hangfire db looks like this -
exec sp_executesql N'delete top (1) JQ
output DELETED.Id, DELETED.JobId, DELETED.Queue
from [HangFire].JobQueue JQ with (readpast, updlock, rowlock, forceseek)
where Queue in (#queues1) and (FetchedAt is null or FetchedAt < DATEADD(second, #timeout, GETUTCDATE()))',N'#queues1 nvarchar(4000),#timeout float',#queues1=N'MYQUEUENAME_master',#timeout=-1800
-- Exactly the same SQL as above is executed about 6 times/second for about 3-4 seconds,
-- then nothing for about 2 seconds, then:
exec sp_getapplock #Resource=N'HangFire:recurring-jobs:lock',#DbPrincipal=N'public',#LockMode=N'Exclusive',#LockOwner=N'Session',#LockTimeout=5000
exec sp_getapplock #Resource=N'HangFire:locks:schedulepoller',#DbPrincipal=N'public',#LockMode=N'Exclusive',#LockOwner=N'Session',#LockTimeout=5000
exec sp_executesql N'select top (#count) Value from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key and Score between #from and #to order by Score',N'#count int,#key nvarchar(4000),#from float,#to float',#count=1000,#key=N'recurring-jobs',#from=0,#to=1596053348
exec sp_executesql N'select top (#count) Value from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key and Score between #from and #to order by Score',N'#count int,#key nvarchar(4000),#from float,#to float',#count=1000,#key=N'schedule',#from=0,#to=1596053348
exec sp_releaseapplock #Resource=N'HangFire:recurring-jobs:lock',#LockOwner=N'Session'
exec sp_releaseapplock #Resource=N'HangFire:locks:schedulepoller',#LockOwner=N'Session'
-- Then nothing is executed for about 8-10 seconds, then:
exec sp_executesql N'update [HangFire].Server set LastHeartbeat = #now where Id = #id',N'#now datetime,#id nvarchar(4000)',#now='2020-07-29 20:09:19.097',#id=N'ps12345:19764:fe362d1a-5ee4-4d97-b70d-134fdfab2b87'
-- Then about 500ms-2s later I get
exec sp_executesql N'delete top (1) JQ ... -- i.e. Same as first query
The update LastHeartbeat query is only there every second time (from just a brief inspection, maybe that’s not exactly right).
It looks like there’s at least 3 threads running the DELETE query against JQ, since I can see several RPC:Starting before the RPC:Completed, suggesting they’re being executed in parallel instead of sequentially.
I don’t know if that’s normal but seems weird as I thought we had just one ‘consumer’ of the jobs.
I only have one Queue in my dev environment, although in live we’d have 20-50 I’d guess.
Any suggestions on where I should look for the configuration that’s causing:
a) the 8-10s pause between checking for jobs
b) the number of threads that are checking for jobs - it seems like I have too many
After writing this I realised we were using an old version so I upgraded from 1.5.x to 1.7.12, upgraded the database, and changed the startup config to this:
app.UseHangfireDashboard();
GlobalConfiguration.Configuration
.UseSqlServerStorage(connstring, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
UseRecommendedIsolationLevel = true,
PrepareSchemaIfNecessary = true, // Default value: true
EnableHeavyMigrations = true // Default value: false
})
.UseAutofacActivator(_container);
JobActivator.Current = new AutofacJobActivator(_container);
but if anything the problem is now worse. Or the same but faster: 20 calls to delete top (1) JQ... happen within about 1s now, then the other queries, then a 15s wait, then it starts all over again.
To be clear, the main problem is that if any jobs are added during that 15s delay then it'll take the remainder of that 15s before my job is executed. A second problem I think is it's hitting SQL Server more than needed: 20 times in a second is a bit much, for my needs at least.
(Cross-posted to hangfire forums)
If you don't set QueuePollInterval then Hangfire with sql server storage defaults to polling every 15s. So the first thing to do if you have this problem is set QueuePollInterval to something smaller, e.g. 1s.
But in my case even when I set that it wasn't having any effect. The reason for that was calling app.UseHangfireServer() before I was calling GlobalConfiguration.Configuration.UseSqlServerStorage() with the SqlServerStorageOptions.
When you call app.UseHangfireServer() it uses the current value of JobStorage.Current. My code had set that:
var storage = new SqlServerStorage(connstring);
JobStorage.Current = storage;
then later called
app.UseHangfireServer()
then later called
GlobalConfiguration.Configuration
.UseSqlServerStorage(connstring, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
UseRecommendedIsolationLevel = true,
PrepareSchemaIfNecessary = true,
EnableHeavyMigrations = true
})
Reordering it to use SqlServerStorageOptions before app.UseHangfireServer() means the SqlServerStorageOptions take effect.
I would suggest checking the Hangfire BackgroundJobServerOptions to see what polling interval you have set up there. This will define the time before the hangfire server will check to see if there are any jobs in queue to execute.
From the documentation
Hangfire Docs
Hangfire Server periodically checks the schedule to enqueue scheduled jobs to their queues, allowing workers to
execute them. By default, check interval is equal to 15 seconds, but you can change it by setting the SchedulePollingInterval property on the options you pass to the BackgroundJobServer constructor:
var options = new BackgroundJobServerOptions
{
SchedulePollingInterval = TimeSpan.FromMinutes(1)
};
var server = new BackgroundJobServer(options);

Way to determine if within a running scheduled task?

How can I tell, within the graph processing logic, if it's being executed through a Scheduled task rather than through user interaction?
PXProcessing doesn't seem to have much, nor does the records within the AUSchedule table
Reason: If I'm in an interactive session, I want to redirect to multiple screens for the document(s) I've created. In a Scheduled task, I don't want to clutter up the server with these Redirects
Take a look through the code repository at SOShipmentEntry. Search for SOInvoiceEntry and it will get you to the Action function where it creates the invoice. You can see that they call the adapter.MassProcess function to see if it is running in a process or not and throws exceptions, sets info, or errors based on the status.
You can also see the AllowRedirect flag in use.
Here is an example of redirecting to the invoice at the end of a shipment invoice creation:
SOInvoiceEntry ie = PXGraph.CreateInstance<SOInvoiceEntry>();
......
......
if (adapter.AllowRedirect && !adapter.MassProcess && created.Count > 0)
{
using (new PXTimeStampScope(null))
{
ie.Clear();
ie.Document.Current = ie.Document.Search<ARInvoice.docType, ARInvoice.refNbr>(((ARInvoice)created[0]).DocType, ((ARInvoice)created[0]).RefNbr, ((ARInvoice)created[0]).DocType);
throw new PXRedirectRequiredException(ie, "Invoice");
}
}

How to fake DateTime in a phpunit test?

ssI write a unittest using Symfony's KernelTestCase and have to test for a functionality, which happens only at a certain tie of the day (extra early or extra late).
So when I let my test run a noon of course nothing happens. How can I fake my system time to pretend that it has a different time and my test case is triggered.
I tried around with Symfony's ClockMock class but it does not work.
https://symfony.com/doc/current/components/phpunit_bridge.html#clock-mocking
This is my test code:
use Symfony\Bridge\PhpUnit\ClockMock;
use \DateTime;
/**
* testUserAchievedEarlyBirdTrophy
* #group time-sensitive
*/
public function testUserAchievedEarlyBirdTrophy()
{
ClockMock::withClockMock(strtotime('2018-11-05 01:00:00'));
echo (new DateTime())->format('Y-m-d H:m:s');
$user8Id = $this->user8->getId();
$progressSaveRequest = new ProgressSaveRequest($user8Id, $this->content_1_1->getId());
$this->progressService->saveProgress($progressSaveRequest);
$this->assertTrue($this->loggerCreator->hasDebugThatContains(
'Early Bird'
));
}
the echo give me today's date: 2019-02-01 16:02:06
I also had the feeling that ClockMock is rather used to skip time for e.g. testing caching instead of sleep().
What am I doing wrong?
The listener configuration is in place in my phpunit.xml
Calling bin/simple-phpunit causes a lot of installation happening.
Can't I use the normal phpunit?
Are there any other options to fake the time of the day?
Following the link you included in your post, the first paragraph says:
The ClockMock class provided by this bridge allows you to mock the
PHP's built-in time functions time(), microtime(), sleep() and
usleep(). Additionally the function date() is mocked so it uses the
mocked time if no timestamp is specified. Other functions with an
optional timestamp parameter that defaults to time() will still use
the system time instead of the mocked time.
(Emphasis added.)
This means that your call of
echo (new DateTime())->format('Y-m-d H:m:s');
is expected to give the system time, not the mocked time.
Change it to
echo date('Y-m-d H:m:s');
in order to match the requirements of ClockMock and get the mocked time.
Note: I've never used ClockMock myself, but just looking at the documentation this should be a first step to see if this resolves your problem.

MarkLogic query taking too long

I am running a query in for loop which does update on a document. I have set the time-limit 36000.
However the query is running for more than 50 hours. I have used 20 threads and all those threads are taken by above query (since it is for loop).
Even after killing each query form admin and even after restarting the ML server, the next 20 query comes into queue and again those query running and occupying all threads.
let $_ := xdmp:set-request-time-limit(36000)
for $each in local:some-update-function()
let $document:= xdmp:invoke-function(function() {
local:another-update-function()
}, $constants:UPDATE_AUTO_COMMIT)
return $document

Get execution time of XQuery in eXist

How can I get a reliable measure of the execution time of an XQuery in eXist-db?
It seems like eXide takes into account even the render of the results in the browser, am I wrong?
eXide measures only the time required to execute the query, not to render the results in the browser or serialize the results. (To confirm, see the eXide source where queries are executed and the duration is measured: https://github.com/wolfgangmm/eXide/blob/develop/controller.xql#L155-L193. The first timestamp taken on line 159 and the 2nd on 189-90.)
You can measure the duration of your own queries using this same technique:
xquery version "3.1";
let $start-time := util:system-time()
let $query-needing-measurement := (: insert query or function call here :)
let $end-time := util:system-time()
let $duration := $end-time - $start-time
let $seconds := $duration div xs:dayTimeDuration("PT1S")
return
"Query completed in " || $seconds || "s."
Another common approach is to log this message or send it to the Monex app's console. For this, use util:log() (built-in) or console:log() (requires Monex, which if not already installed can be installed from the Dashboard > Package Manager).
Also, see the XQuery Wikibook's entry, https://en.wikibooks.org/wiki/XQuery/Timing_a_Query.
Note: Updated with suggestion by Gunther from comments below.

Resources