I need a lucid explanation of why XDMP-EXTIME happens in Marklogic. In my case it's happening during a search(read operation). In the exception message a line from the code is being printed:
XDMP-EXTIME: wsssearch:options($request, $req-config) -- Time limit exceeded
This gives me the impression that the execution does not go beyond that line. But it seems that it's a pretty harmless line of code ,it does not fetch any data from the DB, just sets certain search options. How can I pin point which part of the code is causing this? I have heard that increasing the max time limit of the task server solves such problems but that's not an option with me. Please let me know how such problems are tackled. It would be very very hard for me to show you the code base.Still hoping to hear something helpful from you guys.
The error message can sometimes put you on the wrong foot because of lazy evaluation. The execution can actually be further down the road than the error message seems to indicate. Could be one line, could be several. Look for where the returned value is being used.
Profiling can sometimes help getting a clearer picture of where most time is spent, but the lazy evaluation can throw things off here as well.
The bottom-line meaning of the message is pretty simple: the execution of your code takes too long. The actual search in which the options are being used is the most likely candidate of where it goes wrong.
If you are using cts:search or search:search under the covers, then that should normally perform well. A search typically gets slow when you end up returning many results, e.g. don't apply pagination. Search:search does that by default however.
A search can also get slow if you are running your search in update mode. You could potentially end up having MarkLogic trying to apply many (unnecessary) read locks. Put the following declaration in your search endpoint code, or xquery main module that does the search:
declare option xdmp:update "false";
HTH!
You could try profiling the code to see what specifically is taking so long. This might require increasing the session time limit temporarily to prevent the timeout from occurring while profiling. Note that unless this is being executed on the Task Server via xdmp:spawn or xdmp:spawn-fucntion, you would need to increase the value on the App Server hosting the script.
If your code is in a module, the easiest thing to do is make a call to the function that times out from Query Console using the Profile tab. Alternatively, you could begin the function with prof:enable(xdmp:request()) and later output the contents of prof:report(xdmp:request()) to a file on the filesystem, or insert it somewhere in the database.
Related
I'm facing a weird phenomenal when putting items to DynamoDB.
It seems like if putting items too fast, DynamoDB can't put the whole data to the table(kinda like a broken data, it has partial attributes but with some weird values)?
I'm using the AWS JavaScript SDK to putting items, no errors shown up, everything seemed to work fine, but once I checked the data from web console, some of the inserted data was broken. Is this is related to write capacity units? (but no errors tell me it's caused by the write capacity units..) I could confirm the spike of my write capacity units was about 60/min, the setting is "on-demand".
I tried to slow down the putting speed with one second interval and with the exactly same data, the data could be inserted correctly...
Anyone knows why and how to fix this..?
The answer is no: If DynamoDB decides to throttle your requests because you exceeded your provisioned capacity or exceed their own hardware's capacity or whatever - it will refuse to do whole requests, or in the case of BatchWriteItems do some of the writes and not do others (and it will tell you which it did and which it didn't). DynamoDB will never write part of a request or corrupt parts of one attribute.
If you are seeing that, the most likely culprit is a bug in your own code that does the write. Maybe your own code is not thread-safe, so if your code is trying to prepare two items for writing concurrently, the code doing this preperation has a data race and results in a corrupt item to be written. Obviously, it is also possible that DynamoDB has a bug causing this, but it can't be as simple a bug as "writing more than 60 items a minute causes corruption" - if this were the case, everyone would have encountered this bug...
The procedure is in a package. When a procedure is called the client function got hanged. So they are asking to set a timeout for the API. Is it possible?
Not directly. However, I believe you should be able to accomplish what you are looking for, though I have never done it and do not have time to put together a proof of concept right now. Here is an outline of what you would need to do:
1) See this answer for a trick (kludge?) to get Oracle's resource manager to limit by execution time: https://stackoverflow.com/a/43532330/5174436
2) Modify the resource plan consumer group mapping to only apply to resource limit to a given MODULE_NAME.
3) Set the given module name in your procedure via DBMS_APPLICATION_INFO.SET_MODULE.
I offer no guarantees it will work. It is merely what I would try if I were faced with your need.
Actually, that is not true.. what I would really do if I were faced with your need is figure out why the procedure is "hanging" and fix it.
I'm making a couple of calls in a loop into a custom API to update a table in an SQL database, and I found that if I perform the second one immediately, the first one does not actually change the database. If I wait one second between calls, it works.
These two calls were originally only made after individual UI button presses, so this is likely the first time anyone ever tried doing it twice in a row so quickly after each other. We had a feature request that now requires it though.
A hardcoded sleep() is good for tracking down the issue, but it really goes against the grain to consider that a solution. So I'd like to know what needs to be done in ODBC to ensure a previous operation on a table has completed so that the next one won't fail. But again, I'm a total ODBC noob, so I'm not familiar with how its API is supposed to be used (and of course the author of this code left the company over 6 months ago).
Tracking through the layers of API code, I found
Everything ends in Windows ODBC calls.
A single handle for a single connection (from SQLAllocHandle ) is used for all calls.
The query in question is roughly UPDATE table_name SET ... BEGIN INSERT INTO table_name (...) VALUES (...); END
The call sequence for each query seems to be:
SQLCancel();
check(SQLPrepare());
check(SQLExecute());
SQLCancel();
Where check() is:
if (code != SQL_SUCCESS && code != SQL_SUCCESS_WITH_INFO) {
exception();
}
The main issue I see here is that warnings will be totally ignored. But my noob reading of things is that if the query is still running at the end of the call, it should have gone into exception() with something like HYT00 (Timeout expired), right?
The only other thing I can think is that another thread might be calling this API on the same connection, and cancelling the operation with its SQLCancel(). I'll go triple check that, but I'm pretty sure that's not happening.
I'm fairly new to xProc and xPath, but I've been asked to solve the following problem:
Step 2 receives data via the secondary port from step 1. Step 2 contains a p:for-each, which saves a document into a folder for each element that passes the for-each.
(Part A)
These documents (let's say I receive 6 documents from for-each) lay in the same directory and get filtered by p:directory-list and are eventually stored in one single document, containing the whole path of every document the for-each created. (Part B)
So far, so good.
The problem is that Part A seems to be too slow. Part B already tries to read the data Step A
stores while the directory is still empty. Meaning, I'm having a performance / synchronization problem.
And now comes the question:
Is it possible to let the pipeline wait and to let it continue as soon as a certain event occurs?
That's what I'm imagining:
Step B waits as long as necessary until the directory, which Step A stores the data in, is no longer empty. I read something about
dbxml:breakpoint, but unfortunately I couldn't find more information than the name and
a short description of what it seems to do:
Set a breakpoint, optionally based upon a condition, that will cause pipeline operation to pause at the breakpoint, possibly requiring user intervention to continue and/or issuing a message.
It would be awesome if you know more about it and could give an example of how it's used. It would also help if you know a workaround or another way to solve this problem.
UPDATE:
After searching google for half an eternity, I found SMIL which's timesheets seem to do the trick. Has anyone experience with throwing XML / xProc and SMIL together?
Back towards the end of 2009 I proposed the concept of 'Orchestrating XProc with SMIL' http://broadcast.oreilly.com/2009/09/xproc-and-smil-orchestrating-p.html in a blog post on the O'Reilly Network.
However, I'm not sure that this (XProc + Time) is the solution to your problem. It's not entirely clear, to me, from you description what's happening. Are you implying that you're trying to write something to disk and then read it in a subsequent step? You need to keep stuff in the pipeline in order to ensure you can connect outputs to subsequent inputs.
Can I call a .xqy page from another .xqy page in Marklogic ?
There are several ways to execute another .xqy, but the most obvious is probably by using xdmp:invoke. That calls the .xqy, waits for its results and returns them on the spot in your code. You can also call a single function using the combination xdmp:function and xdmp:apply. You could also mess around with xdmp:eval, but that is usually a last resort.
Another strategy could be to use xdmp:http-get, but then the execution runs in a different transaction, so would always commit. You would also need to know the url of the other .xqy, which need some knowledge about whether, and how url are rewritten in the app server (not by default).
Running other .xqy without waiting for results is also possible with xdmp:spawn. Particularly usefull for dispatching heavy load of for instance content processing. Dispatching batches of 100 to 1000 docs is quite common. Keep an eye on the task queue size though..
HTH!