I am using below code to copy documents related to specified collection from one database to another database.
In my try block I am returning to delete all the documents after successful copy. And in the catch just to print the exception .
But it gives me a syntax error which I am not able to figure out. Can someone please help me in this -
xquery version "1.0-ml";
for $doc in cts:search(doc(), cts:collection-query('Test'))
let $docuri := $doc/base-uri()
let $collections := xdmp:document-get-collections($docuri)
let $permissions := xdmp:document-get-permissions($docuri)
let $document-insert-options :=
<options xmlns="xdmp:document-insert">
<permissions>{$permissions}</permissions>z
<collections>{
<collection>{$collections}</collection>
}
</collections>
</options>
let $db-name := "Final"
let $invoke-function-options :=
<options xmlns="xdmp:eval">
<database>{ xdmp:database($db-name) }</database>
<commit>auto</commit>
</options>
return
try {
xdmp:invoke-function(
function(){ xdmp:document-insert($docuri, $doc, $document-insert-options)},
$invoke-function-options
);
return xdmp:document-delete($doc)
}
catch ($exception) {
"Problem loading file, received the following exception: ",
$exception }
Michael and Rob both have good points, but there are a few more issues.
MarkLogic does complain about the extra 'z' in the $document-insert-options. Get rid of that.
MarkLogic will also complain about the fact you are passing in $doc instead of $docuri into xdmp:document-delete. So change that as well.
Last is the semi-colon bit. MarkLogic does support that, but only on top-level. Using Michael suggestion, insert and delete might happen in parallel. Using Rob's suggestion of an extra 'let $_ :=', and removing ';' will cause MarkLogic to first insert, and delete second. I think that is what you are after.
Running the following against a random database from QC ('Fab' will do), works fine on my end:
xquery version "1.0-ml";
xdmp:document-insert('/test.json', object-node{"foo":"bar"}, map:entry("collections", "Test"))
;
xquery version "1.0-ml";
for $doc in cts:search(doc(), cts:collection-query('Test'))
let $docuri := $doc/base-uri()
let $collections := xdmp:document-get-collections($docuri)
let $permissions := xdmp:document-get-permissions($docuri)
let $document-insert-options :=
<options xmlns="xdmp:document-insert">
<permissions>{$permissions}</permissions>
<collections>{
<collection>{$collections}</collection>
}
</collections>
</options>
let $db-name := "Documents"
let $invoke-function-options :=
<options xmlns="xdmp:eval">
<database>{ xdmp:database($db-name) }</database>
<commit>auto</commit>
</options>
return
try {
let $_ := xdmp:invoke-function(
function(){ xdmp:document-insert($docuri, $doc, $document-insert-options)},
$invoke-function-options
)
return xdmp:document-delete($docuri)
}
catch ($exception) {
"Problem loading file, received the following exception: ",
$exception }
HTH!
What is the error?
Offhand this block looks wrong:
let $document-insert-options :=
<options xmlns="xdmp:document-insert">
<permissions>{$permissions}</permissions>z
<collections>{
<collection>{$collections}</collection>
}
</collections>
</options>
You have a z floating out there after </permissions> which could be breaking schema validation of the options node.
This also could be wrong:
catch ($exception) {
"Problem loading file, received the following exception: ",
$exception }
You may need to explicitly return a Sequence like so:
catch ($exception) {
("Problem loading file, received the following exception: ",
$exception)}
This also seems odd to me:
try {
xdmp:invoke-function(
function(){ xdmp:document-insert($docuri, $doc, $document-insert-
options)},
$invoke-function-options
);
return xdmp:document-delete($doc)
}
This breaks FLWOR by trying to use a multi-statement transaction with the ; .
Instead try:
try {
let $_ := xdmp:invoke-function(
function(){ xdmp:document-insert($docuri, $doc, $document-insert-
options)},
$invoke-function-options
)
return xdmp:document-delete($doc)
}
XQuery doesn't have semicolon-delimited statements, nor does it have any instruction or expression that starts with the keyword "return".
xdmp:invoke-function(
function(){ xdmp:document-insert($docuri, $doc, $document-insert-options)},
$invoke-function-options
);
return xdmp:document-delete($doc)
}
catch ($exception) {
"Problem loading file, received the following exception: ",
$exception }
Replace the "; return" by a comma (",").
Marklogic may also have special rules for invoking functions with side-effects; I wouldn't know about that area.
Related
I have written an Xquery to that gets executed at the time of when incremental backup is in progress. I know the backup status returns three possible values -
completed, in-progress and failed. Not sure the exact value of last one but anyways this is my xquery -
xquery version "1.0-ml";
declare function local:escape-for-regex
( $arg as xs:string? ) as xs:string {
replace($arg,
'(\.|\[|\]|\\|\||\-|\^|\$|\?|\*|\+|\{|\}|\(|\))','\\$1')
} ;
declare function local:substring-before-last
( $arg as xs:string? ,
$delim as xs:string ) as xs:string {
if (matches($arg, local:escape-for-regex($delim)))
then replace($arg,
concat('^(.*)', local:escape-for-regex($delim),'.*'),
'$1')
else ''
} ;
let $server-info := doc("/config/server-info.xml")
let $content-database :="xyzzy"
let $backup-directory:=$server-info/configuration/server-info/backup-directory/text()
let $backup-latest-dateTime := xdmp:filesystem-directory(fn:concat( $backup-directory,'/',$content-database))/dir:entry[1]/dir:filename/text()
let $backup-latest-date := fn:substring-before($backup-latest-dateTime,"-")
let $backup-info := cts:search(/,cts:element-value-query(xs:QName("directory-name"),$backup-latest-date))
let $new-backup := if($backup-info)
then fn:false()
else fn:true()
let $db-bkp-status := if($new-backup)
then (xdmp:database-backup-status(())[./*:forest/*:backup-path[fn:contains(., $backup-latest-dateTime)]][./*:forest/*:incremental-backup eq "false"]/*:status)
else (xdmp:database-backup-status(())[./*:forest/*:backup-path[fn:contains(., $backup-latest-dateTime)]][./*:forest/*:incremental-backup eq "true"][./*:forest/*:incremental-backup-path[fn:contains(., fn:replace(local:substring-before-last(xs:string(fn:current-date()), "-"), "-", ""))]]/*:status)
return $db-bkp-status
We maintain a configuration file that stores backup status. If there is a new full backup day then $backup-info will return nothing. If it is daily incremental backup day then it will return the config. I'm using it just to check if todays backup is new full or incremental. For incremental day $backup-info is false and so it goes to the last line i.e. else condition. this doesn't return anything for incremental backups. Neither completed nor in-progress. I wonder how markLogic picks up the timestamp. Please assist on this.
Feel free to provide your own xquery from scratch. I can update mine.
I even took out the Job id and search in the output of the function xdmp:database-backup-status(()) but that job id too doesn't exist in the result set.
MarkLogic provides the Admin modules to provide much of the information you are attempting to get via other methods. The Admin UI modules (typically found in /opt/MarkLogic/Modules/MarkLogic/Admin/Lib) contains a lot of helpful code that can be adapted to get these sorts of details. In this case I would refer to database-status-form.xqy
define function db-mount-state(
$fstats as node()*,
$fcounts as node()*,
$dbid as xs:unsignedLong)
{
let $times := $fstats/fs:last-state-change,
$ls := max($times),
$since :=
if (not(empty($ls)))
then concat(" since ", longDate($ls), " ", longTimeSecs($ls))
else ""
return concat(database-status($dbid,$fstats,$fcounts),$since)
}
define function backup-recov-state($fstats as node()*)
{
if(empty($fstats/fs:backups/fs:backup)
and
empty($fstats/fs:restore))
then
"No backup or restore in progress"
else
if(empty($fstats/fs:backups/fs:backup))
then
"Restore in progress (see below for details)"
else
"Backup in progress (see below for details)"
}
... Call the functions against your database, then pull the details from the elements you want:
let $last-full-backup := max($fstats/fs:last-backup)
let $last-incremental-backup : = max($fstats/fs:last-incr-backup
return ($last-full-backup, $last-incremental-backup)
This is just some sample code snippets, not executable, but it should get you moving in the right direction.
Below is the code snippet I am using for one of the functionality
declare function local:matchCounts($Id as xs:string, $status as xs:string) as xs:int {
xdmp:estimate(cts:search(/count, cts:and-query((
cts:element-attribute-value-query(xs:QName("count"), xs:QName("Id"), $Id, "exact"),
cts:element-attribute-value-query(xs:QName("child"), xs:QName("MatchStatus"), $status, "exact")
)), "unfiltered"))
};
declare function local:saveCountsMatchC($Id as xs:string) {
let $evenCount := local:matchCounts($Id, "even")
let $oddCount := local:matchCounts($Id, "odd")
return ($evenCount, $oddCount)
};
declare function local:matchingProcess($Id as xs:string) {
let $total-records := 1000
let $batch-size := 50
let $pagination := 0
let $bs :=
for $records in 1 to fn:ceiling($total-records div $batch-size )
let $start := fn:sum($pagination + 1)
let $end := fn:sum($batch-size + $pagination)
let $_ := xdmp:set($pagination, $end)
return
xdmp:spawn-function
(
function() {
for $each at $pos in ($start to $end)
let $id := sem:uuid-string()
let $xml := if(($pos mod 2) eq 0) then <count Id='{$Id}'><child MatchStatus='even'></child></count>
else <count Id='{$Id}'><child MatchStatus='odd'></child></count>
return xdmp:document-insert(concat("/", $id, ".xml"), $xml)
},
<options xmlns="xdmp:eval"><result>{fn:true()}</result><commit>auto</commit><update>true</update></options>
)
let $_ := $bs
return local:saveCountsMatchC($Id)
};
local:matchingProcess("1")
The requirement over here is to iterate 1000 documents using batch size of 50, so basically I am using spawn function to create 20 batches of size 50 which inserts 1000 documents in my database.
Once those documents are inserted, I need to read those documents in same transaction. Here 500 documents have MatchStatus='odd' and 500 documents have MatchStatus='even'
The query should return (500,500) as output; Instead it returns (0,0)
I am using <result>{fn:true()}</results> option so that my next statement waits for all spawn task to be completed, but its not happeneing.
Can anybody help me with the requirement?
Note: Need to insert 1000 documents and then read them in same function call only
Your code that executes the spawns does not perform updates itself, so will run in so-called query mode. In query mode only updates from before the start of the code are visible.
You could try running in update mode (declare option xdmp:transaction-mode "update";), but usually it is easier to just spawn or eval the counting/reading of your updates as well. E.g. wrap the xdmp:estimate in an xdmp:spawn-function with result true as well.
HTH!
I am trying to update a document in a different Database then my current DB. But it is giving me the below error-
XDMP-UPEXTNODES: xdmp:node-replace(fn:doc("/C:/Users/Downloads/abc.csv-0-2")/*:envelope/*:root/*:Status, <Status>1000</Status>) -- Cannot update external nodes
I am using the below code-
let $temp :=
for $i in $result
let $error := $i/*:envelope/*:ErrorMessage
let $status := $i/*:envelope/*:Status
return
if(fn:exists($i) eq fn:true()) then (
xdmp:invoke-function(
function() {
xdmp:node-replace($status,<Status>1000</Status>),
xdmp:node-replace($error,<ErrorMessage>Change Error in other Database-2</ErrorMessage>)
},
<options xmlns="xdmp:eval">
<database>{xdmp:database("DATABASE-2")}</database>
</options>))
else ()
I want to update the Error and Status node of my Database-2.
$result is the document i fetched from Database-2.
This code i am running from Database-1
Any Suggestions ?
You cannot pass database nodes as variable for updating purposes like that. Instead you should pass through the database uri, and get a fresh copy of the element you'd like to update inside the invoked function. Maybe you can push a bit more logic inside the invoked function to make that easier. Something like:
for $i in $result
let $uri := xdmp:node-uri($i)
return xdmp:invoke-function(function() {
let $doc := fn:doc($uri)
let $error := $doc/*:envelope/*:ErrorMessage
let $status := $doc/*:envelope/*:Status
return if(fn:exists($doc) eq fn:true()) then (
xdmp:node-replace($status, <Status>1000</Status>),
xdmp:node-replace($error, <ErrorMessage>Change Error in other Database-2</ErrorMessage>)
) else ()
}, map:entry("database", xdmp:database("DATABASE-2")))
Be careful though. It sounds like $i is pointing to the actual document in Database-2 as well, and it could easily result in dead-locks; the invoking query could be putting a read lock on $i, causing the invoked function to be unable to update it.
HTH!
I have a REST endpoint and it needs to proces a long list of codes. Because this may trigger time-outs I try to use spawn-function and do the magic in the background. But it looks like the spawn-function is holding the 200 OK response from my REST endpoint, so it's not really spawning.
I've added the log lines to check where it strands. All log lines pop up in the debug log.
With small amounts of data, this works fine. With a larger set (60k codes) it fails.
After changing the code to spawn the function for each item in $text, so 60k spawns, I get this error:
2015-07-28 10:20:02.326 Debug: Forest::insert: STRLF3-content-001-1 XDMP-INMMFULL: In-memory storage full; list: table=5%, wordsused=3%, wordsfree=95%, overhead=1%; tree: table=8%, wordsused=3%, wordsfree=97%, overhead=0%
Inserted data:
{
ProjectID: 102124,
Text: "2311\n2253\n2312\n6626\n2253\n1234"
}
Calling the spawn proces:
(: ======================================================================= :)
(: ! Load Transactions into seperate XML files :)
(: ======================================================================= :)
declare
%roxy:params("")
function strlf:post(
$context as map:map,
$params as map:map,
$input as document-node()*
) as document-node()?
{
map:put($context, "output-types", "application/json"),
xdmp:set-response-code(200, "OK"),
document {
(: Get project ID :)
let $_ := xdmp:log('TransTest - stap1', 'debug')
let $project := json:transform-from-json($input)/ns:ProjectID
let $_ := xdmp:log('TransTest - stap2', 'debug')
let $codes := json:transform-from-json($input)/ns:Text
(: Clean current project :)
let $_ := xdmp:log('TransTest - stap3', 'debug')
let $uridir := fn:concat('/app/transactie/', $project/text(), '/', '*')
let $_ := xdmp:log('TransTest - stap4', 'debug')
let $kill := xdmp:document-delete(cts:uri-match($uridir))
(: Spawn the trannies :)
let $_ := xdmp:log('TransTest - stap5', 'debug')
(: return 'ja' :)
let $_ := xdmp:spawn-function(strlf:spawner($project, $codes, $uridir),
<options xmlns="xdmp:eval">
<transaction-mode>update-auto-commit</transaction-mode>
</options>)
return 'done'
}
};
Function strlf:spawner:
declare private function strlf:spawner(
$project,
$codes,
$uridir
)
{
(: Tokenize on lines :)
let $text := fn:tokenize($codes, fn:codepoints-to-string(10))
let $loop :=
for $regel in $text
let $tokregel := fn:tokenize($regel, ",")
let $intvalue :=
if (fn:contains($regel, ","))
then fn:substring-after($regel, "€")
else 1
let $code :=
if (fn:contains($regel, ","))
then $tokregel[1]
else $regel
(: Build map of maps, p4 should be postcode :)
let $map := map:map()
let $_ := map:put($map, 'code', $code)
let $_ := map:put($map, 'p4', fn:substring($code[1], 1, 4))
let $_ := map:put($map, 'value', $intvalue)
let $_ := map:put($map, 'projectid', $project/text())
(: Create unverified random doc id :)
let $docid := fn:string(xdmp:random(1000000000000))
(: Build URI :)
let $uridoc := fn:concat('/app/transactie/', $project/text(), '/', $docid, '.xml')
(: Save transaction document and skip header :)
return
(if (map:get($map, 'code') != 'CODE')
then xdmp:document-insert
(
$uridoc,
<transaction xmlns='http://www.dikw.nl/transactions' projectid='{map:get($map, 'projectid')}' code='{map:get($map, 'code')}' p4='{map:get($map, 'p4')}'>
<value>{map:get($map, 'value')}</value>
</transaction>
)
else ())
(: Empty return :)
return $loop
};
Correct, you have strlf:spawner($project, $codes, $uridir) as first argument to xdmp:spawn-function, causing it to get executed, and the result being passed into xdmp:spawn-function. And since the spawner function returns an empty sequence, no error is being thrown by spawn-function.
The fix is pretty simple, wrap your spawner call in an anonymous function:
let $_ := xdmp:spawn-function(function () { strlf:spawner($project, $codes, $uridir) },
<options xmlns="xdmp:eval">
<transaction-mode>update-auto-commit</transaction-mode>
</options>)
HTH!
i try to split my incoming documents using "Information Studio Flows" (MarkLogic v 8.0-1.1). The problem is in "Transform" section.
This is my importing documents. For simplicity i reduce it content to one stwtext-element
<docs>
<stwtext id="RD-10-00258" update="03.2011" seq="RQ-10-00001">
<head>
<ti>
<i>j</i>
</ti>
<ff-list>
<ff id="0103"/>
</ff-list>
</head><p>
Symbol für die
<vw idref="RD-19-04447">Stromdichte</vw>
.
</p>
</stwtext>
</docs>
This is my "xquery transform" content:
xquery version "1.0-ml";
(: Copyright 2002-2015 MarkLogic Corporation. All Rights Reserved. :)
(:
:: Custom action. It must be a CPF action module.
:: Replace this text completely, or use it as a template and
:: add imports, declarations,
:: and code between START and END comment tags.
:: Uses the external variables:
:: $cpf:document-uri: The document being processed
:: $cpf:transition: The transition being executed
:)
import module namespace cpf = "http://marklogic.com/cpf"
at "/MarkLogic/cpf/cpf.xqy";
(: START custom imports and declarations; imports must be in Modules/ on filesystem :)
(: END custom imports and declarations :)
declare option xdmp:mapping "false";
declare variable $cpf:document-uri as xs:string external;
declare variable $cpf:transition as node() external;
if ( cpf:check-transition($cpf:document-uri,$cpf:transition))
then
try {
(: START your custom XQuery here :)
let $doc := fn:doc($cpf:document-uri)
return
xdmp:eval(
for $wpt in fn:doc($doc)//stwtext
return
xdmp:document-insert(
fn:concat("/rom-data/", fn:concat($wpt/#id,".xml")),
$wpt
)
)
(: END your custom XQuery here :)
,
cpf:success( $cpf:document-uri, $cpf:transition, () )
}
catch ($e) {
cpf:failure( $cpf:document-uri, $cpf:transition, $e, () )
}
else ()
by running of snippet, i take the error:
Invalid URI format
and long description of it:
XDMP-URI: (err:FODC0005) fn:doc(fn:doc("/8122584828241226495/12835482492021535301/URI=/content/home/admin/Vorlagen/testing/v10.new-ML.xml")) -- Invalid URI format: "
j
Symbol für die
Stromdichte
"
In /18200382103958065126.xqy on line 37
In xdmp:invoke("/18200382103958065126.xqy", (xs:QName("trgr:uri"), "/8122584828241226495/12835482492021535301/URI=/content/home/admi...", xs:QName("trgr:trigger"), ...), <options xmlns="xdmp:eval"><isolation>different-transaction</isolation><prevent-deadlocks>t...</options>)
$doc = fn:doc("/8122584828241226495/12835482492021535301/URI=/content/home/admin/Vorlagen/testing/v10.new-ML.xml")
In /MarkLogic/cpf/triggers/internal-cpf.xqy on line 179
In execute-action("on-state-enter", "http://marklogic.com/states/initial", "/8122584828241226495/12835482492021535301/URI=/content/home/admi...", (xs:QName("trgr:uri"), "/8122584828241226495/12835482492021535301/URI=/content/home/admi...", xs:QName("trgr:trigger"), ...), <options xmlns="xdmp:eval"><isolation>different-transaction</isolation><prevent-deadlocks>t...</options>, (fn:doc("http://marklogic.com/cpf/pipelines/14379829270688061297.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline), fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline/p:state-transition[1]/p:default-action, fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline/p:state-transition[1])
$caller = "on-state-enter"
$state-or-status = "http://marklogic.com/states/initial"
$uri = "/8122584828241226495/12835482492021535301/URI=/content/home/admi..."
$vars = (xs:QName("trgr:uri"), "/8122584828241226495/12835482492021535301/URI=/content/home/admi...", xs:QName("trgr:trigger"), ...)
$invoke-options = <options xmlns="xdmp:eval"><isolation>different-transaction</isolation><prevent-deadlocks>t...</options>
$pipelines = (fn:doc("http://marklogic.com/cpf/pipelines/14379829270688061297.xml")/p:pipeline, fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline)
$action-to-execute = fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline/p:state-transition[1]/p:default-action
$chosen-transition = fn:doc("http://marklogic.com/cpf/pipelines/15861601524191348323.xml")/p:pipeline/p:state-transition[1]
$raw-module-name = "/18200382103958065126.xqy"
$module-kind = "xquery"
$module-name = "/18200382103958065126.xqy"
In /MarkLogic/cpf/triggers/internal-cpf.xqy on line 320
i thought, it was a problem with "Document setting" in "load" section of "Flow editor"
URI=/content{$path}/{$filename}{$dot-ext}
but if i remove it, i recive the same error.
i have no idea what to do. i am really new. please help
First of all, Information Studio has been deprecated in MarkLogic 8. I would also recommend very much looking in to the aggregate_record feature of MarkLogic Content Pump:
http://docs.marklogic.com/guide/ingestion/content-pump#id_65814
Apart from that, there are several issues with your code. You are calling fn:doc twice, effectively trying to interpret the doc contents as a uri. There is an unnecessary xdmp:eval wrapping the FLWOR statement, which expects a string as first param. I think you can shorten it to (showing inner part of the action only):
(: START your custom XQuery here :)
let $doc := fn:doc($cpf:document-uri)
for $wpt in $doc//stwtext
return
xdmp:document-insert(
fn:concat("/roempp-data/", fn:concat($wpt/#id,".xml")),
$wpt
)
(: END your custom XQuery here :)
HTH!
very many thanks #grtjn and this is my approach. Practically it is the same solution
(: START your custom XQuery here :)
xdmp:log(fn:doc($cpf:document-uri), "debug"),
let $doc := fn:doc($cpf:document-uri)
return
xdmp:eval('
declare variable $doc external;
for $wpt in $doc//stwtext
return (
xdmp:document-insert(
fn:concat("/roempp-data/", fn:concat($wpt/#id,".xml")),
$wpt,
xdmp:default-permissions(),
"roempp-data"
)
)'
,
(xs:QName("doc"), $doc),
<options xmlns="xdmp:eval">
<database>{xdmp:database("roempp-tutorial")}</database>
</options>
)
(: END your custom XQuery here :)
Ok, now it works. It is fine, but i found, that after the loading is over, i see in MarkLogic two documents:
my splited document "/rom-data/RD-10-00258.xml" with one root element "stwtext" (as desired)
origin document "URI=/content/home/admin/Vorlagen/testing/v10.new-ML.xml" with root element "docs"
is it possible to prohibit insert of origin document ?