Zorba collections: a simple directory of xml files - collections

I have a directory of xml files I'd like to use as a collection, specifically, to look at $my_collection/of/things where each file has <things>.
I am using Zorba.
This results in error retrieving resource (syntax problem?):
variable $my_collection := fn:collection("file://localhost/path/to/my/*.xml");
I also read over the documentation on Zorba's Data Definition Facility... seems like overkill for this singular purpose. Or, not?

Edited question - putting everything into an actual Zorba collection as opposed to a nodelist. You might want to choose between static and dynamic collections for your purposes.
F.xq:
module namespace X = "urn:xml-files";
import module namespace ddl = "http://www.zorba-xquery.com/modules/store/static/collections/ddl";
import module namespace dml = "http://www.zorba-xquery.com/modules/store/static/collections/dml";
import module namespace file = "http://expath.org/ns/file";
import module namespace x = "http://www.zorba-xquery.com/modules/xml";
declare namespace ann = "http://www.zorba-xquery.com/annotations";
declare collection X:files as node()*;
declare variable $X:uri := xs:QName('X:files');
declare %ann:nondeterministic function X:get-xml-list($dir-name as xs:string) {
let $files := file:list($dir-name, true())
for $filename in $files
where ends-with($filename, ".xml")
return $filename
};
declare %ann:sequential function X:load-files($names as xs:string*) {
ddl:create($X:uri);
dml:insert-nodes($X:uri, for $filename in $names return x:parse(file:read-text($filename),()));
};
declare %ann:sequential function X:load-directory($dir-name as xs:string) {
X:load-files(X:get-xml-list($dir-name))
};
x.xq:
xquery version "3.0" encoding "utf-8";
import module namespace X = "urn:xml-files" at "F.xq";
import module namespace dml = "http://www.zorba-xquery.com/modules/store/static/collections/dml";
X:load-directory("C:\Projects-dir...-with-xmls\");
dml:collection($X:uri)
References:
http://www.zorba.io/documentation/2.9/zorba/xqddf.html
http://en.wikipedia.org/wiki/Zorba_(XQuery_processor)
http://www.w3.org/TR/xpath-functions-30/#func-ends-with
http://www.zorba.io/documentation/2.9/modules/expath.org_ns_file.html
create collection, add document to zorba using command line?

Try to use
collection("file:///localhost/path/to/my?select=*.xml;recurse=yes")

Solution added by user Scala William is the way to go but his solution is a bit outdated. Here's his code updated for zorba 3.0:
F.xq:
module namespace X = "urn:xml-files";
import module namespace ddl = "http://zorba.io/modules/store/static/collections/ddl";
import module namespace dml = "http://zorba.io/modules/store/static/collections/dml";
import module namespace fs = "http://expath.org/ns/file";
import module namespace x = "http://zorba.io/modules/xml";
declare namespace an = "http://zorba.io/annotations";
declare collection X:files as node()*;
declare variable $X:uri := xs:QName('X:files');
declare %an:nondeterministic function X:get-xml-list($dir-name as xs:string)
{
let $files := fs:list($dir-name, fn:false(),"*.xml")
for $filename in $files
return concat($dir-name,$filename)
};
declare %an:sequential function X:load-files($names as xs:string*)
{
ddl:create($X:uri);
dml:insert($X:uri, for $filename in $names return x:parse(fs:read-text($filename),()));
};
declare %an:sequential function X:load-directory($dir-name as xs:string)
{
X:load-files(X:get-xml-list($dir-name))
};
Please note the call to concat(...) instead of just $filename in the return at X:get-xml-list(). Also, the call to insert-nodes(...) has changed to insert(...) at X:load-files().
Also please consider changing fn:false() to fn:true() in fs:list(...) at X:get-xml-list() if you need your listing recursive.

Related

How to write Content Transformations using xquery in marklogic?

I have a javascript function(.sjs) which will return a string.
Now, I've to write a xquery transformation which will be triggered through dmsdk. Xquery function will accept a json doc, inside this transformation i need to trigger that javascript function which will return a string. Now, I need to use that string as a uri to load documents..
I've no idea on xquery!!
myXquery transformation function(jsonDoc)
{
/////////////////
myJavaScript function- which return a string
load the doc using this string as uri
}
Use Below query:
declare namespace local ="local";
declare function local:transformation($json-input, $json-uri as xs:string)
{
xdmp:save(concat("D:\", $json-uri), $json-input)(:Here I saved file on local path, If you want to save on Marklogic-DB use-: xdmp:document-insert( $json-uri, $json-input) :)
};
(:1:)
let $input := '{"menu":{"id":"file", "value":"File"}}'
for $json-input in $input
(:2 Call here your Javascript function for json-uri :)
let $json-uri :="json_first.json"
return local:transformation(xdmp:unquote($json-input), $json-uri)

docFX - Remove namespace prefix in API Reference TOC

I am generating API reference from C# project.
The project is part of big solution and has long name convention for assemblies and namespaces, so every namespace in project goes like [CompanyName].[System].[Area].[Module].[...] (e.g. MyBiz.CRM.Sales.Analytics.Persistence.Common and MyBiz.CRM.Sales.Analytics.Persistence.Sql).
Since all namespaces in project start with MyBiz.CRM.Sales. and I generate reference for each system and area separately, I want to exclude MyBiz.CRM.Sales. in TOC on left side and only mention it in title/header.
Is it possible in docFX or I need to write custom server side post-build event script?
Thanks in advance
I also had the same problem and solved it by writing a template. The only thing it does is overriding the preTransform hook which you need to specify in a file named toc.extension.js. There I strip the long namespace prefix. As I have a TOC in several layers, I do that recursively. You can most probably hard-code it for the level you need. My code looks as follows:
exports.preTransform = function (model) {
// replace recursively the root namespace by ""
transformItem(model, 1);
return model;
function transformItem(item, level) {
if (item.name) {
item.name = item.name.replace("Some.Very.Long.Namespace.Prefix.", '');
} else {
item.name = null;
}
if (item.items && item.items.length > 0) {
var length = item.items.length;
for (var i = 0; i < length; i++) {
transformItem(item.items[i], level + 1);
};
}
}
}
Afterwards, I simply specify the own template in addition to the default template in docfx.json which causes the hook to be called. That works as follows:
"build": {
"template": [
"default",
"/path/to/your/template/folder"
]
}
References: Docfx: How to create a custom template

Extbase: Command Controller is not getting Registered

I am new to Extbase. There are two xml files that I want to access. One is abc.dll?type=xml from which I select the ID and then use this Id to fetch the values of other XML file(xyzzy.dll?type=xml and save all the data to the db. I want to create a task in Extbase and run it from command line.
Below is my code:
ext_localconf.php
$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['extbase']['commandControllers'][] = 'TYPO3\Example\Command\XMLFetcherCommandController';
XMLFetcherCommandController:
namespace TYPO3\Example\Command;
class XMLFetcherCommandController extends \TYPO3\CMS\Extbase\Mvc\Controller\CommandController{
/**
* xmlload
*
* #return void
*/
public function findCommand(){
$path="http://abc.dll?type=room&content=xml";
$readXML=file_get_contents($path);
$xml = simplexml_load_string($readXML, "SimpleXMLElement",LIBXML_NOCDATA); $objectManager=\TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Extbase\\Object\\ObjectManager');
$classesRepository = $objectManager->get('TYPO3\\Example\\Domain\\Repository\\ClassesRepository');
$json=json_encode($xml);
$xmlArray=json_decode($json,true);
$serialized_array = serialize($xmlArray);
$unserialized_array = unserialize($serialized_array);
$rooms = $unserialized_array['Rooms']['Room'];
foreach($rooms as $room){
$fetchXML= \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\Example\\Domain\\Model\\Classes');
$fetchXML->setRoomKey($room['Key']);
$fetchXML->setRoomID($room['ID']);
$classesRepository->add($fetchXML);
$newpaths[]='http:/xyz.dll?content=xml&type=room&id='.$room['ID'];
foreach($newpaths as $newpath){
$readLessons=file_get_contents($newpath);
$xmlLessons= simplexml_load_string($readLessons, "SimpleXMLElement",LIBXML_NOCDATA);
$objectManager=\TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Extbase\\Object\\ObjectManager');
$classesRepository = $objectManager->get('Example\\Example\\Domain\\Repository\\ClassesRepository');
$json=json_encode($xml);
$xmlArray=json_decode($json,true);
$serialized_array = serialize($xmlArray);
$unserialized_array = unserialize($serialized_array);
$Lessons = $unserialized_array['Lesson'];
foreach ($Lessons as $Lesson) {
$fetchXMLNew= \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('Example\\Example\\Domain\\Model\\Classes');
$date=date('Ymd',strtotime($Lesson['Date']));
$start=date('Hi',strtotime($Lesson['Start']));
$finish=date('Hi',strtotime($Lesson['Finish']));
$startdatetime=date('YmdHi',strtotime("$date$start"));
$finishdatetime=date('YmdHi',strtotime("$date$finish"));
$fetchXMLNew->setStartdatetime($startdatetime);
$fetchXMLNew->setEnddatetime($finishdatetime);
$classesRepository->add($fetchXML);
}
}
}
$classesRepository->persistAll();
}
}
}
?>
When I run- php cli_dispatch.phpsh extbase help - from cmd, I could not see my command controller which means it is not registered properly. Could you suggest if this is the right way to do it? At First, I created a Service and tried to call it, but since there was a lot of data it was taking a lot of time.
Maybe it does not work, becouse your class can not be loaded with autoloader?
If a class is not available, it will not be registerd.
(#see CommandManager->getAvailableCommands !class_exists($className)).
I think you need to escape the back slashes (=> Double Slashes)
$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['extbase']['commandControllers'][]
= 'TYPO3\\Example\\Command\\XMLFetcherCommandController';
If this not work, you can also check the Namespace of your class. Why does it start with "TYPO3"?
If this also not work, you can try load your class hardcoded with require_once in AdditionalConfiguration.php to figure out whats wrong

Does eXist-db compression:zip function add XML declaration

I have an XQuery function to convert a group of XML files to HTML and Zip them. It runs a trasform on each file to create <entry> elements.
Starting with that function:
declare function xport:make-sources( $path as xs:string) as item()* {
for $article in collection(xmldb:encode-uri($path))
let $docnum := $article/article/div[#class = 'content']/#doc/string()
return
<entry name="{concat($docnum,'.html')}" type='text' method='store'>
{transform:transform($article, doc("/db/EIDO/data/edit/xsl/doc-html.xsl"), <parameters/>)}
</entry>
} ;
Given the input, I run the XQuery to just show me the result of the transformation ... and I see this (exactly what I would expect):
<entry name="LS01.html" type="text" method="store">
<html>
<head>
<style>
body {
font-family: Arial;
}
article img {
width:50%;
}
...
You will note the this entry and all of them have no XML Declaration at all.
But now let's put it all together and send those entries to compression. This is all inside a web application. The full XQuery is this:
xquery version "3.0";
import module namespace transform = "http://exist-db.org/xquery/transform";
declare namespace xport = "http://www.xportability.com";
declare function xport:make-sources( $path as xs:string) as item()* {
for $article in collection(xmldb:encode-uri($path))
let $docnum := $article/article/div[#class = 'content']/#doc/string()
return
<entry name="{concat($docnum,'.html')}" type='text' method='store'>
{transform:transform($article, doc("/db/EIDO/data/edit/xsl/doc-html.xsl"), <parameters/>)}
</entry>
} ;
let $path := request:get-parameter("path", "")
let $filename := request:get-parameter("filename", "")
let $col := xport:make-sources($path)
return
response:stream-binary(
xs:base64Binary(compression:zip($col,true()) ),
'application/zip',
$filename
)
Everything works, I get a ZIP file of all the documents that have been transformed to HTML from the XML.
BUT, when I look at the actually file in the ZIP, it has this:
<?xml version="1.0" encoding="UTF-8"?>
<html>
<head>
The XML Declaration is not on any of the entries to ZIP. It does not exist anywhere (as it couldn't) in the list of entries. But the action of zipping them apparently is adding the declaration. I see no other reason or way. Even specifying omit-xml-declaration or changing the output type in the XSL to text or HTML makes no difference. And this is of course, because the entry list to zip is shown above and that shows the declaration is not there after the transformation.
The files in the ZIP have an added XML declaration, period.
Is there some workaround?
The XML declaration is introduced implicitly in your query when the contents of your zip-bound <entry> elements are passed to the compression:zip() function. I'd advise setting serialization options explicitly using the fn:serialize() function. Here is sample code showing how to achieve the result you describe:
xquery version "3.1";
let $node := <html><head/><body><div><h1>Hello World!</h1></div></body></html>
let $serialized := serialize($node, map { "method": "xml", "indent": true(),
"omit-xml-declaration": true() })
let $entries := <entry name="test.html" type="text" method="store">{$serialized}</entry>
let $filename := "test.zip"
return
response:stream-binary(
compression:zip($entries, true()),
'application/zip',
$filename
)
Saving this query into the database at a location like /db/apps/my-app/test.xq and calling it by pointing your web browser at http://localhost:8080/exist/apps/my-app/test.xq will cause your browser to download test.zip. Opening this zip file will reveal a test.html file absent the XML declaration:
<html>
<head/>
<body>
<div>
<h1>Hello World!</h1>
</div>
</body>
</html>
Stepping back to the fundamentals, the presence or absence of the XML declaration in XQuery is toggled via the omit-xml-declaration serialization parameter. To omit the XML declaration globally for an entire query, you can place this set of declarations in the prolog of your query:
declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization";
declare option output:method "xml";
declare option output:omit-xml-declaration "yes";
Or, when serializing locally within a portion of a query, you can pass this same set of parameters to the fn:serialize function as a map (the method used in the code sample above):
fn:serialize($node, map { "method": "xml", "omit-xml-declaration": true() } )
(There is also an XML syntax for the 2nd options parameter.)
The current version of eXist (v4.0.0) and recent versions (probably since v3.6.0 or so) support all of the options above, and all versions support a somewhat more compact eXist-specific serialization facility, using the exist:serialize option expressed as a string consisting of key=value pairs:
declare option exist:serialize "method=xml omit-xml-declaration=yes";
You can set eXist's default serialization behavior in your conf.xml configuration file. The defaults in conf.xml can be overridden with the methods above. Serialization behavior over different interfaces in eXist, such as WebDAV or XML-RPC, typically respect the defaults set in conf.xml, but these defaults can be overridden on a per-interface basis; for example, see the documentation on serialization over eXist's WebDAV interface.

Manually use precompiled handlebars templates

How to manually use the precompiled handlebars.js templates?
Let's say, we have
source = "<p>Hello, my name is {{name}}</p>"
data = { name: "Joe" }
Currently, I have
template = Handlebars.compile(source)
render: -> template(data)
The source is coming from the database, and in order to cut down on the compilation time, I want to use a compilation step, precompiling the template server side with Handlebars.precompile(source) and then using something like:
template = precompiled_template
render: -> precompiled_template(data)
The precompiled_template is a string with function definition, so that doesn't work.
Also, I've found that Hanlebars.compile(source)() == Handlebars.precompile(source), but after browsing the source codes of handlebars, it's compilers and runtime, I'm still not sure how to achieve this.
if you did not find the right question till now, the answer is pretty simple.
Handlebars comes with a C pre-compiler on the command line, if you can access your shell you can simple just compile your templates each separated or merge them together into one file.
you can install Handlebars via npm / or build it on your system.
on the shell you can access the help file
$> Handlebars [ENTER]
You will see a help file like >
- f --output Output File etc etc ..
- m --min Minimize Output
$> Handlebars mysupertemplate.handlebars -f compiled.js -m ("-m" if
you want to minify the js file)
To run Handlebars.compile in the browser is a huge loss in performance, so it's worth a try to precompile on the server before sending the file to the browser.
To register Handlebars templates in your browser you have to load them like this:
var obj = {"foo":"bar"}
var template = require("./mytemplate-file.js") // require.js example
template = template(Handlebars) // Pass Handlebars Only if the template goes mad asking for his Father
var html = Handlebars.templates[template-name](obj)
For example if you have more then one template registered in the "templates-file" you will be able to access after the require call all templates by name using
var html = Handlebars.templates["templateName"]({"foo":"bar"});
You can go even further by register all the know helper within the file and / or making custom helpers for partials like so..
*// This will be the helper in you app.js file*
Handlebars.registerHelper("mypartials", function(partialName, data) {
var partial = Handlebars.registerPartial(partialName) || data.fn
Handlebars.partials[partialName] = partial
})
And in your template file you can put this...
{{#mypartial "divPartial"}}
<div id="block"><h2>{{foo}}</h2><p>{{bar}}</p></div>
{{/mypartial}}
{{#mypartial "formPartial"}}
<form id="foo"><input type="text" value="{{foo}}" name="{{bar}}"></form>
{{/mypartial}}
Now you can access this files by calling
var html = Handlebars.partials["divPartial"]({"foo":"bar","bar":"foo"})
var formHtml = Handlebars.partials["formPartial"]({"bar":"bar","foo":"foo"})
Hope this helped a bit ..
This speed test http://jsperf.com/handlebars-compile-vs-precompile/3 gave the answer.
Apparently, one solution is to eval() that resulting string and it will work.
The code is
var data = { name: "Greg" };
var source = "<p>Howdy, {{ name }}</p>";
eval("var templateFunction = " + Handlebars.precompile(source));
var template = Handlebars.template(templateFunction);
template(data);
=> "<p>Howdy, Greg</p>"
Of course one needs to be careful with eval and probably a better solution exists.

Resources