mapreduce javascript function error - riak

I was unable to get the erlang function running so trying with Javascript as below :
curl -XPOST http://localhost:8098/mapred \
-H "Content-Type: application/json" \
-d #- \
<<EOF
{
"inputs":"logs",
"query":[{
"map":{
"language":"javascript",
"source":"function(riakObject, keydata, arg) {
var m = riakObject.values[0].data.match(/^INFO.*cart/);
return [(m ? m.length : 0 )];
}"
},
"reduce":{
"language":"javascript",
"source":"function(values, arg){
return [values.reduce(
function(total, v){ return total + v; }, 0)
];
}"
}
}]
}
EOF
seems not working with JS as well. Shell just hangs and does not return at all. Please suggest.
** UPDATE **
Today when i tried i see the following error :
An error occurred parsing the "query" field.
["Unrecognized format of query phase:\n ",
[123,
[34,<<"map">>,34],
58,
[123,
[34,<<"language">>,34],
58,
[34,<<"javascript">>,34],
44,
[34,<<"source">>,34],
58,
[34,
<<"function(riakObject, keydata, arg) { var m = riakObject.values[0].data.match(/^INFO.*Milk/); return [(m ? m.length : 0 )]; }">>,
34],
125],
44,
[34,<<"reduce">>,34],
58,
[123,
[34,<<"language">>,34],
58,
[34,<<"javascript">>,34],
44,
[34,<<"source">>,34],
58,
[34,
<<"function(values, arg){ return [values.reduce( function(total, v){ return total + v; }, 0) ]; }">>,
34],
125],
125],
"\n\nValid formats are:\n {\"map\":{...spec...}}\n {\"reduce\":{...spec...}}\n {\"link:{...spec}}\n"]

The query element should be a list of objects, with "map" and "reduce" being in discreet objects. Your JSON has them as properties of the same object.
This worked for me:
curl -XPOST http://localhost:8098/mapred -H "Content-Type: application/json" -d '{
"inputs":"logs",
"query":[
{"map":{
"language":"javascript",
"source":"function(riakObject, keydata, arg) {
var m = riakObject.values[0].data.match(/^INFO.*cart/);
return [(m ? m.length : 0 )];
}"
}},
{"reduce":{
"language":"javascript",
"source":"function(values, arg){
return [values.reduce(
function(total, v){ return total + v; }, 0)
];
}"
}}
]
}'

Related

Accessing field with jq that can be string or array

I have a large dump of data in json that looks like:
[{
"recordList" : {
"record" : [{
"Production" : {
"creator" : {
"name" : "A"
}
}
},
{
"Production" : {}
},
{
"Production" : [{
"creator" : {
"name" : "B"
},
"creator" : {
"name" : "C"
}
}]
}]
}
}]
I need to check if there is at least one creator in a record or not. If there is I give a 1 else a 0 for that field in a CSV-file.
My code:
jq -r '.[].recordList.record[]|"\(if ((.Production.creator.name)? // (.Production[]?.creator.name)?) == null or ((.Production.creator.name)?|length // (.Production[]?.creator.name)?|length) == 0 then 0 else 1 end),"' file.json
The problem is that the field 'Production' is only an array when there are multiple creators.
The result I want to get in this case is:
1,
0,
1,
jq solution:
jq -r '.[].recordList.record[].Production
| "\(if ((type == "array" and .[0].creator.name !="")
or (type == "object" and .creator.name and .creator.name !=""))
then 1 else 0 end),"' file.json
The output:
1,
0,
1,
Simplified jq solution:
jq -r '.[].recordList.record[].Production
| ((type == "array" and .[0].creator.name) or .creator.name)
| if . then "1," else "0," end' file.json

Data Formatting from Text File

I have data in below format stored in a file.
ABC:9804
{
"count" : 492,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
ABC:95023
{
"count" : 865,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
ABCC:128
{
"count" : 479,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
I am trying to get the output like
ABC:9804 , 492
ABC:95023 , 865
ABCC:128 , 479
I tried using awk to get the 1st like and 3rd line but that is not working .
awk solution:
awk '/^ABC.*:/{ abc=$0 }$0~/"count"/{ gsub(/[^0-9]+/,"",$0); print abc" , "$0 }' file
The output:
ABC:9804 , 492
ABC:95023 , 865
ABCC:128 , 479
Input
$ cat infile
ABC:9804
{
"count" : 492,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
ABC:95023
{
"count" : 865,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
ABCC:128
{
"count" : 479,
"_shards" : {
"total" : 19,
"successful" : 19,
"failed" : 0
}
}
Bye
Output
Gawk
$ awk -F':' -v RS='[{},\n]' '/ABC.*|\"count"/{gsub(/[\n\t ]+/,""); printf /\"/? ", " $2 "\n": $0}' infile
ABC:9804, 492
ABC:95023, 865
ABCC:128, 479
Better Readable
awk -F':' -v RS='[{},\n]' '
/ABC.*|\"count"/{
gsub(/[\n\t ]+/,"");
printf /\"/? ", " $2 "\n": $0
}
' infile
non-gawk
$ awk -F'[ ,]' -v OFS=", " '/^[ \t]+(ABC.*|\"count\"[ ]?):/{ sub(/^[ \t]+/,""); printf /\"/ ? OFS $(NF-1) RS: $0 }' infile
ABC:9804, 492
ABC:95023, 865
ABCC:128, 479
Better Readable
awk -F'[ ,]' -v OFS=", " '
/^[ \t]+(ABC.*|\"count\"[ ]?):/{
sub(/^[ \t]+/,"");
printf /\"/ ? OFS $(NF-1) RS: $0
}
' infile

Using a key within a map

I have this function:
def getMap(value = null) {
[
"SomeTitle": [ Param: 9, Size: 2, Default: 150, Val: value ]
]
}
and I can use getMap(152).SomeTitle.Val
What I want to do is use the key Size while calculating the Key Val, something like:
def getMap(value = null) {
[
"SomeTitle": [ Param: 9, Size: 2, Default: 150, Val: value * Size ]
]
}
Is there a way to use the Value of a Key within a map as a variable while calculating the Value of a another Key
This in Java/Groovy
Groovy can't autoreference the map on it's initialization, but you can use a with {} method to do some post-initialization processing:
def getMap(value = null) {
[
"SomeTitle": [ Param: 9, Size: 2, Default: 150 ].with {
put('Val', value * get('Size'))
it
}
]
}
assert getMap(10).SomeTitle.Val == 20

Groovy recursive deep object compare not reporting failures

The answer to this question can be found in this link provided by #bdkosher:
kousenit.wordpress.com/2014/04/16/the-closure-of-no-return
I am attempting to write a recursive function that compares to complex objects (JSON objects) and reports on the differences of each. The code outputs correctly but the failure is not reported as subsequent comparisons take precedent over an earlier failure.
I'm new to groovy so I'm sure there are groovier ways to write it.
The code:
public boolean diffObjects(Object left, Object right)
{
if (left == null || right == null) {
println "Object comparison failure: One or both object null."
return false
}
if (left.getClass() != right.getClass()) {
println "Object comparison failure: Mismatch object classes."
return false
}
if (isArray(left)) {
if (left.size() != right.size()) {
println "Array comparison failure: Object size mismatch."
println "Left has " + left.size() + " items. Right has " + right.size() + " items."
println "Left Object:"
println left
println "Right Object:"
println right
return false
}
for(int i=0; i < left.size(); i++) {
// May detect matching items here if sort of objects is problem
diffObjects(left[i], right[i])
}
} else if (isLazyMap(left)) {
String[] leftKeys = left.keySet()
String[] rightKeys = right.keySet()
if (leftKeys != rightKeys) {
println "Map comparison failure: Left keys do not match right keys."
println "Left " + leftKeys.toString()
println "Right " + rightKeys.toString()
return false
}
leftKeys.each {
if (isArray(left[it])) {
// May detect matching items here if sort of objects is problem
diffObjects( left[it], right[it])
} else {
if (isValue(left[it])) {
if (left[it].toString() != right[it].toString()) {
println "String comparison failure: Left " + it + " value does not match right value."
println "Left " + left[it]
println "Right " + right[it]
return false
}
}
}
}
}
}
Sample data:
[
{
"productId": "141810",
"sizes": [
{
"sku": "11926",
"size": "L",
"gtin": "008206",
"localizedSize": "L",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "M",
"gtin": "0082065234",
"localizedSize": "M",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "S",
"gtin": "0082065234",
"localizedSize": "S",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "XL",
"gtin": "0082065234",
"localizedSize": "XL",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": true
}
]
}
]
Any help would be appreciated. Thanks in advance.
Here's an alternative method:
def a = ['a', 'b', 'c']
def b = ['a', 'z', 'c']
def c = [
[sku: 'abc', size: 'L'],
[sku: '123', size: 'S']
]
def d = [
[sku: 'abc', size: 'L'],
[sku: 'xyz', size: 'S']
]
use(DiffMixin) {
assert a.diff(b) == [
[
[self:'b', other:'z'],
[self:[name:'bytes', value:[98]], other:[name:'bytes', value:[122]]]
]
]
println c.diff(d)
}
class DiffMixin {
static List diff(Object self, Object other) {
def diffs = []
if(self != other) {
diffs << [
'self': self.toString(),
'other': other.toString()
]
}
self.properties.inject(diffs) {list, entry ->
def key = "$entry.key"
if(self."$key" != other."$key") {
list << [
'self': [name: key, value: self."$key"],
'other': [name: key, value: other."$key"]
]
}
return list
}
}
static List diff(Collection self, Collection other) {
([self] + [other])
.transpose()
.inject([]) {list, pair ->
def diffs = diff(pair[0], pair[1])
if(diffs) list << diffs
return list
}
}
}
I couldn't get a working assert expression for the c-d comparison. The output looks like this:
[[[self:[sku:123, size:S], other:[sku:xyz, size:S]]]]
Explanation
The DiffMixin mixin implements comparison methods for the classes you want to compare. I only implemented two, Object and Collection, but it should give you the idea. The Object comparison compares toString() output and property values. The Collection comparison basically delegates most of the work, while collecting the diffs. These methods return a list describing the differences.
Think of this as a demo :)

How To Control Woo-Commerce JSON API DATA?

Right now , I am working on a project. I have to build some rest full api's for android application using Woo-Commerce CLIENT REST API.
Everything is working fine, but the data i'm getting is unnecessary. So can anybody tell me, how to overcome from this problem.
e.g. I am getting this data
{
product_categories: [
{
id: 8,
name: "Cakes",
slug: "cakes",
parent: 0,
description: "Love is like a good cake; you never know when it's coming, but you'd better eat it when it does!",
count: 11
},
{
id: 9,
name: "Breads",
slug: "breads",
parent: 0,
description: "All sorrows are less with bread. ",
count: 3
},
{
id: 10,
name: "Pastries",
slug: "pastries",
parent: 0,
description: "I'm not a vegetarian! I'm a pastries-ian!",
count: 6
}
but i do not want slug,parent, description parameters.
Thanks in advance.
Your Problems is " " missing in key Ex: "ID", "name" and
You use this function json_pretty() for json format.
function json_pretty($json, $html = false) {
$out = ''; $nl = "\n"; $cnt = 0; $tab = 4; $len = strlen($json); $space = ' ';
if($html) {
$space = ' ';
$nl = '<, $html = false)br/>';
}
$k = strlen($space)?strlen($space):1;
for ($i=0; $i<=$len; $i++) {
$char = substr($json, $i, 1);
if($char == '}' || $char == ']') {
$cnt --;
$out .= $nl . str_pad('', ($tab * $cnt * $k), $space);
} else if($char == '{' || $char == '[') {
$cnt ++;
}
$out .= $char;
if($char == ',' || $char == '{' || $char == '[') {
$out .= $nl . str_pad('', ($tab * $cnt * $k), $space);
}
}
return $out;
}
How to user this function ?
$pre = '{"status": 1,"message": "My Collection downloaded successfully.","myCollections":';
$postd = ' }';
$jsa_data = json_encode($res_arr); // pass your Array
echo $finalJson = json_pretty($pre.$jsa_data.$postd);
Out put
{
"status": 1,
"message": "All Post downloaded successfully",
"postData": [
{
"id": 8,
"name": "Cakes",
"slug": "cakes",
"parent": 0,
"description": "Love is like a good cake; you never know when it's coming, but you'd better eat it when it does!",
"count": 11
},
{
"id": 9,
"name": "Breads",
"slug": "breads",
"parent": 0,
"description": "All sorrows are less with bread. ",
"count": 3
},
{
"id": 10,
"name": "Pastries",
"slug": "pastries",
"parent": 0,
"description": "I'm not a vegetarian! I'm a pastries-ian!",
"count": 6
}
]
}
Check it Json LINT http://jsonlint.com/

Resources