I installed elasticsearch 5.0.1 and corresponding ingest attachment. Tried indexing pdf document from shell script as below
#!/bin/ksh
var=$(base64 file_name.pdf)
var1=$(curl -XPUT 'http://localhost:9200/my_index4/my_type/my_id?pipeline=attachment&pretty' -d' { "data" : $var }')
echo $var1
I got error as
{ "error" : { "root_cause" : [ { "type" : "exception", "reason" :
"java.lang.IllegalArgumentException: ElasticsearchParseException[Error parsing document in field
[data]]; nested: IllegalArgumentException[Illegal base64 character 24];",
"header" : { "processor_type" : "attachment" } } ]...
Can anyone please help on resolving the above issue ... Not sure whether I am passing invalid base64 character ?
Please note that when I pass like this, It works !
var1=$(curl -XPUT 'http://localhost:9200/my_index4/my_type/my_id?pipeline=attachment&pretty'
-d' { "data" : "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" }')
I guess the issue has to shell not expanding the variables within single-quotes, you need to double-quote to expand it. i.e.
change -d' { "data" : $var }'
to
-d '{"data" : "'"$(base64 file_name.pdf)"'"}'
directly to pass the base64 stream.
(or)
-d '{"data" : "'"$var"'"}'
More about quoting and variables in ksh here.
Related
I am using kislyuk/yq - The more often talked about version, which is a wrapper over jq, written in Python using the PyYAML library for YAML parsing
The version is yq 2.12.2
My jq is jq-1.6
I'm using ubuntu and bash scripts to do my parsing.
I wrote this as bash
alias=alias1
token=abc
yq -y -i ".tokens += { $alias: { value: $token }}" /root/.github.yml
I get the following error
jq: error: abc/0 is not defined at <top-level>, line 1:
.tokens += { alias1: { value: abc }}
I don't get it. Why would there be a /0 at the end?
The problem is abc is not interpreted as a literal string, when the double quotes are expanded by the shell. The underlying jq wrapper tries to match with abc as a standard built-in or a user-defined function which it was not able to resolve to, hence the error.
A JSON string (needed for jq) type needs to be quoted with ".." to be consistent with the JSON grammar. One way would be to pass the arg via command line with the --arg support
yq -y -i --arg t "$token" --arg a "$alias" '.tokens += { ($a): { value: $t } }' /root/.github.yml
Or have a quoting mess like below, which I don't recommend at all
yq -y -i '.tokens += { "'"$alias"'": { value: "'"$token"'" }}' /root/.github.yml
Let's pretend I'm running something like this:
jq -nr --arg target /tmp \
'(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"' \
| sh
Everything is fine unless I forgot to pass variable $target:
$ jq -nr '(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"'
jq: error: $target is not defined at <top-level>, line 1:
(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"
jq: 1 compile error
How can I catch this and use default value?
I've tried:
$target?
($target)?
try $target catch null
$target? // null
But it seems to be parsing-time error, which obviously can't be caught at runtime. Have I've missed any dynamic syntax?
I've found that command-line arguments can be found in $ARGS.name, but there are two drawbacks:
This was introduced in version 1.6, but I have 1.5 on CentOS 7.
It doesn't catch locally defined variables.
Assuming you need to do something more useful with jq than write 'Hello World' over a text file. I propose the following,
Maybe we can learn some programming tips from Jesus:
"Give to Caesar what belongs to Caesar, and give to God what belongs to God"
Suppose that Caesar is bash shell and God is jq, bash is appropriate to work and test the existence of files, directories and environment variables, jq is appropriate to process information in json format.
#!/bin/bash
dest_folder=$1
#if param1 is not given, then the default is /tmp:
if [ -z $dest_folder ]; then dest_folder=/tmp ; fi
echo destination folder: $dest_folder
#check if destination folder exists
if [ ! -d $dest_folder ]
then
echo "_err_ folder not found"
exit 1
fi
jq -nr --arg target $dest_folder '(["echo","Hello, world"]|#sh)+">\($target)/sample.txt"' | sh
#if the file is succesfully created, return 0, if not return 1
if [ -e "$dest_folder/sample.txt" ]
then
echo "_suc_ file was created ok"
exit 0
else
echo "_err_ when creating file"
exit 1
fi
Now you can include this script as a step in a more complex batch, because it is congruent with linux style, returning 0 on success.
I'd like to create a zsh-function that will put a computed string into the line editor, e.g.
insert_foo() {
# Do something that will put the string "foo" in the line editor
}
$ insert_foo
$ foo # <- "foo" appears in the line editor, without actually executing it
I know this is possible within a zle-widget, but can it also be it's own command?
We could use print -z:
insert_foo() {
print -z "foo"
}
Here is the zsh doc's copy for print -z:
print [ -abcDilmnNoOpPrsSz ] [ -u n ] [ -f format ] [ -C cols ]
...
-z Push the arguments onto the editing buffer stack, separated by spaces.
--- zshbuiltin(1), shell builtin commands, print
I installed elasicsearch 5.0.1 with ingest attachment and tried indexing pdf in elasticsearch from shell script using below command
#!/bin/ksh
var=$(base64 sample.pdf | perl -pe 's/\n/\\n/g')
var1=$(curl -XPUT 'http://localhost:9200/my_index5/my_type/my_id?pipeline=attachment&pretty' -d' { "data" : "'$var'" }')
echo $var1
Got an error as
{ "error" : { "root_cause" : [ { "type" : "exception", "reason" : "java.lang.IllegalArgumentException: ElasticsearchParseException[Error parsing document in field [data]]; nested: IllegalArgumentException[Illegal base64 character a];", "header" : { "processor_type" : "attachment" } } ]
Can anyone please help resolving the above error
Rectified the error.
Cause for this error is, In the base64 encoded content \n was present which caused the "Illegal format exception".
As a solution, When tried like below it worked
var=$(base64 sample.pdf | perl -pe 's/\n//g')
I need to edit a .jam file used by boost-build for a specific kind of projects. The official manual on BJAM language says:
One of the toolsets that cares about DEF files is msvc. The following line should be added to it. flags msvc.link DEF_FILE
;
Since the DEF_FILE variable is not used by the msvc.link action, we need to modify it to be: actions link bind DEF_FILE { $(.LD) ....
/DEF:$(DEF_FILE) .... } Note the bind DEF_FILE part. It tells bjam to
translate the internal target name in DEF_FILE to a corresponding
filename in the link
So apparently just printing DEF_FILE with ECHO wouldn't work. How can it be expanded to a string variable or something that can actually be checked?
What I need to do is to print an error message and abort the build in case the flag is not set. I tried:
if ! $(DEF_FILE)
{
errors.user-error "file not found" ;
EXIT ;
}
but this "if" is always true
I also tried putting "if ! $_DEF_FILE {...}" inside the "actions" contained but apparently it is ignored.
I am not sure I understand the global task you have. However, if you wanted to add checking for non-empty DEF_FILE -- expanding on the documentation bit you quote, you need to add the check in msvc.link function.
If you have a command line pattern (specified with 'actions') its content is what is passed to OS for execution. But, you can also have a function with the same name, that will be called before generating the actions. For example, here's what current codebase have:
rule link.dll ( targets + : sources * : properties * )
{
DEPENDS $(<) : [ on $(<) return $(DEF_FILE) ] ;
if <embed-manifest>on in $(properties)
{
msvc.manifest.dll $(targets) : $(sources) : $(properties) ;
}
}
You can modify this code to additionally:
if ! [ on $(<) return $(DEF_FILE) ] {
ECHO "error" ;
}