I would like escape a character % in file : parameters.yml, because my password begin by %
database_driver3: pdo_sqlsrv
database_host3: 192.168.10.10
database_port3: 1455
database_name3: MyDB
database_user3: sa
database_password3: %ABC$Admin_2012%
What I really should be doing ?
The easiest way is to simply add quotes to the values you want to escape
database_password3: "%ABC$Admin_2012%"
OR by using double % credit - skowron-line
database_password3: %%ABC$Admin_2012%%
Related
I have below records into txt file.
000D3A|"RiFR Botnets" AD||83634C|dk
000D3|Ries Bidvest" AD||8364A3C|dhh
000D3A|"Ra Boots D"||83634C|gft
here I want to add double quotes for those records which having " into the line using AWK Unix command.
expected output which I want to write to file:
000D3A|""RiFR Botnets" AD"||83634C|dk
000D3|"Ries Bidvest" AD"||8364A3C|dhh
000D3A|""Ra Boots""||83634C|gft
I have tried using AWS command and AWK -F "|" but how do i search double quotes " here into the every line of file.
You can try:
awk -F"|" -v OFS="|" '{
for (i=1;i<=NF;i++) # for every field...
if (match($i,"\"")) # check for "
$i="\"" $i "\"" # add quotes to that field
}
1 # print' file
000D3A|""RiFR Botnets" AD"||83634C|dk
000D3|"Ries Bidvest" AD"||8364A3C|dhh
000D3A|""Ra Boots D""||83634C|gft
Or, you could use this sed:
sed -E 's/\|([^|]*"[^|]*)\|/|"\1"|/g' file
# same output
(As noted in comments, the result is not valid csv. You requested this, but "Ries Bidvest" AD" is not valid quoting and will break csv parsing...)
I don't use Lua, but need to use it with Openresty (nginx) as provided in the link.
Openresty has a lua module which I managed to install and run the Openresty nginx version correctly, the website working.
This answer shows how to concatenate headers into a string $request_headers:
set_by_lua $request_headers '
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
end
return request_headers_all
';
I changed the format from ""..k..": "..v..";" to "["..k..": "..v.."]" in the lua function above.
Log format:
log_format log_realip 'Host: "$host", Via : "$http_via", RemoteAddr: "$remote_addr", XForwardedFor: "$h
ttp_x_forwarded_for", 'RealIPRemoteAddr: "$realip_remote_addr" - $status - "$request_uri" - **"[HEADERS]" - $request_headers';**
Host: "192.168.1.4", Via : "-",
//trimmed just to show the [HEADERS]
....
"[HEADERS]" - [sec-ch-ua: \x22Chromium\x22;v=\x2288\x22, \x22Google Chrome\x22;v=\x228
8\x22, \x22;Not A Brand\x22;v=\x2299\x22][sec-ch-ua-mobile: ?0][cookie: __utmz=abcdef; frontend=abcdef; adminhtml=abcdef
08; TestName=Some Value][upgrade-insecure-requests: 1][accept-language: en-US,en;q=0.9][user-agent: Mozilla/5.0
(Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36][accept
-encoding: gzip, deflate, br][accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/we
bp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9][sec-fetch-dest: document][host: 192.168.1.4][se
c-fetch-user: ?1][connection: keep-alive][sec-fetch-mode: navigate][cache-control: max-age=0][sec-fetch-site: n
one
When using log_format with $request_headers string I get all the headers in one line, but I am trying to create a newline \n to break the string into lines. The example above is where I added \n but doesn't seem to output break to the log file.
I understand the request_headers_all .. concatenates the string, but what is happening here with the key k and value v : ""..k..": "..v..\n""?
What is the "..variablename.." doing, is this how variables are always used inside Lua strings?
How would I be able to create a line break in that string? Or is it possible that nginx(openresty) doesn't output the newline?
you add the \n to a wrong place, you can change to
request_headers_all = request_headers_all .. "["..k..": "..v.."]\n" for a newline log.
In lua, the .. is a concat operator, to concat to strings, for example:
print("hello" .. "world")
get the result helloworld.
your code \n"]" have syntax error, because \n not in a string.
lua strings can not directly use variables, usually, lua use string.format for complex string. for example:
local test = "hello"
string.format("%s world",test) -- hello world
you can use string.format for you string concat.
also, you can use table.concat to concat strings.
for example:
local test = {}
table.insert(test, "hello")
table.insert(test, "world")
local concated_string = table.concat(test, ' ')
print(concated_string) -- hello world
request_headers_all = request_headers_all .. "["..k..": "..v..\n"]"
contains a syntax error. replace "["..k..": "..v..\n"]" with "["..k..": "..v.."]\n"
Newline needs to be inside the quotes as it is part of the string and it will probably make sense to add the new line after the bracket.
What is the "..variablename.." doing, is this how variables are always
used inside Lua strings?
using the concatenation operator on a variable concatenates a string value, concatenates its string representation if it is a number or invokes __concat or raises an error if neither of those is true.
Read https://www.lua.org/manual/5.4/manual.html#3.4.6
The above answers gave me some guidance, but the formats suggested still didn't work. After playing around with string.format("%s %s\n", k, v), string.format("%s %s\\n", k, v) I still got unfinished string errors or no newline output. (Tried to escape string in second example).
Based on the answers given I assumed the answers gave correct lua information, so decided most likely lua + openresty does something different.
I will attempt to update the title to reflect more specific requirements
TLDR
Openresty + lua string manipulation with special characters might not work as expected, even when using string.format()
Change from set_by_lua which returns a string to set_by_lua_block which allows string.format() or string concatenation better.
Update nginx configuration with your custom/existing log_format and add the switch escape=none.
Full Explanation
Investigating the provided link answer set_by_lua function documentation :
NOTE Use of this directive is discouraged following the v0.9.17 release. Use the set_by_lua_block directive instead.
So from the original set_by_lua from the link:
set_by_lua $request_headers '
return 'stringvalue'
';
I changed to set_by_lua_block function:
this directive inlines the Lua source directly inside a pair of curly braces ({}) instead of in an Nginx string literal (which requires special character escaping)
set_by_lua_block $request_headers{
local h = ngx.req.get_headers()
local request_headers_all = ""
for k, v in pairs(h) do
local rowtext = ""
rowtext = string.format("[%s %s]\n", k, v)
request_headers_all = request_headers_all .. rowtext
end
return request_headers_all
}
The important part is that this _block {} function escapes the special characters correctly.
After that I received output in the log files as : x0A (newline character literal).
The final step then is to update the nginx.conf file with the custom log_format and add escape=none:
log_format log_realip escape=none "$nginx_variables"
The main question/problem here is how does the wrapper software handle newline characters/escapes, is it expecting "\n"? or is it expecting "\r\n"?
Ultimately the new line does not actually exist until it is interpreted and printed and you are creating one massive string that gets returned from the Lua engine to the wrapper, so it is up to the wrapper software on how to interpret new lines.
Edit: I missed that this was already answered by using the other parsing function.
Additionally the docs state to use the original parsing function escapes need to be double escaped.
Here, \\\\d+ is stripped down to \\d+ by the Nginx config file parser and this is further stripped down to \d+ by the Lua language parser before running.
I wrote this function in zsh
function test() {
test="a\tb\tc"
while IFS=$'\t' read -r user host key; do
echo "user: $user"
echo "host: $host"
echo "key: $key"
done <<< "$test"
}
The output was:
user: a b c
host:
key:
if instead of
... IFS=$'\t' read -r ...
I change it to
... IFS='\t' read -r ...
the output is
user: a
host:
key: b c
Just what is going on?
I would like to read the tab separated line and set my variables accordingly.
Changing the double-quotes to $'...' (single quotes preceded by a $) could rescue for the variable $test:
test=$'a\tb\tc'
Here is the zsh manual for QUOTING (double-quoting and $'...'):
QUOTING
...
A string enclosed between $' and ' is processed the same way as the string arguments of the print builtin
...
Inside double quotes (""), parameter and command substitution occur, and \ quotes the characters \, `, ", $, and the first character of $histchars (default !).
--- zshmisc(1), QUOTING
For example:
"\$" -> $, "\!" -> ! etc.
"\t" -> \t (zsh does not recognize as tab this case), "\a" -> \a etc.
It does not treat the escape sequence \t as tab when it is used inside double quotes, so "a\tb\tc" does not mean "atabbtabc". (But things are a little more complicated: builtin echo recognizes the escape sequence \t.)
(1) ... IFS=$'\t' read -r ... (the original form)
Because expanding "$test" dose not have any tab characters, so read assigns the whole line to $user:
user: a b c
host:
key:
(But echo recognizes the escape sequence \t as the tab.)
(2) ... IFS='\t' read -r ...
Again, expanding "$test" does not have any tab characters, so read separate the field by \ and t according $IFS.
a\t\b\tc splits into a (to $user), \(separator), `` (empty to $host), t(separator), and the rest of the line (b\tc to $key):
user: a
host:
key: b c
(But again, echo recognizes the escape sequence \t as the tab.)
Here is the code changed from test="..." to test=$'...':
function test() {
test=$'a\tb\tc'
while IFS=$'\t' read -r user host key; do
echo "user: $user"
echo "host: $host"
echo "key: $key"
done <<< "$test"
}
test
The output is:
user: a
host: b
key: c
PS: it is worth reading POSIX's Quoting specification, which is simpler than zsh's (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02)
I'm trying to modify the <config-profile> section of a ossc.conf file, including a grains content.
something like:
ossec-profiles:
- profile1
- profile2
and I want to modify the section <config-profile> from
<config-profile>centos, centos7</config-profile>
to
<config-profile>centos, centos7, profile1, profile2</config-profile>
in the ossec.conf file
Any idea?
This can be done by using file.replace module which makes you able to change a text in a file based on a pattern. So in your case you can do the following:
You need to select the pattern as regex group so you can use it later as shown below
configure_ossec:
file.replace:
- name: /path/to/ossec.conf
- pattern: '((<config-profile>.*?)[^<]*)'
- repl: {{ '\\1, ' + pillar['ossec-profiles'] | join(', ') }}
Or you might use this pattern to match only whatever inside config-profile tags then you will be able to call it again in the repl parameter:
(?<=<config-profile>)(.*)(?=<\/config-profile>)
Note: As pillar['ossec-profiles'] should return a list of profiles
then you have to use the join filter in order to separate the values
with comma as a delimiter
And finally the output expected to be something like this:
Changes:
----------
diff:
---
+++
## -1 +1 ##
-<config-profile>centos, centos7</config-profile>
+<config-profile>centos, centos7, profile1, profile2</config-profile>
I'm making a custom zsh theme for when you open up your terminal. I'm not that new to using oh-my-zsh but I am new to zsh documentation.
Here's what I'm having problems with:
wolf='
.-'''''-.
.' `.
: :
: :
: _/| :
: =/_/ :
`._/ | .'
( / ,|...-'
\_/^\/||__
_/~ `""~`"` \_
__/ -'/ `-._ `\_\__
/ /-'` `\ \ \-.\\
'
print -P $wolf
When I try it in terminal I get two errors:
/Users/User/.oh-my-zsh/themes/wolf.zsh-theme:4: no such file or directory: . .\n : :\n : :\n : _/| :\n : =/_/ :\n._/ |
/Users/User/.oh-my-zsh/themes/wolf.zsh-theme:31: parse error near `\n
I'm guessing that these errors have to do with the apostrophes in the ASCII art.
The issue here (as jdv noted in his comment) is quoting.
In zsh:
it is not possible to use single-quotes (') withing single-quoted text. Any text within two single-quotes is taken verbatim, no additional quoting is possible. Single quotes can only be quoted with a preceding backslash or inside double quotes.
Within double-quotes (") backticks (`), dollar signs ($) and backslashes (\) are treated specially, and need to be quoted with a preceding backslash.
As quite a lot of these characters appear in your ASCII-art, you have to make sure, that they are all quoted properly.
Additionally, you use the builtin print, which by default also interprets \ as escape character.
There are two basic solutions:
Fully quote the string and use print -r or echo -E to print the text. In both cases the parameter disables escape sequence handling.
The easiest way to achieve full quoting is probably to keep the surrounding single-quotes and replace any ' within with '\''. At every occurrence this closes the previous single-quoted text, adds a quoted single-quote and starts a new single-quoted text. Any other special character is then quoted within single quotes.
wolf='
.-'\'''\'''\'''\'''\''-.
.'\'' `.
: :
: :
: _/| :
: =/_/ :
`._/ | .'\''
( / ,|...-'\''
\_/^\/||__
_/~ `""~`"` \_
__/ -'\''/ `-._ `\_\__
/ /-'\''` `\ \ \-.\\
'
print -r $wolf
Use a here-document with cat to avoid the whole quoting issue:
cat <<'END'
.-'''''-.
.' `.
: :
: :
: _/| :
: =/_/ :
`._/ | .'
( / ,|...-'
\_/^\/||__
_/~ `""~`"` \_
__/ -'/ `-._ `\_\__
/ /-'` `\ \ \-.\\
END
Note the single quotes around 'END'. This needs to be done in order to disable parameter substitution ($FOO) and command substitution ($(command) or `command`) inside the here-document.