First question may take care of this. When capturing in tshark using fields, like "-e tcp.flags", is there a way to have the output the flag label, like "FIN", instead of "0x1"? I've done a few searches through documentation. Probably right under my nose.
If not, then I need a function in my data pipeline to convert the hex into the labels. I thought about having a dictionary like "{'0x1':'FIN'}" and map it, but I'm not sure of all the flag combos that might appear.
So I am taking the hex string, converting it to an integer, then to a binary string. I turn that into a list "[0,0,0,0,0,1]" and use that like a filter against a label list like "[u, a, p, r, s, f]" that returns any labels joined, like "f" or "a_s". Using Python.
Is this function necessary? Is there a more efficient/ elegant way to convert the hex to labels?
Normally I would suggest using -e tcp.flags.str, but this doesn't display properly for me at least running on Windows 10 w/TShark (Wireshark) 3.3.0 (v3.3.0rc0-1433-gcac1426dd6b2). For example, here's what I get for what should be a "SYN" indication only:
tshark.exe -r tcpfile.pcap -c 1 -T fields -e frame.number -e tcp.flags -e tcp.flags.str
1 0x00000002 A·A·A·A·A·A·A·A·A·A·SA·
You can try it on your system and maybe it'll display as intended (In Wireshark, it's displayed correctly as ··········S·, so it may be a tshark bug or a problem with my shells - tried with both cmd and powershell.) In any case, if it doesn't display properly for you on your system, you can try using the tcp-flags-postdissector.lua dissector that Didier Stevens wrote, which was inspired by Snort and which I believe served as the inspiration for the Wireshark built-in tcp.flags.str field. I personally preferred a '.' instead of '*' for flag bits that aren't set, so I tweaked the Lua dissector to behave that way. Use it as is, or tweak it anyway you choose. With the Lua dissector, I get the expected output:
tshark.exe -r tcpfile.pcap -c 1 -T fields -e frame.number -e tcp.flags tcpflags.flags
1 0x00000002 ........S.
Since the same incorrect string is displayed in both cmd and powershell, it looks like a tshark bug to me, so I filed Wireshark Bug 16649.
Related
I am attempting to bend zsh, my shell of choice, to my will, and am completely at a loss on the syntax and operation of completions.
My use case is this: I wish to have completions for 'ansible-playbook' under the '-e' option support three variations:
Normal file completion: ansible-playbook -e vars/file_name.yml
Prepended file completion: ansible-playbook -e #vars/file_name.yml
Arbitrary strings: ansible-playbook -e key=value
I started out with https://github.com/zsh-users/zsh-completions/blob/master/src/_ansible-playbook which worked decently, but required modifications to support the prefixed file pathing. To achieve this I altered the following lines (the -e line):
...
"(-D --diff)"{-D,--diff}"[when changing (small files and templates, show the diff in those. Works great with --check)]"\
"(-e --extra-vars)"{-e,--extra-vars}"[EXTRA_VARS set additional variables as key=value or YAML/JSON]:extra vars:(EXTRA_VARS)"\
'--flush-cache[clear the fact cache]'\
to this:
...
"(-D --diff)"{-D,--diff}"[when changing (small files and templates, show the diff in those. Works great with --check)]"\
"(-e --extra-vars)"{-e,--extra-vars}"[EXTRA_VARS set additional variables as key=value or YAML/JSON]:extra vars:__at_files"\
'--flush-cache[clear the fact cache]'\
and added the '__at_files' function:
__at_files () {
compset -P #; _files
}
This may be very noobish, but for someone that has never encountered this before, I was pleased that this solved my problem, or so I thought.
This fails me if I have multiple '-e' parameters, which is totally a supported model (similar to how docker allows multiple -v or -p arguments). What this means is that the first '-e' parameter will have my prefixed completion work, but any '-e' parameters after that point become 'dumb' and only allow for normal '_files' completion from what I can tell. So the following will not complete properly:
ansible-playbook -e key=value -e #vars/file
but this would complete for the file itself:
ansible-playbook -e key=value -e vars/file
Did I mess up? I see the same type of behavior for this particular completion plugin's '-M' option (it also becomes 'dumb' and does basic file completion). I may have simply not searched for the correct terminology or combination of terms, or perhaps in the rather complicated documentation missed what covers this, but again, with only a few days experience digging into this, I'm lost.
If multiple -e options are valid, the _arguments specification should start with * so instead of:
"(-e --extra-vars)"{-e,--extra-vars}"[EXTR ....
use:
\*{-e,--extra-vars}"[EXTR ...
The (-e --extra-vars) part indicates a list of options that can not follow the one being specified. So that isn't needed anymore because it is presumably valid to do, e.g.:
ansible-playbook -e key-value --extra-vars #vars/file
I am following the steps in this knowledge base article to strip off code and intermediate info from an opencl binary. How do I verify that this information has indeed been removed?
You must make sure that the following sections are absent from the resulting binary:
.source
.llvmir
.amdil
For that, run readelf -a stripped_kernel.bin | grep section, replacing "section" with the name of each of the sections above. If the binary has been properly stripped, grep should produce no result.
I'm trying to run this terminal command from within R using system(mess):
mess <- "sed -i -e '62i\ \\\usepackage[margin=2cm]{geometry}' intro-spatial-rl.tex"
But it keeps failing with the following error:
Error: '\u' used without hex digits in character string starting ""sed -i -e '62i\ \\\u"
I've seen paste used for system commands also, but this fails also.
Could use a different regex program, but thought this may be useful to others and improve my understanding of how R deals with characters. Thank you!
Your problem is the unequal number of \ in your escape sequence.
R sees two escape sequences here: \\ and \u. The second one is invalid and gives an error. You probably want to escape the second backslash as well, yielding \\\\. Likewise, you probably meant to escape the previous \ in \ as well, leaving you with \\ .
All that being said, would replace the sed invocation completely by R code in this instance. The way I understand it you just want to insert a line of text. That’s easy in R (although it’s not clear what your input and output here is).
I am really amazed by the functionality of GREP in shell, earlier I used to use substring method in java but now I use GREP for it and it executes in a matter of seconds, it is blazingly faster than java code that I used to write.(according to my experience I might be wrong though)
That being said I have not been able to figure out how it is happening? there is also not much available on the web.
Can anyone help me with this?
Assuming your question regards GNU grep specifically. Here's a note from the author, Mike Haertel:
GNU grep is fast because it AVOIDS LOOKING AT EVERY INPUT BYTE.
GNU grep is fast because it EXECUTES VERY FEW INSTRUCTIONS FOR EACH
BYTE that it
does look at.
GNU grep uses the well-known Boyer-Moore algorithm, which looks first
for the final letter of the target string, and uses a lookup table to
tell it how far ahead it can skip in the input whenever it finds a
non-matching character.
GNU grep also unrolls the inner loop of Boyer-Moore, and sets up the
Boyer-Moore delta table entries in such a way that it doesn't need to
do the loop exit test at every unrolled step. The result of this is
that, in the limit, GNU grep averages fewer than 3 x86 instructions
executed for each input byte it actually looks at (and it skips many
bytes entirely).
GNU grep uses raw Unix input system calls and avoids copying data
after reading it. Moreover, GNU grep AVOIDS BREAKING THE INPUT INTO
LINES. Looking for newlines would slow grep down by a factor of
several times, because to find the newlines it would have to look at
every byte!
So instead of using line-oriented input, GNU grep reads raw data into
a large buffer, searches the buffer using Boyer-Moore, and only when
it finds a match does it go and look for the bounding newlines
(Certain command line options like
-n disable this optimization.)
This answer is a subset of the information taken from here.
To add to Steve's excellent answer.
It may not be widely known but grep is almost always faster when grepping for a longer pattern-string than a short one, because in a longer pattern, Boyer-Moore can skip forward in longer strides to achieve even better sublinear speeds:
Example:
# after running these twice to ensure apples-to-apples comparison
# (everything is in the buffer cache)
$ time grep -c 'tg=f_c' 20140910.log
28
0.168u 0.068s 0:00.26
$ time grep -c ' /cc/merchant.json tg=f_c' 20140910.log
28
0.100u 0.056s 0:00.17
The longer form is 35% faster!
How come? Boyer-Moore consructs a skip-forward table from the pattern-string, and whenever there's a mismatch, it picks the longest skip possible (from last char to first) before comparing a single char in the input to the char in the skip table.
Here's a video explaining Boyer Moore (Credit to kommradHomer)
Another common misconception (for GNU grep) is that fgrep is faster than grep. f in fgrep doesn't stand for 'fast', it stands for 'fixed' (see the man page), and since both are the same program, and both use Boyer-Moore, there's no difference in speed between them when searching for fixed-strings without regexp special chars. The only reason I use fgrep is when there's a regexp special char (like ., [], or *) I don't want it to be interpreted as such. And even then the more portable/standard form of grep -F is preferred over fgrep.
It seems like the only way to do this is to pass the -i parameter in when you initially run less. Does anyone know of some secret hack to make something like this work
/something to search for/i
You can also type command -I while less is running. It toggles case sensitivity for searches.
You can also set the environment variable LESS
I use LESS=-Ri, so that I can pump colorized output from grep into it, and maintain the ANSI colour sequences.
Another little used feature of less that I found is starting it with +F as an argument (or hitting SHIFT+F while in less). This causes it to follow the file you've opened, in the same way that tail -f <file> will. Very handy if you're watching log files from an application, and are likely to want to page back up (if it's generating 100's of lines of logging every second, for instance).
Add-on to what #Juha said: Actually -i turns on Case-insensitive with SmartCasing, i.e if your search contains an uppercase letter, then the search will be case-sensitive, otherwise, it will be case-insensitive. Think of it as :set smartcase in Vim.
E.g.: with -i, a search for 'log' in 'Log,..' will match, whereas 'Log' in 'log,..' will not match.
It appears that you can summon this feature on a per search basis like so:
less prompt> /search string/-i
This option is in less's interactive help which you access via h:
less prompt> h
...
-i ........ --ignore-case
Ignore case in searches that do not contain uppercase.
-I ........ --IGNORE-CASE
Ignore case in all searches.
...
I've not extensively checked but the help in less version 487 on MacOS as well as other Linux distros lists this option as being available.
On MacOS you can also install a newer version of less via brew:
$ brew install less
$ less --version
less 530 (POSIX regular expressions)
Copyright (C) 1984-2017 Mark Nudelman
References
less is always case-insensitive
When using -i flag, be sure to enter the search string completely in lower case, because if any letter is upper case, then its an exact match.
See also: the -I (capital i) flag of less(1) to change this behavior.