I've looked this up a thousand times, and I always forget it, so, here for eternity:
Solaris has a bit of an awkward syntax for tail.
How do I do the equivalent of BSD's tail -nN?
What I want are the last N lines from tail's input.
Just remove the "n"
tail -100
Or you can use:
/usr/xpg4/bin/tail
which does behave like you want (tail -nN).
xpg4 = Xopen Portability Guide Issue 4, contains binaries strictly compliant with several POSIX and other standards. The differences with the former ones are usually details in options supported and behavior.
According to your distribution, there is also /usr/xpg6/bin, /usr/openwin/bin (OpenWindows commands), /usr/dt/bin (CDE desktop commands), /usr/sfw/bin (Solaris freeware) and various other.
For instance, Solaris Express is introducing /usr/gnu/bin to provide Gnu binaries with their custom extensions and specificities.
Cross-platform variant of tail -n 10 for scripts:
sed -e :a -e '$q;N;11,$D;ba' file
This works the same for Linux and Solaris.
Related
First question may take care of this. When capturing in tshark using fields, like "-e tcp.flags", is there a way to have the output the flag label, like "FIN", instead of "0x1"? I've done a few searches through documentation. Probably right under my nose.
If not, then I need a function in my data pipeline to convert the hex into the labels. I thought about having a dictionary like "{'0x1':'FIN'}" and map it, but I'm not sure of all the flag combos that might appear.
So I am taking the hex string, converting it to an integer, then to a binary string. I turn that into a list "[0,0,0,0,0,1]" and use that like a filter against a label list like "[u, a, p, r, s, f]" that returns any labels joined, like "f" or "a_s". Using Python.
Is this function necessary? Is there a more efficient/ elegant way to convert the hex to labels?
Normally I would suggest using -e tcp.flags.str, but this doesn't display properly for me at least running on Windows 10 w/TShark (Wireshark) 3.3.0 (v3.3.0rc0-1433-gcac1426dd6b2). For example, here's what I get for what should be a "SYN" indication only:
tshark.exe -r tcpfile.pcap -c 1 -T fields -e frame.number -e tcp.flags -e tcp.flags.str
1 0x00000002 A·A·A·A·A·A·A·A·A·A·SA·
You can try it on your system and maybe it'll display as intended (In Wireshark, it's displayed correctly as ··········S·, so it may be a tshark bug or a problem with my shells - tried with both cmd and powershell.) In any case, if it doesn't display properly for you on your system, you can try using the tcp-flags-postdissector.lua dissector that Didier Stevens wrote, which was inspired by Snort and which I believe served as the inspiration for the Wireshark built-in tcp.flags.str field. I personally preferred a '.' instead of '*' for flag bits that aren't set, so I tweaked the Lua dissector to behave that way. Use it as is, or tweak it anyway you choose. With the Lua dissector, I get the expected output:
tshark.exe -r tcpfile.pcap -c 1 -T fields -e frame.number -e tcp.flags tcpflags.flags
1 0x00000002 ........S.
Since the same incorrect string is displayed in both cmd and powershell, it looks like a tshark bug to me, so I filed Wireshark Bug 16649.
This is the less I am using:
less 458 (POSIX regular expressions)
Copyright (C) 1984-2012 Mark Nudelman
In Vim it is \< and \>, in most other regex it is \b.
The character classes [[:<:]] and [[:>:]] match beginning and end of word, respectively, in system less on OS X 10.11.5. I haven't found a way to make the documented short forms \<, \>, or \b work.
(Thanks to denis for the suggestion to check man 7 re_format.)
Your version of less was built with posix regular expressions, as if:
wget http://ftp.gnu.org/gnu/less/less-451.tar.gz
tar zxf less-451.tar.gz
cd less-451
./configure --with-regex=posix
make
However, apparently the cause of whether \< works or not does NOT depend on this:
In Debian/Linux, \< will work fine even if you build with the above commands, with posix regex
In Mac OS X, I tried all possible values of --with-regex except pcre, and \< doesn't work with any of them. If I build with pcre, then \b works, instead of \<.
To conclude, I don't know how to make it work with \<. But you can build yourself with pcre and then it should work with \b. If you are not a sysadmin, you probably want to use a --prefix to install under your home directory, for example --prefix=$HOME/opt. After the make step, confirm it works with ./less /path/to/some/file. If looks good, then finish with make install.
First see if man 7 re_format on your computer has an "Enhanced features"
section which lists \< etc.
If it does, change one line in less-451/pattern.h:
#define REGCOMP_FLAG REG_ENHANCED // not REG_EXTENDED
Then ./configure --with-regex=posix; make less will understand \< .
This works on Macosx 10.8; on other systems, try following /usr/include/regex.h .
(Gnu.org has a round dozen
Regular-expression-syntaxes ?! )
less generally uses vi syntax, i.e. \< and \> unless it has been compiled with the --with-regex=none configure option or if the regular expression library found at compilation time doesn't provide word boundary search. Your system might also provide a different syntax.
I am really amazed by the functionality of GREP in shell, earlier I used to use substring method in java but now I use GREP for it and it executes in a matter of seconds, it is blazingly faster than java code that I used to write.(according to my experience I might be wrong though)
That being said I have not been able to figure out how it is happening? there is also not much available on the web.
Can anyone help me with this?
Assuming your question regards GNU grep specifically. Here's a note from the author, Mike Haertel:
GNU grep is fast because it AVOIDS LOOKING AT EVERY INPUT BYTE.
GNU grep is fast because it EXECUTES VERY FEW INSTRUCTIONS FOR EACH
BYTE that it
does look at.
GNU grep uses the well-known Boyer-Moore algorithm, which looks first
for the final letter of the target string, and uses a lookup table to
tell it how far ahead it can skip in the input whenever it finds a
non-matching character.
GNU grep also unrolls the inner loop of Boyer-Moore, and sets up the
Boyer-Moore delta table entries in such a way that it doesn't need to
do the loop exit test at every unrolled step. The result of this is
that, in the limit, GNU grep averages fewer than 3 x86 instructions
executed for each input byte it actually looks at (and it skips many
bytes entirely).
GNU grep uses raw Unix input system calls and avoids copying data
after reading it. Moreover, GNU grep AVOIDS BREAKING THE INPUT INTO
LINES. Looking for newlines would slow grep down by a factor of
several times, because to find the newlines it would have to look at
every byte!
So instead of using line-oriented input, GNU grep reads raw data into
a large buffer, searches the buffer using Boyer-Moore, and only when
it finds a match does it go and look for the bounding newlines
(Certain command line options like
-n disable this optimization.)
This answer is a subset of the information taken from here.
To add to Steve's excellent answer.
It may not be widely known but grep is almost always faster when grepping for a longer pattern-string than a short one, because in a longer pattern, Boyer-Moore can skip forward in longer strides to achieve even better sublinear speeds:
Example:
# after running these twice to ensure apples-to-apples comparison
# (everything is in the buffer cache)
$ time grep -c 'tg=f_c' 20140910.log
28
0.168u 0.068s 0:00.26
$ time grep -c ' /cc/merchant.json tg=f_c' 20140910.log
28
0.100u 0.056s 0:00.17
The longer form is 35% faster!
How come? Boyer-Moore consructs a skip-forward table from the pattern-string, and whenever there's a mismatch, it picks the longest skip possible (from last char to first) before comparing a single char in the input to the char in the skip table.
Here's a video explaining Boyer Moore (Credit to kommradHomer)
Another common misconception (for GNU grep) is that fgrep is faster than grep. f in fgrep doesn't stand for 'fast', it stands for 'fixed' (see the man page), and since both are the same program, and both use Boyer-Moore, there's no difference in speed between them when searching for fixed-strings without regexp special chars. The only reason I use fgrep is when there's a regexp special char (like ., [], or *) I don't want it to be interpreted as such. And even then the more portable/standard form of grep -F is preferred over fgrep.
We are having a discussion at work, what is the best UNIX command tool that to view log files. One side says use LESS, the other says use MORE. Is one better than the other?
A common problem is that logs have too many processes writing to them, I prefer to filter my log files and control the output using:
tail -f /var/log/<some logfile> | grep <some identifier> | more
This combination of commands allows you to watch an active log file without getting overwhelmed by the output.
I opt for less. A reason for this is that (with aid of lessopen) it can read gzipped log (as archived by logrotate).
As an example with this single command I can read in time ordered mode dpkg log, without treating differently gzipped ones:
less $(ls -rt /var/log/dpkg.log*) | less
Multitail is the best option, because you can view multiple logs at the same time. It also colors stuff, and you can set up regex to highlight entries you're looking for.
You can use any program: less, nano, vi, tail, cat etc, they differ in functionality.
There are also many log viewers: gnome-system-log, kiwi etc (they can sort log by date / type etc)
Less is more. Although since when I'm looking at my logs I'm typically searching for something specific or just interested in the last few events I find myself using cat, pipes and grep or tail rather than more or less.
less is the best, imo. It is light weight compared to an editor, it allows forward and backward navigation, it has powerful search capabilities, and many more things. Hit 'h' for help. It's well worth the time getting familiar with it.
On my Mac, using the standard terminal windows, there's one difference between less and more, namely, after exiting:
less leaves less mess on my screen
more leaves more useful information on my screen
Consequently, if I think I might want to do something with the material I'm viewing after the viewer finishes (for example, copy'n'paste operations), I use more; if I don't want to use the material after I've finished, then I use less.
The primary advantage of less is the ability to scroll backwards; therefore, I tend to use less rather than more, but both have uses for me. YMMV (YMWV; W = Will in this case!).
As your question was generically about 'Unix systems', keep into account that
in some cases you have no choice, for old systems you have only MORE available,
but not LESS.
LESS is part of the GNU tools, MORE comes from the UCB times.
Turn on grep's line buffering mode.
Using tail (Live monitoring)
tail -f fileName
Using less (Live monitoring)
less +F fileName
Using tail & grep
tail -f fileName | grep --line-buffered my_pattern
Using less & grep
less +F fileName | grep --line-buffered my_pattern
Using watch & tail to highlight new lines
watch -d tail fileName
Note: For linux systems.
It seems like the only way to do this is to pass the -i parameter in when you initially run less. Does anyone know of some secret hack to make something like this work
/something to search for/i
You can also type command -I while less is running. It toggles case sensitivity for searches.
You can also set the environment variable LESS
I use LESS=-Ri, so that I can pump colorized output from grep into it, and maintain the ANSI colour sequences.
Another little used feature of less that I found is starting it with +F as an argument (or hitting SHIFT+F while in less). This causes it to follow the file you've opened, in the same way that tail -f <file> will. Very handy if you're watching log files from an application, and are likely to want to page back up (if it's generating 100's of lines of logging every second, for instance).
Add-on to what #Juha said: Actually -i turns on Case-insensitive with SmartCasing, i.e if your search contains an uppercase letter, then the search will be case-sensitive, otherwise, it will be case-insensitive. Think of it as :set smartcase in Vim.
E.g.: with -i, a search for 'log' in 'Log,..' will match, whereas 'Log' in 'log,..' will not match.
It appears that you can summon this feature on a per search basis like so:
less prompt> /search string/-i
This option is in less's interactive help which you access via h:
less prompt> h
...
-i ........ --ignore-case
Ignore case in searches that do not contain uppercase.
-I ........ --IGNORE-CASE
Ignore case in all searches.
...
I've not extensively checked but the help in less version 487 on MacOS as well as other Linux distros lists this option as being available.
On MacOS you can also install a newer version of less via brew:
$ brew install less
$ less --version
less 530 (POSIX regular expressions)
Copyright (C) 1984-2017 Mark Nudelman
References
less is always case-insensitive
When using -i flag, be sure to enter the search string completely in lower case, because if any letter is upper case, then its an exact match.
See also: the -I (capital i) flag of less(1) to change this behavior.