I am looking for extracting some information from log using awk and based on the information returned i want to grep the whole file and write all the output from gerp and awk to a file. I was able to extract few information form awk but while using grep inside awk i am not able to extract information. Please find the logs as follow.
2014-04-10 13:55:59,837 [WebContainer : 4] [com.cisco.ata.service.AtAService] WARN - AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest;APP_ID=CDCSDSATAUser.gen;VIEW_NAME=/EntitlementView[CCOID="frhocevar"]REQUEST_ID_STRING=-105411838, took 100 milliseconds.
Based on the REQUEST_ID_STRING i have to get usecaseID.
2014-04-10 13:55:59,800 [Thread-66] [com.cisco.ata.cla.CLAManager] INFO - CLAManager.getAttributeFromCLAMapping() took 6 ms, for useCaseID - UC41, condition= (CCOID=frhocevar), requestID= -105411838
i am extracting REQUEST_ID_STRING using awk but i am not able to extract "useCaseID" using grep.
Below is the command i am using.
grep -i -r 'AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest' /opt/httpd/logs/apps/atasvc/prod1/was70/*/*.log* |
awk 'BEGIN{count=0;}{if($14>1000){print $0}}' |
awk 'BEGIN{ FS=";"}
{a = substr($3,8)}
{b = substr($4,index($4,"/")+1,index($4,"]R")-index($4,"/"))}
{c = substr($4,index($4,"G=")+2,index($4,", took")-index($4,"G=")-2);}
{d = substr($1,0,index($1,":")-1)}
{e=grep command which will extract usecaseid from $d having file name}
{ print a","b","c","d","e} '
Please help me in this issue.
Thanks in advance
I'm incredibly tired, so this is likely not the best solution, but it uses some basic "awkisms" that make for some pretty good boilerplate starting points for a lot of stuff.
AirBoxOmega:~ d$ cat log
2014-04-10 13:55:59,837 [WebContainer : 4] [com.cisco.ata.service.AtAService] WARN - AtAService::AtAServiceRequest DetailMessage - module=ataservice;service=ataservicerequest;APP_ID=CDCSDSATAUser.gen;VIEW_NAME=/EntitlementView[CCOID="frhocevar"]REQUEST_ID_STRING=-105411838, took 100 milliseconds.
2014-04-10 13:55:59,800 [Thread-66] [com.cisco.ata.cla.CLAManager] INFO - CLAManager.getAttributeFromCLAMapping() took 6 ms, for useCaseID - UC41, condition= (CCOID=frhocevar), requestID= -105411838
AirBoxOmega:~ d$ cat stackHelp.awk
{
if ($0 ~ /AtAService::AtAServiceRequest DetailMessage/ && $(NF - 1) > 99) {
split($0, tmp, "[-,]")
slow[tmp[7]]++
}
if (slow[substr($NF,2)]) {
split($0, tmp, "[-,]")
print $NF tmp[8]
}
}
AirBoxOmega:~ d$ gawk -f stackHelp.awk log
-105411838 UC41
This uses a pretty basic awk concept where if you know that there is something common among your log lines (a sessionID, or some such) you create an array for that based on certain conditions (in this case that the log line contains a given string and that the next to last column is > 99). Then later when you run into that same sessionID, you can check to see if an array exists for it, and if so, pull out even more info.
You may need/want to add something to the second if statement so it's only checking log lines you care about, but honestly, awk is so fast that it probably won't matter. (I'm using gawk [via brew] as the version of awk that comes with OSX is somewhat lacking, but this code is basic enough that awk or gawk should work.)
If you need a better explanation of the code, I'll try to explain better.
Ninja edit: Few quit tips:
Don't use grep -i unless you really don't know the case you're looking for. Case insensitivity will make your searches MUCH slower
If you're not using any kind of regular expressions, use fgrep instead of grep. It's much faster out the box.
Learn how to ask questions efficiently. Your question was pretty clear, but use tags to make the log lines more readable, and remember that every technical question should include:
What your input is
What your output should be
What you tried
What you expected
What you got
Get good at awk. The world is slowly moving away from command line centric stuff, and people may say it's not worth it, but once you understand basic concepts in awk, it's easy to apply them elsewhere, be it python, log utilities, or just thinking in terms of data aggregation.
Related
I want to run certain actions on a group of lexicographically named files (01-09 before 10). I have to use a rather old version of FreeBSD (7.3), so I can't use yummies like echo {01..30} or seq -w 1 30.
The only working solution I found is printf "%02d " {1..30}. However, I can't figure out why can't I use $1 and $2 instead of 1 and 30. When I run my script (bash ~/myscript.sh 1 30) printf says {1..30}: invalid number
AFAIK, variables in bash are typeless, so how can't printf accept an integer argument as an integer?
Bash supports C-style for loops:
s=1
e=30
for i in ((i=s; i<e; i++)); do printf "%02d " "$i"; done
The syntax you attempted doesn't work because brace expansion happens before parameter expansion, so when the shell tries to expand {$1..$2}, it's still literally {$1..$2}, not {1..30}.
The answer given by #Kent works because eval goes back to the beginning of the parsing process. I tend to suggest avoiding making habitual use of it, as eval can introduce hard-to-recognize bugs -- if your command were whitelisted to be run by sudo and $1 were, say, '$(rm -rf /; echo 1)', the C-style-for-loop example would safely fail, and the eval example... not so much.
Granted, 95% of the scripts you write may not be accessible to folks executing privilege escalation attacks, but the remaining 5% can really ruin one's day; following good practices 100% of the time avoids being in sloppy habits.
Thus, if one really wants to pass a range of numbers to a single command, the safe thing is to collect them in an array:
a=( )
for i in ((i=s; i<e; i++)); do a+=( "$i" ); done
printf "%02d " "${a[#]}"
I guess you are looking for this trick:
#!/bin/bash
s=1
e=30
printf "%02d " $(eval echo {$s..$e})
Ok, I finally got it!
#!/bin/bash
#BSD-only iteration method
#for day in `jot $1 $2`
for ((day=$1; day<$2; day++))
do
echo $(printf %02d $day)
done
I initially wanted to use the cycle iterator as a "day" in file names, but now I see that in my exact case it's easier to iterate through normal numbers (1,2,3 etc.) and process them into lexicographical ones inside the loop. While using jot, remember that $1 is the numbers amount, and the $2 is the starting point.
I have downloaded the 250G dump of freebase data. I don't want to iterate my development on the big data. I want to extract a small subset of the data (may be a small domain or some 10 personalities and their information). This small subset will make my iterations faster and easier.
What's the best approach to partition the freebase data?
Is there any subset download provided by Google/Freebase?
This is feedback that we've gotten from many people using the data dumps. We're looking into how best to create such subsets. One approach would be to get all the data for a single domain like Film.
Here's how you'd get every RDF triple from the /film domain:
zgrep '\s<http://rdf\.freebase\.com/ns/film.' freebase-rdf-{date}.gz | gzip > freebase-films.gz
The tricky part is that this subset won't contain the names, images or descriptions which you most likely also want. So you'll need to get those like this:
zgrep '\s<http://rdf\.freebase\.com/ns/(type\.object|common\.topic)' freebase-rdf-{date}.gz | gzip > freebase-topics.gz
Then you'll possibly want to filter that subset down to only topic data about films (match only triples that start with the same /m ID) and concatenate that to the film subset.
It's all pretty straight-forward to script this with regular expressions but a lot more work than it should be. We're working on a better long-term solution.
I wanted to do a similar thing and I came up with the following command line.
gunzip -c freebase-rdf-{date}.gz | awk 'BEGIN { prev_1 = ""} { if (prev_1 != $1) { print '\n' } print $0; prev_1 = $1};' | awk 'BEGIN { RS=""} $0 ~ /type\.object\.type.*\/film\.film>/' > freebase-films.txt
It will give you all the triplets for all subjects that has the type film. (it assumes all subjects come in sorted order)
After this you can simply grep for the predicates that you need.
Just one remark for accepted post, variant for topics don't work for me, because if we want use regex we need to set -E parameter
zgrep -E '\s<http://rdf\.freebase\.com/ns/(type\.object|common\.topic)' freebase-rdf-{date}.gz | gzip > freebase-topics.gz
grep -F -f file1 file2
file1 is 90 Mb (2.5 million lines, one word per line)
file2 is 45 Gb
That command doesn't actually produce anything whatsoever, no matter how long I leave it running. Clearly, this is beyond grep's scope.
It seems grep can't handle that many queries from the -f option. However, the following command does produce the desired result:
head file1 > file3
grep -F -f file3 file2
I have doubts about whether sed or awk would be appropriate alternatives either, given the file sizes.
I am at a loss for alternatives... please help. Is it worth it to learn some sql commands? Is it easy? Can anyone point me in the right direction?
Try using LC_ALL=C . It turns the searching pattern from UTF-8 to ASCII which speeds up by 140 time the original speed. I have a 26G file which would take me around 12 hours to do down to a couple of minutes.
Source: Grepping a huge file (80GB) any way to speed it up?
So what I do is:
LC_ALL=C fgrep "pattern" <input >output
I don't think there is an easy solution.
Imagine you write your own program which does what you want and you will end up with a nested loop, where the outer loop iterates over the lines in file2 and the inner loop iterates over file1 (or vice versa). The number of iterations grows with size(file1) * size(file2). This will be a very large number when both files are large. Making one file smaller using head apparently resolves this issue, at the cost of not giving the correct result anymore.
A possible way out is indexing (or sorting) one of the files. If you iterate over file2 and for each word you can determine whether or not it is in the pattern file without having to fully traverse the pattern file, then you are much better off. This assumes that you do a word-by-word comparison. If the pattern file contains not only full words, but also substrings, then this will not work, because for a given word in file2 you wouldn't know what to look for in file1.
Learning SQL is certainly a good idea, because learning something is always good. It will hovever, not solve your problem, because SQL will suffer from the same quadratic effect described above. It may simplify indexing, should indexing be applicable to your problem.
Your best bet is probably taking a step back and rethinking your problem.
You can try ack. They are saying that it is faster than grep.
You can try parallel :
parallel --progress -a file1 'grep -F {} file2'
Parallel has got many other useful switches to make computations faster.
Grep can't handle that many queries, and at that volume, it won't be helped by fixing the grep -f bug that makes it so unbearably slow.
Are both file1 and file2 composed of one word per line? That means you're looking for exact matches, which we can do really quickly with awk:
awk 'NR == FNR { query[$0] = 1; next } query[$0]' file1 file2
NR (number of records, the line number) is only equal to the FNR (file-specific number of records) for the first file, where we populate the hash and then move onto the next line. The second clause checks the other file(s) for whether the line matches one saved in our hash and then prints the matching lines.
Otherwise, you'll need to iterate:
awk 'NR == FNR { query[$0]=1; next }
{ for (q in query) if (index($0, q)) { print; next } }' file1 file2
Instead of merely checking the hash, we have to loop through each query and see if it matches the current line ($0). This is much slower, but unfortunately necessary (though we're at least matching plain strings without using regexes, so it could be slower). The loop stops when we have a match.
If you actually wanted to evaluate the lines of the query file as regular expressions, you could use $0 ~ q instead of the faster index($0, q). Note that this uses POSIX extended regular expressions, roughly the same as grep -E or egrep but without bounded quantifiers ({1,7}) or the GNU extensions for word boundaries (\b) and shorthand character classes (\s,\w, etc).
These should work as long as the hash doesn't exceed what awk can store. This might be as low as 2.1B entries (a guess based on the highest 32-bit signed int) or as high as your free memory.
We are having a discussion at work, what is the best UNIX command tool that to view log files. One side says use LESS, the other says use MORE. Is one better than the other?
A common problem is that logs have too many processes writing to them, I prefer to filter my log files and control the output using:
tail -f /var/log/<some logfile> | grep <some identifier> | more
This combination of commands allows you to watch an active log file without getting overwhelmed by the output.
I opt for less. A reason for this is that (with aid of lessopen) it can read gzipped log (as archived by logrotate).
As an example with this single command I can read in time ordered mode dpkg log, without treating differently gzipped ones:
less $(ls -rt /var/log/dpkg.log*) | less
Multitail is the best option, because you can view multiple logs at the same time. It also colors stuff, and you can set up regex to highlight entries you're looking for.
You can use any program: less, nano, vi, tail, cat etc, they differ in functionality.
There are also many log viewers: gnome-system-log, kiwi etc (they can sort log by date / type etc)
Less is more. Although since when I'm looking at my logs I'm typically searching for something specific or just interested in the last few events I find myself using cat, pipes and grep or tail rather than more or less.
less is the best, imo. It is light weight compared to an editor, it allows forward and backward navigation, it has powerful search capabilities, and many more things. Hit 'h' for help. It's well worth the time getting familiar with it.
On my Mac, using the standard terminal windows, there's one difference between less and more, namely, after exiting:
less leaves less mess on my screen
more leaves more useful information on my screen
Consequently, if I think I might want to do something with the material I'm viewing after the viewer finishes (for example, copy'n'paste operations), I use more; if I don't want to use the material after I've finished, then I use less.
The primary advantage of less is the ability to scroll backwards; therefore, I tend to use less rather than more, but both have uses for me. YMMV (YMWV; W = Will in this case!).
As your question was generically about 'Unix systems', keep into account that
in some cases you have no choice, for old systems you have only MORE available,
but not LESS.
LESS is part of the GNU tools, MORE comes from the UCB times.
Turn on grep's line buffering mode.
Using tail (Live monitoring)
tail -f fileName
Using less (Live monitoring)
less +F fileName
Using tail & grep
tail -f fileName | grep --line-buffered my_pattern
Using less & grep
less +F fileName | grep --line-buffered my_pattern
Using watch & tail to highlight new lines
watch -d tail fileName
Note: For linux systems.
I have a text file (more correctly, a “German style“ CSV file, i.e. semicolon-separated, decimal comma) which has a date and the value of a measurement on each line.
There are stretches of faulty values which I want to remove before further work. I'd like to store these cuts in some script so that my corrections are documented and I can replay those corrections if necessary.
The lines look like this:
28.01.2005 14:48:38;5,166
28.01.2005 14:50:38;2,916
28.01.2005 14:52:38;0,000
28.01.2005 14:54:38;0,000
(long stretch of values that should be removed; could also be something else beside 0)
01.02.2005 00:11:43;0,000
01.02.2005 00:13:43;1,333
01.02.2005 00:15:43;3,250
Now I'd like to store a list of begin and end patterns like 28.01.2005 14:52:38 + 01.02.2005 00:11:43, and the script would cut the lines matching these begin/end pairs and everything that's between them.
I'm thinking about hacking an awk script, but perhaps I'm missing an already existing tool.
Have a look at sed:
sed '/start_pat/,/end_pat/d'
will delete lines between start_pat and end_pat (inclusive).
To delete multiple such pairs, you can combine them with multiple -e options:
sed -e '/s1/,/e1/d' -e '/s2/,/e2/d' -e '/s3/,/e3/d' ...
Firstly, why do you need to keep a record of what you have done? Why not keep a backup of the original file, or take a diff between the old & new files, or put it under source control?
For the actual changes I suggest using Vim.
The Vim :global command (abbreviated to :g) can be used to run :ex commands on lines that match a regex. This is in many ways more powerful than awk since the commands can then refer to ranges relative to the matching line, plus you have the full text processing power of Vim at your disposal.
For example, this will do something close to what you want (untested, so caveat emptor):
:g!/^\d\d\.\d\d\.\d\d\d\d/ -1 write tmp.txt >> | delete
This matches lines that do NOT start with a date (the ! negates the match), appends the previous line to the file tmp.txt, then deletes the current line.
You will probably end up with duplicate lines in tmp.txt, but they can be removed by running the file through uniq.
you are also use awk
awk '/start/,/end/' file
I would seriously suggest learning the basics of perl (i.e. not the OO stuff). It will repay you in bucket-loads.
It is fast and simple to write a bit of perl to do this (and many other such tasks) once you have grasped the fundamentals, which if you are used to using awk, sed, grep etc are pretty simple.
You won't have to remember how to use lots of different tools and where you would previously have used multiple tools piped together to solve a problem, you can just use a single perl script (usually much faster to execute).
And, perl is installed on virtually every unix/linux distro now.
(that sed is neat though :-)
use grep -L (print none matching lines)
Sorry - thought you just wanted lines without 0,000 at the end