I'm looking to feed grep a pattern file with -f, but add a literal tab before and after each pattern (from the file). This method will allow me to grep for an exact column match, since I am dealing with a tab-separated file.
Yes, I have to use Grep. Awk and perl can't seem to handle my large pattern file.
And Yes, I could just add tabs to the pattern file, but I have many pattern files, so that would take a long time, but if all fails, that is what I'll do.
This should be useful for anyone looking to do an exact column match inside of a tsv file.
I'd try adding the tabs "on the fly":
grep -f <(sed 's/^/\\t/; s/$/\\t/' patterns) [args...]
to let grep interpret \t as tabs.
Related
Is there a way to get XML or equivalent output of grep command that can be passed on to other programs.
For example, grep can give the file names, line numbers and context of the pattern matched.
Filename and line number extraction can be done using some split command with delimiter ':'. However, if the filename contains ':' character (I know it is weird, but there is a possibility), it would need lot more processing.
With the context (grep -C option), it becomes even more complex. If the context of two matches overlaps, grep optimizes the output and it will be difficult to separate.
So I am wondering if grep command can simply generate an XML or JSON like output that other programs can just load.
There is an option -Z to grep which produces unambiguous output, by using Nul characters.
I'm searching through text files using grep and sed commands and I also want the file names displayed before my results. However, I'm trying to remove part of the file name when it is displayed.
The file names are formatted like this: aja_EPL_1999_03_01.txt
I want to have only the date without the beginning letters and without the .txt extension.
I've been searching for an answer and it seems like it's possible to do that with a sed or a grep command by using something like this to look forward and back and extract between _ and .txt:
(?<=_)\d+(?=\.)
But I must be doing something wrong, because it hasn't worked for me and I possibly have to add something as well, so that it doesn't extract only the first number, but the whole date. Thanks in advance.
Edit: Adding also the working command I've used just in case. I imagine whatever command is needed would have to go at the beginning?
sed '/^$/d' *.txt | grep -P '(^([A-ZÖÄÜÕŠŽ].*)?[Pp][Aa][Ll]{2}.*[^\.]$)' *.txt --colour -A 1
The results look like this:
aja_EPL_1999_03_02.txt:PALLILENNUD : korraga üritavad ümbermaailmalendu kaks meeskonda
A desired output would be this:
1999_03_02:PALLILENNUD : korraga üritavad ümbermaailmalendu kaks meeskonda
First off, you might want to think about your regular expression. While the one you have you say works, I wonder if it could be simplified. You told us:
(^([A-ZÖÄÜÕŠŽ].*)?[Pp][Aa][Ll]{2}.*[^\.]$)
It looks to me as if this is intended to match lines that start with a case insensitive "PALL", possibly preceded by any number of other characters that start with a capital letter, and that lines must not end in a backslash or a dot. So valid lines might be any of:
PALLILENNUD : korraga üritavad etc etc
Õlu on kena. Do I have appalling speling?
Peeter Pall is a limnologist at EMU!
If you'd care to narrow down this description a little and perhaps provide some examples of lines that should be matched or skipped, we may be able to do better. For instance, your outer parentheses are probably unnecessary.
Now, let's clarify what your pipe isn't doing.
sed '/^$/d' *.txt
This reads all your .txt files as an input stream, deletes any empty lines, and prints the output to stdout.
grep -P 'regex' *.txt --otheroptions
This reads all your .txt files, and prints any lines that match regex. It does not read stdin.
So .. in the command line you're using right now, your sed command is utterly ignored, as sed's output is not being read by grep. You COULD instruct grep to read from both files and stdin:
$ echo "hello" > x.txt
$ echo "world" | grep "o" x.txt -
x.txt:hello
(standard input):world
But that's not what you're doing.
By default, when grep reads from multiple files, it will precede each match with the name of the file from whence that match originated. That's also what you're seeing in my example above -- two inputs, one x.txt and the other - a.k.a. stdin, separated by a colon from the match they supplied.
While grep does include the most minuscule capability for filtering (with -o, or GNU grep's \K with optional Perl compatible RE), it does NOT provide you with any options for formatting the filename. Since you can'd do anything with the output of grep, you're limited to either parsing the output you've got, or using some other tool.
Parsing is easy, if your filenames are predictably structured as they seem to be from the two examples you've provided.
For this, we can ignore that these lines contain a file and data. For the purpose of the filter, they are a stream which follows a pattern. It looks like you want to strip off all characters from the beginning of each line up to and not including the first digit. You can do this by piping through sed:
sed 's/^[^0-9]*//'
Or you can achieve the same effect by using grep's minimal filtering to return every match starting from the first digit:
grep -o '[0-9].*'
If this kind of pipe-fitting is not to your liking, you may want to replace your entire grep with something in awk that combines functionality:
$ awk '
/[\.]$/ {next} # skip lines ending in backslash or dot
/^([A-ZÖÄÜÕŠŽ].*)?PALL/ { # lines to match
f=FILENAME
sub(/^[^0-9]*/,"",f) # strip unwanted part of filename, like sed
printf "%s:%s\n", f, $0
getline # simulate the "-A 1" from grep
printf "%s:%s\n", f, $0
}' *.txt
Note that I haven't tested this, because I don't have your data to work with.
Also, awk doesn't include any of the fancy terminal-dependent colourization that GNU grep provides through the --colour option.
I am very new to Unix.
I have a parameter file Parameter.prm containing following lines.
$$ErrorTable1=ErrorTable1
$$Filename1_New=FileNew.txt
$$Filename1_Old=FileOld.txt
$$ErrorTable2=ErrorTable2
$$Filename2_New=FileNew.txt
$$Filename2_Old=FileOld.txt
$$ErrorTable3=ErrorTable3
$$Filename3_New=FileNew.txt
$$Filename3_Old=FileOld.txt
I want get the output as
$$ErrorTable1=ErrorTable1
$$ErrorTable2=ErrorTable2
$$ErrorTable3=ErrorTable3
Basically, I need to delete line starting with $$Filename.
Since $ is a keyword, I am not able to interpret it as a string. How can I accomplish this using sed?
With sed:
$ sed '/$$Filename/d' infile
$$ErrorTable1=ErrorTable1
$$ErrorTable2=ErrorTable2
$$ErrorTable3=ErrorTable3
The /$$Filename/ part is the address, i.e., for all lines matching this, the command following it will be executed. The command is d, which deletes the line. Lines that don't match are just printed as is.
Extracting information from a textfile based on pattern search is a job for grep:
grep ErrorTable file
or even
grep -F '$$ErrorTable' file
-F tells grep to treat the search term as a fixed string instead of a regular expression.
Just to answer your question, if a regular expression needs to search for characters which have a special meaning in the regex language, you need to escape them:
grep '\$\$ErrorTable' file
I have a a text file which contains E-mail addresses surrounded by a lot of garbage.
I need to separate just the E-mail adresses, and write each address in a separate line (or separate between them with a comma).
the text file looks like that:
per#netvision.net אירית שנהב;רוני אשכול 99; מרכז האולפן 99; דפני אלפר; תים רון; (eina#gmail.com) אינה דגן 9303; (ori#gmail.com) אילן דור 9406; 9304 אורי
I think to "catch" all the words that start with [A-Z] and end with [A-Z] because all the garbage around is not letter in English ([A-Z]).
Can someone show me how to write this script?
I would do this using grep -o. It's not precisely "in" tcsh, but you can use grep from any script. The -o option causes grep to return only the text matched by the regexp.
It looks as if your input file currently separates records using a semi-colon. This is important, since grep reads things line-by-line. So we will use tr to replace your record separators with newlines to ensure that grep sees each record.
ghoti#pc> cat strip_email
#!/bin/tcsh
setenv inputfile emails.txt
setenv re_email '[[:alnum:]][[:alnum:]_%=+-]*#([[:alnum:]]([[:alnum:]-])+\.)+[[:alnum:]]{2,}'
tr ';' '\n' < $inputfile | grep -Eo "$re_email"
ghoti#pc> ./strip_email
per#netvision.net
eina#gmail.com
ori#gmail.com
ghoti#pc>
You can adapt the regular expression to whatever suits you. On a unix or linux system, you can check man pages: re_format(7) or regex(7) for documentation. The [:alnum:] piece is documented there, as well as isalnum.
Ok, So here is the issue. I am trying to create an awk program that adds a few characters to a column in a file. Simple enough, but the problem is the file contains characters awk interprets as escape or special characters, such as \ ^ & and /... I want awk to act as if all characters in between the field separator (or any non field or new record character actually) are simply supposed to be there and don't convey special informatoin. i don't want it to interpret any of the file in any special way. Is there a way to do this?
Judging from your comments, it seems that you are telling awk to use the file as if it were a program rather than treating it as data. Try:
awk -F\| '{print $2}' NH3