Extract text from variable in netCDF file using ncks - netcdf
I am trying to extract the variable "flash_lon" from a file and output to a text file in plain text - using ncks.
When I use the following command, it displays the variables I need on screen and outputs to a file.
ncks -v flash_lon -x file.nc output.txt
However, the file is not in readable text. In the documentation for ncks, it says that "ncks will print netCDF data in ASCII format ".
What do I need to do in order to simply extract the variable to text? It is just text. I have attached an image below showing the data in the command line working, surely there must be a way to get it to output. I am on Windows 10.
If you have ncdump and sed you can output just the data only like this
ncdump -v flash_lon file.nc | sed -e '1,/data:/d' -e '$d' > output.txt
A solution I use frequently and found here:
https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2011/msg00317.html
If you don't want even the first lines with the variable name, you can cut those with tail:
ncdump -v flash_lon file.nc | sed -e '1,/data:/d' -e '$d' | tail -n +3 > output.txt
Related
ncdump - How do I convert .nc to text without including nodata values?
I have a .nc file that I convert to text using the following command : ncdump in.nc > out.txt However, this particular file has thousands of "-999" values (for nodata). Is there a way to stop ncdump from adding these to the output?
Use sed, e.g., ncdump in.nc | sed -e 's/ -999,//g' > out.txt ncks in.nc | sed -e 's/_, //g' -e 's/ -999,//g' > out.txt Add -e commands until all unwanted patterns disappear.
Unix - How do you pass / pipe the output of a sed command to a ls command in Unix
I am able to determine the size of an input file using the following command :
ls -l employee.txt | awk '{print $5}'
This will print only the file size.
And I am able to print the contents of a file ignoring first and last line (refers to header & trailer) of the file. Below command does this work for me :
sed '1d;$d' employee.txt
But how to combine both these commands in such a way that it should determine the size of the file ignoring header and trailer. At the same time the header and trailer should not be removed from the input file.
I am able to achieve this by writing two statements. One to copy the full file into a new file except header and trailer and then doing a ls on the new file as below :
sed '1d;$d' employee.txt > employee1.txt
ls -l employee1.txt
I tried to do it in a single statement as below but to no avail. Any inputs will be helpful.
sed '1d;$d' employee.txt | ls -l employee.txt
ls -l `sed '1d;$d' employee.txt`
sed '1d;$d' employee.txt |xargs ls -l $1
Don't parse ls to get the file size. To get the size of a file on disk, use stat -c '%s' filename, or to get the size of a stream of characters, use wc -c. # size of employee.txt stat -c '%s' employee.txt # size without header and footer sed '1d;$d' employee.txt | wc -c
You can do it through cat someFilename.txt | wc -c The first command outputs the content of the file and the second one counts the size of the input. This does not remove the header and the trailer. For that you can use 'sed' command as before.
I think you're approaching this in the wrong way.
My suggestion: query the total file size using the "stat" command:
stat --format=%s employee.txt
You can store the result in a variable like this (assuming you're running a Bash shell):
fileSize="$(stat --format=%s employee.txt)"
Now you retrieve the first line (assuming Bash shell again):
firstLine="$(head -n 1 employee.txt)"
Next you retrieve the last line (assuming Bash shell again):
lastLine="$(tail -n 1 employee.txt)"
Now compute the size of the first and the last line and subtract it from the file size:
echo "Total adjusted size is:" $((fileSize - ${#firstLine} - ${#lastLine}))
using sed -n with variables
I am having a log file a.log and i need to extract a piece of information from it. To locate the start and end line numbers of the pattern i am using the following. start=$(sed -n '/1112/=' file9 | head -1) end=$(sed -n '/true/=' file9 | head -1) i need to use the variables (start,end) in the following command: sed -n '16q;12,15p' orig-data-file > new-file so that the above command appears something like: sed -n '($end+1)q;$start,$end'p orig-data-file > new-file I am unable to replace the line numbers with the variables. Please suggest the correct syntax. Thanks, Rosy
When I realized how to do it, I was looking for anyway to get line number into a file containing the requested info, and display the file from that line to EOF.
So, this was my way.
with
PATTERN="pattern"
INPUT_FILE="file1"
OUTPUT_FILE="file2"
line number of first match of $PATTERN into $INPUT_FILE can be retrieved with
LINE=`grep -n ${PATTERN} ${INPUT_FILE} | awk -F':' '{ print $1 }' | head -n 1`
and the outfile will be the text from that $LINE to EOF. This way:
sed -n ${LINE},\$p ${INPUT_FILE} > ${OUTPUT_FILE}
The point here, is the way how can variables be used with command sed -n:
first witout using variables
sed -n 'N,$p' <file name>
using variables
LINE=<N>; sed -n ${LINE},\$p <file name>
Remove the single quotes thus. Single quotes turn off the shell parsing of the string. You need shell parsing to do the variable string replacements.
sed -n '('$end'+1)q;'$start','$end''p orig-data-file > new-file
How to remove blank lines from a Unix file
I need to remove all the blank lines from an input file and write into an output file. Here is my data as below. 11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321 11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003 11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032 11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001 11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701 11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326 11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383 11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955 11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746
sed -i '/^$/d' foo This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original: sed '/^$/d' foo > foo.tmp mv foo.tmp foo If you also want to remove lines consisting only of whitespace (not just empty lines) then use: sed -i '/^[[:space:]]*$/d' foo Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too: sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo
awk 'NF' filename awk 'NF > 0' filename sed -i '/^$/d' filename awk '!/^$/' filename awk '/./' filename The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.
Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($): grep -v '^$' infile.txt > outfile.txt If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways: grep -P -v '^\s*$' infile.txt > outfile.txt or, without Perl regular expressions: grep -v '^[[:space:]]*$' infile.txt > outfile.txt
sed -e '/^ *$/d' input > output Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option. If I execute the above command still I have blank lines in my output file. What could be the reason? There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then: sed -e 's/ *$//' -e '/^ *$/d' input > output The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs. Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return: tr -d '\015' < input | sed -e 's/ *$//' -e '/^ *$/d' > output If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against: head -n 2 input | od -c Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.) You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with: echo "Hello World" | sed 's/[[:space:]][[:space:]]*/ /' If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.
grep . file grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.
with awk awk 'NF > 0' filename
To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it: cat file.txt | perl -lane "print if /\S/" Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do. Cheers
You can sed's -i option to edit in-place without using temporary file: sed -i '/^$/d' file
use of grep commands in unix
I have a file and i want to sort it according to a word and to remove the special characters. The grep command is used to search for the characters -b Display the block number at the beginning of each line. -c Display the number of matched lines. -h Display the matched lines, but do not display the filenames. -i Ignore case sensitivity. -l Display the filenames, but do not display the matched lines. -n Display the matched lines and their line numbers. -s Silent mode. -v Display all lines that do NOT match. -w Match whole word but How to use the grep command to do the file sort and remove the special character and number.
grep searches inside all the files to find matching text. It doesn't really sort and it doesn't really chop and change output. What you want is probably to use the sort command sort <filename> and the output sent to either the awk command or the sed command, which are common tools for manipulating text. sort <filename> | sed 's/REPLACE/NEW_TEXT/g' something like above I'd imagine.
The following command would do it. sort FILE | tr -d 'LIST OF SPECIAL CHARS' > NEW_FILE