I have a daily netcdf file (20060101) and the time dimension has 144 steps, so 10 minutes time steps within the file. Now I want to create one netcdf file for each 10 min time step with nco. So at the end it should create 144 files.
I already found similar questions where they use
ncks -d time,min,max,stride infile.nc outfile.nc
But I don't know how to write a code creating these 144 files.
Can someone help me with this?
Thanks for your help!
Found Solution:
start=1131580800 # first time step in seconds
for f in *_test.nc; do #$files
echo 'Processing' $f
for i in {0..17}; do
time=$(expr $start + 600 \* $i) # calculates the next time step --> 600s correspond to 10 minute steps I wanted to have
time_new=$(date -d #$time '+_%H%M') # extracting the time variable from my file and convert it to use them in the file names
outfile=$(echo $f|sed "s/_level2/$time_new/g")
ncks -d time,$i,$i $f $outfile # split my original file to 10 min steps
done
done
I haven't tested it but I think you can do it with one line in CDO:
cdo splitsel,1 input.nc output
The "1" means split the file with 1 timestep in each output file, which I think is what you were asking for.
Related
I am using monthly global potential evapotranspiration data from TerraClimate from 1958-2020 (available as 1 nc per year) and planning to concatenate all into single nc file.
The data has a variable pet and three dimension ppt(time,lat,lon).
I managed to combine all of the data using cod mergetime TerraClimate_*.nc and generate around 100GB of output file.
For analysis purpose in Windows machine, I need single netCDF file with order lat,lon,time. What I have done is as follows:
Reorder the dimension from time,lat,lon into lat,lon,time using ncpdq command
for fl in *.nc; do ncpdq -a lat,lon,time $fl ../pet2/$fl; done
Loop all file in the folder to make time the record dimension/variable used for concatenating files using ncks command
for fl in *.nc; do ncks -O --mk_rec_dmn time $fl $fl; done
Concatenates all nc files in the folder into one nc file using ncrcat command
ncrcat -h TerraClimate_*.nc -O TerraClimate_pet_1958_2020.nc
It's worked, but the result is not what I expected, it generate 458KB size of file, when I check the result using Panoply it provide wrong information, all have value -3276.7. See below picture.
I have check the files from step 1 and 2, and everything is correct.
I also try to concatenate only 2 files, using 1958 and 1959 data (each file 103MB), but the result still not what I expected.
ncrcat -h TerraClimate_pet_1958.nc TerraClimate_pet_1959.nc -O ../TerraClimate_pet_1958_1959.nc
Did I missed something on the code or write the wrong code? Any suggestion how to solve the problem?
UPDATE 1 (22 Oct 2021):
Here's the metadata of original data downloaded from above link.
UPDATE 2 (23 Oct 2021):
Following suggestion from Charlie, I did unpack for all the data from point 2 above using below command.
for fl in *.nc4; do ncpdq --unpack $fl ../unpack/$fl; done
Here's the example metadata from unpack process.
And the data visualised using Panoply.
Then I did test to concatenate again using 2 data from unpack process (1958 and 1959)
ncrcat -h TerraClimate_pet_1958.nc TerraClimate_pet_1959.nc -O ../TerraClimate_pet_1958_1959.nc
Unfortunately the result remain same, I got result with size 1MB. Below is the metadata
And visualised the ncrcat result using Panoply
Your commands appear to be correct, however I suspect that the data in the input files is packed. As explained in the ncrcat documentation here, the input data should be unpacked (e.g., with ncpdq --unpack) prior to concatenating all the input files (unless they all share the same values of scale_factor and add_offset). If that does not solve the problem, then (1) there is likely an issue with _FillValue and (2) please post the pet metadata from a sample input file.
I have two NetCDF files of the Greenland ice sheet velocities, one from 2015 and one from 2016. These files contain grided data where the velocity is plotted with x,y coordinates. However, no time dimension is included. How can I merge these two files into 1, where the final file has a time dimension? So in stead of two separate x,y,z grids, I would like to have one x,y,z,t data structure, where time = 2.
Thanks!
If the files contain the same variables and are the same size, try ncecat
ncecat -u time file1.nc file2.nc out.nc
You can add a time dimension to a file with ncap2:
ncap2 -s 'defdim("time",1);time[time]=74875.0;time#long_name="Time"; etc.etc.etc.' -O ~/nco/data/in.nc ~/foo.nc
I suggest reading this thread for more details: https://sourceforge.net/p/nco/discussion/9830/thread/cee4e1ad/
After you have done that you can merge them together either using the ncrcat command (see https://linux.die.net/man/1/ncrcat) or also with cdo
cdo mergetime file1.nc file2.nc combined_file.nc
Here is what my log file look like
[BCDID::16 T::LIVE_ANALYZER GCDID::16] {"t":"20:50:05","k":"7322","h":"178","s":-53.134575556764}
[BCDID::16 T::LIVE_ANALYZER GCDID::16] {"t":"20:50:06","k":"2115","h":"178","s":-53.134575556764}
[BCDID::16 T::LIVE_ANALYZER GCDID::16] {"t":"20:50:07","k":"1511","h":"178","s":-53.134575556764}
There are multiple log files with similar entries and they are updated every second.
here "t" : "20:50:05" is Time.
What I want to do is, get all logs between specific time from all files from the end of the files.
I tried with tail files*.log | grep -e "20:50:07 | 20:50:05" but it does not return anything.
How do I get get all log entries between given time, starting from the end of file from all logs files?
If you're looking for a range for records, and the format of the lines is consistent, the easiest way is probably to isolate the time field, strip out the colons, and leverage the power of arithmetic operators.
A one-liner awk solution, for example:
tail files*.log | awk -v from="205006" -v to="205007" -F"\"" '{ timeasint=$4; gsub(":","",timeasint); if (timeasint >= from && timeasint <= to) print $0 }'
would get you:
[BCDID::16 T::LIVE_ANALYZER GCDID::16] {"t":"20:50:06","k":"2115","h":"178","s":-53.134575556764}
[BCDID::16 T::LIVE_ANALYZER GCDID::16] {"t":"20:50:07","k":"1511","h":"178","s":-53.134575556764}
Of course you couldn't span across midnight (i.e., 25:59:59 - 00:00:01), but for that you'd need dates as well as times in your log anyway.
If you had dates, my suggestion would be converting them to epoch stamps (using date -d "string" or some other suitable method) and comparing the epoch stamps as integers.
I have a requirement to run a unix script multiple times for different dates. Script should be executed first for 2017-01-01, then for 2017-02-01 and so on. We are checking the data on a monthly basis. Can someone help me with this please. Thanks.
You can use a for loop for iteration, in general. You can generate a sequence of months with {01..12} or seq -w 1 12, as below:
for date in 2017-{01..12}-01; do
echo Processing data for $date
# Use $date in some way
done
or
for month in $(seq -w 1 12); do
date=2017-$month-01
echo Processing data for $date
# use $date in some way
done
The first will work in Bash, zsh, and many other shells, but I don’t think is guaranteed to work for any of them; the second should work for any sh-like shell.
I have a year of data as monthly NetCDF files, and I'd like to turn them into daily files. I know I can use ncks to extract 24 hours, like this
ncks -d time,0,23 -o output.nc input.nc
but I'd like to automatically chunk the whole month into days, without having to worry about the number of days in each month and whatnot. Is there an easy way to do this? (I could of course write a python script or similar that calls ncks, but it would be more elegant to avoid that.)
The NCO -d hyperslab switch understands dates in UDUnits format. If your input file contains well-formatted units attribute for time, there should be no problem in writing twelve commands each with a date hyperslab like
ncks -d time,1918-01-01,1918-01-31 in.nc jan.nc
Other than that, there is no more elegant method currently supported by NCO.
From the question it is not clear to me if your monthly files contain daily data. i.e. you have 12 files and each file contains daily (or finer) time resolution information. If this is the case then I think what you want to do is very easy with cdo using
cdo splitday input.nc output.nc
you will end up with a number of files, each with a day of data, and the number of days in each month is handled automatically for you.