My netcdf4 file has the following header for the time variable
float time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:units = "hours since 1900-01-01 00:00:00.0" ;
time:calendar = "gregorian" ;
time:axis = "T" ;
With the following command
ncdump -tv time infile.nc
time format looks like
"2009-12-31 22"
I want to make it so that it is like
"2020-12-31 22:00"
How do I achieve this with either nco or cdo??
If the file is not huge you could try
cdo -outputtab,date,time infile.nc | sort | uniq
I think it also assumes that time is the unlimited dimension, and may produce errors otherwise.
Related
2021-08-04T22:55:12+0000
I want to convert the above string to local time.
But fromdateiso8601 does not work on this format. What is the best way to convert this kind of string to local time?
EDIT: I tried the following but the part of 23:55:12 does not change when the timezone is changed. I'd expect that this should be changed as TZ is changed.
$ TZ=America/New_York jq '.x | gsub("[+]0000"; "Z") | fromdateiso8601| gmtime | strftime("%Y-%m-%dT%H:%M:%S%Z")' <<< '{"x": "2021-08-04T22:55:12+0000" }'
"2021-08-04T23:55:12EST"
$ TZ=Europe/Madrid jq '.x | gsub("[+]0000"; "Z") | fromdateiso8601| gmtime | strftime("%Y-%m-%dT%H:%M:%S%Z")' <<< '{"x": "2021-08-04T22:55:12+0000" }'
"2021-08-04T23:55:12CET"
The first problem here is that jq's handling of timezones (TZ) is buggy; the second is that jq's built-ins do not recognize timezone offsets.
Unfortunately, there is to my knowledge not much that can easily be
done about jq's TZ-related bugginess other than using gojq, the Go
implementation of jq, instead.
Fortunately, time offsets can be handled quite easily, e.g. using the datetime_to_seconds filter as defined below.
So, for the time being, the following solution to the stated problem assumes the use of gojq rather than stedolan/jq. It has two main steps:
Use the generic filter datetime_to_seconds to convert the timestamp with an offset to "seconds since the epoch";
Use strflocaltime, which recognizes the environment variable TZ.
The solution is embedded in a bash script to facilitate a comparison between different versions of jq and gojq.
bash script
#!/bin/bash
# Syntax: go TZ
function go {
TZ="$1" $jq -Rr '
# Convert a timestamp with a possibly empty timezone offset to seconds since the Epoch.
# Input should be a string of the form yyyy-mm-ddThh:mm:ss or yyyy-mm-ddThh:mm:ss<OFFSET>
# where <OFFSET> is Z, or has the form [-+]hh:mm or [-+]hhmm
# If no timezone offset is explicitly given, it is taken to be Z.
def datetime_to_seconds:
if test("[-+]")
then
sub("(?<s>[-+])(?<d1>[0-9]{2})(?<d2>[0-9]{2})$"; "\(.s)\(.d1):\(.d2)")
| capture("(?<datetime>^.*T[0-9:]+)(?<s>[-+])(?<hh>[0-9]+):?(?<mm>[0-9]*)")
| (.datetime +"Z" | fromdateiso8601) as $seconds
| (if .s == "+" then -1 else 1 end) as $plusminus
| (.mm | if . == "" then 0 else . end) as $mm
| ([.hh,$mm] | map(tonumber) |.[0] *= 60 | add * 60 * $plusminus) as $offset
| ($seconds + $offset)
else . + (if test("Z") then "" else "Z" end) | fromdateiso8601
end;
datetime_to_seconds
| strflocaltime("%Y-%m-%dT%H:%M:%S %Z")
'
}
for jq in jq-1.6 jqMaster gojq ; do
echo $jq is $($jq --version)
done
echo
for TZ in America/New_York Europe/Madrid ;do
for jq in jq-1.6 jqMaster gojq ; do
for time in 2021-08-04T22:55:12+0000 ; do
echo $jq $TZ $time
echo $time | go $TZ
echo
done
done
done
Output
Here is the output with some "#" annotations.
jq-1.6 is jq-master-2e01ff1
jqMaster is jq-1.6-129-g80052e5-dirty
gojq is gojq 0.12.4 (rev: 244f9f7/go1.16.4)
jq-1.6 America/New_York 2021-08-04T22:55:12+0000
2021-08-04T19:55:12 EST # wrong
jqMaster America/New_York 2021-08-04T22:55:12+0000
2021-08-04T18:55:12 EST # wrong
gojq America/New_York 2021-08-04T22:55:12+0000
2021-08-04T18:55:12 EDT # correct
jq-1.6 Europe/Madrid 2021-08-04T22:55:12+0000
2021-08-05T01:55:12 CET # wrong
jqMaster Europe/Madrid 2021-08-04T22:55:12+0000
2021-08-05T00:55:12 CET # wrong
gojq Europe/Madrid 2021-08-04T22:55:12+0000
2021-08-05T00:55:12 CEST # correct
I have a netcdf file, with a daily time step, that I wish to convert to a monthly time step.
The time is formatted as follow:
double time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:bounds = "time_bnds" ;
time:units = "days since 2000-01-01" ;
time:calendar = "standard" ;
time:axis = "T" ;
When I convert to monthly time step using the command:
cdo monmean input.nc output.nc
Everything works fine except that the time output is strange:
time = "2000-01-16", "2000-02-15", "2000-03-16", "2000-04-15 12",
"2000-05-16", "2000-06-15 12", "2000-07-16", "2000-08-16",
"2000-09-15 12", "2000-10-16", "2000-11-15 12", "2000-12-16";
I wish to replace the day on the monthly value by the first day of the month and also remove those odd 12's for the time that appear. The desired output:
time = "2000-01-01", "2000-02-01", "2000-03-01", "2000-04-01",
"2000-05-01", "2000-06-01", "2000-07-01", "2000-08-01",
"2000-09-01", "2000-10-01", "2000-11-01", "2000-12-01";
Any hints is appreciate
cdo --timestat_date first monmean input.nc output.nc
works for me, I hope it's helpful! It places the timestamp at the first step of the averaging period, whereas the default is in the middle. (There is also a --timestat_date last if one want to do the opposite and put it at the last step of the window)
I have a file having a column with start time in the format shown below.
2019-10-30T08:04:30Z
2019-10-30T08:04:25Z
2019-10-30T08:04:30Z
Also I have a file having run time of the jobs executed in the format of milliseconds.
2647ms
360ms
10440ms
.
.
.
How do I add the respective rows in both columns in those files and produce the end time as a result in seperate file?
paste start_time.txt execution_time.txt | while IFS="$(printf '\t')" read -r f1 f2
do
# convert execution_time in seconds from ms
execution_time_ms="$(echo $f2 | cut -d'm' -f1)"
execution_time_ms="$(echo $execution_time_ms | cut -d's' -f1)"
execution_time_s=$(awk "BEGIN {printf \"%.10f\",$execution_time_ms/1000}")
# create dateformat string to add seconds
date_format=$f1
date_format+=" +$execution_time_s"
date_format+="seconds"
# create end date using dateformat string
end_time=`date --date="$date_format" +'%Y-%m-%d %H:%M:%S'`
end_time="${end_time/ /T}"
end_time+='Z'
# append end time
echo $end_time >> end_time.txt
done
This is a script which solves your problem.
I have used 3 files as per you mentioned.
start_time.txt (Input file)
execution_time.txt (Input file)
end_time.txt (Output file)
I have a netcdf file containing 4-D variables:
variables:
double maxvegetfrac(time_counter, veget, lat, lon) ;
maxvegetfrac:_FillValue = 1.00000002004088e+20 ;
maxvegetfrac:history = "From Topo.115MaCTRL_WAM_360_180" ;
maxvegetfrac:long_name = "Vegetation types" ;
maxvegetfrac:missing_value = 1.e+20f ;
maxvegetfrac:name = "maxvegetfrac" ;
maxvegetfrac:units = "-" ;
double mask_veget(time_counter, veget, lat, lon) ;
mask_veget:missing_value = -1.e+34 ;
mask_veget:_FillValue = -1.e+34 ;
mask_veget:long_name = "IF MYVEG4 EQ 10 AND I GE 610 AND J GT 286 THEN 16 ELSE MYVEG4" ;
mask_veget:history = "From desert_115Ma_3" ;
I'd like to use the variable "mask_veget" as a mask to alter values of the variable "maxvegetfrac" over specific regions, and over chosen values of its "veget" dimension.
To do so I am using ncap2. For example, if I want to set maxvegetfrac values over the 5th rank of veget dimension to 500 where mask_veget equals 6, I do :
> ncap2 -s "where (mask_veget(:,:,:,:)== 6) maxvegetfrac(:,5,:,:) = 500" test.nc
My problem is that in the resulting test.nc file, maxvegetfrac has been modified at the first rank of "veget" dimension, not the 5th one. And I get the same result if I run the script over the entire veget dimension:
ncap2 -s "where (mask_veget(:,:,:,:)== 6) maxvegetfrac(:,:,:,:) = 500" test.nc
So I am mistaking somewhere, but... where ?
Any help appreciated !
A couple of things you may not be aware of
you shouldn't be hyperslabbing a variable in the where body -it makes no sense at the moment.
It is ok to hyperslab in the where statement proving its a single index
as a dim with a single value collapses
Try this:
/*** hyper.nco *****/
maxvegetfrac5=maxvegetfrac(:,5,:,:);
where( mask_veget(:,5,:,:)== 6 )
maxvegetfrac5=500.0;
/* put the hyperslab back in */
maxvegetfrac(:,5,:,:)=maxvegetfrac5;
/* script end *****/
run the script now with the command
ncap2 -v -O -S hyper.nco test.nc out.nc
...Henry
I want to know about how to represent the 'datetime' format such as "yyyy-MM-dd hh:mm" using EBNF.
One possible way of formulating this using EBNF is shown below. The expression parses only legal years, months and timestamps. However, it allows any month to have up to 31 days.
Timestamp = [ "-" ] Year "-" Month "-" Day " " Time ;
Year = Digit Digit Digit Digit ;
Month = "0" Digit | "1" "0".."2" ;
Day = "0".."2" Digit | "3" "0".."1" ;
Time = Hour ":" Minute ;
Hour = "0".."1" Digit | "2" "0".."3" ;
Minute = "0".."5" Digit ;
Digit = "0".."9" ;