I have a netcdf file containing maximum daily air temperature, time, lat and lon. I successfully got maximum temperature from a netcdf of 6 hourly temperatures using the nco command:
ncra -y max --mro -d time,,,4,4 6hourly.nc max.nc"
The only problem is, my time steps are still split into quarter days:
variables:
double lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "lon" ;
lon:standard_name = "longitude" ;
double lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "lat" ;
lat:standard_name = "latitude" ;
double time(time) ;
time:units = "days since 0850-01-01 00:00:00" ;
time:long_name = "time" ;
time:calendar = "proleptic_gregorian" ;
double tair(time, lat, lon) ;
tair:units = "K" ;
tair:_FillValue = 1.e+30 ;
tair:long_name = "2 meter air temperature" ;
tair:days\ since\ 850 = 0., 0.25, 0.5, 0.75, 1., 1.25, 1.5, 1.75, 2., 2.25, 2.5, 2.75, 3., 3.25, 3.5, 3.75, 4., 4.25, 4.5, 4.75, 5., 5.25, 5.5, 5.75, 6., 6.25, 6.5, 6.75, 7., 7.25, 7.5, 7.75, 8., 8.25, 8.5, 8.75, 9., 9.25, 9.5, 9.75, 10., 10.25, 10.5, 10.75, 11., 11.25, 11.5...
My question is, how do I change the time step for the 'days\ since\ 850' attribute in the variable tair, to whole numbers?
Thanks!
Charlotte
ncap2 can work with attributes. However, you have a particularly thorny issue because your attribute name contains whitespace, and the attribute value is an array. I think in this case you need to first rename the attribute, then manipulate it. (Then you can rename it back if desired.):
ncrename -O -a "tair#days since 850",tair#days_since_850 in.nc foo.nc
ncap2 -O -s 'tair#days_since_850=int(tair#days_since_850)' foo.nc out.nc
Edit 20210209 in answer to comment below:
To copy an attribute from one variable to another, try
ncap2 -s 'var1#att1=var2#att2' in.nc out.nc
In case anyone has a similar problem, this ended up working for me:
ncap2 -s 'time=array(0,1,$time)' outmax.nc outmax2.nc
ncap2 -s 'time=array(0,1,$time)' outmin.nc outmin2.nc
Related
I have a list of NetCDF files that I would like to open with the xarray.open_mfdataset function.
This would normally be trivial, however I am running into an issue because the files I cam trying to open do not have any "time" dimension included in them:
data
Out[51]:
<xarray.Dataset>
Dimensions: (lat: 850, lon: 1500)
Coordinates:
* lat (lat) float64 54.98 54.94 54.9 54.86 ... 21.14 21.1 21.06 21.02
* lon (lon) float64 -126.0 -125.9 -125.9 -125.9 ... -66.1 -66.06 -66.02
Data variables:
Data (lat, lon) float32 ...
When I try to open my list of files with open_mfdataset, I of course get an error:
xr.open_mfdataset(files)
ValueError: Could not find any dimension coordinates to use to order the datasets for concatenation
I however do have a list of dates corresponding to each file:
dates
Out[54]:
array([datetime.datetime(2009, 1, 1, 0, 0),
datetime.datetime(2009, 1, 2, 0, 0),
datetime.datetime(2009, 1, 3, 0, 0), ...,
datetime.datetime(2019, 12, 29, 0, 0),
datetime.datetime(2019, 12, 30, 0, 0),
datetime.datetime(2019, 12, 31, 0, 0)], dtype=object)
I assume there is some way I add a time dimension to each file and open them all with open_mfdataset, possibly with the "preprocess" argument.
Thanks for any help.
Here is my solution:
Create a function which adds a time dimension to a DataArray, and fill it with a arbitrary date:
def add_time_dim(xda):
xda = xda.expand_dims(time = [datetime.now()])
return xda
Then, pass this function to the preprocess argument when running the open_mfdataset functions:
data = xr.open_mfdataset(files, preprocess = add_time_dim)
Finally, fill the time dimension with my dates:
data['time'] = dates
I'm currently working on a frequency-substitution cipher in F#. Meaning that I count all the occurrences of each letter in a text and when that is done I want to replace the letters based on the letter frequency in the english alphabet.
What i've done so far is that i've created a (char * float * char) list that contains (letter, frequency percentage, recommended letter). Let's say the letter P is the most occurring letter in my ciphered text (13.5 percent of letters are P) and E is the most used letter in english texts our list element will look like this ('P', 13.5, 'E'). This procedure is done all letters in the text, so we will end up with a list of all letters and their recommended replacement.
The problem I have is that I don't really know how to replace the letters in the cipher text with their recommended replacements.
Letter frequency in the english alphabet.
[(' ', 20.0); ('E', 12.02); ('T', 9.1); ('A', 8.12); ('O', 7.68); ('I', 7.31);
('N', 6.95); ('S', 6.28); ('R', 6.02); ('H', 5.92); ('D', 4.32); ('L', 3.98);
('U', 2.88); ('C', 2.71); ('M', 2.61); ('F', 2.3); ('Y', 2.11); ('W', 2.09);
('G', 2.03); ('P', 1.82); ('B', 1.49); ('V', 1.11); ('K', 0.69); ('X', 0.17);
('Q', 0.11); ('J', 0.1); ('Z', 0.07)]
Letter frequency in cipher.
[('W', 21.18); ('Z', 8.31); ('I', 7.7); ('P', 6.96); ('Y', 5.5); ('H', 5.48);
('G', 5.35); ('K', 5.3); ('N', 4.31); ('O', 4.31); ('M', 3.66); (' ', 2.83);
('A', 2.58); ('T', 2.38); ('Q', 2.22); ('B', 2.11); ('F', 2.11); ('.', 2.04);
('R', 1.62); ('S', 1.37); ('E', 1.06); ('X', 0.97); ('U', 0.25); ('L', 0.16);
('V', 0.11); ('J', 0.07); ('C', 0.02); ('D', 0.02)]
Recommended letter changes.
[('W', 21.18, ' '); ('Z', 8.31, 'E'); ('I', 7.7, 'T'); ('P', 6.96, 'A');
('Y', 5.5, 'O'); ('H', 5.48, 'I'); ('G', 5.35, 'N'); ('K', 5.3, 'S');
('N', 4.31, 'R'); ('O', 4.31, 'H'); ('M', 3.66, 'D'); (' ', 2.83, ' ');
('A', 2.58, 'L'); ('T', 2.38, 'U'); ('Q', 2.22, 'C'); ('B', 2.11, 'M');
('F', 2.11, 'F'); ('.', 2.04, 'Y'); ('R', 1.62, 'W'); ('S', 1.37, 'G');
('E', 1.06, 'P'); ('X', 0.97, 'B'); ('U', 0.25, 'V'); ('L', 0.16, 'K');
('V', 0.11, 'X'); ('J', 0.07, 'Q'); ('C', 0.02, 'J'); ('D', 0.02, 'Z')]
If anyone have any ideas that would put me in the right direction on how to tackle the problem i'd be very appreciative since i've been stuck on this problem for some while now.
I believe you are missing . frequency in the English alphabet (should be between D and L. When you'll add missing value to alphaFreq list both lists will be of same length and you'll be able to produce recommended changes map by zipping two ordered lists:
let changes =
alphaFreq // list with letter frequency in the English alphabet
|> List.zip cipherFreq // zipping with cipher frequency list
|> List.map (fun ((cipherLetter,_), (alphaLetter,_)) -> (alphaLetter, cipherLetter))
|> Map.ofList
Encoding test:
"HELLO WORLD" |> String.map (fun ch -> changes.[ch]) |> printfn "%s"
// OZAAYWRYNAM
To get a decoder map just swap letter order -> (cipherLetter, alphaLetter)
Most likely you want a mapping operation. Something along the lines of myString |> String.map mappingFunction. Note that mapping function can also be a functor, or a curried higher order function.
The functor approach would allow you to put the frequencies into the object state.
The curried function would allow you to pass the frequencies as the parameter.
It's up to you to choose which approach makes more sense in your application and/or looks more natural.
Say I want to apply an array of functions to an array of objects. Something like this:
my $a = sub ($) { $_ * 2 };
my $b = sub ($) { $_ / 2 };
my #funcs = ($a, $b);
my #ops = #funcs.roll(4);
I could do
say ^10 ».&» #ops;
but .& is a postfix operator; same thing for >>., which could be used for a single function, however. Using
say ^10 Z. #ops;
with . actually being the infix operator for calling a method, yields an error about mistaking it for the concatenation operator. I don't think I have any other option left. Any idea?
You could use the &infix:« o » / &infix:« ∘ » operator, combined with a lambda/closure factory.
sub lambda-factory ( +in ) {
in.map: -> \_ { ->{_} }
}
my #funcs = (
* * 2,
* / 2
);
(#funcs X∘ lambda-factory ^10)».()
# (0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5)
(#funcs «∘» lambda-factory ^10)».()
# [0, 0.5, 4, 1.5, 8, 2.5, 12, 3.5, 16, 4.5]
( (|#funcs xx *) Z∘ lambda-factory ^10 )».()
# (0, 0.5, 4, 1.5, 8, 2.5, 12, 3.5, 16, 4.5)
If you have defined values, you can use the infix operator andthen.
say (^10 »andthen» (* * 2, * / 2).roll(4))
If you roll your own infix equivalent of .& the construct you are trying to create seems to work fine.
$ perl6 -e 'my sub infix:<foo> ($a, $b) { $b($a) }; say ^10 Z[foo] (-> $_ { $_*2 }, -> $_ { $_/2 }).roll(4);'
(0 0.5 1 6)
Let's say I have time data that looks like this in an xarray Dataset:
ds = xr.Dataset({'time': pd.date_range('2000-01-01', periods=10)})
ds.to_netcdf('asdf.nc')
xarray's to_netcdf() method outputs the time dimension as int64:
$ ncdump -v time asdf.nc
netcdf asdf {
dimensions:
time = 10 ;
variables:
int64 time(time) ;
time:units = "days since 2000-01-01 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
data:
time = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ;
}
Because I'm working with a THREDDS server which does not support int64, I would like for these time data to be int32. Is this possible to do using xarray?
You can specify the data type of each output variable via the encoding property or the encoding keyword argument to to_netcdf. In your example, this would simply look like:
ds.to_netcdf('asdf.nc', encoding={'time': {'dtype': 'i4'}})
More information on writing encoded data can be found in the xarray documentation: http://xarray.pydata.org/en/latest/io.html#writing-encoded-data
I need to know the precision of the data stored in a NetCDF file.
I think that it is possible to know this precision because, when I dump a NetCDF file using ncdump, the number of significant digits displayed depends on the particular NetCDF file that I am using.
So, for one file I get:
Ts = -0.2121478, -0.08816089, -0.4285178, -0.3446428, -0.4800949,
-0.4332879, -0.2057121, -0.06589077, -0.001647412, 0.007711744,
And for another one:
Ts = -2.01, -3.6, -1, -0.53, -1.07, -0.7, -0.56, -1.3, -0.93, -1.41, -0.83,
-0.8, -2.13, -2.91, -1.13, -1.2, -2.23, -1.77, -2.93, -0.7, -2.14, -1.36,
I also have to say that there is no information about precision in any attribute, neither global nor local to the variable. You can see this in the dump of the header of the NetCDF file:
netcdf pdo {
dimensions:
time = UNLIMITED ; // (809 currently)
variables:
double time(time) ;
time:units = "months since 1900-01-01" ;
time:calendar = "gregorian" ;
time:axis = "T" ;
double Ts(time) ;
Ts:missing_value = NaN ;
Ts:name = "Ts" ;
// global attributes:
:Conventions = "CF-1.0" ;
}
Does anybody know how can I get the number of significant digits of the data stored in a NetCDF file?.
This is a tricky question: what ncdump (and many other pretty number generators) does is simply strip the trailing zeros from the fractional part, but does that say anything about the real (observed/calculated/..) precision of the values? Something measured with three decimals accuracy might be 1.100, yet ncdump will still print it as 1.1. If you want to know the true (physical?) significance, it would indeed have to be included as an attribute, or documented elsewhere.
For a large set of numbers, counting the maximum number of significant digits in the fractional part of the numbers could be a first indication of the precision. If that is what you are looking for, something like this might work in Python:
import numpy as np
a = np.array([1.01, 2.0])
b = np.array([1.10, 1])
c = np.array([10., 200.0001])
d = np.array([1, 2])
def count_max_significant_fraction(array):
# Return zero for any integer type array:
if issubclass(array.dtype.type, np.integer):
return 0
decimals = [s.rstrip('0').split('.')[1] for s in array.astype('str')]
return len(max(decimals, key=len))
print( count_max_significant_fraction(a) ) # prints "2"
print( count_max_significant_fraction(b) ) # prints "1"
print( count_max_significant_fraction(c) ) # prints "4"
print( count_max_significant_fraction(d) ) # prints "0"
I suggest you adopt the convention NCO uses and name the precision attribute "number_of_significant_digits" and/or "least_significant_digit". Terms are defined in the lengthy precision discussion that starts here.