one little question. Is there an easy way to convert netcdf file data to a grid which can be used in pygmt?
Pygmt has as default example the following dataset:
<xarray.DataArray 'elevation' (lat: 120, lon: 120)>
array([[-3539.5, -3510.5, -3557.5, ..., -3013. , -2982. , -2981.5],
[-3583.5, -3467.5, -3575.5, ..., -2941. , -2967. , -2982.5],
[-3663. , -3497. , -3452.5, ..., -2900.5, -2976. , -2941. ],
...,
[-3600.5, -3568. , -3572.5, ..., -2558. , -2985. , -3190. ],
[-3535.5, -3499. , -3542. , ..., -2604. , -3016. , -3091.5],
[-3480. , -3455. , -3488.5, ..., -2758. , -3155. , -3248.5]],
dtype=float32)
Coordinates:
* lon (lon) float64 -92.46 -92.38 -92.29 -92.21 ... -82.71 -82.62 -82.54
* lat (lat) float64 -2.958 -2.875 -2.792 -2.708 ... 6.792 6.875 6.958
Attributes:
long_name: elevation relative to the geoid
units: meters
vertical_datum: EMG96
horizontal_datum: WGS84
After using to common nc-read syntax I get an masked array.
Greetings
I have a easy solution:
nc = nc4.Dataset('filename.nc', mode='r') # Or from siphon.ncss
dataset = xr.open_dataset(xr.backends.NetCDF4DataStore(nc))
PyGMT has a function pygmt.load_dataarray() which can take an .nc file name as input and should return a xr.dataarray
example_grid = pygmt.load_dataarray('example.nc')
Related
while Im executing this code in google colab,
def read_raster(raster_file):
"""
Function
--------
read_raster
Given a raster file, get the pixel size, pixel location, and pixel value
Parameters
----------
raster_file : string
Path to the raster file
Returns
-------
x_size : float
Pixel size
top_left_x_coords : numpy.ndarray shape: (number of columns,)
Longitude of the top-left point in each pixel
top_left_y_coords : numpy.ndarray shape: (number of rows,)
Latitude of the top-left point in each pixel
centroid_x_coords : numpy.ndarray shape: (number of columns,)
Longitude of the centroid in each pixel
centroid_y_coords : numpy.ndarray shape: (number of rows,)
Latitude of the centroid in each pixel
bands_data : numpy.ndarray shape: (number of rows, number of columns, 1)
Pixel value
"""
raster_dataset = gdal.Open(raster_file, gdal.GA_ReadOnly)
# get project coordination
proj = raster_dataset.GetProjectionRef()
bands_data = []
# Loop through all raster bands
for b in range(1, raster_dataset.RasterCount + 1):
band = raster_dataset.GetRasterBand(b)
bands_data.append(band.ReadAsArray())
no_data_value = band.GetNoDataValue()
bands_data = np.dstack(bands_data)
rows, cols, n_bands = bands_data.shape
# Get the metadata of the raster
geo_transform = raster_dataset.GetGeoTransform()
(upper_left_x, x_size, x_rotation, upper_left_y, y_rotation, y_size) = geo_transform
# Get location of each pixel
x_size = 1.0 / int(round(1 / float(x_size)))
y_size = - x_size
y_index = np.arange(bands_data.shape[0])
x_index = np.arange(bands_data.shape[1])
top_left_x_coords = upper_left_x + x_index * x_size
top_left_y_coords = upper_left_y + y_index * y_size
# Add half of the cell size to get the centroid of the cell
centroid_x_coords = top_left_x_coords + (x_size / 2)
centroid_y_coords = top_left_y_coords + (y_size / 2)
return (x_size, top_left_x_coords, top_left_y_coords, centroid_x_coords, centroid_y_coords, bands_data)
Helper function to get the pixel index of the point
def get_cell_idx(lon, lat, top_left_x_coords, top_left_y_coords):
"""
Function
--------
get_cell_idx
Given a point location and all the pixel locations of the raster file,
get the column and row index of the point in the raster
Parameters
----------
lon : float
Longitude of the point
lat : float
Latitude of the point
top_left_x_coords : numpy.ndarray shape: (number of columns,)
Longitude of the top-left point in each pixel
top_left_y_coords : numpy.ndarray shape: (number of rows,)
Latitude of the top-left point in each pixel
Returns
-------
lon_idx : int
Column index
lat_idx : int
Row index
"""
lon_idx = np.where(top_left_x_coords < lon)[0][-1]
lat_idx = np.where(top_left_y_coords > lat)[0][-1]
return lon_idx, lat_idx
raster_file = '/content/image.tif'
x_size, top_left_x_coords, top_left_y_coords, centroid_x_coords, centroid_y_coords, bands_data = read_raster(raster_file)
save the result in compressed format
np.savez('/content/nightlight.npz', top_left_x_coords=top_left_x_coords, top_left_y_coords=top_left_y_coords, bands_data=bands_data)
I got this error
RuntimeError Traceback (most recent call last)
in
2
3 raster_file = '/content/image.tif'
----> 4 x_size, top_left_x_coords, top_left_y_coords, centroid_x_coords, centroid_y_coords, bands_data = read_raster(raster_file)
5
6 # save the result in compressed format
1 frames
/usr/local/lib/python3.8/dist-packages/osgeo/gdal.py in Open(*args)
3017 def Open(*args):
3018 """Open(char const * utf8_path, GDALAccess eAccess) -> Dataset"""
-> 3019 return _gdal.Open(*args)
3020
3021 def OpenEx(*args, **kwargs):
RuntimeError: TIFFReadDirectory:Failed to read directory at offset 725954560
Can anyone help me out for solving this problem?
When I used the MODIS data,I got a problem about the Type.
I got the coord by xarray(python 3.9),but I couldn't get the same type as 'GEO2D'.
xr_tmp = xr.Dataset(
{
'Gpp_500m' : (('XDim', 'YDim','time'), Gpp_500m),
'Npp_500m' : (('XDim', 'YDim','time'), Npp_500m),
'Npp_QC_500m' : (('XDim', 'YDim','time'), Npp_QC_500m),
},
coords=dict(
lon = (['XDim', 'YDim'],lon),
lat = (['XDim', 'YDim'],lat),
time = _time_
# 'latitude': lat,
# 'longitude': lon,
# 'time': _time_,
)
)
The figures are shown bellow,Could U help me? (use xarray or pyModis if necessary)
You can see the difference from the pictures as follow.
I am trying to use scipy.integrate.quad with Tensorflow as following.
time and Lambda are two Tensors with shape (None, 1).
def f_t(self, time, Lambda):
h = Lambda * self.shape * time ** (self.shape - 1)
S = tf.exp(-1 * Lambda * time ** self.shape)
return h * S
def left_censoring(self, time, Lambda):
return tf.map_fn(lambda x: integrate.quad(self.f_t,
0.0,
x[0], # it is not a float before evaluation
args=(x[1],)),
tf.concat([time, Lambda], 1))
However, I get an error as below:
File "J:\Workspace\Distributions.py", line 30, in <lambda>
args=(x[1],)),
File "I:\Anaconda3\envs\tensorflow\lib\site-packages\scipy\integrate\quadpack.py", line 323, in quad
points)
File "I:\Anaconda3\envs\tensorflow\lib\site-packages\scipy\integrate\quadpack.py", line 388, in _quad
return _quadpack._qagse(func,a,b,args,full_output,epsabs,epsrel,limit)
TypeError: a float is required
X[0] is a Tensor with shape=(). It is not a float value before evaluation. Is it possible to solve the problem? How should I calculate integration in Tensorflow?
If you have at least TensorFlow 1.8.0, you're probably best off using tf.contrib.integrate.odeint_fixed() like this code (tested):
from __future__ import print_function
import tensorflow as tf
assert tf.VERSION >= "1.8.0", "This code only works with TensorFlow 1.8.0 or later."
def f( y, a ):
return a * a
x = tf.constant( [ 0.0, 1.0, 2, 3, 4 ], dtype = tf.float32 )
i = tf.contrib.integrate.odeint_fixed( f, 0.0, x, method = "rk4" )
with tf.Session() as sess:
res = sess.run( i )
print( res )
will output:
[ 0. 0.33333334 2.6666667 9. 21.333334 ]
properly integrating x2 over the intervals of [ 0, 0 ], [ 0, 1 ], [ 0, 2 ], [ 0, 3 ], and [ 0, 4 ] as per x = [ 0, 1, 2, 3, 4 ] above. (The primitive function of x2 is ⅓ x3, so for example 43 / 3 = 64/3 = 21 ⅓.)
Otherwise, for earlier TensorFlow versions, here's how to fix your code.
So the main issue is that you have to use tf.py_func() to map a Python function (scipy.integrate.quad() in this case) on a tensor. tf.map_fn() will map other TensorFlow operations and passes and expects tensors as operands. Therefore x[ 0 ] will never be a simple float, it will be a scalar tensor and scipy.integrate.quad() will not know what to do with that.
You can't completely get rid of tf.map_fn() either, unless you want to manually loop over numpy arrays.
Furthermore, scipy.integrate.quad() returns a double (float64), whereas your tensors are float32.
I've simplified your code a lot, because I don't have access to the rest of it and it looks too complicated compared to the core of this question. The following code (tested):
from __future__ import print_function
import tensorflow as tf
from scipy import integrate
def f( a ):
return a * a
def integrated( f, x ):
return tf.map_fn( lambda y: tf.py_func(
lambda z: integrate.quad( f, 0.0, z )[ 0 ], [ y ], tf.float64 ),
x )
x = tf.constant( [ 1.0, 2, 3, 4 ], dtype = tf.float64 )
i = integrated( f, x )
with tf.Session() as sess:
res = sess.run( i )
print( res )
will also output:
[ 0.33333333 2.66666667 9. 21.33333333]
I have a large Float64 array x and often treat a slice of it as matrix before changing exactly that slice. Can I somehow refer to this slice as y which already has the correct shape. To
x=zeros(10000)
y=x[10:18]
reshape!(y,(3,3))
y=y+eye(3) # this doesn't change x
This does not work as x[10:18] creates a copy. I had a look at pointer_to_array but I couldn't work it out.
The usual tool of producing no-copy slices in julia is SubArrays:
x = zeros(10000)
y = sub(x, 10:18)
now changing y would change the corresponding elements of x. (you can also use slice, which behaves differently with higher-dimensional arrays).
You can also use y = sub(x, A) where A is an array of indices. Unfortunately reshape(y, 3, 3) produces a copy of the values by converting the SubArray to an Array (currently discussed in Julia issue 9874, so that may change in the future). Using y = sub(x, A) with A a matrix of indices of the right shape does not work.
Maybe for your particular application, it is sufficient to do
x = zeros(10000)
y = sub(x, 10:18)
y[:] += vec(eye(3))
which changes x as you wanted it, without producing a slice of different shape.
I guess the following will do the job:
y = pointer_to_array( pointer( x, 10 ), (3,3) ) # make a slice starting from the 10th element
This can be tested, for example, as
x = zeros( 8 )
p = pointer_to_array( pointer( x, 3 ), (3,2) )
p[:,1] = 100.0
p[:,2] = 200.0
#show x # => [ 0.0, 0.0, 100.0, 100.0, 100.0, 200.0, 200.0, 200.0 ]
If the size of x is a multiple of the slice size, reshape() can also be used directly for modifying a slice. For example,
x = [ i for i=1:8 ]
s = reshape( x, (2,2,2) )
s[:,:,2] = 1000
#show x # => [1,2,3,4,1000,1000,1000,1000]
I have a function to compute latitude and longitude points in order to create a "ring" around a center location. The problem is that the results print to the screen but are not stored anywhere. My function creates a dataframe with 0 columns and 0 rows. I want to be able to take these coordinates and use them elsewhere. I would like to be able to nest this function as well, but I can't really nest it when it doesn't return anything.
My end goal is to create kml code. I already have the kml code but need to repeat it many times. The kml code creates radius rings, fills them with color, and adds a name for the place. I want to generate the files automatically by using a list of locations in lat/lon.
My question is, how can I get this function to return the list of coordinates that I want so that I may paste them in the kml code accordingly? I can get it to loop using adply and get printed results for the 3 coordinates, but nothing is created.
I am also quite new to coding, please be gentle. Thanks in advance.
make.ring.file=function(dist,df)
{
R = 6378.14 #Radius of the Earth
d = dist*1.609344 #Distance of ring radius in km
lat1 = df$lat*(pi/180) #Current lat point converted to radians
lon1 = df$lon*(pi/180) #Current lon point converted to radians
num3=0
index=seq(from=0,to=360,by=120)
bear=NULL
lat=NULL
lon=NULL
z=NULL
coordlist=NULL
for(n in 1:length(index))
{
bear[n]=index[n]*(pi/180)
lat[n]=(asin(sin(lat1)*cos(d/R) + cos(lat1)*sin(d/R)*cos(bear[n])))*(180/pi)
lon[n]=(lon1 + atan2(sin(bear[n])*sin(d/R)*cos(lat1),
cos(d/R)-sin(lat1)*sin(lat[n]*(pi/180))))*(180/pi)
z[n]=0
coordlist[n]=paste(lon[n],lat[n],z[n],sep=",")
}
return(data.frame(cat(coordlist,"\n","\n")))
}
> head(x1)
lat lon
1 38.86095 -86.51672
2 30.63275 -84.41614
3 31.53697 -87.88780
> results=adply(x1,1,make.ring.file,dist=30)
-86.51672,39.2946592897837,0 -86.0358241901732,38.6431079084023,0 -86.9976158098268,38.6431079084023,0 -86.51672,39.2946592897837,0
-84.41614,31.0664592897837,0 -83.9805971533182,30.4151694949636,0 -84.8516828466818,30.4151694949636,0 -84.41614,31.0664592897837,0
-87.8878,31.9706792897837,0 -87.4481292235866,31.3193631233201,0 -88.3274707764134,31.3193631233201,0 -87.8878,31.9706792897837,0
> str(results)
'data.frame': 0 obs. of 0 variables
> is.data.frame(results)
[1] TRUE
I think this is what you want:
make.ring.file=function(dist,df)
{
R = 6378.14 #Radius of the Earth
d = dist*1.609344 #Distance of ring radius in km
lat1 = df$lat*(pi/180) #Current lat point converted to radians
lon1 = df$lon*(pi/180) #Current lon point converted to radians
num3=0
index=seq(from=0,to=360,by=120)
bear=NULL
lat=NULL
lon=NULL
z=NULL
coordlist=NULL
for(n in 1:length(index))
{
bear[n]=index[n]*(pi/180)
lat[n]=(asin(sin(lat1)*cos(d/R) + cos(lat1)*sin(d/R)*cos(bear[n])))*(180/pi)
lon[n]=(lon1 + atan2(sin(bear[n])*sin(d/R)*cos(lat1),
cos(d/R)-sin(lat1)*sin(lat[n]*(pi/180))))*(180/pi)
z[n]=0
coordlist[n]=paste(lon[n],lat[n],z[n],sep=",")
}
return(data.frame(out=paste(coordlist,collapse=" ")))
}
The key addition is:
return(data.frame(out=paste(coordlist,collapse=" ")))
The function cat, prints to console, so you can't assign it to anything - you were merely making an empty data.frame and printing.
The paste command works by pasting together the 4 lines of coordlist, with a separator of " ". You can modify this as required for your downstream code.