I have a large number of files in .hdf format. These are sadly not the hdf5 files that I know are readable in R. Is there a way to load and read hdf files in R? Or is there a way to convert .hdf to hdf5? I downloaded the C-based h4toh5 tool, but it is not useful. Are there any other ways to convert? Many thanks.
I wouldn't describe hdf4 as a legacy format #mdsummer. NASA still provides a lot of its data in hdf4.
This is similar to a few posts where people (myself included) haven't been sure whether their .hdf files are hdf4 or hdf5. You can use a program called HDFView which is free to download and really easy to use to find out if you've got hdf4/5 files. Just load a file in and look under "properties".
If you've got hdf4 files then you should check out the gdal_translate from the gdalUtils package. There's some more info here including the code I used for my hdf files:
Reading hdf files into R and converting them to geoTIFF rasters
You can use ncdf4 or rgdal packages as we explain here:
https://hdfeos.org/software/r.php
If you don't like h4toh5, h4tonccf is another tool that you can try:
http://hdfeos.org/software/h4cflib.php
Related
Similar to the question below, I was wondering whether there is a way to open .yxdb files in R?
Open Alteryx .yxdb file in Python?
YXDB is Alteryx's own native format. I haven't seen or heard of anything else that can open it.
You could change your workflow to write to a CSV (or other compatible file) as well as writing to the YXDB file.
AFAIK there is no way yet for R to read yxdb files. I also export my Alteryx workflows to CSVs or use the R tool, read.Alteryx, and saveRDS to save it as a fast-loading binary file.
I have seen several questions about writing .bin (binary) files from R, but I am wondering what function or package R has that could read in .bin files?
Is R capable of reading in .bin files?
You need the hexView package, and you need to know a lot about the content of the files.
Dump the bytes using hexView::viewRaw(readRaw(filename)), then figure out the structure using what you know about the file. Then figure out what is in there, and use a more specific dump function like viewFormat. See the package documentation.
It's a lot of work, and requires a lot of knowledge specific to the creation of the file. You're unlikely to succeed, but that's how you should do it.
I'm in a big trouble. I've downloaded a GeoTIFFF Dataset from http://earthexplorer.usgs.gov/ ; my problem is that I need the dataset in HDFv4 format, because I've to open it in IDL (please don't tell me "IDL can open GeoTIFF", I NEED HDFv4 format) . May you please suggest me a tool that does this conversion?
Thanks a lot.
Just to get you started, you could read in the image and its GEOTIFF tags using the following command:
file = FILEPATH('boulder.tif', SUBDIR=['examples','data'])
data_variable=READ_TIFF(file, GEOTIFF=GeoKeys)
HELP, GeoKeys, /STRUCTURE
You would then need to pull apart the geotiff structure and write the data back to an HDF4 file. I don't quite understand why you need HDF4, and I'm also not sure how you're going to write the GEOTIFF data into the HDF4 file, since HDF4 doesn't have anything "specific" about map projections.
See the docs for more details:
http://www.harrisgeospatial.com/docs/read_tiff.html
Here's a really bad way to do the conversion:
https://www.hdfgroup.org/HDF5-FAQ.html#gtifftohdf
Basically, in that case you are only saving the image data, not the geotiff-specific data.
Good luck!
I would like to read a hdf5 file in a zipped file. The issue here is that this hdf5 file is also double zipped as a bzip2ed (.bz2) file.
Please refer to the figure shown below.
The zip file is "g2_BIOPAR_SWI_201012250000_GLOBE_ASCAT_V2_0_0.ZIP".
The target bz2 file is "g2_BIOPAR_SWI_201012250000_GLOBE_ASCAT_V2_0_0.h5.bz2".
Could someone show me some tips or guidances in how to do it?
Give a look at this question:
Extract bz2 file in R
It explains how to read a.bz2 file and I think the last answer is the right one for you,; or just type ?bzfile.
After that for hdf5 files you can give a look at rhdf5 package from Bioncoductor; here is the link:
http://www.bioconductor.org/packages/release/bioc/html/rhdf5.html
I am using R to work with meteorological data. I proceed in two steps:
convert grib to netcdf using the command line function ncl_convert2nc from ncar command language
use package ncdf in R to import the netcdf data.
I still have one problem:
2- For some particular grib files, the conversion with ncar tool does not work. Is there other ways or trick (other than transcription into netcdf) to read grib files in R ?
Problem Answered by Dirk: 1- I would like to do automatic treatment of many files within R. Can I call ncl_convert2nc within R ? (answered by Dirk Eddelbuettel below )
Regarding question 1, the answer is 'Yes' -- see help(system) and the internal=TRUE option if you want to capture results.
rgdal also can do it, but is less flexible and requires more care and detail than ncdf or RNetCDF - and depends of your GDAL/rgdal built including the GRIB driver.
ncl_convert2nc seems to be the best solution. However, if the structure of data is a little bit more complicated I use GrADS to convert GRIB file to ASCII (e.g. .csv) and then it is possible to create NetCDF file using ncdf4 package dedicated for R. GrADS also provides support for re-writing GRIB to NetCDF, but there is limitation to only 1 variable.
As an alternative to calling ncl_convert2nc from R, there are two alternatives I can suggest:
1. CDO conversion
Another quick and easy command line solution is to use cdo to convert to netcdf to read in:
cdo -f nc copy file.grb file.nc
If you want to output a netcdf4 file you specify "-f nc4".
One potential glitch with this approach is if your grib file has more than one time axis (e.g. for multiple seasonal forecasts) which can cause issues with the conversion.
2. ECCODES conversion
Instead eccodes offers a grib converter that is very robust and can handle all cases of multiple time axes which usually cause CDO and NCL based conversions to fail.
The command is called grib_to_netcdf
grib_to_netcdf -o output.nc input_grib.grb
So far, grib_to_netcdf has been able to handle every grib file I have thrown at it without problems.
Another solution is to use the wgrib/wgrib2 software (http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/) and dump your GRIB-1/GRIB-2 file directly to CSV format, e.g.:
/path/to/your/wgrib2 input_file.grb -csv output_file.csv
Then it may be read directly in R...