I've download weather forecast files from the ARPEGE that arrived in a file format like this "GRIBHS020+0000:00.jpeg". According to the person who managed the data, it is still a grib file that has been altered with this function "grib_set -r -s 'packingType=grid_jpeg' $f ${f}.jpeg. I need to convert this file to netcdf, but the data manager does not know how to do this. I've tried nearly everything (cdo: cdo -f nc copy, Eccodes: grib_to_netcdf, NCL ncl_convert2nc), but nothing works. For instance, this is the grib_to_netcdf output.
grib_to_netcdf -o test.nc grid.arpege-forecast.hs020+0024:00.grib
grib_to_netcdf: Version 2.12.3
grib_to_netcdf: Processing input file 'grid.arpege-forecast.hs020+0024:00.grib'.
grib_to_netcdf: Found 1018 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'test.nc'
grib_to_netcdf: NetCDF library version: 4.6.2 of Dec 17 2018 19:36:03 $
grib_to_netcdf: Creating large (64 bit) file format.
ECCODES ERROR : Cannot handle fields for different levtypes.
ECCODES ERROR : Please split input data into different files. Exiting!
Any suggestions are highly welcomed!
You can convert using netcdf-java:
java -cp toolsUI-5.4.2.jar ucar.nc2.write.Nccopy -i GRIBHS020+0114_00.jpeg -o GRIBHS020+0114_00.nc -st standard -sh -f netcdf4_classic
The netcdf-java can be downloaded from:
https://www.unidata.ucar.edu/downloads/netcdf-java/
and you will need to setup the netcdf-c library to write as netCDF4 using the nccopy featur:
https://docs.unidata.ucar.edu/netcdf-java/current/userguide/cdm_utility_programs.html#nccopy
Related
I concatenaded 31 netcdf files using ncrcat.I then oppend the output file in jupyter notebook and tried to create a plot in respect with time,but I get the following ,any fixes?
It appears your time coordinate values are all identical. Be sure you are using a recent version of ncrcat and that NCO was built with UDUnits enabled. Otherwise the time concatenation, aka rebasing, of your files cannot be expected to work correctly.
zender#sastrugi:~$ ncrcat -r
NCO netCDF Operators version 4.9.6-alpha03 "Osiris Rex" built by zender on sastrugi at Oct 26 2020 12:50:42
ncrcat version 4.9.6-alpha03
Linked to netCDF library version 4.7.4 compiled Oct 5 2020 21:03:41
Copyright (C) 1995--2020 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABSOLUTELY NO WARRANTY
You may redistribute and/or modify NCO under the terms of the
3-Clause BSD License with exceptions described in the LICENSE file
BSD: https://opensource.org/licenses/BSD-3-Clause
LICENSE: https://github.com/nco/nco/tree/master/LICENSE
Homepage: http://nco.sf.net
Code: http://github.com/nco/nco
Build-engine: Autoconf
User Guide: http://nco.sf.net/nco.html
Configuration Option: Active? Meaning or Reference:
Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
DAP support Yes http://nco.sf.net/nco.html#dap
Debugging: Custom No Pedantic, bounds checking (slowest execution)
Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb)
GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
HDF4 support No http://nco.sf.net/nco.html#hdf4
Internationalization No http://nco.sf.net/nco.html#i18n (pre-alpha)
Logging No http://nco.sf.net/nco.html#dbg
netCDF3 64-bit offset Yes http://nco.sf.net/nco.html#lfs
netCDF3 64-bit data Yes http://nco.sf.net/nco.html#cdf5
netCDF4/HDF5 support Yes http://nco.sf.net/nco.html#nco4
OpenMP SMP threading Yes http://nco.sf.net/nco.html#omp
Regular Expressions Yes http://nco.sf.net/nco.html#rx
**UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits**
You might also try doing the merge with
cdo mergetime infiles*.nc merged_file.nc
Check the output dates from the command line with
cdo showdate merged_file.nc
(This is not a duplicate of Reading Rds file from git)
Is there a way of reading an .rds file in R (or any other file) from a specific git commit without having to checkout the commit nor create a temporary file (not from GitHub, but for example from a bare or non-bare repo saved locally or on a server)?
I tried the following, but not surprisingly it does not work (assuming you have a git repo with some commits and a file a.rds):
b <- readRDS(system("git show 9358:a.rds"))
> Error in readRDS(system("git show 9358:a.rds")) :
invalid 'description' argument
Had the same problem with a text file a few days ago. I solved by saving the file (in the system call) before reading it. I guess in your case it would be something in the lines of:
# Target file output
output_dest <- "~/path/to/output.rds"
# Output stdout to file (in this case 3 commits before HEAD)
system(sprintf('show HEAD~3:your_repo/file.rds > %s', output_dest))
readRDS(output_dest)
We have generated a parquet files, one in Dask (Python) and another with R Drill (using the Sergeant packet ). They use a different implementations of parquet see my other parquet question
We are not able to cross read the files (the python can't read the R file and vice versa).
When reading the Python parquet file in the R environment we receive the following error: system error: Illegalstatexception: UTF8 can only annotate binary filed .
When reading the R/Drill parquet file in Dask we get an FileNotFoundError: [Error 2] no such file or directory ...\_metadata (which is self explanatory).
What are the options to cross read parquet files between R and Python?
Any insights would be appreciated.
To read drill-like parquet data-sets with fastparquet/dask, you need to pas a list of the filenames, e.g.,
files = glob.glob('mydata/*/*.parquet')
df = dd.read_parquet(files)
The error from going in the other direction might be a bug, or (gathering from your other question), may indicate that you used fixed-length strings, but drill/R doesn't support them.
I am trying to get a large dataset (3+ GB) from the following server:
ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/L4/GLOB/JPL/MUR
I know RCurl is a good package for getting data from FTP. The file is a compressed netcdf file. I need to uncompress it to read it into R using ncdf4. It's compressed as bz2.
Importantly, the file is larger than I want on my hard drive, so saving a copy locally is not an ideal option. How can I access data on the file without saving a copy to my disk first?
Here's my attempt so far:
library(RCurl); library(ncdf4)
d = getURL('ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/L4/GLOB/JPL/MUR/2015/144/20150524-JPL-L4UHfnd-GLOB-v01-fv04-MUR.nc.bz2')
d = bzfile(d, open = 'r')
d = nc_open(d)
But I'm stuck at this cryptic error after the first line:
Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) :
embedded nul in string: 'BZh91AY&SY¦ÁÀÉ\0033[ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿáåÏ\035\017)³îÎ\u009dÍØcn]sw7½ÎkÜÞõï=uÎׯv]ìçn\u009dÎn½îê·±Þìê÷wSM\u008có·+ÎçW¹Ý=Ù×¹\u009cγÜëÞs½ÛN¹²w;\u009buÍÝ]{·k^çuªnìº-³6«[+Üå;\033m»Û½ow:w¹ïo{uyîî\u00937¬\\Ƶl¶½\u009dÖVìç¯{ÎõïoSm]Ý×\u009eî\u008dæî®î®î\vÛÕïgW\036î®wqîÝ\\ïw«6½Þï\036Ýrë§=¬Fg·\\íåÔÙº÷gu·3\u009bKmÛ\027Þ»\u0092îî\016îêwwm»\u009b·s;MÞÁ½½ÎóÍso^»q¯o;k\033iµ\u009bÛuyÝÞní5w:ï]ÓuÎo[«\033:åÞvEÜíÎç½ÝË\u009eìQNöÔ\u008e\u0094vmÝȯg»e lÍ^\u008a©'
It seems to be an encoding issue based on other similar problems but I tried both .encoding = 'UTF-8' and .encoding = 'ISO-8859-1' as shown in the getURL() documentation but neither work.
I've seen other answers to problems like this but they all seem to involve editing the source file. I don't have write access to this file, however. Any help?
I'd use httr for this
library("httr")
library("ncdf4")
url <- 'ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/L4/GLOB/JPL/MUR/2015/144/20150524-JPL-L4UHfnd-GLOB-v01-fv04-MUR.nc.bz2'
res <- GET(url, write_disk(basename(url)))
# uncompress - I used OSX's default compression tool
nc_open(sub("\\.bz2", "", res$request$output$path))
the only step i didn't sort out programatically is un-compressing the bz2 file, just did that with OSX's default tool
I don't know much at all about R, but you should be able to do this with curl in FTP mode by changing the output to stdout rather than a local filename and then using bz2 to uncompress the file you want from its standard input.
So, for example, I can do this:
curl --output - --user user:password 'ftp://127.0.0.1/somefile.bz2' | bz2 ...
Maybe you can start that from within R? Or make a fifo with:
mkfifo fifo
curl ....
and then read from the file called fifo in R.
Or maybe R has a system() command, and you could do:
system('mkfifo fifo; curl ..... | bz2 .... > fifo &')
and then read from the file called fifo in R.
I'm running a script with input parameters that are referenced in the code to automate the directory creation, download of file and untar of file. I would be fine with unzip, however this particular file I want to analyze is .tar.gz. I manually unpacked and it was tar.gz, unpacked to .tar file. Would that be the problem?
Full error: Error in untar2(tarfile, files, list, exdir) : unsupported entry type ‘’
Running Windows 10, 64 bit, R set to: [Default] [64-bit] C:\Program Files\R\R-3.2.2
Script notes one solution found (issues, lines 28-31), but I don't really understand it.
I did install 7-zip on my computer, restart and of course restart R:
`#DOWNLOADING AND UNZIPPING TAR FILE
#load required packages.
#If there is a load package error, use install.packages("[package]")
library(dplyr)
library(lubridate)
library(XML) # HTML processing
options(stringsAsFactors = FALSE)
#Set directory locations, data file and fetch data file from internet
#enter full url including file name between ' ' marks
mainDir<-"C:/R/BEES/"
subDir<-"C:/R/BEES/Killers"
Fetch<-'http://dds.cr.usgs.gov/pub/data/nationalatlas/afrbeep020_nt00218.tar.gz'
ArchFile<-basename(Fetch)
download.file<-(ArchFile)
#Check for file directories and create if directory if it doesn't exist
if(!file.exists(mainDir)){dir.create(mainDir)}
if(!file.exists(subDir)){dir.create(subDir)}
#set the working directory
setwd(file.path(subDir))
#check if file exists and download if it doesn't exist.
if(!file.exists(ArchFile))
{download.file (url=Fetch,destfile=ArchFile,method='auto')}
#unpack and view file list
untar(path.expand(ArchFile),list=TRUE,exdir=subDir,compressed="gzip")
list.files(subDir)
#Error: Error in untar2(tarfile, files, list, exdir) :
# unsupported entry type ‘’
#Need solution to use tar/untar app
#instructions here: https://stevemosher.wordpress.com/step-10-build/`
Appreciate feedback - I've been lurking around StackOverflow for some time to use other people's solutions.