I have a folder containing many netcdf files for individual station data (about 8,000 files total). When I attempt to read in all of these files using xarray.open_mfdataset, I get an error that reads "OSError: no files to open." I am using the code below:
station_data = xr.open_mfdataset('~/Data/station_data/all_stations/*.nc')
In contrast, I do not have any issue opening individual station data (one file) using xarray.open_dataset, as below:
station1 = xr.open_dataset('~/Data/station_data/all_stations/hadisd.3.1.0.2019f_19310101-20200101_010010-99999.nc')
I have been playing with other ways to express the path, with no luck. Any suggestions on how to properly read in all station data at once are appreciated!
Related
I am trying to convert multiple (250 or so) .avi video files into .jpeg files with R.
I have managed to convert single .avi files using function av_video_images() from library av, but I would love to know how to iterate this over multiple input files.
av::av_video_images("FILE001.AVI", destdir = "Site_1_JPEG", format = "jpg", fps = 1)
I have the 250 .avi files in a folder and would like all frames produced in the output folder Site_1_JPEG.
This is not a complete solution since I cannot reproduce your issue, but I think it will get you closer. Your example suggests that the files you want to process are in your current working directory. Secondly, your code will not produce the desired results because av_video_images names the .jpg files as "image_000001.jpg", "image_000002.jpg", "image_000003.jpg" and I see no way to alter the names of the extracted jpg's. That means your code will successively overwrite the previous files and at the end you will only have the final set of jpg's. To prevent that you need to create separate folders for each video file. Here is one solution:
library(av)
sapply(flist[1:3], function(x) av_video_images(paste0(path, x), x, fps=.5))
To test the code I specified that only the first 3 files will be processed to check things out. There are two differences between my code and yours. First my video files are located in a different directory (path) so I pasted the path onto the file name. Second I provided a different destination directory for each file which is just the file name. This produced three folders with jpg files in each.
The error message could indicate that one or more of the .avi files is corrupt. You can get the directory information on all of the files with
file.info(flist)
The main thing to look at is the size column to make sure the size is large enough.
Does anyone know a good way to read .vrl files from Vemco acoustic telemetry receivers directly into r as an object. Converting .vrl files to .csv files in the program VUE prior to analyzing the data in r seems like a waste of time if there is a way to bring them in directly. My internet searches have not turned up anything that worked for me.
I figured out a way using glatos to convert all .vrl files to .csv and then reading the .csv files in and binding them.
glatos has to be installed from github.
Convert all .vrl files to .csv files using vrl2csv. The help page has info on finding the path for vueExePath
library(glatos)
vrl2csv(vrl = "VRLFileInput",outDir = "VRLFilesToCSV", vueExePath = "C:/Program Files (x86)/VEMCO/VUE")
The following will pull in all .csv files in the output folder from vrl2csv and rbind them together. I had to add the paste0 function to create the full file path for each .csv in the list.
library(data.table)
AllDetections <- do.call(rbind, lapply(paste0("VRLFilesToCSV/", list.files(path = "VRLFilesToCSV")), read.csv))
I'm trying to download all the zipped csv file from the following pg: http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey
I've started by trying to download one file as an example before I move on to downloading multiple. This site contains prices from specific locations in Texas - interesting, given recent power outages due to cold weather in Texas.
url <- "http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey/cdr.00012301.0000000000000000.20210220.141704636.SPPHLZNP6905_20210220_1415_csv.zip"
temp <- tempfile()
download.file(url,temp, mode = "wb")
data <- read.csv(unzip(temp, "cdr.00012301.0000000000000000.20210220.141704.SPPHLZNP6905_20210220_1415.csv"))
unlink(temp)
Keep receiving the following error message: "error 1 in extracting from zip file."
I'm relatively new to R, so any advice would be helpful.
Edit: If the link above doesn't work, another way to get to the link is following this: http://www.ercot.com/mktinfo/rtm and going to "Real-Time Price Reports" and selecting the last option "Settlement Point Prices at Resource Nodes, Hubs, and Load Zones." Might look a little overwhelming, but my goal for right now is just to download and open the first zipped csv file on there (and ignore all the other files on there)
I was wondering if y'all could help me.
I'm trying to read data from a directory containing 332 compressed zips all with about 1000 lines (or more) of data into a data frame.
I have it so that the directory is set to the file with the zips in it. I figured this way when I tried using the dir() function, I could do something like:
d1 <- read.csv(dir()[1:332])
to read all the data and then find the mean of say "columnA" in that data table by using something like mean(d1$columnA).
Though, so far this has not worked. Any ideas?
I'm operating in R with a Mac OSX.
I'd like to use R to extract data from zip files located in zip files (i.e. preform some ZIP file inception).
An example "directory" of one of my datapoints looks like this:
C:\ZipMother.zip\ZipChild.zip\index.txt
My goal is to read in the "index.txt" from each ZipChild.zip. The issue is that I have 324 ZipMother.zip files with an average of 2000 ZipChild.zip files, therefore unzipping the ZipMother.zip files is a concern due to memory constraints (the ZipMother files are about 600 megabytes on average).
With the unzip package, I can successfully get the filepaths of each ZipChild located in the ZipMother, but I cannot use it to list the files located in the ZipChild folders.
Therefore,
unzip("./ZipMother.zip",list=TRUE)
works just fine, but...
unzip("./ZipMother.zip/ZipChild.zip",list=TRUE)
gives me the following error
Error in unzip("./ZipMother.zip/ZipChild.zip", list = TRUE) :
zip file './ZipMother.zip/ZipChild.zip' cannot be opened
Is there any way to use unzip or another method to extract the data from the ZipChild files?
Once I get this to work, I plan on using the ldply function to compile the index.txt files into a dataset.
Any input is very much appreciated. Thank you!
A reproducible example (i.e. a link to a zip file with the appropriate structure) would be useful, but how about:
tmpd <- tempdir()
## extract just the child
unzip("./ZipMother.zip",
files="zipChild.zip",exdir=tmpd)
ff <- file.path(tmpd,"zipChild.zip")
index <- unzip(ff,list=TRUE)
unlink(ff)
This could obviously be packaged into a function for convenience.
It could be slow, but it means you never have to unpack more than one child at a time ...