I have a script to loop through a selection of net cdf files. The files are opened, data extracted, then closed again. I have used this many times before and it works with no issue. I was recently sent a new selection of files to run through the same code. I can check the files individually using the ncdf4 package and nc_open() function. The files look fine and are not corrupt. However, when I run through the loop the function will not let me open the files and I get this error:
Error in R_nc4_open: NetCDF: HDF error
When I run though the loop to check, all is fine and the file opens. It just cannot open in the loop. There is no issue with the code.
Has anyone come across this before with non-corrupt net cdf files getting this error only on occasion. Even outside the loop I can run the code and get the error first time, then run it again without changing anything and the connection works.
Not sure how to trouble shoot this one, so just looking for advice as to why this might be happening.
Code snippet:
targetYear <- '2005-2019'
variables <- c('CHL','SSH')
ncNam <- list.files(folderdir, '.nc', recursive = TRUE)
for(v in 1:length((variables)))
{
varNam <- unlist(unique(variables))[v]
# Get names corresponding to variable
varLs <- ncNam[grep(varNam, basename(ncNam))]
varLs <- varLs[grep(targetYear, varLs)]}
varLs <- varLs[1]
export <- paste0(exportdir,varNam,'/')
dir.create(export, recursive = TRUE)
if(varNam == 'Proximity1km' | varNam == 'Proximity200m'| varNam ==
'ProximityCoast'| varNam == 'Bathymetry'){
fileNam <- varLs
ncfilename <- paste0(folderdir, fileNam)
print(ncfilename)
# Read ncfile
ncfile <- nc_open(ncfilename)
nc_close(ncfile)
gc()
} else {
fileNam <- varLs
ncfilename <- paste0(folderdir, fileNam)
print(ncfilename)
# Read ncfile
ncfile <- nc_open(ncfilename)
nc_close(ncfile)
gc()}`
I figured out the issue. It was to do with the error detection filer in the .nc files.
I removed the filter and the files work fine inside the loop. Still a bit strange.
Perhaps the ncdf4 package is not up to date with this filtering.
Related
I am currently trying to write a function to batch process a number of PNGs I have in a file into SVGs. I wanted to get this up and running as it is something I usually have to do and find the "free" online alternatives to be lacking. I have written the following script to do this:
image.list <- list.files("D:/R Exports/06 Graphics")
image.select <- list()
batch.svg.convert <- function(image.list){
for (i in 1:length(image.list)){
image.select[i] <- paste0("D:/R Exports/06 Graphics/",image.list[i])
image.read <- image_read(image.select[i])
image.svg <- image_convert(image.read,format = "svg")
image.write <- image_write(image.svg, paste0("D:/R Exports/Graphics SVG",image.list[i]))
}
}
batch.svg.convert(image.list)
However, when doing so I get the following error in response:
Error in image_read(image.select[i]) :
path must be URL, filename or raw vector
I am unsure why this is occurring as when I run tests such as:
image_list2 <- list.files("D:/R Exports/06 Graphics")
image_test <- paste0("D:/R Exports/06 Graphics/",image_list2[6])
image_test2 <- image_read(image_test)
The image is read as intended. What exactly am I missing here? Is this a limitation of the magick package or am I missing something from my code to get this function working correctly?
Thanks in advance,
Jim
I made a for-loop that loops through a folder. The folder is called Ultrasonic data - Plots and it contains subfolders. The names of the subfolders are consistent and the data in the subfolders as well. When I run the code it runs very well for the majority of the loop, but for some subfolders it gives an error.
The error is: file(file, "rt"): cannot open the connection.
The answers that I have read normally state the problem as a wrongly states working directory. However, I think it is not the case in this example. Could anything else cause this error message?
I have already checked the consitency of the names of the subfolders and .txt data files within them.
'''R
parent.folder <- "//home.org.aalto.fi/meijsl1/data/Documents/GAGS/Ultrasonic data/Ultrasonic data - Plots"
sub.folders <- list.dirs(parent.folder, recursive=FALSE)
filt.folders <- sub.folders[grepl("SV-30-[^_]*_S[12]", sub.folders)]
for(i in filt.folders) {
setwd(i)
AIC("SV-30", 20, 40) #This is a function that picks the S-wave onset of an ultrasonic signal
} #End for loop over all specimens
'''
'''PArt of the AIC function where the error occurs: (read.table)
for (n in 1:length(filelist)){
#
# Read the file into R
#
file.path('./out/Processed', basename(filelist[n])) -> procpath
read.table(file=procpath, sep="\t", stringsAsFactors=FALSE, check.names = FALSE) -> temp
assign(paste(substr(basename(filelist[n]), 1, nchar(basename(filelist[n])) - 4)), temp)
'''
The code should be running smoothly, as it does for most of the subfolders, but apperantly there must be something wrong. I am out of ideas what it could be. Consistency was the only thing that could cause this trouble in my point of view. I hope anyone can help.
Cheers
I created a function to save data to a specific location which is loaded as part of a package when launching rstudio:
save_data <- function(fileName, file_name){
file_path <- paste0("~/", file_name)
save(list=deparse(substitute(fileName)), file=file_path)
}
but when calling this function from inside another function it returns an "Error saving the following file: ...".
To reproduce the issue I created a savetestfunction:
savetest <- function(fileName){
data1 <- fileName * 10
save_data(data1, file_name = "test.RData")
data1
}
and a small savetestscript:
source("savetestfunction.R")
x <- c(1:10)
data1 <- savetest(x)
I spent a long time assuming the issue stemmed from the environments and what was being seen from each (e.g. save_data function loaded on startup couldn't see the temporary environment created when calling the savetestfunction) but as a test I tried adding a print(fileName) to the save_data function and to my surprise it could in fact see what fileName was at this point.
The Fix: I updated the function so that it assigned fileName within the environment created by the save_data function and it now functions as intended...
save_data <- function(fileName, file_name){
file_path <- paste0("//placeholder//", file_name)
assign(deparse(substitute(fileName)), fileName)
save(list=deparse(substitute(fileName)), file=file_path)
}
The Confusion: This then led me to believe that fileName was perhaps a promise when it was being saved however save() contains an argument eval.promises which is True by default so it can't be this!
The Question: What was causing this issue? (Confusion related to environments? Promises?) What could have been done to avoid this?
Edit: Tried to use saveRDS but didn't have success with this. More interested in why the save doesn't work in the first place rather than an actual fix as the assign within the save_data function already lets it function.
I have some code which compacts and repairs a number of MS Access databases:
library(RDCOMClient)
library(stringr)
accfolders <- list.dirs('C:\\users\\username\\accessdb\\',recursive = FALSE,full.names=F)[-1] #need -1 to exclude current dir
accfolders <- paste0("C:\\users\\username\\accessdb\\",accfolders)
#launch access
oApp <- COMCreate("Access.Application")
for (folder in accfolders) {
accfiles <- list.files(path=folder, pattern="\\.mdb", full.names=TRUE)
print(paste("working in dir", folder))
for (file in accfiles){
print (paste("working in db", file))
bkfile <- sub(".mdb", "_bk.mdb", file)
oApp$CompactRepair(file, bkfile, FALSE)
file.copy(bkfile, file, overwrite = TRUE)
file.remove(bkfile)
}
#print(paste("completed", folder))
}
oApp$quit()
gc()
However, sometimes the code returns this following error:
<checkErrorInfo> 80020009
Error: Exception occurred.
This error seems to happen somewhat randomly and it happens on the call oApp$CompactRepair during the second for loop
I can't seem to figure out why this happens and it happens with random .mdb files rather than a specific one. Sometimes I run the code and there is no issue at all, other times it produces the error.
Seeing as I can't figure it out, I'm wondering if I could capture this error somehow and just skip that element in the for loop? That way the code will not break down
I am trying to automate the data ingestion process in an R script that pulls data from a directory that updates regularly.
The general framework follows this process
library(sp)
library(rgdal)
library(raster)
f1.t1.cir <- stack("../raster/field1/f1_cir_t1.tif")
f1.t1.NDVI <- stack("../raster/field1/f1_ndvi_t1.tif")
f1.t1.RGB <- stack("../raster/field1/f1_ndvi_t1.tif")
f1.dat <- c(f1.t1.cir, f1.t1.NDVI, f1.t1.RGB)
for (i in f1.dat){
plotRGB(i)
}
I would like to generate each f1.t1.cir type object from the directory directly such that when I add a new TIFF file f1_cir_t2.tif, the r script will create an object f1.cir.t2.
I am trying to use something like
a <- list.files(path= "../raster/field1", pattern = "\\.tif$")
b <- gsub("_", "\\.", a)
for (i in a) {
assign(get(b[(which(a==i))]), stack((paste("../raster/field1/", i,sep=""))))
}
At this point, I would have all tiff files as stacked multiband raster objects in the R workspace.
I am getting the following error,
Error in get(b[(which(a == i))]) : object 'f1_t1_DSM.tif' not found
I can not figure out if this is a get() problem, or something else.
for reference
> a
[1] "f1_t1_DSM.tif" "f1_t1_NDVI.tif"
> b
[1] "f1.t1.DSM.tif" "f1.t1.NDVI.tif"
so that much is working, I think.
Any suggestions?
#joran, great suggestion...
f1.t1<-list()
for(i in list.files(path= "../raster/field1", pattern = "\\.tif$")){
f1.t1[[i]]<-stack((paste("../raster/field1/", i, sep="")))
}
Worked very well, no need to change the names.
Thank you.