-
I am beginner in R and I have made a script where I convert a .grd file into .tif file in a for loop (I have quite a lot of .grd files that should be converted). The script is as you can see below.
# Set your work directory
setwd("xxx")
library(rgdal)
library(sp)
library(raster)
# Data: .grd files in order to reproduce the code below
g1 <- raster(ncol=10, nrow=10)
vals <- 1:100
g1 <- setValues(g1, vals)
writeRaster(g1, filename="TEST_G1.grd", overwrite=TRUE)
g2 <- raster(ncol=50, nrow=50)
vals <- 1
g2 <- setValues(g2, vals)
writeRaster(g2, filename="TEST_G2.grd", overwrite=TRUE)
# Convert .grd to geotif in a for loop
rlist <- list.files(pattern=".grd$")
for (i in rlist)
{
#Read raster
x <- raster(rlist[i])
#make new file name
filename <- rlist[i]
n <- unlist(strsplit(filename, split='.', fixed=TRUE))[1]
name <- paste(n, ".tif", sep="")
#write the raster as GTiff
writeRaster(x, filename=name, format="GTiff", overwrite=TRUE)
}
I have run all sentences in the script separately and managed to get geotif file. However, when I run the lobe I get the following error:
Error in .local(.Object, ...) :
Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)
The file does exist, and when I just run the last line separately the tif file is created... so I don't understand what is wrong with the loop.
If I e.g. write "for (i in 1:2)" instead of "for (i in rlist)" it works. but I would prefer not to count the number of files in every directory, which is the reason I was trying to use "for (i in rlist)"
Thanks a lot for you help!
Related
I have a bunch of text files in a folder. The script should read the whole list and do a calculation on each file. The result should be written in a "results.txt" file. I also want to have the name of the processed file in the results table and the result. But this line is still missing. But I don't know how to add it.
I am pretty far, but now I am stuck:
library(data.table)
ldf <- list() # creates a list
list_txt <- dir(pattern = "*.txt")
for (k in 1:length(list_txt)){
ldf[[k]] <- fread(list[k], select = c("Count"))
br=c(0,1,3,9,15,500) #Set breaks
bins=c(0,1,2,3,4) #Set bins
freq=hist(ldf[[k]]$Count, breaks=br, plot=FALSE)
df=data.frame(bins, frequency=freq$counts)
df$pct <- df$frequency*100 / sum(df$frequency)
df$pct<-round(df$pct,digits=0)
df$hscore<-df$pct * df$bins
hscore=sum(df$hscore)
cat(df$hscore,file="results.txt",sep="\n")
}
The error code I get is:
Error in hist.default(ldf[[k]]$Count, breaks = br, plot = FALSE) :
some 'x' not counted; maybe 'breaks' do not span range of 'x'
Any suggestions?
I tried a bit more and came to this code, wich works without error messages:
library(data.table)
ldf <- list() # creates a list
list_txt <- dir(pattern = "*.txt")
for (k in 1:length(list_txt)){
ldf[[k]] <- fread(list_txt[k], select = c("Count"))
br=c(-Inf,1,3,9,15,Inf) #Set breaks
bins=c(0,1,2,3,4) #Set bins
freq=hist(ldf[[k]]$Count, breaks=br, plot=FALSE)
df=data.frame(bins, frequency=freq$counts)
df$pct <- df$frequency*100 / sum(df$frequency)
df$pct<-round(df$pct,digits=0)
df$hscore<-df$pct * df$bins
hscore=sum(df$hscore)
cat(df$hscore,file="results.txt",sep="\n")
}
BUT, it creates a results.txt file with only 5 entries. When I call the list_txt, there are 164 files.
What could be the problem?
I am trying to loop the
ncin_old<-nc_open("filename", write=TRUE, readunlim=TRUE, verbose=FALSE,
auto_GMT=TRUE, suppress_dimvals=FALSE )
function like this
library(ncdf.tools)
library(ncdf4)
library(ncdf4.helpers)
library(RNetCDF)
library(abind)
setwd("D:/Rwork/Project") # set working folder
# This is the directory where the file for analysing are
dir("D:/Rwork/Project/MASTER_FILES")-> xlab
filelist <- paste("MASTER_FILES/", dir("MASTER_FILES"), sep="")
N <- length(filelist) # Loop over the individual files
for(j in 1:N) {
ncin_old <- nc_open("filelist[j]", write=TRUE, readunlim=TRUE, verbose=FALSE,
auto_GMT=TRUE, suppress_dimvals=FALSE )
}
But I get this error
Error in nc_open("filelist[j]", write = TRUE, readunlim = TRUE,
verbose = FALSE, : Error in nc_open trying to open file
filelist[j]
If I drop everything after filelist[j] the lat file in the loop opens
but the nc_open(x, write) does not seem to like being looped.
I have fixed some issues of your code as below. I think now it is correct.
library(ncdf4)
# set the folder with the files
setwd("D:/Rwork/Project/MASTER_FILES")
# you need the files path, not the directory path
# list only the files with the .nc extension
filelist <- list.files(pattern = "\\.nc$")
# Loop over the individual files
# The filelist cannot be between quotation marks as in your code
N <- length(filelist)
for(j in 1:N) {
ncin_old <- nc_open(filelist[j], write=TRUE, readunlim=TRUE, verbose=FALSE,
auto_GMT=TRUE, suppress_dimvals=FALSE)
}
I've used lapply:
library(ncdf4)
# set the folder that contains all the files
setwd("C:/...")
# create a list with the files with the .nc extension
filelist <- list.files(pattern = "*.nc")
filelist # It contains all the files .nc
# To open all files: Loop over the individual files
for (i in 1:length(filelist)) {
all_nc_files <- lapply(filelist, nc_open)
}
Running that, I obtain "all_nc_files", which contains all .nc files opened and now I can work with them.
Hope it works!
My code works and displays the correct values on screen for print(obser) but will not write the .csv and when I tried str(obser) it gave error object 'obser' not found. I have tried various online help and books and the function is correctly written.
If instead of running the function in the console in RStudio, I run line by line in the scrip screen will the csv then be created?
complete <- function(directory= "specdata", id = 1:332){
# directory <- "specdata"
# id <- 1:332
files_list <- list.files(path=directory,full.names=T)[id]
NumOfFiles <- length(files_list)
obser <- data.frame()
indivFile <-data.frame()
nobserv <- vector(mode= "integer", length = NumOfFiles)
for (i in 1:NumOfFiles){
indivFile <- read.csv(files_list[i]) # read data file into df inc NA's
indivFile <- na.omit(indivFile) # removes NA prev file
x <- nrow(indivFile[1])
nobserv[i] <- x
}
x_name <-"ID"
y_name <-"nobs"
obser <- data.frame(id, nobserv)
return(obser) # object returned
print(obser)
wd <- getwd()
setwd(wd)
write.csv(obser, file="Observations2.csv")
}
Try and save the object returned from the function. Ergo:
output<-complete("specdata",id=1:332)
write.csv(output,"Observations2.csv",row.names=F)
I have a huge .txt file named SDN_1 with more than 1 million rows. I would like to split this file into smaller .txt files (10,000 rows each) using R.
I used this code to load the file into R:
SDN_1 <- read.csv("C:/Users/JHU/Desktop/rfiles/SDN_1.csv", header=FALSE)
Then I used this code to split the table:
chunk <- 10000
n <- nrow(SDN_1)
r <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(SDN_1,r)
Next I would like to save the output of the split function into separate files as .txt and encode as UTF8. The files need to be named in the following format: test_YYYMMDD_HHMMSS.txt
I'm new to R and any help would be appreciated.
UPDATE:
Hack-R suggested the code below to create the .csv file. The code below worked once then started giving me the error message below:
Code Hack-R suggested:
n <- 1
for(i in d){
con <- file(paste0("file",n,"_", gsub("-
","",gsub(":","",gsub("","_",Sys.time()))), "_",".csv"),encoding="UTF-8")
write.csv(tmp, file = con)
n <- n + 1
}
The error message I'm getting:
Error in is.data.frame(x) : object 'tmp' not found
Using the code you already have:
SDN_1 <- mtcars # this represents your csv, to make it reproducible
chunk <- 10 # scaled it down for the example
n <- nrow(SDN_1)
r <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(SDN_1,r)
n <- 1 # this part is optional
for(i in d){
con <- file(paste0("file",n,"_", gsub("-","",gsub(":","",gsub(" ","_",Sys.time()))), "_",".csv"),encoding="UTF-8")
write.csv(tmp, file = con)
n <- n + 1
}
More generally, let's say a and b represent the splits of a larger object or any collection of objects in the environment you want to write out programmatically:
a <- "a"
b <- "b"
You can get a vector containing their names:
files <- ls()
Then loop through and programmatically write them to a UTF-8 encoded csv file as follows, appending the date and time in the format you requested:
for(i in files){
tmp <- get(i)
con <- file(paste0(tmp,"_", gsub("-","",gsub(":","",gsub(" ","_",Sys.time()))), "_",".csv"),encoding="UTF-8")
write.csv(tmp, file = con)
}
I used Sys.time() for the timestamp with nested gsub()s to format the way you wanted. I encoded the file to UTF-8 as explained in this post.
I started recently to work with R so this question has probably a simple solution.
I have some .tif satellite images from different scenes. I can create a test raster brick with it but the process needs to be automatised because of the huge amount of files. Therefore I have been trying to create a function to read the list of .tif files and to output a list of rasters.
You can find here below the code I have been using:
# Description: Prepare a raster brick with ordered acquisitions
# from all the scenes of the study area
library(raster)
library(rgdal)
library(sp)
library(rtiff)
rm(list = ls())
setwd=getwd()
# If you want to download the .tif files of the 2 scenes from dropbox:
dl_from_dropbox <- function(x, key) {
require(RCurl)
bin <- getBinaryURL(paste0("https://dl.dropboxusercontent.com/s/", key, "/", x),
ssl.verifypeer = FALSE)
con <- file(x, open = "wb")
writeBin(bin, con)
close(con)
message(noquote(paste(x, "read into", getwd())))
}
dl_from_dropbox("lndsr.LT52210611985245CUB00-vi.NDVI.tif", "qb1bap9rghwivwy")
dl_from_dropbox("lndsr.LT52210611985309CUB00-vi.NDVI.tif", "sbhcffotirwnnc6")
dl_from_dropbox("lndsr.LT52210611987283CUB00-vi.NDVI.tif", "2zrkoo00ngigfzm")
dl_from_dropbox("lndsr.LT42240631992198XXX02-vi.NDVI.tif", "gx0ctxn2mca3u5v")
dl_from_dropbox("lndsr.LT42240631992214XXX02-vi.NDVI.tif", "pqnjw2dpz9beeo5")
dl_from_dropbox("lndsr.LT52240631986157CUB02-vi.NDVI.tif", "rrka10yaktv8la8")
# 1- Create a list of .tif files with names ordered chronologically (for time series analysis later on)
pathdir= # change
# List all the images from any scene in that folder and
# make a dataframe with a column for the date
a <- list.files(path=pathdir,pattern="lndsr.LT", all.files=FALSE,full.names=FALSE)
a1 <- as.data.frame(a, row.names=NULL, optional=FALSE, stringsAsFactors=FALSE) # class(a1$a) # character
# Create date column with julean date and order it in ascending order
a1$date <- substr(a1$a, 16, 22) # class(a1$date) = character
a1 <- a1[order(a1$date),]
# Keep only the column with the name of the scene
a1 <- subset(a1, select=1) # class(a1$a): character
# retrieve an ordered list from the dataframe
ord_dates <- as.list(as.data.frame(t(a1$a))) # length(ord_dates): 4 (correct)
# class(odd_dates) # list
# 2- Create rasters from elements of a list
for (i in 1:(length(ord_dates))){
# Point to each individual .tif file
tif_file <- ord_dates[i] # Problem: accesses only the first item of ord_dates
# Make a raster out of it
r <- raster(tif_file) # we cant use here a list as an input. Gives error:
# Error in .local(x, ...) : list has no "x"
# Give it a standardised name (r1,r2,r3, etc)
name <- paste("r", 1:length(ord_dates),sep = "")
# Write the raster to file
writeRaster (r , filename = name,format = "GTiff", overwrite =T )
}
I have also tried to use lapply() without much success.
r = lapply(ord_dates, raster)
Can you give me an advice on what concept to follow? I am guessing I should be using matrices but I don't really understand which are their advantages here or in what step they are required.
Any help is really appreciated!
Thanks in advance
Assuming ord_dates is a list of file names (that have full path or are in your getwd()), you can apply a (any) function to this list using lapply. I haven't tested this, unfortunately.
convertAllToRaster <- function(tif_file) {
r <- raster(tif_file)
# Give it a standardised name (r1,r2,r3, etc)
name <- paste("r", 1:length(ord_dates),sep = "")
# Write the raster to file
writeRaster (r , filename = name,format = "GTiff", overwrite =T )
message("Eeee, maybe it was written successfully.")
}
lapply(ord_dates, FUN = convertAllToRaster)
After solving the issues with factors and with the name, this is the code that worked for me. I added a for loop also inside the function you proposed, Roman. Thankyou very much for your kind help!!
convertAllToRaster <- function(ord_dates) {
for (i in 1:(length(ord_dates))){
tif_file <- ord_dates[i]
r <- raster(tif_file)
# Keep the original name
name <- paste(tif_file, ".grd", sep ="")
# Write the raster to file
writeRaster (r , filename = name,format = "raster", overwrite =T ) # in .grd format
}
}
lapply(ord_dates, FUN = convertAllToRaster)