I am looking a way where I can upload files with in a nested folder using R Programming
I tried below functions which uploads either file or folder from aws.s3 package
library("aws.s3")
put_object("pathoftheobject", object = "filename", bucket = "bucketname")
put_folder("foldername", bucket = "bucketname")
Folder Structure and Files :
ParentFolder
SubFolder1
File1
File2
SubFoler2
File3
File4
............
SubFoldern
Filen
Any guidance here will be really useful.
I'm not an R developer but in C# instead of creating folders under buckets, I use filename with / so S3 recognizes that as sub-folder.
my file name when uploading:
ParentFolder/SubFolder1/Fil1
ParentFolder/SubFolder1/Fil2
ParentFolder/SubFolder2/Fil3
aws.s3::s3sync function does the needful.
library(aws.s3)
Sys.setenv("AWS_ACCESS_KEY_ID" = access_key_id,
"AWS_SECRET_ACCESS_KEY" = secret_access_key,
"AWS_DEFAULT_REGION" = "eu-central-1",
"AWS_SESSION_TOKEN" = session_token)
s3sync(files =dir(paste0(getwd(), "/Folder1Name/","Folder2Name"),recursive = T),
bucket = "BucketName", direction = "upload",verbose = TRUE)
Related
I am trying to download a .shp file that is inside a .zip file. This .zip file is on https://drive.google.com/drive/u/2/folders/1u5oO0h9YKL1Nfi_pLGJgcEGGvUBJnE7D. I tried:
url = "https://drive.google.com/drive/u/2/folders/1u5oO0h9YKL1Nfi_pLGJgcEGGvUBJnE7D"
dir = tempdir()
temp = tempfile(fileext = ".zip", tmpdir = dir)
googledrive::drive_download(url, path = temp, overwrite = T)
And I got this error:
Error in `cli::cli_abort()`:
! Cannot export Google file of type:
* application/vnd.google-apps.folder
as a file of type:
* application/zip
How do I download it?
I'm new to R and having trouble with optimizing a function.
My function is to:
create a directory specified in the function
download the zip file from the link inside the function and extract it to the directory
move extracted files to the main directory if files are extracted under a new subfolder
delete the subfolder
It works but consumes a lot of memory and takes 30mins to do such an easy job on a 2.7MB zip file.
Thank you in advance!
create_dir <- function(directory) {
path <- file.path(getwd(), directory)
if (!file.exists(path)) {
dir.create(path)
}
link <-
"https://d396qusza40orc.cloudfront.net/rprog%2Fdata%2Fspecdata.zip"
temp <- tempfile()
download.file(link, temp, mode = "wb")
unzip(temp, exdir = path)
unlink(temp)
existing_loc <- list.files(path, recursive = TRUE)
for (loc in existing_loc) {
if (length(grep("/", loc))) {
file.copy(file.path(path, loc), path)
file.remove(file.path(path, loc))
}
}
dirs <- list.dirs(path)
rm_dirs <- dirs[dirs != path]
if (length(rm_dirs)) {
for (dir in rm_dirs) {
unlink(rm_dirs, recursive = TRUE)
}
}
}
create_dir("testDirectory")
Thanks, I found the problem. It's because of setting a working directory on OneDrive that syncs for every extraction, moving, and deletion of 332 files processed by the function. AntiVirus also run along with OneDrive and caused my PC to freeze for 30 mins by using 70% of CPU.
I have a folder with multiples R functions. I would like to source all these functions in R studio. I tried the following code, but it does not work. I found the following code here ()
sourceFolder <- function(folder, recursive = FALSE, ...)
{
files <- list.files(folder, pattern = "[.][rR]$",
full.names = TRUE, recursive = recursive)
if (!length(files))
stop(simpleError(sprintf('No R files in folder "%s"', folder)))
src <- invisible(lapply(files, source, ...))
message(sprintf('%s files sourced from folder "%s"', length(src), folder))
}
sourceFolder("C:/Users/Admin/Desktop/autots",recursive = TRUE))
However, I got this error:
> sourceFolder("C:/Users/Admin/Desktop/autots",recursive = TRUE)
Error in environment(lstar) : object 'lstar' not found
Called from: environment(lstar)
I do not know how to fix it. The package that I try to install it as a source can be found here http://stat.snu.ac.kr/heeseok/autots
Any help, please?
I am trying to extract(unzip) folder (namely "pakistan.zip" which contains 5 files Pak_admin0.shp, Pak_admin0.shx, Pak_admin0.dbf, Pak_admin0.prj, Pak_admin0.qpj) and copying the files of .shp, .shx, .dbf files from that folder to destination folder using Rstudio 0.99.451 version with the following codes:
for(j in list(".shp", ".shx", ".dbf"))
{
fname <- unzip(file=paste("pakistan", j, sep=""), zipfile= "pakistan.zip")
file.copy(fname, paste("./pakistan", j, sep="/"), overwrite=TRUE)
}
unlink("pakistan.zip")
but it gives me following error
Warning messages:
1: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
2: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
3: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
Please provide any possible solution to deal with this error.
These are actual codes which I have found but zip.file.extract function is no longer part of R:
for(j in list(".shp", ".shx", ".dbf")){
fname <- zip.file.extract(file=paste("zupanije", j, sep=""),
zipname="zupanije.zip")
file.copy(fname, paste("./zupanije", j, sep=""), overwrite=TRUE)
}
unlink("zupanije.zip")
I want to automate the structure of downloading the shape file from website and unzip it and place into another folder then will display it using maptools library using readShapePoly() function.
Your code works for me for a zip file that contains those files. The error suggests those files are not contained in the zip file. Since you say you are trying to extract a "directory" perhaps they are in a subdirectory in the zipfile? For example, if I put the files in a "temp" directory and then create a zip file of that directory, I must add the directory to the file path, like this:
f <- "test.zip"
for(j in list(".shp", ".shx", ".dbf"))
{
# note "pakistan" directory added to path
# unzip pakistan/zupanije.shp (or .shx or .dbf) out of test.zip
fname <- unzip(file=paste("pakistan/zupanije", j, sep=""), zipfile= f)
#copy extracted file to destination directory
file.copy(fname, paste("./destination", j, sep="/"), overwrite=TRUE)
}
If you are in a Linux like environment, you could try the following command to inspect the zip file and ensure it contains what you think it contains and at the path you expect:
unzip -vl pakistan.zip
By the way, your code will output the file "./pakistan/.dbf", "./pakistan/.shx" and "./pakistan/.shp". Is that what you want? Or do you perhaps want "pakistan.shx", etc. in which case this change is needed:
-file.copy(fname, paste("./pakistan", j, sep="/"), overwrite=TRUE)
+file.copy(fname, paste("./pakistan", j, sep=""), overwrite=TRUE)
I have 30 files: f1.csv, f2.csv, ... f30.csv.
I would like to upload all files with R, about as:
ftpUpload(c(f1.csv, f2.csv, ... f30.csv), http://..., ...)
How can I to upload with the command ftpUpload many files?
As #Soheil mentions, why not just save the files first, then upload?
Any reason you can't just do a for loop?
Something like:
files = c("f1.csv", "f2.csv", "f30.csv")
for (file in files){
ftpUpload(file,
paste("ftp://...",file,sep = ""),
)
}