I have 30 files: f1.csv, f2.csv, ... f30.csv.
I would like to upload all files with R, about as:
ftpUpload(c(f1.csv, f2.csv, ... f30.csv), http://..., ...)
How can I to upload with the command ftpUpload many files?
As #Soheil mentions, why not just save the files first, then upload?
Any reason you can't just do a for loop?
Something like:
files = c("f1.csv", "f2.csv", "f30.csv")
for (file in files){
ftpUpload(file,
paste("ftp://...",file,sep = ""),
)
}
Related
I'm new to R and having trouble with optimizing a function.
My function is to:
create a directory specified in the function
download the zip file from the link inside the function and extract it to the directory
move extracted files to the main directory if files are extracted under a new subfolder
delete the subfolder
It works but consumes a lot of memory and takes 30mins to do such an easy job on a 2.7MB zip file.
Thank you in advance!
create_dir <- function(directory) {
path <- file.path(getwd(), directory)
if (!file.exists(path)) {
dir.create(path)
}
link <-
"https://d396qusza40orc.cloudfront.net/rprog%2Fdata%2Fspecdata.zip"
temp <- tempfile()
download.file(link, temp, mode = "wb")
unzip(temp, exdir = path)
unlink(temp)
existing_loc <- list.files(path, recursive = TRUE)
for (loc in existing_loc) {
if (length(grep("/", loc))) {
file.copy(file.path(path, loc), path)
file.remove(file.path(path, loc))
}
}
dirs <- list.dirs(path)
rm_dirs <- dirs[dirs != path]
if (length(rm_dirs)) {
for (dir in rm_dirs) {
unlink(rm_dirs, recursive = TRUE)
}
}
}
create_dir("testDirectory")
Thanks, I found the problem. It's because of setting a working directory on OneDrive that syncs for every extraction, moving, and deletion of 332 files processed by the function. AntiVirus also run along with OneDrive and caused my PC to freeze for 30 mins by using 70% of CPU.
I have a folder with multiples R functions. I would like to source all these functions in R studio. I tried the following code, but it does not work. I found the following code here ()
sourceFolder <- function(folder, recursive = FALSE, ...)
{
files <- list.files(folder, pattern = "[.][rR]$",
full.names = TRUE, recursive = recursive)
if (!length(files))
stop(simpleError(sprintf('No R files in folder "%s"', folder)))
src <- invisible(lapply(files, source, ...))
message(sprintf('%s files sourced from folder "%s"', length(src), folder))
}
sourceFolder("C:/Users/Admin/Desktop/autots",recursive = TRUE))
However, I got this error:
> sourceFolder("C:/Users/Admin/Desktop/autots",recursive = TRUE)
Error in environment(lstar) : object 'lstar' not found
Called from: environment(lstar)
I do not know how to fix it. The package that I try to install it as a source can be found here http://stat.snu.ac.kr/heeseok/autots
Any help, please?
I am looking a way where I can upload files with in a nested folder using R Programming
I tried below functions which uploads either file or folder from aws.s3 package
library("aws.s3")
put_object("pathoftheobject", object = "filename", bucket = "bucketname")
put_folder("foldername", bucket = "bucketname")
Folder Structure and Files :
ParentFolder
SubFolder1
File1
File2
SubFoler2
File3
File4
............
SubFoldern
Filen
Any guidance here will be really useful.
I'm not an R developer but in C# instead of creating folders under buckets, I use filename with / so S3 recognizes that as sub-folder.
my file name when uploading:
ParentFolder/SubFolder1/Fil1
ParentFolder/SubFolder1/Fil2
ParentFolder/SubFolder2/Fil3
aws.s3::s3sync function does the needful.
library(aws.s3)
Sys.setenv("AWS_ACCESS_KEY_ID" = access_key_id,
"AWS_SECRET_ACCESS_KEY" = secret_access_key,
"AWS_DEFAULT_REGION" = "eu-central-1",
"AWS_SESSION_TOKEN" = session_token)
s3sync(files =dir(paste0(getwd(), "/Folder1Name/","Folder2Name"),recursive = T),
bucket = "BucketName", direction = "upload",verbose = TRUE)
I am trying to extract(unzip) folder (namely "pakistan.zip" which contains 5 files Pak_admin0.shp, Pak_admin0.shx, Pak_admin0.dbf, Pak_admin0.prj, Pak_admin0.qpj) and copying the files of .shp, .shx, .dbf files from that folder to destination folder using Rstudio 0.99.451 version with the following codes:
for(j in list(".shp", ".shx", ".dbf"))
{
fname <- unzip(file=paste("pakistan", j, sep=""), zipfile= "pakistan.zip")
file.copy(fname, paste("./pakistan", j, sep="/"), overwrite=TRUE)
}
unlink("pakistan.zip")
but it gives me following error
Warning messages:
1: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
2: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
3: In unzip(file = paste("zupanije", j, sep = ""), zipfile = "pakistan.zip") : requested file not found in the zip file
Please provide any possible solution to deal with this error.
These are actual codes which I have found but zip.file.extract function is no longer part of R:
for(j in list(".shp", ".shx", ".dbf")){
fname <- zip.file.extract(file=paste("zupanije", j, sep=""),
zipname="zupanije.zip")
file.copy(fname, paste("./zupanije", j, sep=""), overwrite=TRUE)
}
unlink("zupanije.zip")
I want to automate the structure of downloading the shape file from website and unzip it and place into another folder then will display it using maptools library using readShapePoly() function.
Your code works for me for a zip file that contains those files. The error suggests those files are not contained in the zip file. Since you say you are trying to extract a "directory" perhaps they are in a subdirectory in the zipfile? For example, if I put the files in a "temp" directory and then create a zip file of that directory, I must add the directory to the file path, like this:
f <- "test.zip"
for(j in list(".shp", ".shx", ".dbf"))
{
# note "pakistan" directory added to path
# unzip pakistan/zupanije.shp (or .shx or .dbf) out of test.zip
fname <- unzip(file=paste("pakistan/zupanije", j, sep=""), zipfile= f)
#copy extracted file to destination directory
file.copy(fname, paste("./destination", j, sep="/"), overwrite=TRUE)
}
If you are in a Linux like environment, you could try the following command to inspect the zip file and ensure it contains what you think it contains and at the path you expect:
unzip -vl pakistan.zip
By the way, your code will output the file "./pakistan/.dbf", "./pakistan/.shx" and "./pakistan/.shp". Is that what you want? Or do you perhaps want "pakistan.shx", etc. in which case this change is needed:
-file.copy(fname, paste("./pakistan", j, sep="/"), overwrite=TRUE)
+file.copy(fname, paste("./pakistan", j, sep=""), overwrite=TRUE)
A password cannot be specified in unzip (utils) function. The other function I am aware of, getZip (Hmisc), only works for zip files containing one compressed file.
I would like to do something like this to unzip all the files in foo.zip in Windows 8:
unzip("foo.zip", password = "mypass")
I found this question very useful but saw that no formal answers were posted, so here goes:
First I installed 7z.
Then I added "C:\Program Files\7-Zip" to my environment path.
I tested that the 7z command was recognized from the command line.
I opened R and typed in system("7z x secure.7z -pPASSWORD") with the appropriate PASSWORD.
I have multiple zipped files and I'd rather not the password show in the source code or be stored in any text file, so I wrote the following script:
file_list <- list.files(path = ".", pattern = ".7z", all.files = T)
pw <- readline(prompt = "Enter the password: ")
for (file in file_list) {
sys_command <- paste0("7z ", "x ", file, " -p", pw)
system(sys_command)
}
which when sourced will prompt me to enter the password, and the zip files will be decompressed in a loop.
I found #Kim 's answer worked for me eventually but not first off. I thought I'd just add a few extra links/steps that helped me get there in the end.
Close and reopen R so that environment path is recognised
If you've already opened R when you do steps 1-3 you need to close and reload R for R to recognise the environment path for 7z. #wush978 's answer to this question r system doesn't work when trying 7zip was informative. I used Sys.getenv("PATH") to check that 7zip was included in the environment paths.
Step 4. I opened R and typed in system("7z x secure.7z -pPASSWORD") with the appropriate PASSWORD.
I actually found this didn't work so I modified it slightly following the instructions in this post which also explains how to specify an output directory https://stackoverflow.com/a/16098709/13678913.
If you have already extracted the files the system command prompts you to choose whether you want to replace the existing file with the file from the archive and provides options
(Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit?
So the modified step 4 (Y allows replacement of files)
system("7z e -ooutput_dir secure.zip -pPASSWORD" Y)
Putting this altogether as a modified set of instructions
Install 7z.
Added "C:\Program Files\7-Zip\" to my environment path using menu options (instructions here https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html)
Closed and reopened R studio. Typed Sys.getenv("PATH") to check path to 7zip recognised in the environment (as per #wush978 's answer to question r system doesn't work when trying 7zip)
Typed in the console system("7z e -oC:/My Documents/output_dir secure.zip -pPASSWORD") with the appropriate PASSWORD (as per instructions here https://stackoverflow.com/a/16098709/13678913)
And here is a modified version of #Kim 's neat function (including specified output directory and check for existing files):
My main script
output_dir <- "C:/My Documents/output_dir " #space after directory name is important
zippedfiles_dir <- "C:/My Documents/zippedfiles_dir/"
file_list <- paste0(output_dir , zippedfiles_dir , list.files(path = zippedfiles_dir, pattern = ".zip", all.files = T))
source("unzip7z.R")
Code inside source file unzip7z.R
pw = readline(prompt = "Enter the password: ")
for (file in file_list) {
csvfile <- gsub("\\.zip", "\\.csv", gsub(".*? ", "", file)) #csvfile name (removes output_dir from 'file' and replaces .zip extension with .csv)
#check if csvfile already exists in output_dir, and if it does, replace it with archived version and if it doesn't exist, continue to extract.
if(file.exists(csvfile)) {
sys_command = paste0("7z ", "e -o", file, " -p", pw, " Y")
} else {
sys_command = paste0("7z ", "e -o", file, " -p", pw)
}
system(sys_command)
}
password <- "your password"
read.table(
text = system(paste0("unzip -p -P ", password, " yourfile.zip ", "yourfile.csv"),
intern = "TRUE"
), stringsAsFactors = FALSE, header = TRUE, sep = ","
)
password <- "your password"
system(
command = paste0("unzip -o -P ", password, " ", "yourfile.zip"),
wait = TRUE
)