I would compare a local list of csv files with another remote list of csv file on online server. I would use R for this, how I can do ?
I'm alreday connect via R to the online server and now I would like to compare the local folder with the online without download the online
Related
I have been given access to a bunch of files on a server and have been tasked with making it so that these files are put on some sort of website where people can visit it and access it. I have been able to SSH to the server using terminal commands and am able to view all of the files on the directory. I have tried using scp_download, however, that downloads all of the files on there which is extremely large and would ideally be avoided. Is there a way where I can take the files on the server and get them directly uploaded to a website using Shiny (R package)? If not, is there another way possible?
It looks like you already discovered the R package ssh. You could ssh::ssh_exec_internal() to execute a command on the server that uploads/copies the files to the website server. How you can do that depends on the type of access you have to the website server.
If you have ssh access to the website server as well, your command could look something like this:
ssh::ssh_exec_internal(ssh_session,
"scp file remote_username#website_server_ip:/directory")
I am working on a remote jupyter notebook located on internal server. I want to save my findings/insight on my local computer to make further analysis.
Example:
Suppose this is the final data I have after analysis
I want to write it to my local disk.
But the code below writes the data to the server's D:\ drive
data.to_csv(r'D:\Team.csv')
This is the sample code for savinng the png file of the dataset which I have train.
plotting data for saving img to local library
plot_model(model, to_file="model.png", show_shapes=True, show_layer_names=True)
I Have a Excel file(xlsl) with multiple worksheets in Sharepoint server which I should read in informatica and load the data into different tables.
Informatica is hosted on unix server
Currently I am thinking of the below work around but I have challenges here:
1.Copying the excel file into unix.(Once i copy the file from sharepoint server to unix using the "curl" command, the format of the file is getting changed to html. I can I retain the original excel format ,I can't install any excel utility on our server)
2.Convert them into multiple CSV files using some script (How can I do this, As I mentioned earlier I don't have any utilities like xls2csv, unoconv )
3.And read the CSV file and load them into tables.
Please let me know if there is any better approach than this.
You can try using wget to download the excel file (or set of files) from the sharepoint to Informatica file server location. It will allow you to specify the directory and target file name as well.
This question already has answers here:
sftp get files with R [closed]
(1 answer)
Schedule weekly Excel file download to an unique name
(1 answer)
Closed 4 years ago.
My company´s data department uploads a .csv file daily to a remote folder that I can only access through my company's network using FileZilla.
Everyday, I use the newest .csv file and process the data in R. I want to automate this process. I want to access the daily .csv file by reading the .csv file from the remote folder using the read.csv function in R.
How can I tell FileZilla to copy the file in the shared folder to a local folder in my PC and do this everyday at 6:00 a.m.? If this isn't possible, how can I access the remote folder through R and read the .csvfile from there?
Thanks in advance!
EDIT:
As seen here, FileZilla does not allow any sort of automation. You can use the client WinSCP instead, write a script to download/upload files from/to a remote SFTP server and schedule the script to run every n days using Windows Task Scheduler.
Now, in order to access an SFTP server from R, you can use the RCrul package. Unfortunately, this closed question (which was closed because it was not even a question to begin with) purges unwanted lines of code from an FTPserver (even though the title says SFTP) and it doesn't specify the user, password nor port specs. Moreover, it uses the write.lines() command, which as I understand, is used to create, not download files.
This question specifically refers to downloading a .csv file from a shared folder using SFTP protocol. Given that FileZilla is no good for this, how can I manage to do this in R using RCurl?
I'm pretty new to cluster computing, so not sure if this is even possible.
I am successfully creating a spark_context in Rstudio (using sparklyr) to connect to our local Spark cluster. Using copy_to I can upload data frames from R to Spark, but I am trying to upload a locally stored CSV file directly to the Spark cluster using spark_read_csv without importing it into the R environment first (it's a big 5GB file). It's not working (even prefixing location with file:///), and it seems that it can only upload files that are ALREADY stored in the cluster.
How do I upload a local file directly to spark without loading it into R first??
Any tips appreciated.
You cannot. File has to be reachable from each machine in your cluster either as a local copy or placed on distributed files system / object storage.
You can upload the files from local to spark by using spark_read_csv() method. Please pass the path properly.
Note: It is not necessary to load the data first into R environment.