I have downloaded a large number of Sentinel-2 SAFE files using the R package 'sen2r', which has implemented a Google Cloud download method to retrieve products stored in Long Term Archive. This has worked for me, but after checking the files I have found a decent number of empty files appended with _.gstmp, which according to this represent partially downloaded temporary files that are supposed to be resumed by gsutil. I have re-run the sen2r() command (with server = "gcloud" setting) but it does not resume and correct the downloads as the folders are already there. I would like to resume downloading just the _.gstmp files as it took over a week to download all of the SAFE products and I don't want to start all over again. I'm guessing I can fix this by using 'gsutil' directly but I'm a bit out of my element as this is my first experience using Google Cloud and the sen2r author as they no longer have time to respond to issues on github. If you have any tips for resuming these downloads manually using gsutil command line it would be much appreciated.
I have searched stack exchange and also the sen2r manual and github issues and have found any other reports of the problem.
Related
Title says it all. I want to be able to read data using a SharePoint link without saving the data locally on my device. Downloading the data is NOT an option. Any help is appreciated. I've already attempted using the Microsoft365R package but ran into an issue since the email authorized to access the data is managed by my company which is notoriously slow at approving access...
Since Rstudio cloud is no longer free, I'd like to download my projects from it. I found a lot of information how to download files from the projects, but I need to download the projects themselves, because I'd like to keep on working on them and be able to use my previous code. Is there any option to reach this, except copying code from every single project as a text?
Thanks in advance for any help
I am a somewhat new/basic git user and I'm having a problem that I can not seem to find an answer for. I am trying to figure out a way I can store two different branches from a github repo locally on my computer. My understanding is that when I clone a repo to my laptop from Github, it also downloads all of the branch and commit history my local machine. I want to continue to use github as a version control/backup for my project. However, I am working with colleagues who understand git less than I do so I am trying to find a way to help keep everything simple for them at the same time.
Here is a description of the situation:
We are developing an analysis in RStudio to examine information about quantitative writing in college students.
I am writing the R analysis scripts and want to maintain a safe backup with github
I am sharing the project files with my colleagues via Google drive since they do not know how to use git/ github
I have reached a point where I am going to change the fundamental file structure of the project. However, I do not want to disrupt their ability to perform analyses while I am making these changes.
My colleagues need to be able to save analysis outputs to the project folder where they are synced back to me via Google Drive and then pushed to GitHub.
I can think of two ways to handle this situation but both seem to have problems that I can't see around.
Create a branch in github, make changes to the branch and then merge the branch with the master
This won't work because I am sharing the files via Google Drive and you can only have one branch of a repo on your local machine at a time. Once clone the branch to my machine, that is what gets shared via Google Drive and any changes I make disrupts everyone else's workflow.
Create a second copy of my repo, make changes there, and then push those changes to the original repo that gets shared to colleagues via Google Drive ##
I have no idea how to do this. Everything I have read discusses how to push/pull between different github users. How can I do this as a single user?
Did I forget anything important?
Any help/suggestions greatly appreciated.
You can do it all in the same directory by adding a second remote origin, if you've already added GitHub as 'origin', then run:
git remote add gdrive https://example.com/path/to/repo.git
then you can push up your changes you've made to the two repos
to push to GitHub:
git push origin
and then to push to GitHub
git push gdrive
See this github doc page
I build transport models for various government agencies. My model is managed through GitHub, and it depends on R to perform certain calculations. I currently have my entire r installation folder in the repository. This can't be the right solution, but here are some of my constraints:
My clients are usually even less sophisticated programmers then I am. When they download/clone the model, it just needs to work.
This needs to be the case 10 years from now - regardless of what the current build of R and all the package dependencies are.
Placing my entire R folder in the repo solves these two problems, but creates some new ones:
The repository is much larger than it needs to be / longer download time.
If the transport model is updated to a new version (say v2.0), I'd want to update R and its packages to the latest versions. I'm afraid this would increase the size of the repo even further.
One solution I understand is submodules. I could place the full R folder in a separate repo and bring it in as a submodule. This, at the very least, cleans up the model repository.
What about zipping the R folder? Some early testing showed that git can diff the zip file, but I don't know if it is doing it as a flat file or reading the contents. Also, is GitHub going to complain about 100MB+ zip file? I'd like to avoid GitLFS if I can, but asking my clients to unzip that file wouldn't be a problem.
I also looked at packrat, but as far as I can tell, that only works for R projects.
Lastly, I don't entirely understand makefiles / recipes, but it would be nice if there was a script I could run that would download specific versions of R and it's libraries. One complicating thing is that some of the R packages are private GitHub repos.
Anyway, I'm happy to provide more info if needed. Thank you for your help!
First I should say that a lot of this is over my head, so I apologize in advance for using incorrect terminology and potentially asking an unclear question. I'm doing my best.
Also, I saw ThisPost; is RCurl the tool I want to use for this task?
Every day for 4 months I'll be analyzing new data, and generating .csv files and .png's that need to be uploaded to a web site so that other team members will be checking. I've (nearly) automated all of the data collecting, data downloading, analysis, and file saving. The analysis is carried out in R, and R saves the files. Currently I use Filezilla to manually upload the new files to the website. Is there a way to use R to upload the files to the web site, so that I don't have to open Filezilla and drag+drop files?
It'd be nice to run my R-code and walk away, knowing that once it finishes running, the newly saved files will be automatically be put on the website.
Thanks for any help!
You didn't specify which protocol you use to upload your files using FileZilla. I assume it is ftp. If so, you can use the ftpUpload function of RCurl:
library(RCurl)
ftpUpload("yourfile", "ftp://ftp.yourserver.foo/yourfile",
userpwd="username:passwd")
RCurl also had methods for scp and should also support sftp using ftpUpload.