I am new to Rand Shyni. I created on my local an app which reads as a csva large file. But each time I run the app, the line read.csv(my_large_file...) takes long time. How can I prevent my app to fetch each time the data but instead store it in some memory cache ?
Just to give You some idea how You can solve this situation:
Read the csv file outside of the shiny App (not in server!) and read it using fread() function from data.table package (it is very fast in reading files)
Or
Use the solution of #ConCave: use save(df, "mydf.RData"), and load("mydf.RData")
Related
I am trying to develop a web scraping code. I need to automate it and run it in the Google Cloud daily. The daily web scrapped data have to be saved in a Google sheet. Following is the relevant part of the code that I have developed to save data in a CSV file and then upload it to an existing Google Sheet.
# Here is a sample data set
apt_link <- c('https://www.immobilienscout24.at/expose/6220b265d188d1cf74252fbb',
'https://www.immobilienscout24.at/expose/622f314859ff6df2ed86c2ee',
'https://www.immobilienscout24.at/expose/619ca702f1a2b400224637d4',
'https://www.immobilienscout24.at/expose/61cc1cf099a6ef002161f721',
'https://www.immobilienscout24.at/expose/606761cd2c34720022d4117f')
rooms <- c(4,5,2,4,3)
Surface <-c(87.09,104.00,44.90,138.00,146.00)
cost <- c(389000,497000,279000,1890000,1600000)
address <-c('1140 Wien','1210 Wien','1210 Wien','1180 Wien','1060 Wien')
# Creating a dataframe with web scrapped data
df_one <- cbind.data.frame(apt_link,rooms,surface,cost, address, Sys.time())
# Saving data as a CSV file in the computer
con <- file('Real_Estate_Wien_Data.csv',encoding="UTF-8")
data <- write.csv('Real_Estate_Wien_Data.csv', file=con, row.names = T)
# Write Google sheets
library(googlesheets4)
library(googledrive)
drive_auth()
# Link to the folder in my google drive
td <- drive_get("https://drive.google.com/drive/u/0/folders/1ZK6vUGXhRfzCPJ9I-gIrj3Xbzu72R1e3")
# Update
drive_put('Real_Estate_Wien_Data.csv', name = "Real_Estate_Wien_Data", type="spreadsheet", path=as_id(td)) # keeps id because of other links
The issue here is that now this code creates a CSV file on my computer. So that when I am going to automate it on the Google Cloud Platform, I think it's not possible to save the CSV file. There has to be another way to directly write the data to a Google Sheet.
Thank you in advance, and your suggestions are much appreciated.
I would recommend using Google Apps Script, as it is specifically built to interact with Sheets and other Google files. It seems to me that you would like to accomplish 3 different tasks, I've summarized them below:
Fetching Drive folders and files: This can be accomplished by Apps Script's DriveApp class. From here you can fetch folders via getFolderById() or getFoldersByName(), as well as fetching individual files with the same dynamic.
Writing data into spreadsheets: You can do that using the SpreadsheetApp class. The are many ways in which a Spreadsheet can be modified via code, here is a simple example of using the Range.setValues() function to write some data in the spreadsheet.
Running the code daily: Within Apps Script, you can easily set up Triggers (read more about them here) that will enable you to automatically run the code daily in the cloud, without interacting in any way with your local computer.
Not sure if you ever found the solution, but you can absolutely use googlesheet4package to write your data to a new or existing spreadsheet. Check out the write_sheet() function here.
I'm currently working on an R Shiny App that utilizes googlesheets4 to pull in a large dataset from GoogleSheets upon app launch. Loading in this dataset to my app takes ~2 minutes, which stalls my entire app's load time.
The only visual in my app is based on this GoogleSheets data, so it is very dependent on this specific dataset. Once the dataset gets pulled into my app, it is filter and therefore becomes much smaller (85,000 rows ---> 1,000 rows). This GoogleSheet data is updated every day, so I don't have the luxury of pre-downloading it once and storing it as a .csv forever.
There are two different fixes for this that I have tried but have been unsuccessful...curious if anyone has any thoughts.
Have a separate app running. My first idea was to create a separate Shiny app entirely, that would have a sole purpose of pulling the GoogleSheets df once a day. Once it pulls it, it would conduct the necessary data cleaning to get it down to ~1,000 rows, and then would push the smaller df to a different GoogleSheet link. Then, my original app with the visual would just always reference that new GoogleSheet (which would take much less time to load in).
The problem I ran into here is that I couldn't figure out how to write a new GoogleSheets doc using googlesheets4. If anyone has any idea how to do that it would be much appreciated.
Temporarily delay the load in of the GoogleSheets data, and let visual populate first. My second idea was to have the code that pulls in the GoogleSheets df be delayed upon launch, letting my visual first populate (using old data) and then have the GoogleSheets pull happen. Once the pull is complete, have the visual re-populate with the updated data.
I couldn't figure out the best/right way to make this happen. I tried messing around with sleep.sys() and futures/promises but couldn't get things to work correctly.
Curious if anyone has any thoughts on my 2 different approaches, or if there's a better approach I'm just not considering...
Thanks!
There is a function called write_sheet that allows you to write data to a google sheet. Does that work for you?
googlesheets4::write_sheet(data = your_data,
ss = spread_sheet_identifier,
sheet = "name_of_sheet_to_write_in")
If you on only want to add something without deleting everything in the sheet, the function is sheet_append
googlesheets4::sheet_append(data = your_data,
ss = spread_sheet_identifier,
sheet = "name_of_sheet_to_write_in")
Not sure you can store the credentials in a save way, but couldn't you use github actions? Or alternatively a cron job on your local computer?
I've created a shiny app that creates a graph based on data that's entered in daily via a shared excel sheet (.xlsx) that is in a shared folder (an L drive).
How would I format or upload the data so that it is able to be refreshed whenever a new daily line of data is entered?
Here is one possible approach along with reference documentations:
Create a workflow to fetch the data using its URL:
read in online xlsx - sheet in R
Make the data retrieval process reactive:
https://shiny.rstudio.com/articles/reactivity-overview.html
Set a reactiveTimer to periodically check for updates:
https://shiny.rstudio.com/reference/shiny/1.0.0/reactiveTimer.html
By doing so, your app will fetch the document on a regular basis to update your graph. If you want real time updates (i.e. every time there is a change in the document), you have to be able to trigger the application from outside, which is more complicated (especially via Excel).
Update:
Following up your comment; you don't need the data to be online. You are fine if you are able to import it into R. Just make this process reactive and set a timer to refresh everyday (see the documentation for examples). Alternatively you can have an actionButton to refresh manually.
I have a shiny app which selects a subset of observations from a large dataframe, and then renders r markdown reports against each observation of that subset, zipping them all these reports at the end and downloading the zip file.
When the subset is small (eg less than 10 reports), all works fine, but a network timeout occurs once it takes more than a certain amount of time to render all the reports in the background (eg in some cases more than 100 reports need to be rendered).
I have tried editing the config file to set app_init_timeout = 3600 and app_idle_timeout =3600 but this does not seem to impact this problem....
Any ideas?
I solved this problem by separating the report creation from the download. I used eventReactive to handle the report creation and the zipping of the files, and then made the downloadHandler conditional on the existence of the zip file, so that it only appeared when the downloaded file was ready.
I'm wondering if it's possible for a shiny app that's not run on the web (i.e. it's only run by a user launching it from their R session) to assign values to objects in the user's global environment. For example, suppose that as part of the app a data.frame is generated and, instead of using a download button to save the data.frame to a file, is it possible to assign it to an object in the user's R session so that when they close the app the data.frame is available to them?
What about automatically save the entire environment (using "save" function) to a temporary file? Then, you can just load it (using "load" function) and your data frame will be in the environment. All this process can be easily automated, without needing to use any save button.