How to cache data in shiny server? - r

I am using R to deploy an app over the web, but the URL from which my app takes data is where my app takes time. Is it possible to cache that data?
I tried to install the packages memoise, R.cache and a few more that were all unsupported by the server.

I recommend trying the DataCache package by Jason Bryer. The package is available through GitHub and I was successful in using it today for a Shiny app that I am developing.
The main function from that package is data.cache. You will need to a define a function that generates a list of objects that you want to cache, and then pass the function you define as an argument to data.cache. I also recommend setting the cache.name parameter of data.cache if you intend on caching more than one list of objects in your application.
For example:
DataCache::data.cache(
function(){
list(
normal_random_numbers = rnorm(10),
uniform_random_numbers = runif(10)
)
},
cache.name = 'my_random_numbers'
)
The above code creates two objects in the local environment, normal_random_numbers and uniform_random_numbers, as well as caches these to the file system. When you run this code again, the cached copies of these objects will be used rather than being regenerated - unless of course the cache expires. The frequency parameter of data.cache is used to set the expiry of the cache, which is set to daily by default.
If you are running the application under Windows then use this slightly modified version of the package. This is to address--- a bug that is apparently due to the cache filename being incompatible with the Windows file system.

This article from Rstudio is quite exhaustive and walks you through different ways to achieve that (i.e diskcache, the storr package or a Redis instance).
The main logic revolves around rendering a cached element and setting up the logic for cache invalidation:
function(input, output) {
renderCachedPlot(
{
# Code for a beautiful plot
},
# A change in the input or the dataframe will invalidate the cache
cacheKeyExpr = list({ input$n, df() })
)
}

Related

Auto-Update Data for R Shiny Dashboard

so I thought I had figured this out thanks to previous help (Updating R shiny data), but I have come across more issues. I didn't notice these issues since I got busy doing other things in my job. But turns out it is not auto-updating the app on shinyapps.io. I'll post the test code I have made in the server section:
dftest<-reactiveFileReader(1000,session,"Crop Registration 2022.xlsx",readFunc = read_excel,sheet = "CSV")
output$table2<-renderTable(
{
dftest2<-filter(dftest(),Variety=="Test1", Week>25 )
dftest2
}
)
What I found is after publishing the app on shinyapps.io any changes made to the excel sheet "CSV" do not show up. What I want is I can add data to the original excel sheet the app on shiny apps auto updates with the new data. Am I approaching this problem completely wrong? Or is it even possible via shinyapps.io ?

Is it possible to clear the Shiny cache when reloading an app?

Is it possible to clear the Shiny cache history when reloading an app? Specifically, I have a textInput for adding a title that doesn't clear when the app is restarted. In other words, all of the titles entered into the app on previous runs have been "remembered" and show up in a dropdown list when the app is rerun or restarted again. I have tried using shinyjs::reset, but that clears the current value and not the ones in the "history". I've also tried updateTextInput with an action button set to " ", but also clears only the current value and not the ones in the "history". I have also tried setting ui as a function using:
ui <- function(req) {
fluidPage(...)
}
as well as:
I found that cache of R Shiny server is updated when the date of creation for the file "app.R" is changed.
So, here is the trick I used:
server <- function(input, output, session) {
Trick file date creation update
onStop(function() {
# File name
p <- paste0(getwd(), "/app.R")
# Update file 'date creation'
Sys.setFileTime(p, now())
}) # onStop
} # server
The idea is to update the date of "app.R" creation after each session.
Neither of these solutions worked for me. Finally, I found this post:
I also found this post:
I have been struggling with this problem for quite a while, and thought I had tried everything, including putting a js button on the shiny sidebar to manually refresh (unfortunately that did not work either). There are two things that did work for me:
Make sure all code to read data from files is NOT in a code chunk with a name OTHER THAN global OR
Manually restart the shiny server when new data is uploaded
Obviously the first one is much more manageable, and a solution that I wish I had known weeks ago when I started playing with workarounds.
But I am not sure what is involved in implementing either, so if someone could clarify that, that would be great.
Any help would be much appreciated.

Configure random proxies with R for scraping

I scraped a website authorizing scraping in robots rules but sometimes I get blocked.
While I contacted the admin to understand why, I want to understand how I can use different proxies within R to keep on scraping without being blocked.
I followed this quick tutorial:
https://support.rstudio.com/hc/en-us/articles/200488488-Configuring-R-to-Use-an-HTTP-or-HTTPS-Proxy
So I edited the environment file:
file.edit('~/.Renviron')
and within this I inserted a list of proxies to be selected randomly:
proxies_list <- c("128.199.109.241:8080","113.53.230.195:3128","125.141.200.53:80","125.141.200.14:80","128.199.200.112:138","149.56.123.99:3128","128.199.200.112:80","125.141.200.39:80","134.213.29.202:4444")
proxy <-paste0('https://', sample(proxies_list, 1))
https_proxy=proxy
But when I scrape with this code:
download.file(url_proxy, destfile ='output.html',quiet = TRUE)
html_output <- read_html('output.html')
I keep being blocked.
Am I not setting the proxies correctly?
Thanks !
M.
You need to set environment variables, not R variables. See ?download.file for more details.
eg
Sys.setenv(http_proxy=proxy)
before anything else happens. Also note the warning in the docs:
These environment variables must be set before the download code is
first used: they cannot be altered later by calling 'Sys.setenv'.

How to make Shiny redirect immediately?

I set up a shiny app that checks for a GET string and presents a link if a file matching the id argument exists. Now what I would like to do is have the page redirect straight to the download file if a valid query is detected in the URL. Does anybody know of the syntax to insert, e.g. a <meta http-equiv=...> header from server.R?
Motivation: I want to be able to download files directly into an R console session from a URL pointing at a Shiny app. So, a non-geeky user specifies their preliminary statistical model using Shiny, then a statistician downloads it into their usual working environment and takes it the rest of the way. I need to do this server-side rather than with something like javascript's window.location because javascript won't be supported client-side.
Here is the server.R
shinyServer(function(input, output, clientData) {
query <- reactive(parseQueryString(clientData$url_search));
revals <- reactiveValues();
## obtain ID from GET string
observe({revals$id <- query()$id});
## alternatively obtain ID from user input if any
observe({input$submitid; if(length(id<-isolate(input$manualid))>0) revals$id <- id;});
## update filename, path, and existance flag
observe({ revals$filename <- filename <- paste0(id<-revals$id,".rdata");
revals$filepath <- filepath <- paste0("backups/",filename);
revals$filexists <- file.exists(filepath)&&length(id)>0; });
## update download handler
output$download <- {downloadHandler(filename=function() revals$filename, content=function(oo) if(revals$filexists) system(sprintf('cp %s %s',revals$filepath,oo)))};
## render the download link (or message, or lack thereof)
output$link <- renderUI({
cat('writing link widget\n');
id<-revals$id;
if(length(id)==0) return(div(""));
if(revals$filexists) list(span('Session download link:'),downloadLink('download',id)) else {
span(paste0("File for id ",id," not found"));}});
});
Here is the ui.R
shinyUI(pageWithSidebar(
headerPanel(div("Banner Text"),"Page Name"),
sidebarPanel(),
mainPanel(
htmlOutput('link'),br(),br(),
span(textInput('manualid','Please type in the ID of the session you wish to retrieve:'),actionButton('submitid','Retrieve')))));
Update:
In trying #jeff-allen 's suggestion, I ran into another problem: how to extract the filesystem path to which the files get copied for downloading and turn it into a valid URL? It's probably possible by screwing with shell scripts and http config settings on my local host, but how to do this in a portable way that doesn't require superuser privileges and is as shiny-native as possible?
Motivation: I want to be able to download files directly into an R
console session from a URL pointing at a Shiny app.
...i.e. this amounts to a very roundabout way of trying to serve static content from a shiny app. Turns out I don't need to redirect or use downloadHandler at all. As this post on the Shiny forum says, any file I create inside the local www directory will be accessible as if it is at the root of my app directory. I.e. if I have my app do save.image(file='www/foo.rdata') then I will be able to access it from [http://www.myhost.com/appname/foo.rdata] if the app itself lives at [http://www.myhost.com/appname/]

Out of memory error when publishing large result from GetListItems

I'm using SDL Tridion 2009 SP1 on a 64-bit server and trying to publish a massive XML of all the Multimedia Components in the system (190K+). I'm using the folder.GetListItems(filter) method with the filter set to Recursive="true".
The template runs for several seconds and then blows up with an out of memory error:
<?xml version="1.0"?>
<tcm:Error xmlns:tcm="http://www.tridion.com/ContentManager/5.0" ErrorCode="7" Category="7" Source="Kernel" Severity="1"><tcm:Line ErrorCode="7" Cause="true"><![CDATA[Out of memory]]></tcm:Line><tcm:Details><tcm:CallStack><tcm:Location>FolderBLST.GetListData</tcm:Location></tcm:CallStack></tcm:Details></tcm:Error>
at Tridion.ContentManager.Interop.TDSBL._IBLOrganizationalItemST.GetListData(UserContext userContext, String URI, EnumListKind listKind, ListColumnFilter columnFilter, String rowFilter)
at Tridion.ContentManager.ContentManagement.OrganizationalItem.GetListItems(Filter filter)
at myNS.myTbb.Transform(Engine engine, Package package)
at Tridion.ContentManager.Templating.Assembly.AssemblyMediator.Transform(Engine engine, Template template, Package package)
at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.RunTemplate(Engine engine, Package package, String templateUri, String className)
at Tridion.Templating.CSharpTemplate.CSharpSourceTemplate.Transform(Engine __engine, Package __package)
at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.Transform(Engine engine, Template template, Package package)
at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
at Tridion.ContentManager.Templating.Compound.CompoundTemplateMediator.Transform(Engine engine, Template templateToTransform, Package package)
at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
at Tridion.ContentManager.Templating.Engine.TransformPackage(Template template, Package package)
at Tridion.ContentManager.Templating.Engine.TransformItem(Template template, IdentifiableObject itemToRender)
at Tridion.ContentManager.Templating.Debugging.DebuggingEngine.Run()
at Tridion.ContentManager.Templating.Debugging.DebugSession.Run()
From the stack trace it looks as if the error is happening in the Business Layer of the CM server. Is there a memory setting that I can increase for this, and if so, which is it?
folder.GetListItems(filter) recursive is going to consume a huge amount of resources in your scenario.
If you have a big amount of multimedia items, it is adding a huge overload to the system.
Even if you try to scale the server, you will face the same issue at some point.
In general, you will face this issue as you are trying to perform a huge operation for data retrieval.
Maybe you can use different techniques for achieve the same scenario (The following are samples)
Scenario 1
Using the Event System, you can add information of the binary (when is created, etc...) to a common repository (example an XML stored in a field of a System Component) and publish that XML once in a while.
If you need just a list of ids for instance, use a Component created for store that information. You can also define a range of ids and create new Components, if required, for not having too many entries in only one (example: ids from 0 to 10000 will be stored in a component named References_0_10000, ids from 10001 to 20000 in a component named References_10001_20000).
Scenario 2
Split the initial load in Sub-Loads (still using the recursive=true) when processing subfolders within the main Folder for instance, and assemble the results.
In this case you minimize the folder.GetListItems(filter) load.
Scenario 3
Use still the folder.GetListItems(filter) multiple times but implement the recursive logic in your code, instead to use that in the filter options and assemble the results returned per each call.
Note:
Check the TimeOut settings of the SDL Tridion Content Manager configuration MMC Snap-in and increase those in case that helps.

Resources