Shiny server docker app runs locally but not when deployed to AWS Fargate - r

I have a containerized R Shiny app that, when run locally with docker run --rm -p 3838:3838 [image] works as expected. The "landing" page appears when I go to localhost:3838 and all is good. However, when this container is deployed to AWS Fargate, things break down. The container appears to start and run, but there's no webpage being served on 3838 even though all ports are pointed to 3838 in Fargate.
I'm using dockerfile
FROM rocker/verse:3.5.1
LABEL Steven "email"
## Add shiny capabilities to container
RUN export ADD=shiny && bash /etc/cont-init.d/add
## Update and install
RUN tlmgr update --self
RUN tlmgr install beamer translator
## Add R packages
RUN R -e "install.packages(c('shiny', 'googleAuthR', 'dplyr', 'googleAnalyticsR', 'knitr', 'rmarkdown', 'jsonlite', 'scales', 'ggplot2', 'reshape2', 'Cairo', 'tinytex'), repos = 'https://cran.rstudio.com/')"
#Copy app dir and them dirs to their respective locations
COPY app /srv/shiny-server/ga-reporter
COPY app/report/themes/SwCustom /opt/TinyTeX/texmf-dist/tex/latex/beamer/
#Force texlive to find my custom beamer thems
RUN texhash
EXPOSE 3838
## Add shiny-server information
COPY shiny-server.sh /usr/bin/shiny-server.sh
COPY shiny-customized.config /etc/shiny-server/shiny-server.conf
## Add dos2unix to eliminate Win-style line-endings and run
RUN apt-get update && apt-get install -y dos2unix
RUN dos2unix /usr/bin/shiny-server.sh && apt-get --purge remove -y dos2unix && rm -rf /var/lib/apt/lists/*
RUN ["chmod", "+x", "/usr/bin/shiny-server.sh"]
CMD ["/usr/bin/shiny-server.sh"]
and shiny-server.conf
# Instruct Shiny Server to run applications as the user "shiny"
run_as shiny;
# Define a server that listens on port 3838
server {
listen 3838;
# Define a location at the base URL
location / {
# Host the directory of Shiny Apps stored in this directory
app_dir /srv/shiny-server/ga-reporter;
# Log all Shiny output to files in this directory
log_dir /var/log/shiny-server;
# When a user visits the base URL rather than a particular application,
# an index of the applications available in this directory will be shown.
directory_index on;
}
}
with shiny-server.sh
#!/bin/sh
# ShinyServer: Make sure the directory for individual app logs exists
mkdir -p /var/log/shiny-server
chown -R shiny.shiny /var/log/shiny-server
# RUN ShinyServer
exec shiny-server >> /var/log/shiny-server.log 2>&1
I have edited the .conf file to display the app (i.e., location /) in /srv/shiny-server/ga-reporter which is also where I've copied the app_dir in the Dockerfile. Shiny is listening on port 3838 and should serve the page there. Again, this happens locally but not when deployed to AWS Fargate. I've tried logging Shiny logs to stdout by using the first answer provided here but have had no luck seeing any errors generated. Server "health checking" is only offered in the "pro" version so I can't check to see if the server is actually running.
On AWS, the container starts and appears to function normally (i.e., the "normal" start up logs appear):
but there is simply no page displayed at the location I expect it to be served.
I found another Shiny app that is on dockerhub running under the same configuration as the Fargate cluster but have had no luck trying to implement anything in the shiny-server.conf or the shiny-server.sh files located there.
What am I missing? Everything on Fargate is pointed to listening on 3838; there must be something I'm missing in the .conf file for this to be failing when deployed.
EDIT
I can't bash in to the running container on Fargate because I don't have access to the server on which docker is running.
Fargate has a UI that accepts host and container ports:
EDIT 2 (2018-08-27)
The engineer that was deploying this has been able to resolve the issue:
"it was the port change, I forgot to can the port on the ALB’s security group, I only updated the cluster’s inbound rules
so the cluster was allowing connections, but the ALB security group wasn’t letting it out"

Related

Dockerized Shiny Appp always shows Hello World not actual app from code while in RStudio its perfect

I have a code of 200 lines in R for shiny app which works perfect when I run from RStudio , while I have to Dockerize the app but when I run docker version of the app , I only see Hello-World app on localhost but not the actual code. I see inside container there are some extra files which docker always copies which is the problem, Can someone help me out ?
this is Dockerfile
FROM rocker/shiny:3.5.1
RUN apt-get update && apt-get install libcurl4-openssl-dev libv8-3.14-dev -y &&\
mkdir -p /var/lib/shiny-server/bookmarks/shiny
# Download and install library
RUN R -e "install.packages(c("library(shiny)","shinydashboard","xlsx", "V8"))"
# copy the app to the image
COPY app /srv/shiny-server/
# make all app files readable (solves issue when dev in Windows, but building in Ubuntu)
RUN chmod -R 755 /srv/shiny-server/
EXPOSE 3838
CMD ["/usr/bin/shiny-server.sh"]
Thanks
Thanks
You could copy additional files / folders individually, e.g.:
COPY modules /srv/shiny-server/modules
COPY R /srv/shiny-server/R
COPY *.R /srv/shiny-server/
COPY www /srv/shiny-server/www
Not sure, but are you copying your shiny-server.sh file over? I usually run my dockerized Shiny Apps with:
CMD ["Rscript", "-e", "shiny::runApp('/srv/shiny-server/', host='0.0.0.0', port=3838)"]

How can I get a golem app to generate logs when dockerized?

We have a dockerized golem app that runs fine except for not creating any output (log statements) when deployed in a docker container. In fact, we aren't even seeing any default shiny-server logs.
Here is app.R for our "AirSensorDataViewer" golem app:
pkgload::load_all(export_all = FALSE, helpers = FALSE, attach_testthat = FALSE)
options(
golem.app.prod = TRUE,
shiny.port = 3838,
shiny.host = '0.0.0.0'
)
AirSensorDataViewer::run_app()
And here is our Dockerfile (that builds on top of of a base image with all necessary packages):
FROM mazamascience/airsensor-dataviewer-base:1.0.1
# Create the build zone, copy the local directory over to the docker image, build and install R package.
RUN mkdir /build_zone
ADD . /build_zone
WORKDIR /build_zone
RUN R -e 'remotes::install_local(upgrade="never")'
# Remove sample apps
RUN rm -rf /srv/shiny-server/
# copy app to image
COPY . /srv/shiny-server/asdv
# add .conf file to image/container to preserve log file
COPY ./shiny-server.conf /etc/shiny-server/shiny-server.conf
# When run image and create a container, this container will listen on port 3838
EXPOSE 3838
# Avoiding running as root --> run container as user 'shiny' instead
# allow permission
RUN sudo chown -R shiny:shiny /srv/shiny-server
RUN chmod -R 755 /srv/shiny-server/asdv
# execute in the following as user --> imortant to give permission before that step
USER shiny
##run app
CMD ["/usr/bin/shiny-server.sh"]
And, lastly, our shiny-server.conf file:
# Instruct Shiny Server to run applications as the user "shiny"
run_as shiny;
# Define a server that listens on port 3838
server {
listen 3838;
# Define a location at the base URL
location /asdv/test/ {
# Host the directory of Shiny Apps stored in this directory
site_dir /srv/shiny-server/asdv;
# Log all Shiny output to files in this directory
log_dir /var/log/shiny-server;
# When a user visits the base URL rather than a particular application,
# an index of the applications available in this directory will be shown.
directory_index on;
}
}
Has anyone had success getting a dockerized golem app to create/write to files inside the docker container?
I hope the posted question might be helpful for those wanting to do this as it turns out everything works fine.
For the last few hours I was mistakenly typing docker run ... when I wanted to check the container log files. This creates a new container.
Instead, log files are indeed found in /var/log/shiny-server/ when I:
docker exec -ti airsensor-dataviewer-desktop /bin/bash
ls /var/log/shiny-server/

How to run R Shiny App in Docker Container

I built a Docker Image for an R Shiny App and ran the corresponding container with Docker Toolbox on Windows 10 Home. When trying to open the App with my web browser, only the index is shown. I don't know why the app isn't executed.
The log shows me this:
*** warning - no files are being watched ***
[2019-08-12T15:34:42.688] [INFO] shiny-server - Shiny Server v1.5.12.1 (Node.js v10.15.3)
[2019-08-12T15:34:42.704] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf"
[2019-08-12T15:34:43.100] [INFO] shiny-server - Starting listener on http://[::]:3838
I already specified the app host-to-container path by executing the following command which refers to a docker hub image:
docker run --rm -p 3838:3838 -v /C/Docker/App/:/srv/shinyserver/ -v /C/Docker/shinylog:/var/log/shiny-server/ didsh123/ps_app:heatmap
My Docker File looks like the following:
# get shiny serves plus tidyverse packages image
FROM rocker/shiny-verse:latest
# system libraries of general use
RUN apt-get update && apt-get install -y \
sudo \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
libssl-dev \
libssh2-1-dev
##Install R packages that are required--> were already succesfull
RUN R -e "install.packages(c('shinydashboard','shiny', 'plotly', 'dplyr', 'magrittr'))"
#Heatmap related packages
RUN R -e "install.packages('gpclib', type='source')"
RUN R -e "install.packages('rgeos', type='source')"
RUN R -e "install.packages('rgdal', type='source')"
# copy app to image
COPY ./App /srv/shiny-server/App
# add .conf file to image/container to preserve log file
COPY ./shiny-server.conf /etc/shiny-server/shiny-server.conf
##When run image and create a container, this container will listen on port 3838
EXPOSE 3838
###Avoiding running as root --> run container as user instead
# allow permission
RUN sudo chown -R shiny:shiny /srv/shiny-server
# execute in the following as user --> imortant to give permission before that step
USER shiny
##run app
CMD ["/usr/bin/shiny-server.sh"]
So when I address the docker ip and the assessed port in the browser, the app should run there, but only the index is displayed. I use the following line:
http://192.168.99.100:3838/App/
I'm glad for every hint or advice. I'm new to Docker, so I'm also happy for detailed explanations.
To use shiny with docker, I suggest you use the golem package. golem provides a framework for builing shiny apps. If you have an app developed according to their framework, the function golem::add_dockerfile() can be used to create dockerfiles automatically.
If you are not interested in a framework, You can still have a look at the source for add_dockerfile() to see how they manage the deployment. Their strategy is to use shiny::runApp() with the port argument. Therefore, shiny-server is not necessary in this case.
The Dockerfile in golem looks roughly like this
FROM rocker/tidyverse:3.6.1
RUN R -e 'install.packages("shiny")'
COPY app.R /app.R
EXPOSE 3838
CMD R -e 'shiny::runApp("app.R", port = 3838, host = "0.0.0.0")'
This will make the app available on port 3838. Of course, you will have to install any other R packages and system dependencies.
Additional tips
To increase reproducibility, I would suggest you use remotes::install_version() instead of install.packages().
If you are going to deploy several applications with similar dependencies (for example shinydashboard), it makes sense to write your own base image that can be used in place of rocker/tidyverse:3.6.1. This way, your builds will be much quicker.
Minimal Reproducible Example
Create an empty directory (can be called anything you want)
Inside it, create two things:
i. A file called Dockerfile
ii. An empty directory called app
Place your shiny app inside the directory called app.
Your shiny app can be a single app.R file, or, for older shiny apps, ui.R and server.R. Either way is fine (see here for more on that).
If unsure about any of the above, just copy the files found here.
Place this in Dockerfile
FROM rocker/shiny:latest
COPY ./app/* /srv/shiny-server/
CMD ["/usr/bin/shiny-server"]
In the terminal, cd to the root of the empty directory you created in step 1, and build the image with
docker build -t shinyimage .
Run the container with
docker run -p 3838:3838 shinyimage
Finally, visit this url to see your shiny app: http://localhost:3838/
Here's a copy of all of the above in case anything's unclear.
Check the logs for any useful information? And exec into the container to verify if the App content is copied to the correct location.
Because the way /App content is copied looks incorrect
The content of /App is copied into the image to /srv/shiny-server/App during the build phase and you are trying to override /srv/shiny-server content using -v option when running the container.
Looks like during runtime the App data copied is being overwritten.
Try without -v /C/Docker/App/:/srv/shinyserver/ or use -v /C/Docker/App/:/srv/shinyserver/App/
docker run --rm -p 3838:3838 -v /C/Docker/shinylog:/var/log/shiny-server/ didsh123/ps_app:heatmap

Deploy shiny app in rocker/shiny docker

Well, I'm new at Docker and I need to implement a Shiny app in a Docker Container.
I have the image from https://hub.docker.com/r/rocker/shiny/, that includes Shiny Server, but I don't know how to deploy my app in the server.
I want to deploy the app in the server, install the required packages for my app into the Docker, save the changes and export the image/container.
As I said, I'm new at Docker and I don't know how it really works.
Any idea?
I guess you should start by creating a Dockerfile in a specific folder which would look like something like this :
FROM rocker/shiny:latest
RUN echo 'install.packages(c("package1","package2", ...), \
repos="http://cran.us.r-project.org", \
dependencies=TRUE)' > /tmp/packages.R \
&& Rscript /tmp/packages.R
EXPOSE 3838
CMD ["/usr/bin/shiny-server.sh"]
Then go into this folder and build your image, giving it a name by using this command :
docker build -t your-tag .
Finally, once your image is built you can create a container, and if you don't forget to map the volume and the port, you should be able to find it at localhost:3838 with the following command launched from the folder containing the srv folder :
docker run --rm -p 3838:3838 -v $PWD/srv/shinyapps/:/srv/shiny-server/ -v $PWD/srv/shinylog/:/var/log/shiny-server/ your-tag
As said in the Docker documentation at the following address https://hub.docker.com/r/rocker/shiny/, you might want to launch it in detached mode with -d option and map it with your host's port 80 for a real deployment.
The link(https://hub.docker.com/r/rocker/shiny/) covers how to deploy the shiny server.
Simplest way would be:
docker run --rm -p 3838:3838 rocker/shiny
If you want to extend shiny server, you can write your own Dockerfile and start with shiny image as base image.(https://docs.docker.com/engine/reference/builder/)
Dockerfile:
FROM rocker/shiny:latest

Docker run results in "host not found in upstream" error

I have a frontend-only web application hosted in Docker. The backend already exists but it has "custom IP" address, so I had to update my local /etc/hosts file to access it. So, from my local machine I am able to access the backend API without problem.
But the problem is that Docker somehow can not resolve this "custom IP", even when the host in written in the container (image?) /etc/hosts file.
When the Docker container starts up I see this error
$ docker run media-saturn:dev
2016/05/11 07:26:46 [emerg] 1#1: host not found in upstream "my-server-address.com" in /etc/nginx/sites/ms.dev.my-company.com:36
nginx: [emerg] host not found in upstream "my-server-address.com" in /etc/nginx/sites/ms.dev.my-company.com:36
I update the /etc/hosts file via command in Dockerfile, like this
# install wget
RUN apt-get update \
&& apt-get install -y wget \
&& rm -rf /var/lib/apt/lists/*
# The trick is to add the hostname on the same line as you use it, otherwise the hosts file will get reset, since every RUN command starts a new intermediate container
# it has to be https otherwise authentification is required
RUN echo "123.45.123.45 my-server-address.com" >> /etc/hosts && wget https://my-server-address.com
When I ssh into the machine to check the current content of /etc/hosts, the line "123.45.123.45 my-server-address.com" is indeed there.
Can anyone help me out with this? I am Docker newbee.
I have solved this. There are two things at play.
One is how it works locally and the other is how it works in Docker Cloud.
Local workflow
cd into root of project, where Dockerfile is located
build image: docker build -t media-saturn:dev .
run the builded image: docker run -it --add-host="my-server-address.com:123.45.123.45" -p 80:80 media-saturn:dev
Docker cloud workflow
Add extra_host directive to your Stackfile, like this
and then click Redeploy in Docker cloud, so that changes take effect
extra_hosts:
'my-server-address.com:123.45.123.45'
Optimization tip
ignore as many folders as possible to speed up process of sending data to docker deamon
add .dockerignore file
typically you want to add folders like node_modelues, bower_modules and tmp
in my case the tmp contained about 1.3GB of small files, so ignoring it sped up the process significantly

Resources