I'm interested in running a simple image like this behind a corporate Zscaler firewall:
FROM rocker/r-base
RUN apt-get update && apt-get install libssl-dev
CMD Rscript -e "install.packages('beepr')"
Building the image with docker build -t test . fails with errors like this:
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: ]
I've tried some of the solutions from here but they don't work. For example:
FROM rocker/r-base
# Add local certificate to Docker
ADD ./zscaler.cer /usr/local/share/ca-certificates/zscaler.crt
# Move the certificate to the cert dir of openssl and update certificates
RUN CERT_DIR=$(openssl version -d | cut -f2 -d \")/certs ; cp /usr/local/share/ca-certificates/zscaler.crt $CERT_DIR ; update-ca-certificates
# Try making https requests
RUN apt-get update && apt-get install libssl-dev
CMD Rscript -e "install.packages('beepr')"
Same errors persist with docker build -t test .. I've read some possible solutions online but all of them continually fail either for apt-get or for installing packages with R. Is there anyone who has experienced this and found a fix?
Apparently, the current advice is slightly wrong. The certificate should not go in /etc/ssl/certs/ (which is the result of CERT_DIR=$(openssl version -d | cut -f2 -d \")/certs) but rather on CERT_DIR=/usr/local/share/ca-certificates/ (at least on this Ubuntu image). After changing that, update-ca-certificates correctly updates the certificate an all HTTPS requests are successful.
This should work now:
FROM rocker/r-base
# Add local certificate to Docker
ADD ./zscaler.pem /usr/local/share/ca-certificates/ZscalerRootCertificate-2048-SHA256.crt
# update certificates
RUN update-ca-certificates
# Try making https requests
RUN apt-get update && apt-get install libssl-dev
CMD Rscript -e "install.packages('beepr')"
Related
I am trying to run a shiny application on an open port on my server. I usually run docker images using command docker run -p 4000:3838 tag_name, assuming that docker container has exposed port, and shiny app is running at this port.
This all works completely fine for any shiny application that is using http. But I need https.
So the Dockerfile I use consists of:
FROM rocker/r-ver:4.0.1
# System libs:
RUN apt-get update
RUN apt-get install -y libcurl4-openssl-dev
RUN apt-get install -y libssl-dev
RUN apt-get install -y zlib1g-dev
RUN apt-get install -y libxml2-dev
# R packages installed
RUN R -e "install.packages('remotes')"
RUN R -e "remotes::install_version('searchConsoleR')"
RUN R -e "remotes::install_version('googleAuthR')"
# [...] more R libraries are installed
# Copy application files to a dir
RUN mkdir /root/app
COPY . /root/app
# Expose and set run command from dir above
EXPOSE 3838
CMD ["R", "-e", "shiny::runApp('/root/app', port = 3838, host = '0.0.0.0')"]
Then I execute docker build -t tag_name . and docker run -p 4000:3838 tag_name.
The page is available at http://server.host:4000
However, since I am using Google Login, I need to use https. But when I visit server's https://server.host:4000 I see an error of page not existing.
Can someone please help?
I'm trying to put my shiny app in docker container. My shiny app works totally fine on my local computer. But after dockerize my shiny app, I always have error message on my localhost like The application failed to start. The application exited during initialization..
I have no idea why that happens. I'm new to docker. How can I find the error logs when I run the docker image? I need the log to know what goes wrong.
Here is my dockfile:
# Install R version 3.6
FROM r-base:3.6.0
# Install Ubuntu packages
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev/unstable \
libxt-dev \
libssl-dev
# Download and install ShinyServer (latest version)
RUN wget --no-verbose https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/VERSION -O "version.txt" && \
VERSION=$(cat version.txt) && \
wget --no-verbose "https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/shiny-server-$VERSION-amd64.deb" -O ss-latest.deb && \
gdebi -n ss-latest.deb && \
rm -f version.txt ss-latest.deb
# Install R packages that are required
# TODO: add further package if you need!
RUN R -e "install.packages(c( 'tidyverse', 'ggplot2','shiny','shinydashboard', 'DT', 'plotly', 'RColorBrewer'), repos='http://cran.rstudio.com/')"
# Copy configuration files into the Docker image
COPY shiny-server.conf /etc/shiny-server/shiny-server.conf
COPY /app /srv/shiny-server/
# Make the ShinyApp available at port 80
EXPOSE 80
# Copy further configuration files into the Docker image
COPY shiny-server.sh /usr/bin/shiny-server.sh
CMD ["/usr/bin/shiny-server.sh"]
I built image and ran like below:
docker build -t myshinyapp .
docker run -p 80:80 myshinyapp
Usually the logs for any (live or dead) container can be found by just using:
docker logs full-container-name
or
docker logs CONTAINERID
(replacing the actual ID of your container)
As first said, this usually works as well even for stopped (not still removed) containers, which you can list with:
docker container ls -a
or just
docker ps -a
However, sometimes you won't even have a log, since the container was never created at all (which I think, by experience, fits more to your case)
And it can be happening simply because the docker engine is unable to allocate all of the resources that your service definition is requiring to have available.
The application failed to start. The application exited during initialization
is usually reflect of your docker engine being unable to get the required resources.
And the most common case for that, is just as simple as your host ports:
If you have another service (being dockerized or not) using (for example) that port that you want to use for your service (in your case, port 80) then Docker would just be unable to start your container.
So... in short... the easiest fix for that situation (and your first try whenever you face this kind of issues) is just to bind any other port from your host (say: 8080), to that 80 port that your service will be listening to internally (inside your container):
docker run -p 8080:80 myshinyapp
The same principle applies to unallocatable volumes (e.g.: trying to bind a volume as read-only that doesn't actually exist in the host)
As an aside comment/trick:
Since you're not setting a name for your container, you will need to use the container id instead when looking for its logs.
But instead of typing (or copy-pasting) the full container id (usually something like: 1283c66babea or even larger) you can just type in a few first digits instead, and it will still work as expected:
docker logs 1283c6 or docker logs 1283 or even docker logs 128
(of course... as long as you don't have any other 128***** container)
I am trying to launch a shiny app using ShinyProxy - something I have done many times before. However, this app is not correctly using any of the CSS or JS files that is required to make it run.
When I run the app manually with docker run -p 3838:3838 my_app everything works perfectly fine. However, when pointing ShinyProxy to the my_app image, the resulting app fails to load any CSS or JS files.
Dockerfile
FROM openanalytics/r-base
MAINTAINER Daniel Beachnau "DannyBeachnau#gmail.com"
# Dependencies outside of R
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libpq-dev \
libmariadb-client-lgpl-dev \
# Might be needed for the archivist R-Library
dbus \
systemd \
# needed for odbc
unixodbc-dev
RUN apt-get install apt-transport-https curl -y
RUN curl http://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get install msodbcsql17 -y
# Download R-Packages
# tidyverse
RUN R -e "install.packages('tidyr')"
RUN R -e "install.packages('dplyr')"
RUN R -e "install.packages('readr')"
# Shiny Packages
RUN R -e "install.packages('shiny')"
RUN R -e "install.packages('shinycssloaders')"
RUN R -e "install.packages('shinydashboard')"
RUN R -e "install.packages('shinyWidgets')"
RUN R -e "install.packages('DT')"
RUN R -e "install.packages('shinyjs')"
RUN R -e "install.packages('flexdashboard')"
# Database Packages
RUN R -e "install.packages('odbc')"
RUN R -e "install.packages('RMySQL')"
# Other
RUN R -e "install.packages('devtools')"
RUN R -e "install.packages('lubridate')"
RUN R -e "install.packages('reshape2')"
RUN R -e "install.packages('grid')"
RUN R -e "install.packages('lemon')"
RUN R -e "install.packages('scales')"
RUN R -e "install.packages('ggthemes')"
RUN R -e "install.packages('ggplot2')"
RUN R -e "devtools::install_bitbucket(repo = 'my_repo/my_package', auth_user = 'my_username', password = 'my_password')"
# copy the app to the image
COPY . /root
# run the script to update the app data
WORKDIR /root
RUN Rscript app_data_update.R
WORKDIR /root/app
COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838
CMD ["R", "-e", "shiny::runApp('/root/app', host='0.0.0.0', port=3838)"]
application.yml
shiny:
proxy:
title: ShinyProxy Server
logo-url: /images/logo-image.png
landing-page: /
heartbeat-rate: 10000
heartbeat-timeout: 60000
container-wait-time: 60000
port: 8080
authentication: ldap
# Docker configuration
docker:
cert-path: /home/none
url: http://localhost:2375
port-range-start: 20000
support:
container-log-path: ./container-logs
mail-to-address: DannyBeachnau#gmail.co,
- name: my_apps_name
display-name: Shiny App
docker-image: dbeachnau/my_app
groups: [Shiny Users Management]
logo-url: /images/logo-image.png
container-volumes: ["/path/to/app:/root/app"]
logging:
file:
shinyproxy.log
Here is how app looks in shiny proxy.
problem
Here is hoe my app looks when running manually.
desired
The console in chrome's inspect tool is replete with errors such as
GET https://myshinyserver.com/container_name/font-awesome-5.3.1/css/all.min.css net::ERR_ABORTED 404 (Not Found)
I do have other apps running on ShinyProxy which display properly, but I cannot solve the difference between how those apps are configured to how this app is configured. Let me know if additional details are required for diagnosing the issue. All feedback is appreciated - thank you.
You're probably seeing this with Shiny v1.3.0, and not with earlier versions. If so, it's probably because of a misconfiguration in your NGINX proxy directives. I've written up the details here, but I'll also post the salient details here.
proxy_set_header Connection "upgrade";
This directive causes NGINX to add a Connection: upgrade header to every HTTP request, when it's only supposed to be used for WebSockets.
This line is recommended by NGINX Inc. themselves, however, those recommendations are intended for proxying of traffic that is exclusively WebSockets, whereas Shiny traffic is a combination of normal HTTP requests and WebSockets. Older versions of shiny/httpuv didn't mind this situation, but the new versions are stricter.
A correct configuration looks something like this:
http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
location / {
proxy_pass http://localhost:3838;
proxy_redirect / $scheme://$http_host/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 20d;
proxy_buffering off;
}
}
}
See the articles linked in the RStudio Community post for other examples.
You will have to install the requested font at the top of your Dockerfile. You can add it to your list "Dependencies outside of R":
sudo apt-get install fonts-font-awesome
I have solved my problem, although, this still may not count as a sufficient answer or explanation, because I cannot account for why this solution makes a difference. I decided to rewrite the Dockerfile using a different base image which now works. Nothing else in my code changed - just the Dockerfile. The working docker file is as follows:
FROM rocker/shiny-verse
# based on debian 9
MAINTAINER Daniel Beachnau "DannyBeachnau#gmail.com"
# Dependencies outside of R
RUN apt-get update && apt-get install -y \
gnupg2 \
apt-utils \
sudo \
gdebi-core \
libxt-dev \
xtail \
wget
# Install ODBC driver from microsoft
RUN apt-get install apt-transport-https curl -y
RUN curl http://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/9/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get install msodbcsql17 -y
# Download R-Packages
# Shiny Packages
RUN R -e "install.packages('shinycssloaders')"
RUN R -e "install.packages('shinydashboard')"
RUN R -e "install.packages('shinyWidgets')"
RUN R -e "install.packages('DT')"
RUN R -e "install.packages('shinyjs')"
RUN R -e "install.packages('flexdashboard')"
# Database Packages
RUN R -e "install.packages('odbc')"
RUN R -e "install.packages('RMySQL')"
# Other
RUN R -e "install.packages('lubridate')"
RUN R -e "install.packages('reshape2')"
RUN R -e "install.packages('scales')"
RUN R -e "install.packages('ggthemes')"
RUN R -e "install.packages('ggplot2')"
RUN R -e "devtools::install_bitbucket(repo = 'my_repo', auth_user = 'my_username', password = 'my_password')"
# copy the app to the image
COPY . /root
# run the script to update the app data
WORKDIR /root
RUN Rscript app_data_update.R
WORKDIR /root/app
COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838
CMD ["R", "-e", "shiny::runApp('/root/app', host='0.0.0.0', port=3838)"]
If anyone has insight to why this behavior is observed I would love to hear it because I am baffled to say the least.
why this solution makes a difference
It seems to be an issue with the Shiny version, changing the base image has very probably fixed that.
See Shiny apps not rendering after updated to v1.3.0
So I've set up an Azure Data Science Virtual Machine on Linux (Ubuntu) and I've executed the following on the terminal to enable Remote R workspace, RStudio Server, R Server Operationalization and hadoop:
sudo apt update
sudo apt -y upgrade
# Hadoop is installed but doesn't seem to appear on the PATH or have its environment variable set by default
sudo echo "" >> ~/.bashrc
sudo echo "export PATH="'$'"PATH:/opt/hadoop/hadoop-2.7.4/bin" >> ~/.bashrc
sudo echo "export HADOOP_HOME=/opt/hadoop/hadoop-2.7.4" >> ~/.bashrc
#
source ~/.bashrc
#Setting up a password as none exists to begin with because of private key selection in the installation
#RStudio Server requires a password though
"MyPassword\nMyPassword\n" | sudo passwd sshuser
#Unfortunately hadoop fails on Data Science Virtual Machine
#error: mkdir: Call From IM-DSonUbuntu/192.168.5.4 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
# hadoop fs -mkdir /user/RevoShare/rserve2
# hadoop fs -chmod uog+rwx /user/RevoShare/rserve2
sudo mkdir -p /var/RevoShare/rserve2
sudo chmod uog+rwx /var/RevoShare/rserve2
# hadoop fs -mkdir /user/RevoShare/sshuser
# hadoop fs -chmod uog+rwx /user/RevoShare/sshuser
sudo mkdir -p /var/RevoShare/sshuser
sudo chmod uog+rwx /var/RevoShare/sshuser
#Setting up R Server Operationalisation
cd /opt/microsoft/mlserver/9.2.1/o16n
sudo dotnet Microsoft.MLServer.Utils.AdminUtil/Microsoft.MLServer.Utils.AdminUtil.dll -silentoneboxinstall MyPassword
#They say this Data Science Virtual Machine already has RStudio Server, but even though the port 8787 is open, it's nowhere to be found! So installing it now, and after the installation it's accessible by refreshing the page that failed before.
#Perhaps it's not installed then? Or a service is not running like it shoudl?
#https://www.rstudio.com/products/rstudio/download-server/
wget https://download2.rstudio.org/rstudio-server-1.1.414-amd64.deb
yes | sudo gdebi rstudio-server-1.1.414-amd64.deb
#They are small, leave them for debug reasons - lets have evidence the script run thus far.
#sudo rm rstudio-server-1.1.414-amd64.deb
# Remote R workspace Service needs dotnet sdk
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt update
sudo apt -y install dotnet-sdk-2.0.0
sudo apt install libxml2-dev
#Downloading and installing the Remote R service
wget -O rtvs-daemon.tar.gz https://aka.ms/r-remote-services-linux-binary-current
tar -xvzf rtvs-daemon.tar.gz
sudo ./rtvs-install -s
sudo systemctl enable rtvsd
sudo systemctl start rtvsd
#sudo rm rtvs-daemon.tar.gz
#sudo rm rtvs-install
#Fixing Remote R: For some reason, even though 'sudo systemctl enable rtvsd' runs, after every reboot the service won't become automatically active. So let's fix that.
wget https://sa0im0general.blob.core.windows.net/general-blob-container/StartRemoteRAfterReboot.sh
sudo mv StartRemoteRAfterReboot.sh /var/RevoShare/StartRemoteRAfterReboot.sh
sudo /sbin/shutdown -r 5
sudo chown root /etc/rc.local
sudo chmod 755 /etc/rc.local
sudo systemctl enable rc-local.service
sudo -s
sudo find /etc/ -name "rc.local" -exec sed -i 's/exit 0//g' {} \;
sudo echo "" >> /etc/rc.local
sudo echo "sh /var/RevoShare/StartRemoteRAfterReboot.sh" >> /etc/rc.local
sudo echo "exit 0" >> /etc/rc.local
exit
I've also tried, one by one, these, to see if it makes any difference to the RStudio Server (it didn't, but even if it did, I want a global solution to work on Remote R Workspace Service and R Server Operationalisation as well, not only RStudio Server):
#Configuring RStudio Server to see the R Server R
sudo echo "rsession-which-r=/opt/microsoft/mlserver/9.2.1/bin/R/R" >> /etc/rstudio/rserver.conf
export RSTUDIO_WHICH_R=/opt/microsoft/mlserver/9.2.1/bin/R/R
sudo echo "RSTUDIO_WHICH_R=/opt/microsoft/mlserver/9.2.1/bin/R/R" >> ~/.profile
source ~/.profile
sudo echo "RSTUDIO_WHICH_R=/opt/microsoft/mlserver/9.2.1/bin/R/R" >> ~/.bashrc
source ~/.bashrc
sudo echo "PATH=$PATH:/opt/microsoft/mlserver/9.2.1/bin/R" >> ~/.bashrc
export PATH=$PATH:/opt/microsoft/mlserver/9.2.1/bin/R
source ~/.bashrc
The problem is that even though "which R" points to R Server's R, i.e. typing "sudo R" will show the message "Loading Microsoft R Server packages, version 9.2.1." and will load packages like RevoScaleR, everything else fails to do so.
Accessing the RStudio Server with http://THE-IP-GOES-HERE.westeurope.cloudapp.azure.com:8787 and logging in with the initial user ("sshuser") (or with any other user for that matter) will NOT load R Server and RevoScaleR rx functions are unavailable
Using my local Visual Studio 2017 to access the remote workspace via "Add connection" on "Workspaces" tab loads MRO and says:
Installed R versions:
[0] Microsoft R Open '3.4.1.1347' (Default)
And finally, when I use R Server's Operationalisation and log in with "mrsdeploy" package's "remoteLogin()" R Server packages like RevoScaleR are not loaded again, so things like "rxSummary(~., data=iris)" fail with error 'could not find function "rxSummary"'
The exact same thing happened when I deployed from azure a "Machine Learning Server 9.2.1 on Linux (Ubuntu)".
I don't want to just use the regular open source R, I want to be able to use the R Server - that's why I deployed this VM. How can I make it so that everything loads R Server's R, not Microsoft R Open? (Like I'm able to do from terminal using "R")
As a result of my having tried all of this and the fact that R Server is loaded in the console, my mind now goes to permissions. Could it be that by default the Data Science VM doesn't have the correct permissions to allow these?
I'm at a loss
RStudio Server is installed on the Ubuntu DSVM, but the service is disabled by default as it does not support SSL. You can enable it with systemctl enable rstudio-server, then start it with systemctl start rstudio-server.
RStudio Server uses the same R as Microsoft R Server, but the .libPaths are different, which is why you cannot load the MRS packages. You will need to manually set the .libPaths so they match.
I had installed nginx and lua in my docker image, but I didn't install nginx on my physical CentOS system. I "docker run my image" and start nginx on my physical CentOS system. So the nginx master and worker process are working. I run an example of nginx-systemtap-toolkit. I run:
sudo ./ngx-active-reqs -p 24945
24945 is worker process id. errors as following:
semantic error: while resolving probe point: identifier 'process' at <input>:6:7
source: probe process("/data1/nginx/sbin/nginx").function("ngx_process_events_and_timers"),
^
semantic error: no match
Pass 2: analysis failed. [man error::pass2]
Number of similar error messages suppressed: 1
In fact, "/data1/nginx/sbin/nginx" is the nginx path in docker image, nginx is not installed on my physical system. So whether I must install nginx on my physical system , or is there other method to use nginx's process function? I don't know how to solve the problem.
To install debug symbol which package is not included in the repositories in the form of -dbg package, you need to add a new repository as detailed in the Debugging Guide:
Add the repositories:
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
Add the GPG key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
Update your package list:
sudo apt-get update
Install the debugging symbols package, in this case coreutils-dbgsym:
sudo apt-get install coreutils-dbgsym