RStudio and Shiny in one dockerfile - r

I am looking into docker to distribute a shiny application that also requires RStudio. The primary goal is easy installation at hospitals under Windows. Everything that requires character input into black boxes will certainly fail during installation by non-IT people.
My previous attempts used vagrant, but installing vagrant alone proved to be a hurdle.
The rocker repository, has an RStudio and a Shiny , and for my own installation both work together. However, I would like to create a combined application for easier installation.
What is the recommended workflow? Start with RStudio, and manually add Shiny?
Or merge the dockerfiles code from both Rockers, starting with r-base? Or use compose tool?

The point of Docker, in general, is isolation of services so that they can be updated/changed without effecting others. My recommendation would be to use docker-compose, instead. Below is an example docker-compose yaml file that serves both rstudio and shiny on the same server at different subdomains using the incredibly useful docker-gen by Jason Wilder. All R docker images used below are courtesy of Rocker or more directly Rocker Docker Hub. These are very very reliable because, well, Dirk Eddelbeutel and Carl Boettiger made them. In this example I've also included some options for RStudio such as setting a user/pass and whether or not the user has root access. There are more instructions on using the Rocker RStudio image available on this wiki page:
Change the following:
your_user to your username on the server
SOME_USER to your desired RStudio username
SOME_PASS to your desired Rstudio password
*.DOMAIN.tld to your domain, don't forget to add A records for your subdomains.
nginx1:
image: nginx
container_name: nginx
ports:
- "80:80"
- "443:443"
volumes:
- /etc/nginx/conf.d
- /etc/nginx/vhost.d
- /usr/share/nginx/html
- /home/your_user/services/volumes/proxy/certs:/etc/nginx/certs:ro
nginx-gen:
links:
- "nginx1"
image: jwilder/docker-gen
container_name: nginx-gen
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- /home/your_user/services/volumes/proxy/templates/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
volumes_from:
- nginx1
entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
rstudio:
links:
- "nginx1"
image: rocker/hadleyverse
container_name: rstudio
ports:
- "8787:8787"
environment:
- VIRTUAL_PORT=8787
- ROOT=TRUE
- VIRTUAL_HOST=rstudio.DOMAIN.tld
- USER=SOME_USER
- PASSWORD=SOME_PASS
shiny:
links:
- "nginx1"
image: rocker/shiny
container_name: shiny
environment:
- VIRTUAL_HOST=shiny.DOMAIN.tld
volumes:
- /home/your_user/services/volumes/shiny/apps:/srv/shiny-server/
- /home/your_user/services/volumes/shiny/logs:/var/log/
- /home/your_user/services/volumes/shiny/packages:/home/shiny/
It's trivial to add more services like a blog, for example, just follow the pattern or search the internet for a docker-compose version of your service and add it.

Interesting question, but I'm not sure I understand the advantage of having the shiny-server and the rstudio-server instances served from the same container.
Is the purpose so that the two containers share the same R libraries (e.g. so a package doesn't need to be installed separately on each) or merely to have one docker container instead of two? Just having to run two docker commands instead of one doesn't seem that onerous, but maybe I'm underestimating.
Sharing the underlying libraries seems like a valid objective though, and I don't think there's an ideal solution available yet.
I feel the most docker-esque solution would be to do this via container orchestration/compose tool as you mention. This is the usual way to combine separate services (e.g. web server and database) without building one on top of the other.
Unfortunately, the tooling for orchestration based on mapping volumes is not nearly as well developed as it is for mapping ports.
Imagine running the rstudio as a volume container:
docker run --name rstudio -v /usr/local/lib/R/site.library rocker/rstudio true
(If you wanted RStudio access at the same time, one could instead run this as:)
docker run --name rstudio -dP -v /usr/local/lib/R/site.library rocker/rstudio
You can then use the the site.library from the rstudio container in place of that on the shiny container with a command like:
docker run --volumes-from rstudio -dP rocker/shiny
Unfortunately, this clobbers the site.library of the shiny container. To work around this, you'd want to mount the library of the rstudio container in a different place, but there's no easy syntax for this like we already have with port links. It can be done though, see:
How to map volume paths using Docker's --volumes-from?
There's an open thread on this issue in the rocker repo too.

I have developed a working single docker for
R
RStudio (server)
Shiny Server (free edition)
I built it exactly for the same reasons mentioned by #Dieter Menne. It may be not ideal for ops, but it great for dev (especially if the team members all use different envs. like mac, windows etc.).
It is on Centos 6 as this is the env. I use at work.
This is the dockerfile:
FROM centos:centos6.7
MAINTAINER enzo smartinsightsfromdata
RUN yum -y install epel-release
RUN yum update -y && yum clean all
# RUN yum reinstall -y glibc-common
RUN yum install -y locales java-1.7.0-openjdk-devel tar
# Misc packages
RUN yum groupinstall -y "Development Tools"
# R devtools pre-requisites:
RUN yum install -y wget git xml2 libxml2-devel curl curl-devel openssl-devel
WORKDIR /home/root
RUN yum install -y R
RUN wget http://cran.r-project.org/src/contrib/rJava_0.9-7.tar.gz
RUN R CMD INSTALL rJava_0.9-7.tar.gz
RUN R CMD javareconf \
&& rm -rf rJava_0.9-7.tar.gz
#-----------------------
# Add RStudio binaries to PATH
# export PATH="/usr/lib/rstudio-server/bin/:$PATH"
ENV PATH /usr/lib/rstudio-server/bin/:$PATH
ENV LANG en_US.UTF-8
RUN yum install -y openssl098e supervisor passwd pandoc
# RUN wget http://download2.rstudio.org/rstudio-server-rhel-0.99.484-x86_64.rpm
# Go for the bleading edge:
RUN wget https://s3.amazonaws.com/rstudio-dailybuilds/rstudio-server-rhel-0.99.697-x86_64.rpm
RUN yum -y install --nogpgcheck rstudio-server-rhel-0.99.697-x86_64.rpm \
&& rm -rf rstudio-server-rhel-0.99.484-x86_64.rpm
RUN groupadd rstudio \
&& useradd -g rstudio rstudio \
&& echo rstudio | passwd rstudio --stdin
RUN R -e "install.packages(c('shiny', 'rmarkdown'), repos='http://cran.r-project.org', INSTALL_opts='--no-html')"
RUN wget https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.4.0.756-rh5-x86_64.rpm
RUN yum -y install --nogpgcheck shiny-server-1.4.0.756-rh5-x86_64.rpm \
&& rm -rf shiny-server-1.4.0.756-rh5-x86_64.rpm
RUN mkdir -p /var/log/shiny-server \
&& chown shiny:shiny /var/log/shiny-server \
&& chown shiny:shiny -R /srv/shiny-server \
&& chmod 777 -R /srv/shiny-server \
&& chown shiny:shiny -R /opt/shiny-server/samples/sample-apps \
&& chmod 777 -R /opt/shiny-server/samples/sample-apps
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
RUN mkdir -p /var/log/supervisor \
&& chmod 777 -R /var/log/supervisor
EXPOSE 8787 3838
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
This is how the supervisord.conf file looks like:
[supervisord]
nodaemon=true
logfile=/var/log/supervisor/supervisord.log
pidfile = /tmp/supervisord.pid
[program:rserver]
user=root
command=/usr/lib/rstudio-server/bin/rserver
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
startsecs=0
autorestart=false
[program:shinyserver]
user=root
command=/usr/bin/shiny-server
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
autorestart=false
It is available at my github page: smartinsightsfromdata
I have also developed a working docker for shiny server pro on centos (using shiny server pro temporary edition, valid 45 days only).

Somewhat unfortunately, there is no definite answer, it all depends on how much reusability you would be looking for and whether an upstream base image is well maintained. The is also images size tradeoff, more layers there are, bigger the resulting image gets.

Related

Shiny app does not appear when deployed using Shinyproxy

I'm trying to learn how to deploy a shiny app using Shinyproxy, and I'm using the templated "euler app" (from this repo), but the application does not appear when I navigate to http://localhost:4445. Here's the most similar question I could find, but unfortunately not helpful to my issue: link.
Background
All installations seem fine, and I successfully installed Docker and Java.
The Dockerfile and Docker image work locally, no issues there. The command docker run --rm -p 3838:3838 shiny-euler-app works.
Here is my Dockerfile (copied from the repo):
FROM openanalytics/r-base
MAINTAINER Tobias Verbeke "tobias.verbeke#openanalytics.eu"
# system libraries of general use
RUN apt-get update && apt-get install -y \
sudo \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
libssl-dev \
libssh2-1-dev \
libssl1.1
# system library dependency for the euler app
RUN apt-get update && apt-get install -y \
libmpfr-dev
# basic shiny functionality
RUN R -e "install.packages(c('shiny', 'rmarkdown'), repos='https://cloud.r-project.org/')"
# install dependencies of the euler app
RUN R -e "install.packages('Rmpfr', repos='https://cloud.r-project.org/')"
# copy the app to the image
RUN mkdir /root/euler
COPY euler /root/euler
COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838
CMD ["R", "-e", "shiny::runApp('/root/euler')"]
As well, Shinyproxy works fine with the default openanalytics/shinyproxy-demo Docker image, as you can see:
Problem
The issue I have is when I try and supply a different Shiny app and its accompanying application.yml. Here is the application.yml file I'm using (I've tried to make it as basic as possible, with no authentication, etc):
proxy:
title: Standalone Docker Engine
port: 4445
authentication: none
docker:
url: http://localhost:2375
specs:
- id: euler
display-name: Euler's number
container-cmd: ["R", "-e", "shiny::runApp('/root/euler')"]
container-image: shiny-euler-app
Unfortunately, when I run java -jar shinyproxy-2.4.2.jar (in the directory which contains the shinyproxy-2.4.2.jar file and the application.yml file) I get this blank webpage:
For some reason, I am able to access the Shinyproxy webpage, but the Dockerized Shiny app does not appear.
Would really appreciate any helpful suggestions on where/how I could try and solve this issue. Thanks!

How to find error logs when my dockerized shiny app does not work

I'm trying to put my shiny app in docker container. My shiny app works totally fine on my local computer. But after dockerize my shiny app, I always have error message on my localhost like The application failed to start. The application exited during initialization..
I have no idea why that happens. I'm new to docker. How can I find the error logs when I run the docker image? I need the log to know what goes wrong.
Here is my dockfile:
# Install R version 3.6
FROM r-base:3.6.0
# Install Ubuntu packages
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev/unstable \
libxt-dev \
libssl-dev
# Download and install ShinyServer (latest version)
RUN wget --no-verbose https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/VERSION -O "version.txt" && \
VERSION=$(cat version.txt) && \
wget --no-verbose "https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/shiny-server-$VERSION-amd64.deb" -O ss-latest.deb && \
gdebi -n ss-latest.deb && \
rm -f version.txt ss-latest.deb
# Install R packages that are required
# TODO: add further package if you need!
RUN R -e "install.packages(c( 'tidyverse', 'ggplot2','shiny','shinydashboard', 'DT', 'plotly', 'RColorBrewer'), repos='http://cran.rstudio.com/')"
# Copy configuration files into the Docker image
COPY shiny-server.conf /etc/shiny-server/shiny-server.conf
COPY /app /srv/shiny-server/
# Make the ShinyApp available at port 80
EXPOSE 80
# Copy further configuration files into the Docker image
COPY shiny-server.sh /usr/bin/shiny-server.sh
CMD ["/usr/bin/shiny-server.sh"]
I built image and ran like below:
docker build -t myshinyapp .
docker run -p 80:80 myshinyapp
Usually the logs for any (live or dead) container can be found by just using:
docker logs full-container-name
or
docker logs CONTAINERID
(replacing the actual ID of your container)
As first said, this usually works as well even for stopped (not still removed) containers, which you can list with:
docker container ls -a
or just
docker ps -a
However, sometimes you won't even have a log, since the container was never created at all (which I think, by experience, fits more to your case)
And it can be happening simply because the docker engine is unable to allocate all of the resources that your service definition is requiring to have available.
The application failed to start. The application exited during initialization
is usually reflect of your docker engine being unable to get the required resources.
And the most common case for that, is just as simple as your host ports:
If you have another service (being dockerized or not) using (for example) that port that you want to use for your service (in your case, port 80) then Docker would just be unable to start your container.
So... in short... the easiest fix for that situation (and your first try whenever you face this kind of issues) is just to bind any other port from your host (say: 8080), to that 80 port that your service will be listening to internally (inside your container):
docker run -p 8080:80 myshinyapp
The same principle applies to unallocatable volumes (e.g.: trying to bind a volume as read-only that doesn't actually exist in the host)
As an aside comment/trick:
Since you're not setting a name for your container, you will need to use the container id instead when looking for its logs.
But instead of typing (or copy-pasting) the full container id (usually something like: 1283c66babea or even larger) you can just type in a few first digits instead, and it will still work as expected:
docker logs 1283c6 or docker logs 1283 or even docker logs 128
(of course... as long as you don't have any other 128***** container)

Docker Theme Jupyter

I recently installed the data science notebook for Jupyter but i cant seem to install any themes on it.
Using the local version i have installed a dark theme and i am used to it.
Following this guide and the install section I tried making /custom/ folder and added a cascading style sheet into the mounted volume. But it doesn't seem to work.
Is there anyway i can install a custom theme on the docker image?
My workaround is: add the custom.css file locally under '~/.jupyter/custom/'. When the docker container runs, it will automatically adopt the theme.
You can actually install jupyter-themes and opt for a theme on Dockerfile itself. Here is a example for reference
Install jupyter-themes using pip
RUN pip3 install jupyterthemes
Opt for a theme
CMD ["bash", "-c", "jt -t solarizedd -T -N && jupyter notebook --port=8888 --no-browser --ip=0.0.0.0 --allow-root --notebook-dir=/home/user/workdir"]
Here is a complete docker file sample for reference
FROM ubuntu:20.04
# Install Python and other dependencies
RUN apt-get update && apt-get install -y \
python3 \
python3-pip \
wget
# Install Jupyter
RUN pip3 install jupyter
RUN pip3 install jupyterthemes
# Create a user with a home directory
RUN useradd --create-home --home-dir /home/user user
USER user
# Mount a volume for the working directory
VOLUME /home/user/workdir
# Set the default command to launch Jupyter
CMD ["bash", "-c", "jt -t solarizedd -T -N && jupyter notebook --port=8888 --no-browser --ip=0.0.0.0 --allow-root --notebook-dir=/home/user/workdir"]
# CMD ["jupyter", "notebook", "--no-browser", "--ip=0.0.0.0", "--allow-root", "--notebook-dir=/home/user/workdir"]
You can also include the said command on a docker-compose file, if you are using a docker image
version: "3"
services:
notebook:
image: jupyter/datascience-notebook
ports:
- "8888:8888"
environment:
JUPYTER_ENABLE_LAB: "yes"
volumes:
- .:/home/user/workdir
command: bash -c "pip install jupyterthemes && jt -t solarizedd -T -N"
Here i have chosen the solarizedd theme, just tweak and include the theme that best suits for you.
Hope this helps

How to install JupyterHub with Docker on a local machine and in a sub domain

I will run JupyterHub in a sub domain. Here is the Dockerfile, jupyterhub_config.py, .gitlab-ci.yml.
My first question is how to configure the jupyter_config.py. How can I load the jupyterhub_config.py on the build in the container?
How do I start Jupyterhub in the .gitlab-ci.yml for tests and how do I copy the application in the sub domain? I wrote a README.md. I need a little help for the JupyterHub. If all works fine, I will write a complete HOWTO Install JupyterHub on a local machine and in a sub domain by a provider.
FROM continuumio/miniconda3
# Updating packages
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends \
git \
nano \
unzip \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install conda and Jupyter
RUN conda update -y conda
RUN conda install -c conda-forge jupyter_nbextensions_configurator \
jupyterhub \
jupyterlab \
matplotlib \
pandas \
scipy
# Setup application
EXPOSE 8000
CMD ["jupyterhub", "--ip='*'", "--port=8000", "--no-browser", "--allow-root"]
The .gitlab-ci.yml
image: docker:latest
variables:
CONTAINER_IMAGE: registry.gitlab.com/joklein
DOCKER_IMAGE: jupyterhub
TAG: 0.1.0
services:
- docker:dind
stages:
- build
- test
- release
- deploy
before_script:
- echo "$GITLAB_PASSWORD" | docker login registry.gitlab.com --username $GITLAB_USER --password-stdin
build:
stage: build
script:
- docker build -t $CONTAINER_IMAGE/$DOCKER_IMAGE .
- docker push $CONTAINER_IMAGE/$DOCKER_IMAGE
test:
stage: test
script:
- docker pull $CONTAINER_IMAGE/$DOCKER_IMAGE
# - docker run $CONTAINER_IMAGE/$DOCKER_IMAGE -dt -p 8000:8000 --name $DOCKER_IMAGE
release:
stage: release
script:
- docker pull $CONTAINER_IMAGE/$DOCKER_IMAGE
- docker tag $CONTAINER_IMAGE/$DOCKER_IMAGE:latest $CONTAINER_IMAGE/$DOCKER_IMAGE:$TAG
- docker push $CONTAINER_IMAGE/$DOCKER_IMAGE:$TAG
only:
- master
deploy:
stage: deploy
image: alpine:latest
before_script:
- apk update && apk add git openssh-client rsync
script:
- mkdir .public
- cp -r * .public
- mv .public public
- mkdir "${HOME}/.ssh"
- echo "${SSH_HOST_KEY}" > "${HOME}/.ssh/known_hosts"
- echo "${SSH_PRIVATE_KEY}" > "${HOME}/.ssh/id_rsa"
- chmod 700 "${HOME}/.ssh/id_rsa"
- rsync -hrvz --delete --exclude=_ public/ user#example.com:www/jupyter/
only:
- master
The jupyterhub_config.py
c = get_config()
# Letsencrypt (https://letsencrypt.org/) to obtain a free, trusted SSL
# certificate.
c.JupyterHub.ssl_key = '/etc/letsencrypt/live/example.com/privkey.pem'
c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/example.com/fullchain.pem'
c.JupyterHub.port = 443
#
# Change from JupyterHub to JupyterLab
c.Spawner.default_url = '/lab'
c.Spawner.debug = True
#
# # Specify users and admin
c.Authenticator.whitelist = {"systemuser"}
c.Authenticator.admin_users = {"systemuser"}
Docker base image of JupyterHub and JupyterLab
JupyterHub is a multi-user server for Jupyter notebooks. JupyterLab is the
next-generation web-based user interface for the Jupyter Project. This
JupyterHub is a Docker base image for JupyterHub and JupyterLab
that works as a stand-alone application and in a (sub) domain.
Images derived from this image can either run as a stand-alone server, or
function as a volume image for your server. You can also use them in a CI/CD
system such as GitLab CI to build your content prior to bundling it into a
standalone server container.
Building your JupyterHub image
Based on this structure, you can easily build an image for your needs. There are two options for using the image you generated:
as a stand-alone image
as a volume image for your webserver
The simplest way to build your own image is to use a Dockerfile. This is only an example. If you need more software packages you can install them with this
Dockerfile and conda.
Build the container
docker build -t juypterhub .
Your JupyterHub with JupyterLab is automatically generated during this build.
Run the container
docker run -p 8000:8000 -d --name jupyterhub jupyterhub jupyterhub
-p is used to map your local port 8000 to the container port 8000
-d is used to run the container in background. JupyterHub will just write
logs so no need to output them in your terminal unless you want to troubleshoot a server error.
-- name jupyterhub names your container jupyterhub
jupyterhub the image
jupyterhub is the last command used to start the jupyterhub server
and your JupyterHub with Jupyterlab is now available of http://localhost:8000.
Start / Stop JupyterHub
docker start / stop juyterhub
Configure JupyterHub
Let's encrypt certificates for JupyterHub
To enable HTTPS on your website, you need to get a certificate (a type of file) from a Certificate Authority (CA). Let’s Encrypt is a CA. In order to get a certificate for your website’s domain from Let’s Encrypt, you have to
demonstrate control over the domain. With Let’s Encrypt, you do this using
software that uses the ACME protocol, which typically runs on your web host.
Change to zerossl.com and generate a certificate for your domain. As the
result you get four files, domain-key.txt, domain-crt.txt, domain-csr.txt, account-key.txt. This files uses base 64, which is readable in
ASCII, not binary format. The certificates are already in PEM format. Just
change the extension to *.pem.
For JupyterHub only the files domain-key.txt and domain-crt are needed.
cp domain-crt.txt fullchain.pem
cp domain-key.txt privkey.pem
Add a System user in the container
By default JupyterHub searches for users on the server. In order to be able to
log in to our new JupyterHub server we need to connect to the JupyterHub docker
container and create a new system user with a password.
docker exec -it jupyterhub bash
useradd --create-home systemuser
passwd systemuser
exit
The command docker exec -it jupyterhub bash will spawn a root shell in your
docker container. You can use the root shell to create system users in the
container. These accounts will be used for authentication in JupyterHub's
default configuration.
The first command useradd creates a new user named systemuser. The second will
ask you for a password.
The all process might be simpler with GitLab 12.0 (June 2019), and its
Git integration for JupyterHub
Deploying JupyterHub via GitLab’s Kubernetes integration provides an easy way to get started with Jupyter notebooks, which can be used to create and share documents that contain live code, visualizations, and even runbooks.
Starting with GitLab 12.0, JupyterLab’s Git extension is automatically provisioned and configured when installing JupyterHub onto your Kubernetes cluster.
This integration enables full version control of your notebooks as well as issuance of Git commands within Jupyter. Git commands can be issued via the Git tab on the left panel or via Jupyter’s command line prompt.
See documentation and gitlab-ce issue 47138.
jupyterhub --generate-config
This is what on the documentation
It created a config.py file in /srv/jupyterhub

Updating a Symfony app with Docker-compose without losing data

I have a multi-container Symfony application that uses docker-compose to handle the relationships between the containers. To simplify a little, i have 4 main services :
code:
image: mycode
web:
image: mynginx
volumes-from:
- code
ports:
- "80:80"
links:
- php-fpm
php-fpm:
image: myphpfpm
volumes-from:
- code
links:
- mongo
mongo:
image: mongo
The "mycode" image contains the code of my application and is built from the following Dockerfile :
FROM composer/composer
RUN apt-get update && apt-get install -y \
libfreetype6-dev \
libmcrypt-dev \
libxml2-dev \
libicu-dev \
libcurl4-openssl-dev \
libssl-dev \
pkg-config
RUN docker-php-ext-install iconv mcrypt mbstring bcmath json ctype iconv posix intl
RUN pecl install mongo \
&& echo extension=mongo.so >> /usr/local/etc/php/conf.d/mongo.ini
COPY . /code
WORKDIR /code
RUN rm -rf /code/app/cache/* \
&& rm -rf /code/app/logs/* \
&& chown -R root /code/app/cache \
&& chown -R root /code/app/logs \
&& chmod -R 777 /code/app/cache \
&& chmod -R 777 /code/app/logs \
&& composer install \
&& rm -f /code/web/app_dev.php \
&& rm -f /code/web/config.php
VOLUME ["/code", "/code/app/logs", "/code/app/cache"]
At first, deploying this application was easy. I just had to do a simple docker-compose up -d and it created all the containers and ran them without any issue. But then i had to deploy a new version.
This configuration uses volumes to store data :
the source code is mounted on the /code volume, and shared between 3
containers (code, web, php-fpm). It has to be replaced by a new version when deploying.
the MongoDb data is on another
volume, mounted only by the mongo container. I have to keep this data between deployments.
When i deploy an update to my code, i publish the new version of the mycode image and re-create the container. But since the /code volume is still used by the web and php-fpm containers, the old volume can't be replaced by the new one. I have to stop all the running services to delete the old volume, and if i use the docker-compose rm -v command, it will delete the mongodb data too !
Can't i replace only one volume with a new version, without any downtime ?
So i'm kind of stuck here. I'm thinking of having a permanent volume to store the code and update it through SSH with Capistrano, old style. This will allow me to run doctrine migrations scripts after deployment too. But i have other issues with it as Capistrano uses symlinks to handle versions so i can't just mount the /current folder to /code.
Do you have a solution to handle the deployment of a Docker application without losing data and without downtime ?
Should i use manual scripts instead of docker-compose ?
the source code is mounted on the /code volume
This is the problem, it is not what you want.
Code never goes into a volume, it should change when the image changes. Volumes are for things that you want to preserve between changes to the image (data, logs, state, etc).
Code is the immutable thing that you want to replace when you change a container. So remove the /code volume from the Dockerfile entirely, and instead do an ADD . /code in the mynginx and myphpfpm Dockerfiles.
With that change, you can deploy with just up -d. It will recreate any container that have changed, and your volumes will be copied over. You don't need an rm anymore.
If you have your Dockerfile for myphpfpm and mynginx in a different directory, you can build using docker build -f path/to/dockerfile .
Using a host volume (as suggested in another answer) is another option, however that's not usually what you want outside of development. With a host volume you would still remove the /code VOLUME from the dockerfile.
Do not copy the code via the Dockerfile, just attach volumes to the 'code' container.
Few edits:
code:
image: mycode
volumes:
- .:/code
- /code
web:
image: mynginx
volumes-from:
- code
ports:
- "80:80"
links:
- php-fpm
php-fpm:
image: myphpfpm
volumes-from:
- code
links:
- mongo
mongo:
image: mongo
Same thing applies to mongo mount it to an external volume so it persists when the container shuts down. Actually there is also another method, they mention it in their dockerhub page https://hub.docker.com/_/mongo/
Where to Store Data
Important note: There are several ways to store data used by
applications that run in Docker containers. We encourage users of the
mongo images to familiarize themselves with the options available,
including:
Let Docker manage the storage of your database data by writing the
database files to disk on the host system using its own internal
volume management. This is the default and is easy and fairly
transparent to the user. The downside is that the files may be hard to
locate for tools and applications that run directly on the host
system, i.e. outside containers.
Create a data directory on the host system (outside the container) and
mount this to a directory visible from inside the container. This
places the database files in a known location on the host system, and
makes it easy for tools and applications on the host system to access
the files. The downside is that the user needs to make sure that the
directory exists, and that e.g. directory permissions and other
security mechanisms on the host system are set up correctly.

Resources