Updating a Symfony app with Docker-compose without losing data - symfony

I have a multi-container Symfony application that uses docker-compose to handle the relationships between the containers. To simplify a little, i have 4 main services :
code:
image: mycode
web:
image: mynginx
volumes-from:
- code
ports:
- "80:80"
links:
- php-fpm
php-fpm:
image: myphpfpm
volumes-from:
- code
links:
- mongo
mongo:
image: mongo
The "mycode" image contains the code of my application and is built from the following Dockerfile :
FROM composer/composer
RUN apt-get update && apt-get install -y \
libfreetype6-dev \
libmcrypt-dev \
libxml2-dev \
libicu-dev \
libcurl4-openssl-dev \
libssl-dev \
pkg-config
RUN docker-php-ext-install iconv mcrypt mbstring bcmath json ctype iconv posix intl
RUN pecl install mongo \
&& echo extension=mongo.so >> /usr/local/etc/php/conf.d/mongo.ini
COPY . /code
WORKDIR /code
RUN rm -rf /code/app/cache/* \
&& rm -rf /code/app/logs/* \
&& chown -R root /code/app/cache \
&& chown -R root /code/app/logs \
&& chmod -R 777 /code/app/cache \
&& chmod -R 777 /code/app/logs \
&& composer install \
&& rm -f /code/web/app_dev.php \
&& rm -f /code/web/config.php
VOLUME ["/code", "/code/app/logs", "/code/app/cache"]
At first, deploying this application was easy. I just had to do a simple docker-compose up -d and it created all the containers and ran them without any issue. But then i had to deploy a new version.
This configuration uses volumes to store data :
the source code is mounted on the /code volume, and shared between 3
containers (code, web, php-fpm). It has to be replaced by a new version when deploying.
the MongoDb data is on another
volume, mounted only by the mongo container. I have to keep this data between deployments.
When i deploy an update to my code, i publish the new version of the mycode image and re-create the container. But since the /code volume is still used by the web and php-fpm containers, the old volume can't be replaced by the new one. I have to stop all the running services to delete the old volume, and if i use the docker-compose rm -v command, it will delete the mongodb data too !
Can't i replace only one volume with a new version, without any downtime ?
So i'm kind of stuck here. I'm thinking of having a permanent volume to store the code and update it through SSH with Capistrano, old style. This will allow me to run doctrine migrations scripts after deployment too. But i have other issues with it as Capistrano uses symlinks to handle versions so i can't just mount the /current folder to /code.
Do you have a solution to handle the deployment of a Docker application without losing data and without downtime ?
Should i use manual scripts instead of docker-compose ?

the source code is mounted on the /code volume
This is the problem, it is not what you want.
Code never goes into a volume, it should change when the image changes. Volumes are for things that you want to preserve between changes to the image (data, logs, state, etc).
Code is the immutable thing that you want to replace when you change a container. So remove the /code volume from the Dockerfile entirely, and instead do an ADD . /code in the mynginx and myphpfpm Dockerfiles.
With that change, you can deploy with just up -d. It will recreate any container that have changed, and your volumes will be copied over. You don't need an rm anymore.
If you have your Dockerfile for myphpfpm and mynginx in a different directory, you can build using docker build -f path/to/dockerfile .
Using a host volume (as suggested in another answer) is another option, however that's not usually what you want outside of development. With a host volume you would still remove the /code VOLUME from the dockerfile.

Do not copy the code via the Dockerfile, just attach volumes to the 'code' container.
Few edits:
code:
image: mycode
volumes:
- .:/code
- /code
web:
image: mynginx
volumes-from:
- code
ports:
- "80:80"
links:
- php-fpm
php-fpm:
image: myphpfpm
volumes-from:
- code
links:
- mongo
mongo:
image: mongo
Same thing applies to mongo mount it to an external volume so it persists when the container shuts down. Actually there is also another method, they mention it in their dockerhub page https://hub.docker.com/_/mongo/
Where to Store Data
Important note: There are several ways to store data used by
applications that run in Docker containers. We encourage users of the
mongo images to familiarize themselves with the options available,
including:
Let Docker manage the storage of your database data by writing the
database files to disk on the host system using its own internal
volume management. This is the default and is easy and fairly
transparent to the user. The downside is that the files may be hard to
locate for tools and applications that run directly on the host
system, i.e. outside containers.
Create a data directory on the host system (outside the container) and
mount this to a directory visible from inside the container. This
places the database files in a known location on the host system, and
makes it easy for tools and applications on the host system to access
the files. The downside is that the user needs to make sure that the
directory exists, and that e.g. directory permissions and other
security mechanisms on the host system are set up correctly.

Related

Docker: How can I have sqlite db changes persist to the db file?

FROM golang:1.8
ADD . /go/src/beginnerapp
RUN go get -u github.com/gorilla/mux
RUN go get github.com/mattn/go-sqlite3
RUN go install beginnerapp/
VOLUME /go/src/beginnerapp/local-db
WORKDIR /go/src/beginnerapp
ENTRYPOINT /go/bin/beginnerapp
EXPOSE 8080
The sqlite db file is in the local-db directory but I don't seem to be using the VOLUME command correctly. Any ideas how I can have db changes to the sqlite db file persisted?
I don't mind if the volume is mounted before or after the build.
I also tried running the following command
user#cardboardlaptop:~/go/src/beginnerapp$ docker run -p 8080:8080 -v ./local-db:/go/src/beginnerapp/local-db beginnerapp
docker: Error response from daemon: create ./local-db: "./local-db" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
EDIT: Works with using /absolutepath/local-db instead of relative path ./local-db
You are not mounting volumes in a Dockerfile.
VOLUME tells docker that content on those directories can be mounted via docker run --volumes-from
You're right. Docker doesn't allow relative paths on volumes on command line.
Run your docker using absolute path:
docker run -v /host/db/local-db:/go/src/beginnerapp/local-db
Your db will be persisted in the host file /host/db/local-db
If you want to use relative paths, you can make it work with docker-compose with "volumes" tag:
volumes:
- ./local-db:/go/src/beginnerapp/local-db
You can try this configuration:
Put the Dockerfile in a directory, (e.g. /opt/docker/myproject)
create a docker-compose.yml file in the same path like this:
version: "2.0"
services:
myproject:
build: .
volumes:
- "./local-db:/go/src/beginnerapp/local-db"
Execute docker-compose up -d myproject in the same path.
Your db should be stored in /opt/docker/myproject/local-db
Just a comment. The content of local-db (if any) will be replaced by the content of ./local-db path (empty). If the container have any information (initialized database) will be a good idea to copy it with docker cp or include any init logic on an entrypoint or command shell script.

Docker run results in "host not found in upstream" error

I have a frontend-only web application hosted in Docker. The backend already exists but it has "custom IP" address, so I had to update my local /etc/hosts file to access it. So, from my local machine I am able to access the backend API without problem.
But the problem is that Docker somehow can not resolve this "custom IP", even when the host in written in the container (image?) /etc/hosts file.
When the Docker container starts up I see this error
$ docker run media-saturn:dev
2016/05/11 07:26:46 [emerg] 1#1: host not found in upstream "my-server-address.com" in /etc/nginx/sites/ms.dev.my-company.com:36
nginx: [emerg] host not found in upstream "my-server-address.com" in /etc/nginx/sites/ms.dev.my-company.com:36
I update the /etc/hosts file via command in Dockerfile, like this
# install wget
RUN apt-get update \
&& apt-get install -y wget \
&& rm -rf /var/lib/apt/lists/*
# The trick is to add the hostname on the same line as you use it, otherwise the hosts file will get reset, since every RUN command starts a new intermediate container
# it has to be https otherwise authentification is required
RUN echo "123.45.123.45 my-server-address.com" >> /etc/hosts && wget https://my-server-address.com
When I ssh into the machine to check the current content of /etc/hosts, the line "123.45.123.45 my-server-address.com" is indeed there.
Can anyone help me out with this? I am Docker newbee.
I have solved this. There are two things at play.
One is how it works locally and the other is how it works in Docker Cloud.
Local workflow
cd into root of project, where Dockerfile is located
build image: docker build -t media-saturn:dev .
run the builded image: docker run -it --add-host="my-server-address.com:123.45.123.45" -p 80:80 media-saturn:dev
Docker cloud workflow
Add extra_host directive to your Stackfile, like this
and then click Redeploy in Docker cloud, so that changes take effect
extra_hosts:
'my-server-address.com:123.45.123.45'
Optimization tip
ignore as many folders as possible to speed up process of sending data to docker deamon
add .dockerignore file
typically you want to add folders like node_modelues, bower_modules and tmp
in my case the tmp contained about 1.3GB of small files, so ignoring it sped up the process significantly

Why I can't see my files inside a docker container?

I'm a Docker newbie and I'm trying to setup my first project.
To test how to play with it, I just cloned one ready-to-go project and I setup it (Project repo).
As the guide claims if I access a specific url, I reach the homepage. To be more specific a symfony start page.
Moreover with this command
docker run -i -t testdocker_application /bin/bash
I'm able to login to the container.
My problem is if I try to go to the application folder through bash, the folder that I shared with my host is empty.
I tried with another project, but the result is the same.
Where I'm wrong?
Here some infos about my env:
Ubuntu 12.04
Docker version 1.8.3, build f4bf5c7
Config:
application:
build: code
volumes:
- ./symfony:/var/www/symfony
- ./logs/symfony:/var/www/symfony/app/logs
tty: true
Looks like you have a docker-compose.yml file but are running the image with docker. You don't actually need docker-compose to start a single container. If you just want to start the container your command should look like this:
docker run -ti -v $(pwd)/symfony:/var/www/symfony -v $(pwd)/logs/symfony:/var/www/symfony/app/logs testdocker_application /bin/bash
To use your docker-compose.yml start your container with docker-compose up. You would also need to add the following to drop into a shell.
stdin_open: true
command: /bin/bash

Docker permissions development environment using a host mounted volume

I'm using docker-compose to set up a portable development environment for a bunch of symfony2 applications (though nothing I want to do is specific to symfony). I've decided to have the source files on the local machine exposed as a data volume with all the other dependencies in docker. This way developers can edit on the local file-system.
Everything works great, except that after running the app my cache and log files and the files created by composer in the /vendor directory are now owned by root.
I've read about this problem and some possible approaches here:
Changing permissions of added file to a Docker volume
But I can't quite quite tease out what changes I have to make in my docker-compose.yml file so that when my symphony container starts with docker-compose up any files that are created have the permissions of the user on the host machine.
I'm posting the file for reference, worker is where php, etc. live:
source:
image: symfony/worker-dev
volumes:
- $PWD:/var/www/app
mongodb:
image: mongo:2.4
ports:
- "27017:27017"
volumes_from:
- source
worker:
image: symfony/worker-dev
ports:
- "80:80"
- mongodb
volumes_from:
- source
volumes:
- "tmp/:/var/log/nginx"
One of the solutions is to execure the commands inside your container. I've tried multiple workarounds for the same issue I faced in the past. I find executing the command inside the container the most user-friendly.
Example command: docker-compose run CONTAINER_NAME php bin/console cache:clear. You may use make, ant or any modern tool to keep the commands short.
Example with Makefile:
all: | build run test
build: | docker-compose-build
run: | composer-install clear-cache
############## docker compose
docker-compose-build:
docker-compose build
############## composer
composer-install:
docker-compose run app composer install
composer-update:
docker-compose run app composer update
############## cache
clear-cache:
docker-compose run app php bin/console cache:clear
docker-set-permissions:
docker-compose run app chown -R www-data:www-data var/logs
docker-compose run app chown -R www-data:www-data var/cache
############## test
test:
docker-compose run app php bin/phpunit
Alternatively, you may introduce a .env file which contains a environment variables and then user one of the variables to run usermod command in the Docker container.

RStudio and Shiny in one dockerfile

I am looking into docker to distribute a shiny application that also requires RStudio. The primary goal is easy installation at hospitals under Windows. Everything that requires character input into black boxes will certainly fail during installation by non-IT people.
My previous attempts used vagrant, but installing vagrant alone proved to be a hurdle.
The rocker repository, has an RStudio and a Shiny , and for my own installation both work together. However, I would like to create a combined application for easier installation.
What is the recommended workflow? Start with RStudio, and manually add Shiny?
Or merge the dockerfiles code from both Rockers, starting with r-base? Or use compose tool?
The point of Docker, in general, is isolation of services so that they can be updated/changed without effecting others. My recommendation would be to use docker-compose, instead. Below is an example docker-compose yaml file that serves both rstudio and shiny on the same server at different subdomains using the incredibly useful docker-gen by Jason Wilder. All R docker images used below are courtesy of Rocker or more directly Rocker Docker Hub. These are very very reliable because, well, Dirk Eddelbeutel and Carl Boettiger made them. In this example I've also included some options for RStudio such as setting a user/pass and whether or not the user has root access. There are more instructions on using the Rocker RStudio image available on this wiki page:
Change the following:
your_user to your username on the server
SOME_USER to your desired RStudio username
SOME_PASS to your desired Rstudio password
*.DOMAIN.tld to your domain, don't forget to add A records for your subdomains.
nginx1:
image: nginx
container_name: nginx
ports:
- "80:80"
- "443:443"
volumes:
- /etc/nginx/conf.d
- /etc/nginx/vhost.d
- /usr/share/nginx/html
- /home/your_user/services/volumes/proxy/certs:/etc/nginx/certs:ro
nginx-gen:
links:
- "nginx1"
image: jwilder/docker-gen
container_name: nginx-gen
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- /home/your_user/services/volumes/proxy/templates/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
volumes_from:
- nginx1
entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
rstudio:
links:
- "nginx1"
image: rocker/hadleyverse
container_name: rstudio
ports:
- "8787:8787"
environment:
- VIRTUAL_PORT=8787
- ROOT=TRUE
- VIRTUAL_HOST=rstudio.DOMAIN.tld
- USER=SOME_USER
- PASSWORD=SOME_PASS
shiny:
links:
- "nginx1"
image: rocker/shiny
container_name: shiny
environment:
- VIRTUAL_HOST=shiny.DOMAIN.tld
volumes:
- /home/your_user/services/volumes/shiny/apps:/srv/shiny-server/
- /home/your_user/services/volumes/shiny/logs:/var/log/
- /home/your_user/services/volumes/shiny/packages:/home/shiny/
It's trivial to add more services like a blog, for example, just follow the pattern or search the internet for a docker-compose version of your service and add it.
Interesting question, but I'm not sure I understand the advantage of having the shiny-server and the rstudio-server instances served from the same container.
Is the purpose so that the two containers share the same R libraries (e.g. so a package doesn't need to be installed separately on each) or merely to have one docker container instead of two? Just having to run two docker commands instead of one doesn't seem that onerous, but maybe I'm underestimating.
Sharing the underlying libraries seems like a valid objective though, and I don't think there's an ideal solution available yet.
I feel the most docker-esque solution would be to do this via container orchestration/compose tool as you mention. This is the usual way to combine separate services (e.g. web server and database) without building one on top of the other.
Unfortunately, the tooling for orchestration based on mapping volumes is not nearly as well developed as it is for mapping ports.
Imagine running the rstudio as a volume container:
docker run --name rstudio -v /usr/local/lib/R/site.library rocker/rstudio true
(If you wanted RStudio access at the same time, one could instead run this as:)
docker run --name rstudio -dP -v /usr/local/lib/R/site.library rocker/rstudio
You can then use the the site.library from the rstudio container in place of that on the shiny container with a command like:
docker run --volumes-from rstudio -dP rocker/shiny
Unfortunately, this clobbers the site.library of the shiny container. To work around this, you'd want to mount the library of the rstudio container in a different place, but there's no easy syntax for this like we already have with port links. It can be done though, see:
How to map volume paths using Docker's --volumes-from?
There's an open thread on this issue in the rocker repo too.
I have developed a working single docker for
R
RStudio (server)
Shiny Server (free edition)
I built it exactly for the same reasons mentioned by #Dieter Menne. It may be not ideal for ops, but it great for dev (especially if the team members all use different envs. like mac, windows etc.).
It is on Centos 6 as this is the env. I use at work.
This is the dockerfile:
FROM centos:centos6.7
MAINTAINER enzo smartinsightsfromdata
RUN yum -y install epel-release
RUN yum update -y && yum clean all
# RUN yum reinstall -y glibc-common
RUN yum install -y locales java-1.7.0-openjdk-devel tar
# Misc packages
RUN yum groupinstall -y "Development Tools"
# R devtools pre-requisites:
RUN yum install -y wget git xml2 libxml2-devel curl curl-devel openssl-devel
WORKDIR /home/root
RUN yum install -y R
RUN wget http://cran.r-project.org/src/contrib/rJava_0.9-7.tar.gz
RUN R CMD INSTALL rJava_0.9-7.tar.gz
RUN R CMD javareconf \
&& rm -rf rJava_0.9-7.tar.gz
#-----------------------
# Add RStudio binaries to PATH
# export PATH="/usr/lib/rstudio-server/bin/:$PATH"
ENV PATH /usr/lib/rstudio-server/bin/:$PATH
ENV LANG en_US.UTF-8
RUN yum install -y openssl098e supervisor passwd pandoc
# RUN wget http://download2.rstudio.org/rstudio-server-rhel-0.99.484-x86_64.rpm
# Go for the bleading edge:
RUN wget https://s3.amazonaws.com/rstudio-dailybuilds/rstudio-server-rhel-0.99.697-x86_64.rpm
RUN yum -y install --nogpgcheck rstudio-server-rhel-0.99.697-x86_64.rpm \
&& rm -rf rstudio-server-rhel-0.99.484-x86_64.rpm
RUN groupadd rstudio \
&& useradd -g rstudio rstudio \
&& echo rstudio | passwd rstudio --stdin
RUN R -e "install.packages(c('shiny', 'rmarkdown'), repos='http://cran.r-project.org', INSTALL_opts='--no-html')"
RUN wget https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.4.0.756-rh5-x86_64.rpm
RUN yum -y install --nogpgcheck shiny-server-1.4.0.756-rh5-x86_64.rpm \
&& rm -rf shiny-server-1.4.0.756-rh5-x86_64.rpm
RUN mkdir -p /var/log/shiny-server \
&& chown shiny:shiny /var/log/shiny-server \
&& chown shiny:shiny -R /srv/shiny-server \
&& chmod 777 -R /srv/shiny-server \
&& chown shiny:shiny -R /opt/shiny-server/samples/sample-apps \
&& chmod 777 -R /opt/shiny-server/samples/sample-apps
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
RUN mkdir -p /var/log/supervisor \
&& chmod 777 -R /var/log/supervisor
EXPOSE 8787 3838
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
This is how the supervisord.conf file looks like:
[supervisord]
nodaemon=true
logfile=/var/log/supervisor/supervisord.log
pidfile = /tmp/supervisord.pid
[program:rserver]
user=root
command=/usr/lib/rstudio-server/bin/rserver
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
startsecs=0
autorestart=false
[program:shinyserver]
user=root
command=/usr/bin/shiny-server
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
autorestart=false
It is available at my github page: smartinsightsfromdata
I have also developed a working docker for shiny server pro on centos (using shiny server pro temporary edition, valid 45 days only).
Somewhat unfortunately, there is no definite answer, it all depends on how much reusability you would be looking for and whether an upstream base image is well maintained. The is also images size tradeoff, more layers there are, bigger the resulting image gets.

Resources