Sunday, June 14, 2015

Series: How to create your own website based on Docker (Part 11 - Run the whole application stack with Docker Compose)

Manage all your created containers with Docker Compose

This is part 11 of the series: How to create your own website based on Docker.

Well, we have created our images now, so it's about time to get everything up and running. As mentioned in the first posting we're using Docker Compose for that.

This is the time where our docker-compose file from part 5 comes back into play. Docker Compose will use this file to figure out which containers need to be started and in what order. I will make sure that all posts are exposed correctly and will mount our directories.


Source code

All files mentioned in this series are available on Github, so you can play around with it! :)


Custom Clean Script

I've created a custom clean script that allows me to clean all existing images and containers. This is comes in pretty handy when you're in the middle of development. I've used this script a thousand times already.
#!/bin/bash
docker rm -f $(docker ps -q -a)
docker rmi -f $(docker images -q)
Source: https://github.com/mastix/project-webdev-docker-demo/blob/master/cleanAll

This script iterates through all containers first and removes them (it doesn't care whether they are up and running or not) and will then remove all images. Don't be afraid... the images will be re-created from the Dockerfiles. So if you've played around with these containers already, I recommend to run this script before reading further. Important: Do not run this script in your production environment! :)

Working with Docker Compose

Docker Compose will only do it's magic when you're in a directory that contains a docker-compose.yml. As mentioned before, I've put all files into /opt/docker/, so we're operating in this directory only.

I'm currently using the following version:
/opt/docker$ docker-compose --version
docker-compose 1.2.0
Make sure that no containers are running:
/opt/docker$ docker-compose ps
Name   Command   State   Ports
------------------------------
Update: If you're running Docker Compose >= 1.3.x then you'll have to run the following command if you've checked out the code from my repository:
docker-compose migrate-to-labels
Since we have our docker-compose.yml ready, the only thing to do is fire docker-compose via the following command:
docker-compose up -d
The -d flag makes sure that Docker Compose starts in daemon mode (in the background).

When you run this command you'll see that it fetches the images and configures them based on our Dockerfile (e.g. like running apt-get, copying files,...). It would be too much to copy here, but you'll see soon how much output it generates.

Let's play around with our containers

Let's see if all containers are up and running:
/opt/docker$ docker-compose ps
           Name                         Command               State               Ports          
--------------------------------------------------------------------------------------------------
docker_mongodb_1             /tmp/entrypoint.sh mongod  ...   Up       3333/tcp                  
docker_nginxreverseproxy_1   /bin/sh -c /etc/nginx/conf ...   Up       443/tcp, 0.0.0.0:80->80/tcp
docker_projectwebdev_1       /bin/sh -c nginx                 Up       8081/tcp                  
docker_projectwebdevapi_1    pm2 start index.js --name  ...   Up       3000/tcp                  
docker_ubuntubase_1          bash                             Exit 0               
It's pretty fine that our ubuntu container already exited, since there is no background task running (e.g. like our nginx server that has to reply to requests) - all other three services are up and running. You can also see that only our nginx reverse proxy exposes its port (80) to the public. All other ports are internal ports.

Let's see if our website is up and running:

Our Person REST API:


Our Person Demo page:



Just a short information: Don't try to access the URLs you see in this image. Since this is just a demo I just uploaded it for demonstration purposes and stopped the containers right after taking these screenshots.

Let's see how much memory our Person API consumes:
/opt/docker$ docker-compose stats docker_projectwebdevapi_1
CONTAINER                   CPU %               MEM USAGE/LIMIT       MEM %               NET I/O
docker_projectwebdevapi_1   0.00%               76.12 MiB/1.954 GiB   3.80%               3.984 KiB/1.945 KiB
Let's stop & start all containers - why? Because we can!
/opt/docker$ docker-compose restart
Restarting docker_projectwebdev_1...
Restarting docker_mongodb_1...
Restarting docker_projectwebdevapi_1...
Restarting docker_nginxreverseproxy_1...
Let's see the stacked images:
docker images --tree
Warning: '--tree' is deprecated, it will be removed soon. See usage.
└─1c3c252d48a5 Virtual Size: 131.3 MB
  └─66b5d995810b Virtual Size: 131.3 MB
    └─b7e7cde90a84 Virtual Size: 131.3 MB
      └─c6a3582257ff Virtual Size: 131.3 MB Tags: ubuntu:15.04
        └─beec7359d06b Virtual Size: 516 MB
          └─56f95e536056 Virtual Size: 516 MB
            └─2e6215be7f22 Virtual Size: 516 MB
              └─0da535016806 Virtual Size: 516 MB Tags: docker_ubuntubase:latest
                ├─22e3ad368e3d Virtual Size: 516.4 MB
  […]
                  └─bc20ce213396 Virtual Size: 679 MB
                │   └─b20c90481a4e Virtual Size: 679 MB Tags: docker_mongodb:latest
                └─419a34bcfcfd Virtual Size: 516 MB
                  ├─2d2525cf28e1 Virtual Size: 537.1 MB
                  │ └─9c9f238dc62d Virtual Size: 558.2 MB
                  │   └─4bf8554af678 Virtual Size: 580.2 MB
                  │     └─9d6fdb379360 Virtual Size: 620.4 MB
                  │       └─02b3cd93208f Virtual Size: 638.1 MB
           […]    
   └─aba65d0f0c06 Virtual Size: 706 MB
                  │           └─9b4b55e323e3 Virtual Size: 706 MB Tags: docker_projectwebdevapi:latest
                  └─466f9910439a Virtual Size: 543.9 MB
                    ├─008ffe8fa738 Virtual Size: 543.9 MB
                    │ └─476a45c16218 Virtual Size: 543.9 MB
                    […]
                    │   └─b53827f8ddfd Virtual Size: 543.9 MB Tags: docker_nginxreverseproxy:latest
                    └─aec75192e11a Virtual Size: 543.9 MB
                      └─eadec9140592 Virtual Size: 543.9 MB
                        └─27b6deeec60a Virtual Size: 543.9 MB
                          └─6f0f6661c308 Virtual Size: 543.9 MB Tags: docker_projectwebdev:latest
As you can see: All our images are based on the Ubuntu Base Image and files are only added on top of the underlying base image.

Let's see all logs to console/stdout:
/opt/docker$ docker-compose logs
Attaching to docker_nginxreverseproxy_1, docker_projectwebdevapi_1, docker_mongodb_1, docker_projectwebdev_1, docker_ubuntubase_1
projectwebdevapi_1  | pm2 launched in no-daemon mode (you can add DEBUG="*" env variable to get more messages)
projectwebdevapi_1  | 2015-06-12 21:29:23: [PM2][WORKER] Started with refreshing interval: 30000
projectwebdevapi_1  | 2015-06-12 21:29:23: [[[[ PM2/God daemon launched ]]]]
[…]
nginxreverseproxy_1 | START UPDATING DEFAULT CONF
nginxreverseproxy_1 | CHANGED DEFAULT CONF
nginxreverseproxy_1 | upstream blog  {
nginxreverseproxy_1 |       server 172.17.0.29:8081; #Blog
nginxreverseproxy_1 | }
nginxreverseproxy_1 |
nginxreverseproxy_1 | upstream blog-api  {
nginxreverseproxy_1 |       server 172.17.0.54:3000; #Blog-API
nginxreverseproxy_1 | }
nginxreverseproxy_1 |
nginxreverseproxy_1 | ## Start blog.project-webdev.com ##
nginxreverseproxy_1 | server {
nginxreverseproxy_1 |     listen  80;
nginxreverseproxy_1 |     server_name  blog.project-webdev.com;
[…]
nginxreverseproxy_1 | }
nginxreverseproxy_1 | ## End blog.project-webdev.com ##
nginxreverseproxy_1 |
nginxreverseproxy_1 | ## Start api.project-webdev.com ##
nginxreverseproxy_1 | server {
nginxreverseproxy_1 |     listen  80;
nginxreverseproxy_1 |     server_name  api.project-webdev.com;
nginxreverseproxy_1 |
[…]
nginxreverseproxy_1 | }
nginxreverseproxy_1 | ## End api.project-webdev.com ##
nginxreverseproxy_1 |
[…]
nginxreverseproxy_1 |  END UPDATING DEFAULT CONF
As you can see, you'll get all stdout output from each container. You'll see the container name in the beginning of the line... in our example we'll only get output from our ioJS container/projectwebdevapi_1 (started with pm2) and our nginx reverseproxy/nginxreverseproxy_1.

Let's check our log directory:
/opt/docker/logs$ ll
total 24
drwxrwxrwx 2 mastix docker 4096 Jun 14 21:21 ./
drwxr-xr-x 9 mastix docker 4096 Jun 12 17:39 ../
-rw-r--r-- 1 root   root      0 Jun 14 21:21 access.log
-rw-r--r-- 1 root   root      0 Jun 14 21:21 error.log
-rw-r--r-- 1 root   root   2696 Jun 14 21:21 mongodb-projectwebdev.log
-rw-r--r-- 1 root   root    200 Jun 14 21:21 nginx-reverse-proxy-blog.access.log
-rw-r--r-- 1 root   root    637 Jun 14 21:21 nginx-reverse-proxy-blog-api.access.log
-rw-r--r-- 1 root   root      0 Jun 14 21:21 nginx-reverse-proxy-blog-api.error.log
-rw-r--r-- 1 root   root      0 Jun 14 21:21 nginx-reverse-proxy-blog.error.log
-rw-r--r-- 1 root   root      0 Jun 14 21:21 pm2-0.log
-rw-r--r-- 1 root   root    199 Jun 14 21:21 project-webdev.access.log
-rw-r--r-- 1 root   root      0 Jun 14 21:21 project-webdev.error.log
Remember: We've told each container to log its files into our /opt/docker/logs directory on the Docker host... And now we have them all in one place.

That's it. I hope you had fun learning Docker with this session. And if you find any bugs, I'm happy to fix them. Just add a comment or create an issue in the Github repository.

Greetz,

Sascha

18 comments:

  1. Can't see the command to see the logs. You probably meant to put 'docker-compose logs'?

    ReplyDelete
  2. Hi Sascha, really great write up! I'm trying out Docker for the first time and you've made an excellent piece! I'm running into a couple issues with starting the compose.

    1)F CONTROL Failed global initialization: FileNotOpen Failed to open "/var/log/mongodb/mongodb-projectwebdev.log"
    Which is causing
    2) Cannot start container 2c08c7415781f411f720fc1c2753c45359ed1790172955f8db76cde4fcba23b4: Cannot link to a non running container: /docker_mongodb_1 AS /docker_projectwebdev_1/db

    I've noticed some typos and corrections I could make to your text, how can I help you with correcting them?

    Any help on the above issue would be greatly appreciated!

    Best,
    Matt

    ReplyDelete
    Replies
    1. Hey Matt,

      have you checked whether the log directory has the correct permissions? (/opt/docker/logs). Maybe just do a "sudo chmod 777 /opt/docker/logs" to see if this is really the issue.

      Regards,

      Sascha

      Delete
  3. I am also getting cannot link to a non running container error..

    ReplyDelete
    Replies
    1. Hey Raj,

      please see my reply to Matt's answer. :)

      Delete
  4. Hi Sascha,

    Container error is resolved after chmod to logs directory and all containers are up and running and I don’t see any errors in the logs but when I try to run below command(api call) I get “404 Not Found”

    I am kind of new to the web technologies and not sure why I am getting this error, also I noticed that robots.txt file missing in the generated code. since I don’t see any errors in logs, how do I debug this?

    command

    curl -X GET -H "Accept: application/json" http://api.project-webdev.com/person/

    Error Msg

    The requested URL /person/ was not found on this server

    Apache/2.2.22 (Ubuntu) Server at api.project-webdev.com Port 80

    ReplyDelete
    Replies
    1. Hey Raj,

      you have to point the nginx configuration to your own domain of course. Project-webdev.com is my domain and it's currently not up and running. So your nginx configuration must specify a server_name that contains your domain. E.g. api.raj-domain.com... :)

      Delete
  5. Hay Sascha,

    Its a great Blog. I hope it will be good if you share something regarding how to build & run two web apps in the same host, same port as well. waiting to see some good thing.

    ReplyDelete
  6. Hello Sascha,

    all the services are up and running .. Yet, I dont see the Data Entry / PERSON creation screen.

    this is my public ip : http://159.203.246.29/

    Can you please suggest what should I do to enter person data ?

    Please note that API is working fine for CRUD operations inside my bare metal OS. .

    Thanks and Regards,
    raj

    ReplyDelete
    Replies
    1. Hello Sascha,
      figured it out .. It worked .. http://blog.icaregplus.com
      And configured subdomains ..
      See the blog output from my DO server ( which may go down, after couple of days ) http://oi65.tinypic.com/j7ffoy.jpg

      Thanks and Regards,
      Raj

      Delete
  7. docker-compose rm is available to stopped containers.
    And hence no need to remember docker rm -f $(docker ps -q -a) command.
    Hope this info is useful to others

    ReplyDelete
  8. Hello! I made steps with your tutorial but configured it for my settings, and i have got some troubles. My domain name is queryme.ru, and if i go to this link i get index.html page from nginx reverse proxy, but if i go to the site.queryme.ru i got the error in browser, but logs are empty. Can you check my project on github https://github.com/EgorkZe/dockerTest . Thank you! Your tutorial is awesome!

    ReplyDelete
  9. Hello Sascha,
    thank you for your great article.
    I have a problem while running first time docker-compose up -d
    It starts downloading the ubuntu image but at the end I got this error:
    Pulling repository docker.io/library/docker_ubuntubase
    [31mERROR [0m: Service 'projectwebdev' failed to build: Error: image library/docker_ubuntubase not found
    Have you got any idea how it could be fixed?

    ReplyDelete
    Replies
    1. You have to do "docker images" to see the name of your ubuntubase image. The docs says it creates the image based on your folder's name, so it should be folder_ubuntubase. You can change your folder's name to docker and it will work.

      Delete
  10. I have read your blog its very attractive and impressive. I like it your blog.

    Angularjs Online Training Angularjs Training Angularjs Training Angularjs Training in Chennai Angularjs Training in Chennai

    ReplyDelete
  11. It was a great information and Its really worth reading it. The author did an mind blowing work by describing each and every concept in detail. Thanks for such an informative post. Please keep up your good work.
    Marine Colleges in Chennai, Nautical Science Colleges In Chennai

    ReplyDelete
  12. This is really a nice and informative, containing all information and also has a great impact on the new technology. Thanks for sharing it Create WordPress Websites

    ReplyDelete