How can I get more precise log sources from my Deis apps/containers? - deis

Question

I have a Deis cluster running in a (hopefully-soon-to-be) Production environment, with quite a few different apps using the Dockerfile deployment method. Everything's running fine, but promoting this system to a true Production environment (that is, converting the DNS over) isn't really possible unless I can get some worthwhile log output. Using the standard Deis logging platform, here's some sample output of a Web hit (with a bit more output, for context):
Feb 10 01:46:04 ip-10-21-2-154.ec2.internal systemd[1]: Starting Generate /run/coreos/motd...
Feb 10 01:46:04 ip-10-21-2-154.ec2.internal systemd[1]: Started Generate /run/coreos/motd.
Feb 10 01:46:08 ip-10-21-2-154.ec2.internal docker[1867]: [info] GET /containers/json
Feb 10 01:46:08 ip-10-21-2-154.ec2.internal docker[1867]: [215084df] +job containers()
Feb 10 01:46:08 ip-10-21-2-154.ec2.internal docker[1867]: [215084df] -job containers() = OK (0)
Feb 10 01:46:09 ip-10-21-2-154.ec2.internal sh[1316]: 2015/02/10 01:46:09 set /deis/services/production-web/production-web_v8.cmd.1 -> 10.21.2.154:49409
Feb 10 01:46:12 ip-10-21-2-154.ec2.internal sh[9844]: 2015-02-10 01:46:12.302721 7f213ae14700 0 mon.ip-10-21-2-154.ec2.internal#4(peon).data_health(58) update_stats avail 80% total 102400 MB, used 17621 MB, avail 82542 MB
Feb 10 01:46:18 ip-10-21-2-154.ec2.internal docker[1867]: [info] GET /containers/json
Feb 10 01:46:18 ip-10-21-2-154.ec2.internal docker[1867]: [215084df] +job containers()
Feb 10 01:46:18 ip-10-21-2-154.ec2.internal docker[1867]: [215084df] -job containers() = OK (0)
Feb 10 01:46:19 ip-10-23-1-151.ec2.internal sh[1521]: [INFO] - [10/Feb/2015:01:46:27 +0000] - 10.21.2.179 - - - 200 - "GET / HTTP/1.1" - 4927 - "-" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36" - "~^production-web\x5C.(?<domain>.+)$" - 10.21.2.154:49409
Feb 10 01:46:19 ip-10-21-2-154.ec2.internal sh[8468]: ===========
Feb 10 01:46:19 ip-10-21-2-154.ec2.internal sh[8468]: HIT TRACKER
Feb 10 01:46:19 ip-10-21-2-154.ec2.internal sh[8468]: SLUG: public/javascripts/bundle.js
Feb 10 01:46:19 ip-10-21-2-154.ec2.internal sh[8468]: ===========
That contains alot of platform information – which is great to have, if only I could filter it out. The problem is all the lines for which the source is sh, but with different PIDs. Those are each completely different containers:
1316 is deis-publisher
9844 is deis-store-monitor
1521 is deis-router
8468 is my web application, production-web
The only way for me to find that out is to ssh into the box and run ps. What's worse, if I had any logs from my other containers at the same time, they would have also shown up as sh – in a production environment with several active apps all logging to the same stream, this situation is obviously untenable. The ideal situation would have sh replaced by the name of the Docker container or, preferably, the Deis app.
I've poured over the documentation and dug into the logspout and logger source code, but I can't find anything to fix this. Any chance I could get some pointers here?

Answer 1

In order to get at the name of the deis container that logged the line, the best way I've found is either:
To run the output of journalctl -f -o short through netcat to a fluentd or logstash tcp listener. You can use these tools to summarize the fields like _SYSTEMD_UNIT that appeal to your needs.
Use ianblenke/fluentd with LOG_DOCKER_JSON defined or fork and modify the autobuild source docker-ianblenke/fluentd. This uses the fluentd-docker plugin to follow the raw docker container json logs.
If you're using CoreOS, I use this fluentd.cloud-init to auto-feed my logs to a local elasticsearch instance on TCP 9200. Will fill find other useful CoreOS cloud-init configs in that project as well.