I decided today to give our early access AWS/DCOS cluster a try and run one of our backend system on it as a test. I installed our Ochothon portal and had to slightly modify the code to reflect how slaves are configured in DCOS (the slave settings are located in /opt/mesosphere which differs from the usual package install setup). Other than that it was a breeze and I ended up with a fairly complex 7 clusters system neatly stacked on my 5 gentoo linux boxes.
A simple ls in my Ochothon remote shell illustrates well we didn't just run some toy container but something more substantial:
I was not able to deploy to a production-esque level on just 5 r3.large machines (I'd need at least 5 Kafka brokers and probably a 10-node API tier or so). The system was however fully functional which was the point of my experiment.
One thing I wanted to try with the latest Marathon release (0.8.2 RC1) is that docker pull flag. We use a set of private repositories on the Docker Hub for our various images and having to manually docker pull them on my nodes is not exactly enjoyable. It turned out to not really work as planned with only public repositories ending up being pulled. I however found a great blog page that gave the trick away: you need to specify the .dockercfg via the "uris" array when invoking the REST API (so for instance something like "uris": ["file:///root/.dockercfg"]). It then works like a charm (don't ask me why).
The best part for me was the very cool DCOS dashboard Mesosphere offers. That thing is really super neat, probably the best dashboard i've seen in a while. Having the compound cpu shares neatly displayed in a graph is extremely useful.
Definitively a great job from the guys at Mesosphere ! I especially love the circle control that reflects the staged/running tasks.
So overall i'm quite happy and using that stack for our Ochothon PaaS was quite easy. I'm still waiting for having the ability to use my own image when deploying the cluster but besides that I could not spot any real blocker. I still need to study how we would dev/ops a CoreOS cluster but that's another story.