Wednesday, 16 August 2017

AWS Spot fleet with Docker

Preferred pattern for applications that is easy for developers, low in cost, simple and very portable:
- Use Docker
- One Docker container per EC2 instance (KISS principle)
- Application runs behind load balancer
- EC2 instance registers with ELB on startup, deregisters from ELB on termination notice
- Application servers are stateless, state in stored in RDS/DynamoDB and/or EFS
- Diversified Spot fleet to save up to 80%+ costs
- Diversified Spot fleet runs across multiple AZs with health checks
- Autoscaling to scale up/down on demand
- Docker container is portable to VMware or any other cloud vendor
- No Orchestration complexity of Mesos/Kubernetes/Compose/Swarm required
- Developers provides:
         - Docker image name
         - Minimum instance type (vCPUs / Memory)
         - Min/Max instances for autoscaling
         - Scaleup policy, Scaledown policy
   Then pipeline creates autoscaling Spot Fleet that provides resilience as many spot markets (in mutiple AZs) are used. 


Here is how to create a docker image from a physical server, VM or EC2 instance:
tar --numeric-owner  --exclude=/proc  --exclude=/sys --exclude=/mnt --exclude=/var/cache --exclude=/usr/share/doc --exclude=/tmp/zoran --exclude=/var/log  -czvf /tmp/zoran/mydocker.tar.gz /
docker import /tmp/zoran/mydocker.tar.gz mydocker

Then simply add supervisord to start the required processes with Dockerfile:

FROM mydocker:latest
MAINTAINER Zoran Gagic <zorang@gmail.com>

RUN pip install supervisor
#RUN echo_supervisord_conf > /etc/supervisord.conf
ADD supervisord.conf /etc

EXPOSE 22
CMD ["supervisord", "-n", "-c", "/etc/supervisord.conf"]

Here is sample supervisord.conf:
[unix_http_server]
file=/tmp/supervisor.sock   ; the path to the socket file
[supervisord]
logfile=/tmp/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB        ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10           ; # of main logfile backups; 0 means none, default 10
loglevel=info                ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false               ; start in foreground if true; default false
minfds=1024                  ; min. avail startup file descriptors; default 1024
minprocs=200                 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:sshd]
directory=/usr/local/
command=/usr/sbin/sshd -D
autostart=true
autorestart=true
redirect_stderr=true

[program:crond]
directory=/usr/local/
command=/usr/sbin/crond
autostart=true
autorestart=true
redirect_stderr=true

No comments:

Post a Comment