Author Archives: wallnerryan

Piping AWS output to Ansible Inventory

piping

I’ve had the opportunity to work with a few different infrastructure automation tools such as Puppet, Chef, Heat and CloudFormation but Ansible just has a simplicity to it that I like, although I admit I do have a strong preference for Puppet because i’ve used it extensively and have had good success with it.

In one of my previous project  I was creating a repeatable solution to create a Docker Swarm cluster (before SwarmKit) with Consul and Flocker. I wanted this to be completely scripted to I climbed on the shoulders of AWS, Ansible and Docker Machine.

The script would do 4 things.

  1. Initialize a security group in an existing VPC and create rules for the given setup.
  2. Create machines using Docker-Machine of Consul and Swarm
  3. Use AWS CLI to output the machines and pipe them to a python script that processes the JSON output and creates an Ansible inventory.
  4. Use the inventory to call Ansible to run something.

This flow can actually be used fairly reliable not only for what I used it for but to automate a lot of things, even expand an existing deployment.

An example of this workflow can be found here.

I’m going to focus on steps #3 and #4 here. First, we use the AWS CLI to output machine information and pass it to a script.

# List only running my-prefix* nodes
$ aws ec2 describe-instances \
   --filter Name=tag:Name,Values=my-prefix* \
   Name=instance-state-code,Values=16 --output=json | \
   python create_flocker_inventory.py

We use the instance-state-code of 16 as it corresponds with Running instances. You can find more codes here: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_InstanceState.html. Then we choose JSON output using –output=json. 

Next, the important piece is the pipe ( `|` ). This signifies we pass the output from the command on the left of the | to the command on the right which is create_flocker_inventory.py so that the output is used as input to the script.

Shell-pipes.png

So what does the python script do with the output? Below is the script that I used to process the JSON output. It first setups up an _AGENT_YML variable that contains YAML for a configuration then the main() function takes the JSON from json.loads() in the script initialization and creates an array of dictionaries that represent instances and opens a file and writes each instance to the Ansible inventory file called “ansible_inventory”. After that the “agent.yml” is written to a file along with some secrets from the environment.

import os
import json
import sys


_AGENT_YML = """
version: 1
control-service:
  hostname: %s
  port: 4524
dataset:
  backend: aws
  access_key_id: %s
  secret_access_key: %s
  region: %s
  zone: %s
"""

def main(input_data):
    instances = [
        {
            u'ip': i[u'Instances'][0][u'PublicIpAddress'],
            u'name': i[u'Instances'][0][u'KeyName']
        }
        for i in input_data[u'Reservations']
    ]

    with open('./ansible_inventory', 'w') as inventory_output:
        inventory_output.write('[flocker_control_service]\n')
        inventory_output.write(instances[0][u'ip'] + '\n')
        inventory_output.write('\n')
        inventory_output.write('[flocker_agents]\n')
        for instance in instances:
            inventory_output.write(instance[u'ip'] + '\n')
        inventory_output.write('\n')
        inventory_output.write('[flocker_docker_plugin]\n')
        for instance in instances:
            inventory_output.write(instance[u'ip'] + '\n')
        inventory_output.write('\n')
        inventory_output.write('[nodes:children]\n')
        inventory_output.write('flocker_control_service\n')
        inventory_output.write('flocker_agents\n')
        inventory_output.write('flocker_docker_plugin\n')

    with open('./agent.yml', 'w') as agent_yml:
        agent_yml.write(_AGENT_YML % (instances[0][u'ip'], os.environ['AWS_ACCESS_KEY_ID'], os.environ['AWS_SECRET_ACCESS_KEY'], os.environ['MY_AWS_DEFAULT_REGION'], os.environ['MY_AWS_DEFAULT_REGION'] + os.environ['MY_AWS_ZONE']))


if __name__ == '__main__':
    if sys.stdin.isatty():
        raise SystemExit("Must pipe input into this script.")
    stdin_json = json.load(sys.stdin)
    main(stdin_json)

After this processes the JSON from the AWS CLI, all that remains is to run Ansible with our newly created Ansible inventory. In this case, we pass the inventory and configuration along with the ansible playbook we want for our installation.

$ ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook \
 --key-file ${AWS_SSH_KEYPATH} \
 -i ./ansible_inventory \
 ./aws-flocker-installer.yml \
 --extra-vars "flocker_agent_yml_path=${PWD}/agent.yml"

 

Conclusion

Overall this flow can be used along with other Cloud CLI tools such as Azure, GCE etc that can output instance state that you can pipe to a script for more processing. It may not be the most effective way but if you want to get a semi complex environment up and running in a repeatable fashion for development needs it has worked pretty well to follow the “pre-setup_get-output_prcocess-output_install_config” flow outlined above.

Advertisements

Docker-based FIO I/O benchmarking

687474703a2f2f692e696d6775722e636f6d2f336f46443358502e706e67

What is FIO?

fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate. – (https://linux.die.net/man/1/fio)

fio can be a great tool for helping to measure workload I/O of a specific application workload on a particular device or file. Fio proves to be a detailed benchmarking tool used for workloads today with many options. I personally came across the tool while working at EMC when needing to benchmark Disk I/O of application running in different Linux container runtimes. This leads me to my next topic.

Why Docker based fio-tools

One of the projects I was working on was using Docker on AWS and various private cloud deployments and we wanted to see how workloads performed on these different cloud environments inside Docker container with various CPU, Memory, Disk I/O limits with various block, flash, or DAS based storage devices.

One way to wanted to do this was to containerize fio and allow users to pass the workload configuration and disk to the container that was doing the testing.

The first part of this was to containerize fio with the option to pass in JOB files by pathname or by a URL such as a raw Github Gist.

The Dockerfile (below) is based on Ubuntu 14 which admittedly can be smaller but we can easily install fio and pass a CMD script called run.sh.

FROM ubuntu:14.10
MAINTAINER <Ryan Wallner ryan.wallner@clusterhq.com>

RUN sed -i -e 's/archive.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
RUN apt-get -y update && apt-get -y install fio wget

VOLUME /tmp/fio-data
ADD run.sh /opt/run.sh
RUN chmod +x /opt/run.sh
WORKDIR /tmp/fio-data
CMD ["/opt/run.sh"]

What does run.sh do? This script does a few things, is checked that you are passing a JOBFILE name (fio job) which without REMOTEFILES will expect it to exist in `/tmp/fio-data` it also cleans up the fio-data directory by copying the contents which may be jobs files out and then back in while removing any old graphs or output. If the user passes in REMOTEFILES it will be downloaded from the internet with wget before being used.

#!/bin/bash

[ -z "$JOBFILES" ] && echo "Need to set JOBFILES" && exit 1;
echo "Running $JOBFILES"

# We really want no old data in here except the fio script
mv /tmp/fio-data/*.fio /tmp/
rm -rf /tmp/fio-data/*
mv /tmp/*fio /tmp/fio-data/

if [ ! -z "$REMOTEFILES" ]; then
 # We really want no old data in here
 rm -rf /tmp/fio-data/*
 IFS=' '
 echo "Gathering remote files..."
 for file in $REMOTEFILES; do
   wget --directory-prefix=/tmp/fio-data/ "$file"
 done 
fi

fio $JOBFILES

There are two other Dockerfiles that are aimed at doing two other operations. 1. Producing graphs of the output data with fio2gnuplot and serving the graphs and output from a python SimpleHTTPServer on port 8000.

All Dockerfiles and examples can be found here (https://github.com/wallnerryan/fio-tools) and it also includes an All-In-One image that will run the job, generate the graphs and serve them all in one which is called fiotools-aio.

How to use it

  1. Build the images or use the public images
  2. Create a Fio Jobfile
  3. Run the fio-tool image
docker run -v /tmp/fio-data:/tmp/fio-data \
-e JOBFILES= \
wallnerryan/fio-tool

If your file is a remote raw text file, you can use REMOTEFILES

docker run -v /tmp/fio-data:/tmp/fio-data \
-e REMOTEFILES="http://url.com/.fio" \
-e JOBFILES= wallnerryan/fio-tool

Run the fio-genplots script

docker run -v /tmp/fio-data:/tmp/fio-data wallnerryan/fio-genplots \
<fio2gnuplot options>

Serve your Graph Images and Log Files

docker run -p 8000:8000 -d -v /tmp/fio-data:/tmp/fio-data \
wallnerryan/fio-plotserve

Easiest Way, run the “all in one” image. (Will auto produce IOPS and BW graphs and serve them)

docker run -p 8000:8000 -v /tmp/fio-data \
-e REMOTEFILES="http://url.com/.fio" \
-e JOBFILES=<your-fio-jobfile> \
-e PLOTNAME=MyTest \
-d --name MyFioTest wallnerryan/fiotools-aio

Other Examples

Important

  • Your fio job file should reference a mount or disk that you would like to run the job file against. In the job fil it will look something like: directory=/my/mounted/volume to test against docker volumes
  • If you want to run more than one all-in-one job, just use -v /tmp/fio-data instead of -v /tmp/fio-data:/tmp/fio-data This is only needed when you run the individual tool images separately

To use with docker and docker volumes

docker run \
-e REMOTEFILES="https://gist.githubusercontent.com/wallnerryan/fd0146ee3122278d7b5f/raw/cdd8de476abbecb5fb5c56239ab9b6eb3cec3ed5/job.fio" \
-v /tmp/fio-data:/tmp/fio-data \
--volume-driver flocker \
-v myvol1:/myvol \
-e JOBFILES=job.fio wallnerryan/fio-tool

To produce graphs, run the fio-genplots container with -t <name of your graph> -p <pattern of your log files>

Produce Bandwidth Graphs

docker run -v /tmp/fio-data:/tmp/fio-data wallnerryan/fio-genplots \
-t My16kAWSRandomReadTest -b -g -p *_bw*

Produce IOPS graphs

docker run -v /tmp/fio-data:/tmp/fio-data wallnerryan/fio-genplots \
-t My16kAWSRandomReadTest -i -g -p *_iops*

Simply serve them on port 8000

docker run -p 8000:8000 -d \
-v /tmp/fio-data:/tmp/fio-data \
wallnerryan/fio-plotserve

To use the all-in-one image

docker run \
-p 8000:8000 \
-v /tmp/fio-data \
-e REMOTEFILES="https://gist.githubusercontent.com/wallnerryan/fd0146ee3122278d7b5f/raw/006ff707bc1a4aae570b33f4f4cd7729f7d88f43/job.fio" \
-e JOBFILES=job.fio \
-e PLOTNAME=MyTest \
—volume-driver flocker \
-v myvol1:/myvol \
-d \
—name MyTest wallnerryan/fiotools-aio

To use with docker-machine/boot2docker/DockerForMac

You can use a remote fit configuration file using the REMOTEFILES env variable.

docker run \
-e REMOTEFILES="https://gist.githubusercontent.com/wallnerryan/fd0146ee3122278d7b5f/raw/d089b6321746fe2928ce3f89fe64b437d1f669df/job.fio" \
-e JOBFILES=job.fio \
-v /Users/wallnerryan/Desktop/fio:/tmp/fio-data \
wallnerryan/fio-tool

(or) If you have a directory that already has them in it. *NOTE*: you must be using a shared folder such as Docker > Preferences > File Sharing.

docker run -v /Users/wallnerryan/Desktop/fio:/tmp/fio-data \
-e JOBFILES=job.fio wallnerryan/fio-tool

To produce graphs, run the genplots container, -p

docker run \
-v /Users/wallnerryan/Desktop/fio:/tmp/fio-data wallnerryan/fio-genplots \
-t My16kAWSRandomReadTest -b -g -p *_bw*

Simply serve them on port 8000

docker run -v /Users/wallnerryan/Desktop/fio:/tmp/fio-data \
-d -p 8000:8000 wallnerryan/fio-plotserve

Notes

  • The fio-tools container will clean up the /tmp/fio-data volume by default when you re-run it.
  • If you want to save any data, copy this data out or save the files locally.

How to get graphs

  • When you serve on port 8000, you will have a list of all logs created and plots created, click on the .png files to see graph (see below for example screen)

687474703a2f2f692e696d6775722e636f6d2f6e6b73516b5a692e706e67

 

Testing and building with codefresh

As a side note, I recently added this repository to build on Codefresh. Right now, it builds the fiotools-aio Dockerfile  which I find most useful and moves on but it was an easy experience that I wanted to add to the end of this post.

Navigate to https://g.codefresh.io/repositories? or create a free account by logging into codefresh with your Github account. By logging in with Github it will have access to your repositories you gave access to and this is where the fio-tools images are.

I added the repository as a build and configured it like so.

screen-shot-2016-12-29-at-2-45-46-pm

This will automatically build my Dockerfile and run any integration tests and unit tests I may have configured in codefresh, thought right now I have none but will soon add some simple job to run against a file as an integration test with a codefresh composition.

Conclusion

I found over my time using both native linux tools and docker-based or containerized tools that there is need for both sometimes and in fact when testing container-native application workloads sometimes it is best to get metrics or benchmarks from the point of view of the application which is why we chose to run fio as a microservice itself.

Hopefully this was an enjoyable read and thanks for stopping by!

Ryan

Service Oriented Architecture vs Modern Microservices: Whats the difference?

 

Images thanks to http://martinfowler.com/articles/microservices.html and https://en.wikipedia.org/wiki/Service-oriented_architecture

I’ve been researching and working in the area of modern microservices for the past ~3 to 4 years and have always seen a strong relationship between Modern Microservices with tools and cultures like Docker and DevOps back to Service-Oriented Architecture (SOA) and design. I traced SOA roots back to Gartner Research in 1996 [2] or at least this is what I could find, feel free to correct me here if I haven’t pegged this. More importantly for this post I will briefly explore SOA concepts and design and how they relate to Modern Microservices.

Microservice Architectures (MSA) (credit to meetups and conversations with folks at meetups), are typically RESTful and based on HTTP/JSON. MSA is an architectural style not a “thing” to conform exactly to. In other words, I view it as more of a guideline. MSAs are derived from multiple code bases and each microservice (MS) has or can have its own language it’s written in. Because of this, MSAs typically have better readability and simpler deployments for each MS deployed which in turn leads to better release cycles as long as the organization surrounding the MS teams is put together effectively (more on that later). An MSA doesn’t NEED to be a polyglot of  but will often naturally become one because teams may be more familiar with one language over the other which helps delivery time especially if the interfaces between microservices are defined correctly, it truly doesn’t matter most of the time. It also enables scale at a finer level instead of worrying about the whole monolith which is more agile. Scaling a 100 lines of Golang that does one thing well can be achieved much easier when you dont have to worry about other parts of your application that dont need or you dont want to scale in the monolith. In most modern MSAs, the REST interfaces mentioned earlier can be considered the “contract” between microservices in an MSA. These contracts should self describing as they can be, meaning using formats like JSON which is human readable and well-organized.

Overall an MSA doesn’t just have technical benefits but could also mean fewer reviews and approvals because of smaller context boundaries for each microservice team. Better acquisition and on-boarding because you dont have to be so strict about language preference, instead of retooling, you can ingest using polyglot.

Motivations for SOA, from what I have learned, is typically business transformation oriented which shouldn’t be surprising. The enterprise based SOA transformation on a large budget but the motivation is different now with modern MSA, now its quick ROI and better technology to help scale using DevOps practices and platforms.

Some things to consider while designing your modern MSA that I’ve heard and stuck with me:

  • Do not create too many services/microservices
  • Try not to manage your own infrastructure if you can
  • Dont make too many dependencies, (e.g. 1 calls 2 calls 3 calls 4 calls 5 ……)
  • Circuit Breaker Pattern, a control point between microservices.
  • Bulkhead, do not allow 1 problem affect the entire boat, each microservice has its own data service / database / connection pool, 1 service does not take down the whole system or other microservices.
  • Chaos testing (Add it to your test suite!)  Example: Chaos Monkey
  • You can do microservices with or without service discovery / catalog. Does it over complicate things?

The referenced text[1] that I use for a comparison or similar concepts and differences in this post talk about a vast number of important topics related to Service-Oriented Architecture. Such topics include the overall challenges of SOA, service reuse, deployment efficiency, integration of application and data, agility, flexibility, alignment, reference architectures, common semantics, semantic pitfalls, legacy application integration, governance, security, service discovery, inventory and registration, best practices and more. This post does not go into depth of each individual part but instead this post aims at looking at some of the similarities and differences of SOA and modern microservices.

Service-Oriented Architecture:

Some of concepts of SOA that I’d like to mention (not fully encompassing):

  • Technologies widely used were SOAP, XML, WSDL, XSD and lots of Java
  • SOAs typically had a Service Bus or ESB (Enterprise Service Bus) a complex middleware aimed at providing access and masking of interfaces.
  • Identification and Inventory
  • Value chain and business model is more about changing the entire business process

Modern Microservces:

  • Technologies widely used are JSON, REST/HTTP and Polyglot services.
  • Communication is done over HTTP and the interfaces are abstracted using RESTful contracts.
  • Service Discovery
  • Value chain and business model is about efficiencies, small teams and DevOps practices while eliminating cilos.
The Bulkhead Analogy

I want to spend a little bit of time on one of the analogies that stuck with me about modern microservices. This was the Bulkhead analogy which I cannot for the life of me remember where I heard it or seem to google a successful author so credit to who or whom ever you are.

The bulkhead analogy is pretty simple actually but has a powerful statement for microservice design. The analogy is such that a MSA, like a large ship is made up of many containers (or in the ships case, bulkheads) that have boundaries between them and hold different component of the ships such as engines, cargo, pumps etc. In MSA, these containers hold different functions or processes that do something wether its handle auth requests, connection to a DB, service a lookup or transformation mechanism it doesn’t matter, just that in both cases you want all containers to be un-damaged for everything to be running the best it can.

The bulkhead analogy goes further to say that if a container gets damaged and takes on water then the entire ship should not sink due to one or few failures. In MSA this can be applied by saying that a few broken microservices should not be designed in a way where there failure would take down your entire application or business process. It essence designs the bulkheads or containers to take damage and remain afloat or “running”.

Again, this analogy is quite simple, but when designing your MSA it’s important to think about these details and is why doing things like proper RESTful design and Chaos testing is worth your time in the long run.

Similarities and Differences or the two architectures / architecture styles:

Given the little glimpse of information I’ve provided above about service oriented architectures and microservices architectures I want to spend a little time talking about the obvious similarities and differences.

Similarities

Both SOA and MSA do the following:

  • Code or service reuse
  • Loose coupling of services
  • Extensibility of the system as a whole
  • Well-defined, self-contained services or functions that overall help the business process or system
  • Services Registries/Catalogs to discover services

Differences

Some of the differences that stick out to me are:

  • Focus on business process, instead of the focus of many services making one important business process MSA focuses on allowing one thing (containerized process) to do one thing and do it well. This allows tighter context boundaries for microservices.
  • SOA tailors towards SOAP, XML, WSDL while MSA favors JSON, REST and Polyglot. This is one of the major differences to me, even though its just a tech difference this RESTful polyglot paradigm enables MSAs to thrive with todays developers.
  • The value chain and business model is more DevOps centric allowing the focus to be on loosely coupled teams that break down cilos and can focus on faster release cycles and CI/CD of their services rather than with SOA teams typically still had one monolithic view of the ESB and services without the DevOps focus.

Conclusion

Overall this post was mainly a complete high-level overview of what I think are some of the concepts and major differences between traditional SOA and Modern Microservices that stemmed from a course I took during my masters that explored SOA while I was in the industry working on Microservices. The main point I would say I have is that SOA and MSA are very similar but MSA being SOA’s offspring in a way using modern tooling and architecture approaches to todays scaleable data center.

Note* by no means did I cover SOA or MSA to do them any real justice, so I suggest looking into some of the topics talked about here or reading through some of the references below if your interested.

Cheers!

[1] Rosen, Michael “Applied SOA: Service-Oriented Design Architecture and Design Strategies”  Wiley, Publishing Inc. 2008

[2] Gartner Research “Service Oriented” Architectures, Part 1:” – //www.gartner.com/doc/code/29201

[3] “SOA fundamentals in a nutshell” Aka Sniv February 2015 http://www.ibm.com/developerworks/webservices/tutorials/ws-soa-ibmcertified/ws-soa-ibmcertified.html

Container Data Management and Production Use-cases

voting-logo

In my last few jobs I have had the pleasure to work on three main focus areas, Software Defined Networking, OpenStack, and Linux Containers. More recently I have been focused on container data management and what it means for persistence and data management to be a first class citizen to containerized applications and microservices. This has done a few things for me, give me an opportunity to work on interesting (and hard) problems, hack, create and apply new solutions and technologies into proof of concept and production environments. In my current role as a Technical Evangelist at ClusterHQ we have been hard at work continuing the Flocker project, creating the Volume Hub and spinning out dvol (git workflows for Docker volumes / data). All of this is aimed at one major thing, helping your team move from development on your laptop to test and Q/A and finally into production seamlessly with persistence.

We’re just at the beginning of the container and data revolution and if your interested in learning more about some of these topics, click the links below and vote for some upcoming talks at OpenStack Austin 2016.

Three Critical Concepts for Containerized Storage Management

https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/8506

Lessons learned running database containers on OpenStack

https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/8501

A special shoutout to Andrew Sullivan and Sumit Kumar for their CFP on “Data Mobility for Docker containers with Flocker”! Please help them spread the word on stateful containers by voting for their talk as well! https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/7859

Cheers!

Ryan – @RyanWallner

Migrating the monolith from EC2 to an ECS-based multi-service Docker app

  

On my spare time I run a website for a tax accounting company. This monolithic app is a largely stateless app (not really but we don’t need persistent stores/volumes), uses one of the best online tax software, and uses ruby on rails and runs on an EC2 instance with Apache and MySQL Server and Client installed. This is what is referenced as a “monolithic app” because all components are installed on 1 single VM. This makes the application more complicated to edit, patch, update etc (even though it barely ever needs to updated apart from making sure the current year’s Tax information and links are updated.) If we want to migrate to a new version of the database or newer version of rails, things are not isolated and can overlap. Now, RVM and other mechanisms can be used to do this, but by using Docker to isolate components and Amazon ECS to deploy the app, we can rapidly develop, and push changes into production in a fraction of the time we used to. This blog post will go through the experience of migrating that “Monolithic” ruby on rails app and converting it into a 2 service Docker application that can be deployed to Amazon ECS using Docker Compose.

First things first:

The first thing I did was ssh into my EC2 instance and figure out what dependencies I had. The following approach was taken to figure this out:

  • Use lsb_release -a to see what OS and release we’re using.
  • Look at installed packages via “dpkg” / “apt
  • Look at installed Gems used in the Ruby on Rails app, take a peek at Gemfile.lock for this.
  • View the history of commands via “history” see any voodoo magic I may have done and forgot about 🙂
  • View running processes via “ps [options]” which helped me remember what all is running for this app. E.g. Apache2, MySQL or Postgres, etc.

This gives us a bare minimum of what we need to think about for breaking our small monolith into separate services, at-least from a main “component” viewpoint (e.g. Database and Rails App). It’s more complicated to try and figure out how we can carve up the actual Rails / Ruby code to implement smaller services that make up the site, and in some cases this is where we can go wrong, if it works, don’t break it. Other times, go ahead, break out smaller services and deploy them, but start small, think of it like the process of an amoeba splitting 🙂

We can now move into thinking about the “design” of the app as it applies to microservices and docker containers. Read the next section for more details.

Playing with Legos:

As it was, we had apache server, rails, and mysql all running in the same VM. To move this into an architecture which uses containers, we need to separate some of these services into separate building blocks or “legos” which you could think of as connecting individual legos together to build a single service or app. SOA terms calls this “Composite Apps” which are similar in thinking, less in technology. We’ll keep this simple as stated above and we’re going to break our app into 2 pieces, a MySQL database container and a “Ruby on Rails” container running our app.

Database Container

First, we’ll take a look at how we connect the database with rails. typically in Rails, a connection to a database is configured in a database.yml file and looks something like this. (or something like this)

    development:
        adapter: mysql
        database: AppName_development
        username: root
        password:
        host: localhost
    test:
        adapter: mysql
        database: AppName_test
        username: root
        password:
        host: localhost
    production:
        adapter: mysql
        database: AppName_production
        username: root
        password:
        host: localhost

Now, since we can deploy a MySQL container with Docker using something like the following:

docker run -d --name appname_db -e MYSQL_ROOT_PASSWORD=<password> mysql

We need a way to let our application know where this container lives (ip address) and how to connect to it (tcp?, username/password, etc). We’re in luck, with docker we can use the –link (read here for more information on how –link works) flag when spinning up our Rails app and this will inject some very useful environment variables into our app so we can reference them when our application starts. By assuming we will use the –link flag and we link our database container with the alias “appdb” (more on this later) we can change our database.yml file to look something like the following (Test/Prod config left out on purpose, see the rest here)

development:
 adapter: mysql2
 encoding: utf8
 reconnect: false
 database: appdb_dev
 pool: 5
 username: root
 password: <%= ENV['APPDB_ENV_MYSQL_ROOT_PASSWORD'] %>
 host: <%= ENV['APPDB_PORT_3306_TCP_ADDR'] %>
 port: <%= ENV['APPDB_PORT_3306_TCP_PORT'] %>
Rails Container

For the ruby on rails container we can now deploy it making sure we adhere to our dynamic database information like this. Notice how we “link” this container to our “appname_db” container we ran above.

docker run -d --link appname_db:appdb -p 80:3000 --name appname wallnerryan/appname

We also map a port to 3000 because we’re running our ruby on rails application using rails server on port 3000.

Wait, let’s back track and see what we did to “containerize” our ruby on rails app. I had to show the database and configuration first so that once you see how it’s deployed it all connects, but now let’s focus on what’s actually running in the rails container now that we know it will connect to our database.

The first thing we needed to do to containerize our rails container was to create an image based on the rails implementation we had developed for ruby 1.8.7 and rails 3.2.8. These are fairly older version of the two and we had deployed on EC2 on ubuntu, so in the future we can try and use the Ruby base image but instead we will use the ubuntu:12.04 base image because this is the path of least resistance, even though we can reduce our total image size with the prior. (more about squashing our image size later in the post)

Doing this we can create Dockerfile that looks like the following. To see the code, look here We actually don’t need all these packages (I dont think) but I haven’t got around to reducing this to bare minimum by removing one by one and seeing what breaks. (This is actually easy and a fun way to get your container just right because we’re using docker, and things build and run so quickly)

Screen Shot 2015-10-15 at 5.04.06 PM

As you can see we use “FROM ubuntu:12.04” to denote the base image and then we continue to install packages, COPY the app, make sure “bundler” is installed and install using the bundler the dependencies. After this we set the RAILS_ENV to use “production” and rake the assets. (We cannot rake with “db:create” because the DB does not exist at docker build time, more on this in runtime dependencies) Then we throw our init script into the container, chmod it and set it as the CMD used when the container is run via Docker run. (If some of this didn’t make sense, please take the time to run over to the Docker Docs and play with Docker a little bit. )

Great, now we have a rails application container, but a little more detail on the init script and runtime dependencies before we run this. See the below section.

Runtime dependencies:

There are a few runtime dependencies we need to be aware of when running the app in this manner, the first is that we cannot “rake db:create” until we know the database is actually running and can be connected to. So in order to make this happen at runtime we place this inside the init script.

The other portion of the runtime dependencies is to make sure that “rake db:create” does not fire off before the database is initialized and ready to use. We will use Docker Compose to deploy this app and while compose allows us to supply dependencies in the form of links there no real control over this if A) they aren’t linked, and B) if there is a time sequence needed. In this case it takes the MySQL container about 10 seconds to initialize so we need to put a “sleep 15” in our init script before firing off “rake db:create” and then running the server.

In the below script you can see how this is implemented.

Screen Shot 2015-10-15 at 5.14.19 PM

Nothing special, but this ensures our app runs smoothly every time.

Running the application

We can run the app a few different ways, below we can see via Docker CLI and via Docker Compose.

Docker CLI
docker run -d --name appname_db -e MYSQL_ROOT_PASSWORD=root mysql
docker run -d --link appname_db:appdb -p 3000:3000 --name appname app_image
Docker Compose

With compose we can create a docker-comose.yml like the following.

app:
 image: wallnerryan/app
 cpu_shares: 100
 mem_limit: 117964800
 ports:
 - "80:3000"
 links:
 - app_mysql:appdb

app_mysql:
 image: mysql
 cpu_shares: 100
 mem_limit: 117964800
 environment:
 MYSQL_ROOT_PASSWORD: XXXXXX

Then run “docker compose up

Running the app in ECS and moving DNS:

We can run this in Amazon ECS (Elastic Container Service) as well, using the same images and docker compose file we just created. If you’re unfamiliar with EC2 or the Container Service, check out the getting started guide.

First you will need the ecs cli installed, and the first command will be to setup the credentials and the ECS cluster name.

ecs-cli configure --region us-east-1 --access-key <XXXXX> --secret-key <XXXXXXXXX> --cluster ecs-cli-dev

Next, you will want to create the cluster. We only create a cluster of size=1 because we don’t need multiple nodes for failover and we aren’t running a load balancer for scale in this example, but these are all very good ideas to implement for your actual microservice application in production so you do not need to update your domain to point to different ECS cluster instances when your microservices application moves around.

ecs-cli up --keypair keypair-name --capability-iam --size 1 --instance-type t2.micro

After this, we can send our docker compose YAML file to ecs-cli to deploy our app.

ecs-cli compose --file docker-compose.yml up

To see the running app, run the following command.

ecs-cli ps

NOTE: When migrating this from EC2 make sure and update the DNS Zone File of your domain name to point at the ECS Cluster Instance.  

Finally, now that the application is running

Let’s back track and squash our image size for the rails app.

There are a number of different ways we can go about shrinking our application such as a different base image, removing unneeded libraries, running apt-get remove and autoclean and a number of others. Some of these taking more effort than others, such as if we change the base image, we would need to make sure our Dockerfile still installs the needed version of gems, and we can alternatively use a ruby base image but the ones I looked at don’t go back to 1.8.7.

The method we use as a “quick squash” is to export, and import the docker image and re-label it, this will squash out images into one single image and re-import the image.

docker export 7c7e6a6fff3b | docker import - wallnerryan/appname:imported

As you can see, this squashed our image down from 256MB to 184MB, not bad for something so simple. Now I can do more, but this image size for my needs is plenty small. Here is a good post from Brian DeHamer on some other things to consider when optimizing image sizes. Below you can see the snapshot of the docker image (taxmatters is the name of the company, I have been substituting this with “appname” in the examples above).

Screen Shot 2015-10-14 at 1.54.42 PM

Development workflow going forward:

So after we finished migrating to a Docker/ECS based deployment it is very easy to make changes, test in either a ECS development cluster or using Local Docker Machine, then deploy to the production closer on ECS when everything checks out. We could also imagine code changes automated in a CI pipeline where a CI pipeline kicks off lambda deployments to development after initial smoke tests triggered by git push, but we’ll leave that for “next steps” :).

Thanks for reading, cheers!

Ryan

A breakdown of layers and tools within the container and microservices ecosystem

I wrote a post not to long ago about creating a microservices architecture from scratch as part of series I am doing on modern microservices. Some colleagues and friends of mine suggested I break a portion of that post out into its own so I can continue to update it as the ecosystem grows. This is an attempt to do so. The portion they were talking about was the breakdown of layer and tools within MSA in my post here which laid the initial pass at this. This post will try and fill these layers out and and continue to add to them, there is just no way I can touch every single tool or definition correctly as I see them so please take this as my opinion based on the experience I have had in this ecosystem and please comment with additions, corrections, comments etc.

  • Applications / Frameworks / App Manifests
  • Scheduling / Scaling
  • Management Orchestration
  • Monitoring (including Health) / Logging / Auditing
  • Runtime Build/Creation (think build-packs and runtimes like rkt and docker)
  • Networking / Load Balancing
  • Service Discovery / Registration
  • Cluster Management / Distributed Systems State
  • Container OS’s
  • Data Services, Data Intelligence, and Storage Pools

To give you an idea of the tools available and technologies that fall into these categories, here is the list again, but with some of the tools and technologies in the ecosystem added. *Keep in mind this is is probably not an exhaustive list, if you see a missing layer or tool please comment!

*Note: Some of these may seem to overlap, if I put Kubernetes under Orchestration, it could easily fit into Cluster Management, or Scheduling because of its underlying technologies, however this is meant to label something with it’s overall “feel” for how to ecosystem views the tool(s), but some tools may appear in more than one section. I will labels these (overlap)

*Note: I will continue to add links as I continue to update the breakdown*

Again, if you see a missing layer or tool (which I’m sure I am) please comment!

Cheers.

Microservices: An architecture from scratch using Docker, Swarm, Compose, Consul, Facter and Flocker

compose consul docker flocker puppet swarm

You might be asking yourself about the information going around about microservices, containers, and the many different tools around building a flexible architecture or “made-of-many-parts” architecture. Well you’re not alone, and there are many tools out there helping (or confusing) you to do so. In this post I’ll talk about some of the different options available like Mesos, Docker Engine, Docker Swarm, Consul, Plugins and more out there. The various different layers involved in a modern microservices architecture have various responsibilities and deciding how you can go about choosing the right pieces to build out those layers can be tough. This post is by far not the only way you can put the layers together and in fact this is MY opinion on the subject given the experience I have had in the ecosystem, it also does not reflect ideas of my employer.

Typically I define modern microservices architecture as having the following layers and responsibilities:

  • Applications / Frameworks / App Manifests
  • Scheduling
  • Orchestration
  • Monitoring / Logging / Auditing
  • Service Discovery
  • Cluster Management / Distributed Systems State
  • Data Services and Intelligence

To give you an idea of the tools available and technologies that fall into these categories, here is the list again, but with some of the projects, products and technologies in the ecosystem added. keep in mind this is not an exhaustive list.

*UPDATE: here is separate post aimed at making this a more exhaustive list

So lets choose a few components, we choose components apart from networking which we leave out here and just use host-only networking with vagrant, but we could add in libnetwork support in Docker directly. For logging and monitoring, we also just spin up DockerUI for this but could also add in loggly, fluentd, sysdig and others.

  • Consul – Service Discovery, DNS, K/V Storage
  • Docker Engine (Runtime)
  • Docker Swarm (Scheduler / Cluster / Distributed State )
  • Docker Compose (Orchestration)
  • Docker Plugins (Volume integration)
  • Flocker (Data Services / Orchestration)

So what do these layers look like all together? We can represent the layers I mentioned above the the following way, including the applications as a logical mapping to the containers running programs and processes above.

microservicesblock

Above, we can put together a microservices architecture with the tools defined above, atop of this we can create applications from manifests and schedule containers onto the architecture once this is all running. I want to pinpoint a few specific areas in this architecture because we can add some extra logic to this to make things a little more interesting.

Service Discovery and Registration:

The registration layer can serve many purposes, it is mostly used to register and allow discovery of services (microservices/containers) that are running on a system/cluster. We can use consul to do this type of registration within its key / value mechanisms. You can use Consul’s built-in service mechanism or there are other ways to talk to consul key/value like registrator. In our example we can use the registry layer for something a little more interesting, in this case we can use consul’s locking mechanisms to lock resources we put in them, allowing schedulers to tap into the registry layer instead of talking to every node in the cluster for updates on CPU, Memory etc.

We can add resource updating scripts to our consul services by adding a service to consul’s service mechanism, these services will import keys and values from Facter and other resources then upload them to the K/V store, consul will also health check these services for us as a added benefit.

Below we can see how Consul registers services, in this case we register an “update service” which updates system resources into the registry layer.

consul-service

We now have the ability to add many system resources, but we also have the flexibility to upload custom resources (facts) like system overload and memory swap free, the below “fact” is a sample of how we can do so giving us a system overload.

custom_fact

Doing so, we can enable the swarm scheduler to use them for scheduling containers. Swarm does not do this today, instead it has only static labels added when the docker engine is started, in this example our swarm scheduler utilizes dynamic labels by getting up-to-date realtime labels that mean something to the system which allows us to schedule containers a little better. What this looks like in swarm is below. See “how to use” section below for how this actually gets used.

swarm info

Data Services Layer:

Docker also allows us to plug into its ecosystem for volumes with volume plugins which enable containers to add data services like RexRay and Flocker. This part of the ecosystem is rapidly expanding and today we can provision fairly basic volumes with a size, and basic attributes. Docker 1.9 has a volume API which introduces options (opts) for more advanced features and metadata passed to data systems. As you will see in the usage examples below, this helps with more interesting workflows for the developer, tester, etc. As a note, the ecosystem around containers will continue to grow fast, use cases around different types of applications with more data needs will help drive this part of ecosystem.

If your wondering what the block diagram may actually looks like on a per node (server/vm) basis, with what services installed where etc, look no more, see below.

ms1

From the above picture, you should get a good idea of what tools sit where, and where they need to be installed. First, every server participating in the cluster needs a Swarm Agent, Flocker-Dataset-Agent, Docker Plugin for Docker, A Consul server (or agent depending how big the cluster, at least 3 servers), and a Docker Engine. The components we talked about above like the custom resource registration has custom facter facts on each node as well so consul and facter can import them appropriately. Now, setting this up by hand is sure to be a pain in the a** if your cluster is large, so in reality we should think about the DevOps pipeline and the roll of puppet, or chef to automate the deployment of a lot of this. For my example I packed everything into a Vagrantfile and vagrant shell scripts to do the install and configuration, so a simple “vagrant up” would do, given I have about 20-30 minutes to watch the cluster come up 🙂

How do I use it!!?

Okay, lets get to actually using this microservices cluster now that we have it all set. This section of the blog post should give you an idea of the use cases and types of applications you can deploy to your microservices architecture and what tooling to use given then above examples and layers we introduced.

Using the Docker CLI with Swarm to schedule new resources via constraints and volume profiles.

This will look for specific load between 0-25% because we have a custom registration layer. *(note, some of the profiles work was in collaboration Mahuri from CHQ and Sean Dell for the Docker Global Hackday #3)

docker run -d -e constraint:system_overload_10min==/[0-2][0-5]/ -e constraint:architecture==x86_64 -e constraint:virtual=virtualbox -e constraint:selinux_enforced=false -v myVol@gold:/data/ redis

Using Docker Compose with Flocker volume driver that supports storage profiles:

This will schedule a redis database container using the flocker volume driver with a “gold” volume, meaning we will get a better IOPS, Bandwidth and other “features” that are considered of more performance and value.

Screen Shot 2015-10-01 at 2.24.40 PM

Using Docker compose with swarm to guarantee a server has a specific volume driver and to schedule a container to a server with a specific CPU Overload:

In this example we also use the overload percentage resource in the scheduler but we also take advantage of the registry layer knowing which nodes are running certain volume plugins and we can schedule to Swarm knowing we will get nodes with a specific driver and hypervisor making sure that it will support the profile we want.

Screen Shot 2015-10-01 at 2.25.34 PM

A use case where a developer wants to schedule to specific resources, start a web service, snapshot an entire database container and its data and view that data all using the Docker CLI.

This example shows how providing the right infrastructure tooling to the Docker CLI and tools allows for a more seamless developer / test workflow. This example allows us to get everything we need via the docker CLI while never leaving our one terminal to run commands on a storage system or separate node.

Start MySQL Container with constraints:

docker run -ti -e constraint:is_virtual==True -e constraint:system_overload_10min==/[0-2][0-5]/ -v demoVol@gold:/var/lib/mysql --volume-driver=flocker -p 3306:3306 --name MySQL wallnerryan/mysql
Start a simple TODO List application:

docker run -it -e constraint:is_virtual==True --rm -e DATABASE_IP=192.168.50.13 -e DATABASE=mysql -p 8005:8080 wallnerryan/todolist
We want to snapshot our original MySQL database, so lets pull the dataset-ID from its mount point and input that into the snapshot profile.

docker inspect --format='{{.Mounts}}' MySQL
[{demoVol@gold /flocker/f23ca986-cb43-43c1-865c-b57b1023ab7e /var/lib/mysql flocker z true}]
Here we place a substring of the the Dataset ID into the snapshot profile in our MySQL-snap container creation, this will create a second MySQL container with a snapshot of production data.

docker run -ti -e affinity:container==MySQL -v demoVol-snap@snapshot-f23ca986:/var/lib/mysql --volume-driver=flocker -p 3307:3306 --name MySQL-snap wallnerryan/mysql

At this point we have pretty much been able to snapshot the entire MySQL application (container and data) and all of its data at a point in time while scheduling the point in time data and container to a specific node in one terminal and few CLI commands where our other MySQL database was running. In the future we could see the option to have dev/test clusters and have the ability to schedule different operations and workflows across shared production data in a microservices architecture that would help streamline teams in an organization.

Cheers, Thanks for reading!