Container Data Management and Production Use-cases

voting-logo

In my last few jobs I have had the pleasure to work on three main focus areas, Software Defined Networking, OpenStack, and Linux Containers. More recently I have been focused on container data management and what it means for persistence and data management to be a first class citizen to containerized applications and microservices. This has done a few things for me, give me an opportunity to work on interesting (and hard) problems, hack, create and apply new solutions and technologies into proof of concept and production environments. In my current role as a Technical Evangelist at ClusterHQ we have been hard at work continuing the Flocker project, creating the Volume Hub and spinning out dvol (git workflows for Docker volumes / data). All of this is aimed at one major thing, helping your team move from development on your laptop to test and Q/A and finally into production seamlessly with persistence.

We’re just at the beginning of the container and data revolution and if your interested in learning more about some of these topics, click the links below and vote for some upcoming talks at OpenStack Austin 2016.

Three Critical Concepts for Containerized Storage Management

https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/8506

Lessons learned running database containers on OpenStack

https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/8501

A special shoutout to Andrew Sullivan and Sumit Kumar for their CFP on “Data Mobility for Docker containers with Flocker”! Please help them spread the word on stateful containers by voting for their talk as well! https://www.openstack.org/summit/austin-2016/vote-for-speakers/presentation/7859

Cheers!

Ryan – @RyanWallner

Migrating the monolith from EC2 to an ECS-based multi-service Docker app

  

On my spare time I run a website for a tax accounting company. This monolithic app is a largely stateless app (not really but we don’t need persistent stores/volumes), uses ruby on rails and runs on an EC2 instance with Apache and MySQL Server and Client installed. This is what is referenced as a “monolithic app” because all components are installed on 1 single VM. This makes the application more complicated to edit, patch, update etc (even though it barely ever needs to updated apart from making sure the current year’s Tax information and links are updated.) If we want to migrate to a new version of the database or newer version of rails, things are not isolated and can overlap. Now, RVM and other mechanisms can be used to do this, but by using Docker to isolate components and Amazon ECS to deploy the app, we can rapidly develop, and push changes into production in a fraction of the time we used to. This blog post will go through the experience of migrating that “Monolithic” ruby on rails app and converting it into a 2 service Docker application that can be deployed to Amazon ECS using Docker Compose.

First things first:

The first thing I did was ssh into my EC2 instance and figure out what dependencies I had. The following approach was taken to figure this out:

  • Use lsb_release -a to see what OS and release we’re using.
  • Look at installed packages via “dpkg” / “apt
  • Look at installed Gems used in the Ruby on Rails app, take a peek at Gemfile.lock for this.
  • View the history of commands via “history” see any voodoo magic I may have done and forgot about🙂
  • View running processes via “ps [options]” which helped me remember what all is running for this app. E.g. Apache2, MySQL or Postgres, etc.

This gives us a bare minimum of what we need to think about for breaking our small monolith into separate services, at-least from a main “component” viewpoint (e.g. Database and Rails App). It’s more complicated to try and figure out how we can carve up the actual Rails / Ruby code to implement smaller services that make up the site, and in some cases this is where we can go wrong, if it works, don’t break it. Other times, go ahead, break out smaller services and deploy them, but start small, think of it like the process of an amoeba splitting🙂

We can now move into thinking about the “design” of the app as it applies to microservices and docker containers. Read the next section for more details.

Playing with Legos:

As it was, we had apache server, rails, and mysql all running in the same VM. To move this into an architecture which uses containers, we need to separate some of these services into separate building blocks or “legos” which you could think of as connecting individual legos together to build a single service or app. SOA terms calls this “Composite Apps” which are similar in thinking, less in technology. We’ll keep this simple as stated above and we’re going to break our app into 2 pieces, a MySQL database container and a “Ruby on Rails” container running our app.

Database Container

First, we’ll take a look at how we connect the database with rails. typically in Rails, a connection to a database is configured in a database.yml file and looks something like this. (or something like this)

    development:
        adapter: mysql
        database: AppName_development
        username: root
        password:
        host: localhost
    test:
        adapter: mysql
        database: AppName_test
        username: root
        password:
        host: localhost
    production:
        adapter: mysql
        database: AppName_production
        username: root
        password:
        host: localhost

Now, since we can deploy a MySQL container with Docker using something like the following:

docker run -d --name appname_db -e MYSQL_ROOT_PASSWORD=<password> mysql

We need a way to let our application know where this container lives (ip address) and how to connect to it (tcp?, username/password, etc). We’re in luck, with docker we can use the –link (read here for more information on how –link works) flag when spinning up our Rails app and this will inject some very useful environment variables into our app so we can reference them when our application starts. By assuming we will use the –link flag and we link our database container with the alias “appdb” (more on this later) we can change our database.yml file to look something like the following (Test/Prod config left out on purpose, see the rest here)

development:
 adapter: mysql2
 encoding: utf8
 reconnect: false
 database: appdb_dev
 pool: 5
 username: root
 password: <%= ENV['APPDB_ENV_MYSQL_ROOT_PASSWORD'] %>
 host: <%= ENV['APPDB_PORT_3306_TCP_ADDR'] %>
 port: <%= ENV['APPDB_PORT_3306_TCP_PORT'] %>
Rails Container

For the ruby on rails container we can now deploy it making sure we adhere to our dynamic database information like this. Notice how we “link” this container to our “appname_db” container we ran above.

docker run -d --link appname_db:appdb -p 80:3000 --name appname wallnerryan/appname

We also map a port to 3000 because we’re running our ruby on rails application using rails server on port 3000.

Wait, let’s back track and see what we did to “containerize” our ruby on rails app. I had to show the database and configuration first so that once you see how it’s deployed it all connects, but now let’s focus on what’s actually running in the rails container now that we know it will connect to our database.

The first thing we needed to do to containerize our rails container was to create an image based on the rails implementation we had developed for ruby 1.8.7 and rails 3.2.8. These are fairly older version of the two and we had deployed on EC2 on ubuntu, so in the future we can try and use the Ruby base image but instead we will use the ubuntu:12.04 base image because this is the path of least resistance, even though we can reduce our total image size with the prior. (more about squashing our image size later in the post)

Doing this we can create Dockerfile that looks like the following. To see the code, look here We actually don’t need all these packages (I dont think) but I haven’t got around to reducing this to bare minimum by removing one by one and seeing what breaks. (This is actually easy and a fun way to get your container just right because we’re using docker, and things build and run so quickly)

Screen Shot 2015-10-15 at 5.04.06 PM

As you can see we use “FROM ubuntu:12.04” to denote the base image and then we continue to install packages, COPY the app, make sure “bundler” is installed and install using the bundler the dependencies. After this we set the RAILS_ENV to use “production” and rake the assets. (We cannot rake with “db:create” because the DB does not exist at docker build time, more on this in runtime dependencies) Then we throw our init script into the container, chmod it and set it as the CMD used when the container is run via Docker run. (If some of this didn’t make sense, please take the time to run over to the Docker Docs and play with Docker a little bit. )

Great, now we have a rails application container, but a little more detail on the init script and runtime dependencies before we run this. See the below section.

Runtime dependencies:

There are a few runtime dependencies we need to be aware of when running the app in this manner, the first is that we cannot “rake db:create” until we know the database is actually running and can be connected to. So in order to make this happen at runtime we place this inside the init script.

The other portion of the runtime dependencies is to make sure that “rake db:create” does not fire off before the database is initialized and ready to use. We will use Docker Compose to deploy this app and while compose allows us to supply dependencies in the form of links there no real control over this if A) they aren’t linked, and B) if there is a time sequence needed. In this case it takes the MySQL container about 10 seconds to initialize so we need to put a “sleep 15” in our init script before firing off “rake db:create” and then running the server.

In the below script you can see how this is implemented.

Screen Shot 2015-10-15 at 5.14.19 PM

Nothing special, but this ensures our app runs smoothly every time.

Running the application

We can run the app a few different ways, below we can see via Docker CLI and via Docker Compose.

Docker CLI
docker run -d --name appname_db -e MYSQL_ROOT_PASSWORD=root mysql
docker run -d --link appname_db:appdb -p 3000:3000 --name appname app_image
Docker Compose

With compose we can create a docker-comose.yml like the following.

app:
 image: wallnerryan/app
 cpu_shares: 100
 mem_limit: 117964800
 ports:
 - "80:3000"
 links:
 - app_mysql:appdb

app_mysql:
 image: mysql
 cpu_shares: 100
 mem_limit: 117964800
 environment:
 MYSQL_ROOT_PASSWORD: XXXXXX

Then run “docker compose up

Running the app in ECS and moving DNS:

We can run this in Amazon ECS (Elastic Container Service) as well, using the same images and docker compose file we just created. If you’re unfamiliar with EC2 or the Container Service, check out the getting started guide.

First you will need the ecs cli installed, and the first command will be to setup the credentials and the ECS cluster name.

ecs-cli configure --region us-east-1 --access-key <XXXXX> --secret-key <XXXXXXXXX> --cluster ecs-cli-dev

Next, you will want to create the cluster. We only create a cluster of size=1 because we don’t need multiple nodes for failover and we aren’t running a load balancer for scale in this example, but these are all very good ideas to implement for your actual microservice application in production so you do not need to update your domain to point to different ECS cluster instances when your microservices application moves around.

ecs-cli up --keypair keypair-name --capability-iam --size 1 --instance-type t2.micro

After this, we can send our docker compose YAML file to ecs-cli to deploy our app.

ecs-cli compose --file docker-compose.yml up

To see the running app, run the following command.

ecs-cli ps

NOTE: When migrating this from EC2 make sure and update the DNS Zone File of your domain name to point at the ECS Cluster Instance.  

Finally, now that the application is running

Let’s back track and squash our image size for the rails app.

There are a number of different ways we can go about shrinking our application such as a different base image, removing unneeded libraries, running apt-get remove and autoclean and a number of others. Some of these taking more effort than others, such as if we change the base image, we would need to make sure our Dockerfile still installs the needed version of gems, and we can alternatively use a ruby base image but the ones I looked at don’t go back to 1.8.7.

The method we use as a “quick squash” is to export, and import the docker image and re-label it, this will squash out images into one single image and re-import the image.

docker export 7c7e6a6fff3b | docker import - wallnerryan/appname:imported

As you can see, this squashed our image down from 256MB to 184MB, not bad for something so simple. Now I can do more, but this image size for my needs is plenty small. Here is a good post from Brian DeHamer on some other things to consider when optimizing image sizes. Below you can see the snapshot of the docker image (taxmatters is the name of the company, I have been substituting this with “appname” in the examples above).

Screen Shot 2015-10-14 at 1.54.42 PM

Development workflow going forward:

So after we finished migrating to a Docker/ECS based deployment it is very easy to make changes, test in either a ECS development cluster or using Local Docker Machine, then deploy to the production closer on ECS when everything checks out. We could also imagine code changes automated in a CI pipeline where a CI pipeline kicks off lambda deployments to development after initial smoke tests triggered by git push, but we’ll leave that for “next steps”🙂.

Thanks for reading, cheers!

Ryan

A breakdown of layers and tools within the container and microservices ecosystem

I wrote a post not to long ago about creating a microservices architecture from scratch as part of series I am doing on modern microservices. Some colleagues and friends of mine suggested I break a portion of that post out into its own so I can continue to update it as the ecosystem grows. This is an attempt to do so. The portion they were talking about was the breakdown of layer and tools within MSA in my post here which laid the initial pass at this. This post will try and fill these layers out and and continue to add to them, there is just no way I can touch every single tool or definition correctly as I see them so please take this as my opinion based on the experience I have had in this ecosystem and please comment with additions, corrections, comments etc.

  • Applications / Frameworks / App Manifests
  • Scheduling / Scaling
  • Management Orchestration
  • Monitoring (including Health) / Logging / Auditing
  • Runtime Build/Creation (think build-packs and runtimes like rkt and docker)
  • Networking / Load Balancing
  • Service Discovery / Registration
  • Cluster Management / Distributed Systems State
  • Container OS’s
  • Data Services, Data Intelligence, and Storage Pools

To give you an idea of the tools available and technologies that fall into these categories, here is the list again, but with some of the tools and technologies in the ecosystem added. *Keep in mind this is is probably not an exhaustive list, if you see a missing layer or tool please comment!

*Note: Some of these may seem to overlap, if I put Kubernetes under Orchestration, it could easily fit into Cluster Management, or Scheduling because of its underlying technologies, however this is meant to label something with it’s overall “feel” for how to ecosystem views the tool(s), but some tools may appear in more than one section. I will labels these (overlap)

*Note: I will continue to add links as I continue to update the breakdown*

Again, if you see a missing layer or tool (which I’m sure I am) please comment!

Cheers.

Microservices: An architecture from scratch using Docker, Swarm, Compose, Consul, Facter and Flocker

compose consul docker flocker puppet swarm

You might be asking yourself about the information going around about microservices, containers, and the many different tools around building a flexible architecture or “made-of-many-parts” architecture. Well you’re not alone, and there are many tools out there helping (or confusing) you to do so. In this post I’ll talk about some of the different options available like Mesos, Docker Engine, Docker Swarm, Consul, Plugins and more out there. The various different layers involved in a modern microservices architecture have various responsibilities and deciding how you can go about choosing the right pieces to build out those layers can be tough. This post is by far not the only way you can put the layers together and in fact this is MY opinion on the subject given the experience I have had in the ecosystem, it also does not reflect ideas of my employer.

Typically I define modern microservices architecture as having the following layers and responsibilities:

  • Applications / Frameworks / App Manifests
  • Scheduling
  • Orchestration
  • Monitoring / Logging / Auditing
  • Service Discovery
  • Cluster Management / Distributed Systems State
  • Data Services and Intelligence

To give you an idea of the tools available and technologies that fall into these categories, here is the list again, but with some of the projects, products and technologies in the ecosystem added. keep in mind this is not an exhaustive list.

*UPDATE: here is separate post aimed at making this a more exhaustive list

So lets choose a few components, we choose components apart from networking which we leave out here and just use host-only networking with vagrant, but we could add in libnetwork support in Docker directly. For logging and monitoring, we also just spin up DockerUI for this but could also add in loggly, fluentd, sysdig and others.

  • Consul – Service Discovery, DNS, K/V Storage
  • Docker Engine (Runtime)
  • Docker Swarm (Scheduler / Cluster / Distributed State )
  • Docker Compose (Orchestration)
  • Docker Plugins (Volume integration)
  • Flocker (Data Services / Orchestration)

So what do these layers look like all together? We can represent the layers I mentioned above the the following way, including the applications as a logical mapping to the containers running programs and processes above.

microservicesblock

Above, we can put together a microservices architecture with the tools defined above, atop of this we can create applications from manifests and schedule containers onto the architecture once this is all running. I want to pinpoint a few specific areas in this architecture because we can add some extra logic to this to make things a little more interesting.

Service Discovery and Registration:

The registration layer can serve many purposes, it is mostly used to register and allow discovery of services (microservices/containers) that are running on a system/cluster. We can use consul to do this type of registration within its key / value mechanisms. You can use Consul’s built-in service mechanism or there are other ways to talk to consul key/value like registrator. In our example we can use the registry layer for something a little more interesting, in this case we can use consul’s locking mechanisms to lock resources we put in them, allowing schedulers to tap into the registry layer instead of talking to every node in the cluster for updates on CPU, Memory etc.

We can add resource updating scripts to our consul services by adding a service to consul’s service mechanism, these services will import keys and values from Facter and other resources then upload them to the K/V store, consul will also health check these services for us as a added benefit.

Below we can see how Consul registers services, in this case we register an “update service” which updates system resources into the registry layer.

consul-service

We now have the ability to add many system resources, but we also have the flexibility to upload custom resources (facts) like system overload and memory swap free, the below “fact” is a sample of how we can do so giving us a system overload.

custom_fact

Doing so, we can enable the swarm scheduler to use them for scheduling containers. Swarm does not do this today, instead it has only static labels added when the docker engine is started, in this example our swarm scheduler utilizes dynamic labels by getting up-to-date realtime labels that mean something to the system which allows us to schedule containers a little better. What this looks like in swarm is below. See “how to use” section below for how this actually gets used.

swarm info

Data Services Layer:

Docker also allows us to plug into its ecosystem for volumes with volume plugins which enable containers to add data services like RexRay and Flocker. This part of the ecosystem is rapidly expanding and today we can provision fairly basic volumes with a size, and basic attributes. Docker 1.9 has a volume API which introduces options (opts) for more advanced features and metadata passed to data systems. As you will see in the usage examples below, this helps with more interesting workflows for the developer, tester, etc. As a note, the ecosystem around containers will continue to grow fast, use cases around different types of applications with more data needs will help drive this part of ecosystem.

If your wondering what the block diagram may actually looks like on a per node (server/vm) basis, with what services installed where etc, look no more, see below.

ms1

From the above picture, you should get a good idea of what tools sit where, and where they need to be installed. First, every server participating in the cluster needs a Swarm Agent, Flocker-Dataset-Agent, Docker Plugin for Docker, A Consul server (or agent depending how big the cluster, at least 3 servers), and a Docker Engine. The components we talked about above like the custom resource registration has custom facter facts on each node as well so consul and facter can import them appropriately. Now, setting this up by hand is sure to be a pain in the a** if your cluster is large, so in reality we should think about the DevOps pipeline and the roll of puppet, or chef to automate the deployment of a lot of this. For my example I packed everything into a Vagrantfile and vagrant shell scripts to do the install and configuration, so a simple “vagrant up” would do, given I have about 20-30 minutes to watch the cluster come up🙂

How do I use it!!?

Okay, lets get to actually using this microservices cluster now that we have it all set. This section of the blog post should give you an idea of the use cases and types of applications you can deploy to your microservices architecture and what tooling to use given then above examples and layers we introduced.

Using the Docker CLI with Swarm to schedule new resources via constraints and volume profiles.

This will look for specific load between 0-25% because we have a custom registration layer. *(note, some of the profiles work was in collaboration Mahuri from CHQ and Sean Dell for the Docker Global Hackday #3)

docker run -d -e constraint:system_overload_10min==/[0-2][0-5]/ -e constraint:architecture==x86_64 -e constraint:virtual=virtualbox -e constraint:selinux_enforced=false -v myVol@gold:/data/ redis

Using Docker Compose with Flocker volume driver that supports storage profiles:

This will schedule a redis database container using the flocker volume driver with a “gold” volume, meaning we will get a better IOPS, Bandwidth and other “features” that are considered of more performance and value.

Screen Shot 2015-10-01 at 2.24.40 PM

Using Docker compose with swarm to guarantee a server has a specific volume driver and to schedule a container to a server with a specific CPU Overload:

In this example we also use the overload percentage resource in the scheduler but we also take advantage of the registry layer knowing which nodes are running certain volume plugins and we can schedule to Swarm knowing we will get nodes with a specific driver and hypervisor making sure that it will support the profile we want.

Screen Shot 2015-10-01 at 2.25.34 PM

A use case where a developer wants to schedule to specific resources, start a web service, snapshot an entire database container and its data and view that data all using the Docker CLI.

This example shows how providing the right infrastructure tooling to the Docker CLI and tools allows for a more seamless developer / test workflow. This example allows us to get everything we need via the docker CLI while never leaving our one terminal to run commands on a storage system or separate node.

Start MySQL Container with constraints:

docker run -ti -e constraint:is_virtual==True -e constraint:system_overload_10min==/[0-2][0-5]/ -v demoVol@gold:/var/lib/mysql --volume-driver=flocker -p 3306:3306 --name MySQL wallnerryan/mysql
Start a simple TODO List application:

docker run -it -e constraint:is_virtual==True --rm -e DATABASE_IP=192.168.50.13 -e DATABASE=mysql -p 8005:8080 wallnerryan/todolist
We want to snapshot our original MySQL database, so lets pull the dataset-ID from its mount point and input that into the snapshot profile.

docker inspect --format='{{.Mounts}}' MySQL
[{demoVol@gold /flocker/f23ca986-cb43-43c1-865c-b57b1023ab7e /var/lib/mysql flocker z true}]
Here we place a substring of the the Dataset ID into the snapshot profile in our MySQL-snap container creation, this will create a second MySQL container with a snapshot of production data.

docker run -ti -e affinity:container==MySQL -v demoVol-snap@snapshot-f23ca986:/var/lib/mysql --volume-driver=flocker -p 3307:3306 --name MySQL-snap wallnerryan/mysql

At this point we have pretty much been able to snapshot the entire MySQL application (container and data) and all of its data at a point in time while scheduling the point in time data and container to a specific node in one terminal and few CLI commands where our other MySQL database was running. In the future we could see the option to have dev/test clusters and have the ability to schedule different operations and workflows across shared production data in a microservices architecture that would help streamline teams in an organization.

Cheers, Thanks for reading!

#DockerCon #DockerCon #DockerCon Production and Persistence for Containers

Screen Shot 2015-06-19 at 1.25.05 PM

It has been a crazy past few months leading up to DockerCon in San Francisco and I wanted to share some thoughts about current events and some of the work we have been participating in and around the Docker community and ecosystem now that we’re post-conference this week.

I have been working in open-source communities for more than five years now across technology domains including software defined networking, infrastructure & platform as a service, and container technologies. Working on projects in the Openflow/SDN, Openstack, and container communities has had its ups and downs but Docker is arguably the hottest tech communities out there right now.

There are so many developments going on in this ecosystem around pluggable architectures, logging, monitoring, migrations, networking and enabling stateful services in containers. Before I talk more in depth about persistence and some of the work my team and partners have been up to I want to highlight some of the major takeaways from the conference and the community right now.

The theme at @DockerCon was “docker in production” and by this I mean docker is ready to be used in production. Depending on who you ask and how docker is being used, production and containers with microservices can either be “hell” as Bryan Cantrill put it (If you haven’t seen Bryan’s Talks on the unix philosophy and production debugging, I highly recommend any of his sessions, especially the ones from the recent Orielly conference and this past weeks DockerCon) or it can really help your applications break down into their bounded domains with highly manageable and efficient teams going through the CI/CD build/ship/deploy process efficiently well. Netflix OSS [http://netflix.github.io/] is always a great example of this done well and many talks by Adrian Cockcroft dig into this sufficiently. You can also see my last post [https://aucouranton.com/2015/04/10/what-would2-microservices-do/] about microservices which will help with some context here.

So, Is Docker ready for production? Here are a list of what you need to make sure you have a handle on before productions use, and I might add that these are also big topics at DockerCon technically and there are many projects and problems still being actively solved in these spaces, I’ll try to list a few as I go through.

• Networking

Docker’s “aquirehire-sition” of the Socketplane startup culminated in the ability to use lib network [https://github.com/docker/libnetwork] which mean you can connect your docker daemons from hosts to host to allow container to easily send IP traffic over layer 2 networks. Libnetwork is maintains outside of the main docker daemon and abstracts the networking nicely and most importantly abstracts too much network knowledge away from the end user and make things “just work”

• Security

Docker is hardening the registry and images as well as the docker engine itself. I got to chat with Eric Windisch from Docker about what he had been focusing on with docker security. The docker engine’s security and hardening is at the forefront of focus because any vulnerabilities there means the rest of your container could be compromised. There is a lot of work going on around basic source hardening as well as other techniques using apparmour and selinux. Looking forward to seeing how to security aspect of Docker unfolds with other projects like Lightwave from VMware.

• Logging & Monitoring & Manageability

Containers are great, but once your running thousands to tens of thousands of them in products the need for great tooling to help debug, audit, troubleshoot and manage is a necessity. There seems to be lots of great tooling coming out to help manage containers. First docker talked about “Project Orca”, an opinionated container stack that aims at combining Docker Engine, Docker Swarm, Networking, GUI, Docker Compose, security, plus tools for installation, deployment, configuration. This of course isn’t always the way docker will be run but would be nice to have a way to get all this up and running quickly with manageability. Other tools like loggly, cadvisor, ruxit, datadog, log entries and others are all competing to be the best options here and quite frankly thats great!

• Pluggability

Docker has given the power to its ecosystem by telling it wants a pluggable, extensible toolset that allows for different plugins to work with their network, auth, and storage stacks. This provides a way similar to openstack for customers and users to say plugin an openvswitch network driver, along with lightwave for auth and EMC ScaleIO for persistent storage. Pretty cool stuff considering docker is only just over 2 years old!

• Stateful / Persistant services

This last bullet here is near and dear to my employer EMC and we have done some really awesome work by partnering up with ClusterHQ (The Data Container Poeple) [https://clusterhq.com/] who’s open-source Flocker project can manage volumes for your containers and enables mobility and HA for those volumes when you want to go ahead and start moving around or recovering your containerized applications, really cools stuff.

Docker’s announcement of native volume extensions/plugins for Docker https://github.com/docker/docker/blob/master/experimental/plugins_volume.md was a popular one at DockerCon. Even though its experimental in the 1.7.0 releases the pluggable nature to persistence allows different options for managing your stateful apps.

I mentioned before the we partnered up with ClusterHQ to delivers some drivers that work with their Flocker [https://github.com/ClusterHQ/flocker]. Flocker itself can also work with docker’s new volume extensions with the —volume-driver=flocker flag if you use the Flocker-Docker Plugin [https://github.com/ClusterHQ/flocker-docker-plugin]

EMC integrated and open-sourced 2 drivers (below), some more information on the partnership can be found here
https://clusterhq.com/2015/06/17/emc-partnership/

EMC ScaleIOhttps://github.com/emccorp/scaleio-flocker-driver
EMC XtremIO https://github.com/emccorp/xtremio-flocker-driver

Below are some videos showing the integrations.

ScaleIO
https://www.youtube.com/watch?v=ufSbF0-pk_Q&list=PLbssOJyyvHuWiBQAg9EFWH570timj2fxt&index=8

XtremIO
https://www.youtube.com/watch?v=QKwNWTEE6hA&list=PLbssOJyyvHuWiBQAg9EFWH570timj2fxt&index=7

 

We had the opportunity to host a meetup at Pivotal Labs in San Francisco to showcase the partnership and drivers, at the Pivotal labs office we had a number of people come out for a few live demos, some beer and great food and conversation. Here is a gist for the ScaleIO demo we ran at the meetup showcasing Flocker + ScaleIO running on Amazon AWS deploying a MEAN stack application that ingested twitter data and placed it into MongoDB using node and express.
https://gist.github.com/wallnerryan/7ccc5455840b76c07a70

The slides from the meetup are also available http://slides.com/ryanwallner/persistence-docker-chq-emc/live#/

Pivotal Meetup

At DockerCon Clint Kitson and I along with some ECS folks had a packed house for our partner tutorial session showcasing the ClusterHQ, RexRay and ECS announcements at DockerCon. The room was pas standing room only and attendees started to fill the floor. We hoped to have a bit more time to let the folks with laptops actually get to hack on some of the work we did but unfortunately pressed into 40 minutes we did what we could!

EMC also integrates through a native Go implementation called RexRay, a way to manage your persistence volumes but without the auto-mobility Flocker gives. RexRay is really cool in the way its working on letting you use multiple backends at the same time, say EC2 EBS Volumes as well as EMC ScaleIO.

More information here:

RexRay

https://github.com/emccode/rexray

Video

https://www.youtube.com/watch?v=rF8Bc3HZnAU&list=PLbssOJyyvHuWiBQAg9EFWH570timj2fxt&index=11

In all, persistence and containers is a here to stay and there are a number of reason why and some items to keep and eye on. First the stateless and 12-factor app was the rage, but its not realistic and people are realizing state exists and running stateful containers like databases is actually important to the microservices world. This is because all containers have state, even if its “stateless” there is in memory state like running programs and open sockets that may need to be dealt with in certain use cases like live migration. If you haven’t seen the live movement of realtime Quake playing and moving between cities across oceans please watch, its great! (I’ll post the video once I find it)

Data is becoming a first class citizen in these containerized environments. As more workloads gets mapped to container architectures we see the need for the import and of data consistency, integrity and availability come into play for services that produce state and need it to persist. Enter enterprise storage into the mix. Over the week and weekend we saw a number of companies and announcements around this including some of our own at EMC. A few offerings that caught my eye are:

A scalable distributed database you install on your application servers

Open-source container volume manager that give you the ability to containerize databases and other stateful applications and move them around without worrying about managing your backend.

  • RedHat

Red Hat announced it has its own integration for persistent storage for container using it RHS(Red Hat Storage)

http://redhatstorage.redhat.com/2015/06/22/red-hat-drives-deeper-integration-of-persistent-storage-for-containerized-environments/

  • Nutanix

Nutanix talks about a “Volume API” for its platform that helps your provision persistence to containers on its platform.

http://itbloodpressure.com/2015/06/22/nutanix-volume-api-and-containers-dockercon/

  • Portworx

A seemingly new startup that competes with the likes of ClusterHQ and what it does. Announcement below talks about how “some” of its platform will be open-source. http://venturebeat.com/2015/06/22/container-storage-startup-portworx-puts-away-8-5m/

  • Kubernetes Support for storage

Kubernetes updated some docs on github that reflect being able to use a Google Cloud persistent disk with k8s.

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/volumes.md#gcepersistentdisk

All in all it was a great DockerCon full of fun events, great people and innovative technology. Hopefully we’ll see you out in Barcelona in November!

EMC at DockerCon – Enabling the Community!

EMC {code} - Blog

Screen Shot 2015-06-19 at 1.25.05 PMAs we prepare for DockerCon 2015 next week, we’re taking a very different approach than EMC usually takes at events. Most events are focused on outbound messaging, product announcements and a heavy dose of sales and marketing. But since EMC World 2015, it’s no longer your father’s EMC.

Next week, our focus is entirely on enablingthe community of developers and end-users that are moving towards Cloud Native applications (or “3rd Platform”) and container-based infrastructure. We fully believe that this is not just a trend, but a major shift in how businesses will interact with their customers, and that software is at the center of this revolution.

Screen Shot 2015-06-19 at 3.27.47 PM

So what does it mean to enable the community and ecosystem of developers? We’re taking a three-legged approach. The overall theme is a focus on Data Persistence and how this evolving space is critical to those Cloud Native applications.

  1. Work within the…

View original post 473 more words

What Would Microservices do!?

Image Credit for the Googling of images

Private/Public IaaS, and PaaS environments are some of the fastest moving technology domains of their kind right now. I give credit to the speed of change and adoption to the communities that surround them, open-source or within the enterprise. As someone who is in the enterprise but contributes to open-source, it is surprising to many to find out that within the enterprise there is a whole separate community around these technologies that is thriving. That being said, I have been working with os-level virtualization technologies now for the past (almost) 3 years and in the past 2 years most people are familiar with the “Docker-boom” and now with the resurgence or SoA/Microservices I think its worth while exploring what modern technologies are involved, what drivers for change and how it affects applications and your business alike.

Containers and Microservices are compelling technologies and architectures, however, exposing the benefits and understanding where they come from is a harder subject to catch onto. Deciding to create a microservice architecture for your business application or understanding which contexts are bound to which functionality can seems like the complexity isn’t worth it in the long wrong. So here I explore some of the knowledge of microservices that is already out there in understanding when and why to turn to microservices and figuring out why, as in many cases, there really isn’t a need to.

In this post I will hopefully talk about some of major drivers and topics of microservices and how I see them in relationship to data-center technologies and applications. These are my own words and solely my opinion, however I hope this post can help those to understand this space a little better. I will talk briefly about, Conway’s Law, what it means to Break Down the Silos, why it is important to Continuous delivery, the importance of the unix philosophy, how to define a microservice, their relationship to SOA, the complexity involved in the architecture, what changes in the organization must happen, what companies and products are involved int this space, how to write a microservice, the importance of APIs and service discovery, and layers of persistence.

Microservices

The best definition in my opinion is “A Microservice fits in your head”. There are other definitions involving an amount of pizza, or a specific amount for lines of code, but I don’t like putting these boundaries on what a microservices is. In the simplest, a microservice is something that is small enough to conceptually fit in your head without really having to think to much about it. You can argue about how much someone can fit in their head etc, but then thats just rubbish and un-important to me.

I like to bring up the unix principle here as ice heard from folks at Joyent and other inn he field, this is the design that programs are designed to do one thing and do that one thing well. Like “ls” or “cat” for instance, typically if you design a microservice this way, you can limit its internal failure domain because it does one thing and exposes and API to do so. Now, microservices is a loaded term, and just like SoA there are similarities in these two architectures. But they are just that, architectures and I will add that you can find similarities in many of their parts but the some of the main differences is that SoA used XML, SOAP, typically a Single Message Bus for communication and a shared data source for services. Microservices uses more modern lightweight protocols like RESTful APIs, JSON, HTTP, RPC and typically a single microservice is attached to its own data source, whether is a copy, shard or a its own distributed database. This helps with multi-tenancy, flexibility and context boundaries that help scale such an architecture like microservices. One of the first things people start to realize when deep divining into microservices is the amount of complexity that comes out of slicing up the monolith because inherently you need to orchestrate, monitor, audit, and log many more processes, containers, services etc than you did with a typical monolithic application. The fact that these architectures are much more “elastic and ephemeral” than others forces technical changes that center around the smallest unit of business logic that helps deliver business value when combine with other services to deliver the end goal. This way each smaller unit can have its own change lifecycle, scale independently and be developed free of other dependencies within the typical monolith.

This drives the necessity to adopt a DevOps culture and change organizationally as each service should be developed by independent, smaller teams that can each release code within their own cycles. Teams still need to adhere to the invisible contracts that are between the services, these contracts are the APIs themselves between the services which talk to one another. I could spend an entire post on this topic but there is a great book called “Migrating to Cloud-Native Application Architectures” by Matt Stine of Pivotal (Which is free, download here) that talks about organizational changes, api-based collaboration, microservices and more. There is also a great post by Martin Fowler (here) that talks about microservices and the way Conways law affects the organization.

Importance of APIs

I want to briefly talk about the importance of orchestration, choreography and the important of the APIs that exist within a microservices architecture. A small note on choreography, this is another terms that may be new but its related to orchestration. Choreography is orchestration turned on its head, instead of an orchestration unit signaling when things happen, the intelligence is pushed to the endpoints and those endpoints react to events of changing environments, therefore each service known its own job. A great comparison of this is (here) in the book “Building Microservices” by Sam Newman. Rest APIs are at the heart of this communication, if an event is received from a customer of user, a choreography chain is then initialized and each endpoint talks to each other via these APIs, therefore, these APIs must remain robust, backward compatible and act as contracts between how services interact. A great post of the Netflix microservices work (here) explains this in a little more detail.

If the last few paragraphs and resources make some sense, you end up with a combination of loosely coupled services, strict boundaries, APIs (contracts), robust choreography and vital health and monitoring for all services deployed. These services can be scaled, monitored and moved independently without risk and react well to failures. Some of the exa plea of tools to hel you do this can be found at http://netflix.github.io. This all sounds great, but without taking the approach of “design for the integrations not the infrastructure/platform” (which I’ve heard a lot but can’t quite figure out who the quote belongs to, coudos to who you are :] ) this can fail pretty easily. There is a lot of detail I didnt cover in the above and I suggest looking into the sources I listed for a start on getting into the details of each part. For now I am going to turn to a few topics within microservices, Service Discovery & Registration and Data Persistence layers in the stack.

Service Discovery and Registration

Distributed systems at scale using microservices need a way to registry and discover what services and endpoints are available, enter Service Discovery tools like Consul, Etcd, Zookeepr, Eureka, and Doozerd. (others not listed) These tools make it easy for services to call this layer and find out a way to consume what else is available. Typically this helps one service find out how to connect and use another service. There are three main processes IMO for applications to use this layer:

  • Registration
    • when a service gets installed or “comes up” it needs to initially be registered with the discovery layer. An example of this is Registrator (https://github.com/gliderlabs/registrator) which reacts to docker containers starting and sends key/value pairs of data to a tool like Consul or Etcd to keep current discovery data about the service. Such information could be IP Endpoint, Port, API URL/Path, Resources, etc that can be used by the service.
  • Discovery
    • Discovery is the other end of Registration, when a service wants to use a (for instance) “proxy”, how will it know where the proxy lives or how to access it? Typically in applications this information is in a configuration file or hard coded into the app, whith service discovery all the app needs to do is know how to implement the information owned by the registration mechanism. For instance, an app can start and immediately say “Where is ‘Proxy'” and the discovery mechanism can respond by saying “Here is the Proxy thats closest to you” or “Here is the first Proxy available” along with the IP and Port of that proxy, the app can then just use those values, typically given in JSON or XML and use them inside the application thus not ever hardcoding any configuration anywhere.
  • Consume
    • Last but smallest is when the applications received the response back from the discovery mechanism it must know how to process and the the data. E.g. if your asking for a proxy or asking for a database the information given back would be different for the proxy versus how you would actually access the database.

Persistence and Backing Services

Today most applications are stateless applications, which means that they do not own any persistence themselves. You can think of a stateless application as a web-server, this web-server processes requests and talks to a database, but the database is another microsevice and this is where all the state lives. We can scale the web-server as much as we want and even actively load balance those endpoints without ever worrying about any state. However, that database I mentioned in the above example is something we should worry about, because we want our data to be available, and protected at all times. Though, you can’t (today) spin up your entire application stack (e.g. MongoDB, Express.js, Angular.js and Node.js) all in different services (containers) and not worry about how your data is stored, if you do this today you need persistent volumes that can be flexible enough to move with your apps container, which is hard to do today, the data container is just not as flexible as we need it be in todays architectures like Mesos and Cloud Foundry. Today persistence is added via Backing Services (http://12factor.net/backing-services) which are persistence / data layers that exist outside of the normal application lifecycle. This mean that in order to use a database one must first create the backing service then bind it to the application. Cloud Foundry does this today via “cf create-service and cf bind-service APPLICATION SERVICE_INSTANCE” where SERVICE_INSTANCE is the backing store, you can see more about that here. I won’t dig into this anymore other to say that this is problem that needs to be solved, and making your data services as flexible as the rest of your microservices architecture is not easy feat. The below link is a great article by Luke Marsden of ClusterHQ that talks about this very issue. http://www.infoq.com/articles/microservices-revolution

I also wanted to mention an interesting note on persistence in the way Netflix deploys Cassandra. All the data that Netflix uses is deployed on Amazon on EC2 instance and they use ephemeral storage! Which means when the node dies all their data is gone. But alas! they don’t worry about this type of issue anymore because Cassandra’s distributed, self healing architecture allows Netflix to move around their persistence layers and automatically scale them out when needed. I found out they do run incremental backups to S3 by briefly speaking with Adrian Cockcroft at offices hours at the Oreilly Software Architecture conference. I found this to be a pretty interesting point to how Netflix runs its operations for its data layers with Cassandra showing that these cloud-native, flexible, and de-coupled applications are actually working in production and remain reliable and resilient.

Major Players

Some of the major players in the field today include: (I may have missed some)

  • Pivotal
  • Apache Mesos
  • Joyent (Manta)
  • IBM Bluemix
  • Cloud Foundry
  • Amazon Elastic Container Service
  • Openshift by RedHat
  • OpenStack Magnum
  • CoreOS (with Kubernetes) see (Project Tectonic)
  • Docker (and Assorted Tools/Binaries)
  • Cononical’s LXD
  • Tutum
  • Giant Swarm

There are many other open source tools at work like Docker Swarm, Consul, Registrator, Powerstrip, Socketplane.io (Now owned by Docker Inc), Docker Compose, Fleet, Weave. Flocker and many more. This is just a token to how this field of technology is booming and were going to see many fast changes in the near future. Its clear that future importance of deploying a service and not caring about the “right layers” or infrastructure will be key. Enabling data flexibility without tight couplings to the service is part of an the overall application design or the data service. These architectures can be powerful for your applications and for your data itself. Ecosystems and communities alike are clearly coming together to help and try to solve problems for these architectures, I’m sure some things are coming so keep posted.

Microservices on your laptop

One way to get some experience with these tools is to run some examples on your laptop, checkout Lattice (https://github.com/cloudfoundry-incubator/lattice/) from Cloud Foundry which allows you to run some microservice-like containerized workloads. This post is more about the high-level thinking, and I hope to have some more technical posts about some of the technologies like Lattice, Swarm, Registrator and others in the future.