Category Archives: innovation

Exploring Powerstrip from ClusterHQ: A Socketplane Adapter for Docker

icon_512x512ClusterHQ_logopowerstrip@2x

sources: http://socketplane.io, https://github.com/ClusterHQ/powerstrip, http://clusterhq.com

Over the past few months one of the areas worth exploring within the container ecosystem is how it works with external services and applications. I currently work in EMC CTO Advanced Development so naturally my interest level is more about data services, but because my background working with SDN controllers and architectures is still one of my highest interests I figured I would get to know what Powerstrip was by working with Socketplane’s Tech Release.

*Disclaimer:

This is not the official integration for powerstrip with sockeplane merged over the last week or so, I was working on this in a rat hole and it works a little differently than the one that Socketplane merged recently.

What is Powerstrip?

Powerstrip is a simple proxy for docker requests and responses to and from the docker client/daemon that allows you to plugin “adapters” that can ingest a docker request, perform an action, modification, service setup etc, and output a response that is then returned to Docker. There is a good explaination on ClusterHQ’s Github page for the project.

Powerstrip is really a prototype tool for Docker Plugins, and a more formal discussion , issues, and hopefully future implementation of Docker Plugins will come out of such efforts and streamline the development of new plugins and services for the container ecosystem.

Using a plugin or adapter architecture, one could imagine plugging storage services, networking services, metadata services, and much more. This is exactly what is happening, Weave, Flocker both had adapters, as well as Socketplane support recently.

Example Implementation in GOlang

I decided to explore using Golang, because at the time I did not see an implementation of the PowerStripProtocol in Go. What is the PowerStripProtocol?

The Powerstrip protocol is a JSON schema that Powerstrip understands so that it can hook in it’s adapters with Docker. There are a few basic objects within the schema that Powerstrip needs to understand and it varies slightly for PreHook and PostHook requests and responses.

Pre-Hook

The below scheme is what PowerStripProtocolVersion: 1 implements, and it needs to have the pre-hook Type as well as a ClientRequest.

{
    PowerstripProtocolVersion: 1,
    Type: "pre-hook",
    ClientRequest: {
        Method: "POST",
        Request: "/v1.16/container/create",
        Body: "{ ... }" or null
    }
}

Below is what your adapter should respond with, a ModifiedClientRequest

{
    PowerstripProtocolVersion: 1,
    ModifiedClientRequest: {
        Method: "POST",
        Request: "/v1.16/container/create",
        Body: "{ ... }" or null
    }
}

Post-Hook

The below scheme is what PowerStripProtocolVersion: 1 implements, and it needs to have the post-hook Type as well as a ClientRequest and a Server Response. We add ServerResponse here because post hooks are already processed by Docker, therefore they already have a response.

{
    PowerstripProtocolVersion: 1,
    Type: "post-hook",
    ClientRequest: {
        Method: "POST",
        Request: "/v1.16/containers/create",
        Body: "{ ... }"
    }
    ServerResponse: {
        ContentType: "text/plain",
        Body: "{ ... }" response string
                        or null (if it was a GET request),
        Code: 404
    }
}

Below is what your adapter should respond with, a ModifiedServerResponse

{
    PowerstripProtocolVersion: 1,
    ModifiedServerResponse: {
        ContentType: "application/json",
        Body: "{ ... }",
        Code: 200
    }
}

Golang Implementation of the PowerStripProtocol

What this looks like in Golang is the below. (I’ll try and have this open-source soon, but it’s pretty basic :] ). Notice we implement the main PowerStripProtocol in a Go struct, but the JSON tag and options likes contain an omitempty for certain fields, particularly the ServerResponse. This is because we always get a ClientRequest in pre or post hooks but now a ServerResponse.

powerstripprotogo

We can implement these Go structs to create Builders, which may be Generic, or serve a certain purpose like catching pre-hook Container/Create Calls from Docker and setting up socketplane networks, this you will see later. Below are generall function heads that return an Marshaled []byte Go Struct to gorest.ResponseBuilder.Write()

buildprehook

builtposthook

Putting it all together

Powerstrip suggests that adapters be created as Docker containers themselves, so the first step was to create a Dockerfile that built an environment that could run the Go adapter.

Dockerfile Snippets

First, we need a Go environment inside the container, this can be set up like the following. We also need a couple of packages so we include the “go get” lines for these.

pwerstripdockerfilego

Next we need to enable our scipt (ADD’ed earlier in the Dockerfile) to be runnable and use it as an ENTRYPOINT. This script takes commands like run, launch, version, etc

runascript

Our Go-based socketplane adapter is laid out like the below. (Mind the certs directory, this was something extra to get it working with a firewall).

codelayout

“powerstrip/” owns the protocol code, actions are Create.go and Start.go (for pre-hook create and post-hook Start, these get the ClientRequests from:

  • POST /*/containers/create

And

  • POST /*/containers/*/start

“adapter/” is the main adapter that processes the top level request and figures out whether it is a prehook or posthook and what URL it matches, it uses a switch function on Type to do this, then sends it on its way to the correct Action within “action/”

“actions” contains the Start and Create actions that process the two pre hook and post hook calls mentioned above. The create hook does most of the work, and I’ll explain a little further down in the post.

actions

Now we can run “docker buid -t powerstrip-socketplane .” in this directory to build the image. Then we use this image to start the adapter like below. Keep in mind the script is actually using the “unattended nopowerstrip” options for socketplane, since were using our own separate one here.

docker run -d --name powerstrip-socketplane \
 --expose 80 \
 --privileged \ 
 --net=host \
 -e BOOTSTRAP=true \
 -v /var/run/:/var/run/ \
 -v /usr/bin/docker:/usr/bin/docker \
 powerstrip-socketplane launch

Once it is up an running, we can use a simple ping REST URL to test if its up: It should respond “pong” if everything is running.

$curl http://localhost/v1/ping
pong

Now we need to create our YAML file for PowerStrip and start our Powerstrip container.

Screen Shot 2015-02-04 at 4.23.59 PM

Screen Shot 2015-02-04 at 4.24.05 PM

If all is well, you should see a few containers running and look somthing like this

dddd151d4076        socketplane/socketplane:latest   "socketplane --iface   About an hour ago   Up About an hour                             romantic_babbage

6b7a63ce419a        clusterhq/powerstrip:v0.0.1      "twistd -noy powerst   About an hour ago   Up About an hour    0.0.0.0:2375->2375/tcp   powerstrip
d698047800b1        powerstrip-socketplane:latest    "/opt/run.sh launch"   2 hours ago         Up About an hour                             powerstrip-socketplane

The adapter will automatically spawn off a socketplane/socketplane:latest container because it installs socketplane brings up the socketplane software.

Once this is up, we need to update our DOCKER_HOST environment variable and then we are ready to go to start issuing commands to docker and our adapter will catch the requests. Few examples below.

export DOCKER_HOST=tcp://127.0.0.1:2375

Next we create some containers with a SOCKETPLANE_CIDR env vairable, the adapter will automatically catch this and process the networking information for you.

docker create --name powerstrip-test1 -e SOCKETPLANE_CIDR="10.0.6.1/24" ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done"
docker create --name powerstrip-test2 -e SOCKETPLANE_CIDR="10.0.6.1/24" ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done”

Next, start the containers.

docker start powerstrip-test1

docker start powerstrip-test2

If you issue an ifconfig on either one of these containers, you will see that it owns an ovs<uuid> port that connects it to the virtual network.

sudo docker exec powerstrip-test2 ifconfig
ovs23b79cb Link encap:Ethernet  HWaddr 02:42:0a:00:06:02

          inet addr:10.0.6.2  Bcast:0.0.0.0  Mask:255.255.255.0

          inet6 addr: fe80::a433:95ff:fe8f:c8d6/64 Scope:Link

          UP BROADCAST RUNNING  MTU:1440  Metric:1

          RX packets:12 errors:0 dropped:0 overruns:0 frame:0

          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:956 (956.0 B)  TX bytes:726 (726.0 B)

We can issue a ping to test connectivity over the newly created VXLAN networks. (powerstrip-test1=10.0.6.2, and powerstrip-test2=10.0.6.3)

$sudo docker exec powerstrip-test2 ping 10.0.6.2

PING 10.0.6.2 (10.0.6.2) 56(84) bytes of data.

64 bytes from 10.0.6.2: icmp_seq=1 ttl=64 time=0.566 ms

64 bytes from 10.0.6.2: icmp_seq=2 ttl=64 time=0.058 ms

64 bytes from 10.0.6.2: icmp_seq=3 ttl=64 time=0.054 ms

So what’s really going on under the covers?

In my implementation of the powerstrip adapater, the adapter does the following things

  • Adapter recognizes a Pre-Hook POST /containers/create call and forwards it to PreHookContainersCreate
  • PreHookContainersCreate checks the client request Body foe the ENV variable SOCKETPLANE_CIDR, if it doesn’t have it it returns like a normal docker request. If it does then it will probe socketplane to see if the network exists or not, if it doesn’t it creates it.
  • In either case, there will be a “network-only-container” created connected to the OVS VXLAN L2 domain, it will then modify the response body in the ModifiedClientRequest so that the NetworkMode gets changed to –net:container:<new-network-only-container>.
  • Then upon start the network is up and the container boots likes normal with the correct network namespace connected to the socketplane network.

Here is a brief architecture to how it works.

diag

Thanks for reading, please comment or email me with any questions.

Cheers!

Advertisements

Docker Remote Host Management with Openstack

Untitled
Screen Shot 2012-07-26 at 6.34.17 PM

So I decided to participate in #DockerGlobalHackday and oh boy was it a learning experience. First off, the hackday started off with great presentations from some of the hackers and docker contributors. One that caught my eye was Host Management (https://www.youtube.com/watch?v=lZGmvGw-mWc) and (https://github.com/docker/docker/issues/8681)

Ben Firshman and contributors thought up and created this feature for Docker that lets you provision remote daemons on demand given cloud providers. It had me thinking that maybe I should hack on a driver for a Local Openstack Deployment. So I did, and this is my DockerHackDayHack.

https://github.com/wallnerryan/docker/tree/host-managment-openstack 

https://github.com/bfirsh/docker/pull/13

*Note, the code is raw, very raw, I haven’t coded in Go until this Hackday 🙂 Which is what I guess it is good for.

*Note this code was developed using Devstack with Flat Network orignaly, so there is some rough edged code for supporting out of the box devstack with nova network but it probably won’t work 🙂 I’ll make an update on this soon.

*Note the working example was testing on Openstack Icehouse with Neutron Networking. Neutron has one public and one private network. The public network is where the floating ip comes from for the docker daemon.

Here are the options now for host-management:

./bundles/1.3.0-dev/binary/docker-1.3.0-dev hosts create
Screen Shot 2014-11-03 at 3.29.50 PM

Notice the areas with  “–openstack-” prefix, this is what was added. If your using neutron network then the network for floating ips is needed. The image can be a Ubuntu or Debian based cloud image, but must support Cloud-Init / Metadata Service. This is how the docker installation is injected.Below is an example of how to kickoff a new Docker OpenStack Daemon: (beware the command is quite long with openstack options, replace X.X.X.X with your keystone endpoint, as well as UUIDs of any openstack resources.) It also includes –openstack-nameserver, this is not required but in my case I was, and will inject a nameserver line into the resolve.conf of the image using Cloud-Init / Metadata Service

In the future I plan on making this so we don’t need as many UUIDs. but rather the driver will take text as input and search for the relevant UUIDs to use. ( limited time to hack on this )

#./bundles/1.3.0-dev/binary/docker-1.3.0-dev hosts create -d openstack

–openstack-image-id=”d4f62660-3f03-45b7-9f63-165814fea55e” \

–openstack-auth-endpoint=”http://X.X.X.X:5000/v2.0” \

–openstack-floating-net=”4a3beafb-2ecf-42ca-8de3-232e0d137931″ \

–openstack-username=”admin” \

–openstack-password=”danger” \

–openstack-tenant-id=”daad3fe7f60e42ea9a4e881c7343daef” \

–openstack-keypair=”keypair1″ \

–openstack-region-name=”regionOne” \

–openstack-net-id=”1664ddb9-8a14-48cd-9bee-a3d4f2fe16a0″ \

–openstack-flavor=”2″ \

-openstack-nameserver=”10.254.66.23″ \

–openstack-secgroup-id=“e3eb2dc6-4e67-4421-bce2-7d97e3fda356” \

openstack-dockerhost-1

The result you will see is: (with maybe some errors if your security groups are already setup)

#[2014-11-03T08:31:51.524125904-08:00] [info] Creating server.
#[2014-11-03T08:32:16.698970304-08:00] [info] Server created successfully.
#%!(EXTRA string=63323227-1c1e-40f6-9c25-78196010936b)[2014-11-03T08:32:17.888292214-08:00] [info] Created Floating Ip
#[2014-11-03T08:32:18.439105902-08:00] [info] “openstack-dockerhost-1” has been created and is now the active host. Docker commands #will now run against that host
If you specified a ubuntu image (I just downloaded the Ubuntu Cloud Img from here)The Docker Daemon will get deployed in a Ubuntu Server in openstack and the Daemon will start on port 2375
Screen Shot 2014-11-03 at 11.36.10 AM
The Instance will look somthing like this in the dashboard of openstack.
Screen Shot 2014-11-03 at 11.36.27 AM
Here is the floating ip association with the Daemon Host.
Screen Shot 2014-11-03 at 11.36.44 AM
And it’s fully functional, see some other Docker Hosts commands below. Here is a video of the deployment (https://www.youtube.com/watch?v=aBG3uL8g124)
(View Hosts)
./bundles/1.3.0-dev/binary/docker-1.3.0-dev hosts
Screen Shot 2014-11-03 at 3.17.30 PM
You can make either the local unix socket or the openstack node the active Daemon and you can use it like any other docker client. This “hosts” command can run locally on your laptop but your containers and daemon run in OpenStack.  One could see this feature replacing something like Boot2Docker.
(docker ps) – Shows containers running in your openstack deployed docker daemon
./bundles/1.3.0-dev/binary/docker-1.3.0-dev ps -a
Screen Shot 2014-11-19 at 10.15.30 AM
(View Remote Info)
./bundles/1.3.0-dev/binary/docker-1.3.0-dev info
Screen Shot 2014-11-03 at 3.18.48 PM
Thanks for the Docker community for putting these events together! Pretty cool! Happy Monday and Happy Dockering.
P.S. I also used a Packer/VirtualBox Setup for DevStack in the begining. Here is the Packer Config and the Preseed.cfg. Just download devstack and run it on there.
{
 "variables": {
 "ssh_name": "yourname",
 "ssh_pass": "password",
 "hostname": "packer-ubuntu-1204"
 },

 "builders": [{
 "type": "virtualbox-iso",
 "guest_os_type": "Ubuntu_64",

 "vboxmanage": [
 ["modifyvm", "{{.Name}}", "--vram", "32"],
 ["modifyvm", "{{.Name}}", "--memory", "2048"],
 ["modifyvm", "{{.Name}}","--natpf1", "web,tcp,,8080,,80"],
 ["modifyvm", "{{.Name}}","--natpf1", "fivethousand,tcp,,5000,,5000"],
 ["modifyvm", "{{.Name}}","--natpf1", "ninesixninesix,tcp,,9696,,9696"],
 ["modifyvm", "{{.Name}}","--natpf1", "eightsevensevenfour,tcp,,8774,,8774"],
 ["modifyvm", "{{.Name}}","--natpf1", "threefivethreefiveseven,tcp,,35357,,35357"]
 ],

 "disk_size" : 10000,

 "iso_url": "http://releases.ubuntu.com/precise/ubuntu-12.04.4-server-amd64.iso",
 "iso_checksum": "e83adb9af4ec0a039e6a5c6e145a34de",
 "iso_checksum_type": "md5",

 "http_directory" : "ubuntu_64",
 "http_port_min" : 9001,
 "http_port_max" : 9001,

 "ssh_username": "{{user `ssh_name`}}",
 "ssh_password": "{{user `ssh_pass`}}",
 "ssh_wait_timeout": "20m",

 "shutdown_command": "echo {{user `ssh_pass`}} | sudo -S shutdown -P now",

 "boot_command" : [
 "<esc><esc><enter><wait>",
 "/install/vmlinuz noapic ",
 "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg ",
 "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
 "hostname={{user `hostname`}} ",
 "fb=false debconf/frontend=noninteractive ",
 "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
 "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
 "initrd=/install/initrd.gz -- <enter>"
 ]
 }]
}


(Preseed.cfg Starts HERE)
# Some inspiration:
# * https://github.com/chrisroberts/vagrant-boxes/blob/master/definitions/precise-64/preseed.cfg
# * https://github.com/cal/vagrant-ubuntu-precise-64/blob/master/preseed.cfg

# English plx
d-i debian-installer/language string en
d-i debian-installer/locale string en_US.UTF-8
d-i localechooser/preferred-locale string en_US.UTF-8
d-i localechooser/supported-locales en_US.UTF-8

# Including keyboards
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layout select USA
d-i keyboard-configuration/variant select USA
d-i keyboard-configuration/modelcode string pc105


# Just roll with it
d-i netcfg/get_hostname string this-host
d-i netcfg/get_domain string this-host

d-i time/zone string UTC
d-i clock-setup/utc-auto boolean true
d-i clock-setup/utc boolean true


# Choices: Dialog, Readline, Gnome, Kde, Editor, Noninteractive
d-i debconf debconf/frontend select Noninteractive

d-i pkgsel/install-language-support boolean false
tasksel tasksel/first multiselect standard, ubuntu-server


# Stuck between a rock and a HDD place
d-i partman-auto/method string lvm
d-i partman-lvm/confirm boolean true
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-auto/choose_recipe select atomic

d-i partman/confirm_write_new_label boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true

# Write the changes to disks and configure LVM?
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto-lvm/guided_size string max

# No proxy, plx
d-i mirror/http/proxy string

# Default user, change
d-i passwd/user-fullname string yourname
d-i passwd/username string yourname
d-i passwd/user-password password password
d-i passwd/user-password-again password password
d-i user-setup/encrypt-home boolean false
d-i user-setup/allow-password-weak boolean true

# No language support packages.
d-i pkgsel/install-language-support boolean false

# Individual additional packages to install
d-i pkgsel/include string build-essential ssh

#For the update
d-i pkgsel/update-policy select none

# Whether to upgrade packages after debootstrap.
# Allowed values: none, safe-upgrade, full-upgrade
d-i pkgsel/upgrade select safe-upgrade

# Go grub, go!
d-i grub-installer/only_debian boolean true

d-i finish-install/reboot_in_progress note

Innovation at the EMC Centers of Excellence

Our CTO sits down and talks about innovation, how its important, why should we do it, and how we start a fire underneath people who can innovate. Interesting talks hosted by Calvin Smith of the OCTO.

“John Roese, Chief Technology Officer, and Vic Bhagat, Chief Information Officer, sit down with Calvin Smith, Principal Manager, Global Innovation & Marketing, to talk about the current state of organic innovation at EMC. – See more at: http://reflectionsblog.emc.com/2014/02/nurturing-organic-innovation.html#prclt-lmhe1Hyp&#8221;