Hacking Docker: Discovering Containers

As more businesses prepare to make a digital transformation, containers have become the choice cloud computing architecture for faster, more portable and reliable deployments. With the growing interest in containerization, the question arises about how containers integrate with existing infrastructure. In this post, we will look at how containerization affects service discovery and present a network routing solution that allows NetflixOSS Eureka to provide unified discovery between both containers and with VM-based services.

Kenzan specializes in cloud technologies with extensive experience in Amazon Web Services (AWS). We have adopted a number of tools from the NetflixOSS stack for use in AWS, such as Zuul, Ribbon, and Eureka. The discovery service feature of Eureka allows us to build dynamically scalable AWS environments without the need to setup fixed routing and load balancing infrastructure.

Docker introduced a new networking layer that changes everything we know about networking in the cloud. This makes discovery and routing with Eureka challenging. Containers have their own IP addresses, belong on a different subnet, and are only routable from the host running the Docker daemon.

We have experimented with tools like flanneld that create virtual networking layers between Docker hosts, allowing for cross-host communication between Docker containers. Flanneld is easy to setup and does as advertised, but requires hosts wishing to network with containers to be running the flannel daemon. All-in-one Docker solutions like Kubernetes do everything from networking containers to orchestrating and managing multiple Docker hosts, but still cannot network with containers from outside the Kubernetes cluster.

What we are looking for is to spin up containers like we do with EC2 instances, have them register with a discovery service, and allow us to send traffic from anywhere in the VPC. Let’s start simple with a cluster of Docker hosts, which is something that EC2 Container Service (ECS) will provide us. We launch a few applications as Docker instances using ECS, but we get containers that can’t talk to each other and can’t be reached from any external service.

As an application in a discovery-based world, you need to tell the discovery service who you are and how others can reach you. This is easy on EC2 instances because the application can provide the host IP address and the port it’s listening on. Containers in Docker are given IP addresses that are only routable to containers running on the same Docker daemon. We can expose internal container ports as host ports, but that port (which port? That = ?)has to be static and non-conflicting with other containers on the same host. e want to be dynamic and not have to remember which ports are used versus which ports are free.

ECS provides an option to expose containers through an Elastic Load Balancer (ELB). Issues with ELB’s include consuming several VPC IP addresses, requiring management of limits on how many ELB’s can be created, and adding additional AWS costs. Our applications now have to remember a series of hostnames representing ELB endpoints for each environment, adding more configuration overhead. ELB’s have the advantage of security groups, which is something we may expect to lose in a container world. The new networking layer on top of Docker does not play nicely with network based firewalls like security groups in AWS.

The goal is to find a non-conflicting dynamic way for containers deployed to a cluster of Docker hosts to identify themselves to a discovery service and have their identity be reachable. To achieve this, we need a tool that will find other containers on the host and route traffic based on a series of rules. Traefik is that tool. Traefik is a discovery based HTTP reverse proxy and load balancer that can discover Docker containers with minimal configuration, as well as several other means of discovery.

Let’s look at what it takes to get Traefik running with a Docker based backend:

$ docker run -d -p 80:80 -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock traefik –web –docker –docker.domain=docker.localhost

That’s it. The front end is now listening on port 80. Notice what we did here is we mounted the docker.sock file as a volume launching the container. This gives Traefik API access to the Docker daemon so that it can find other containers.

Alright, now it is time to add some containers for Traefik to find:

$ docker run -d –name nginx -l traefik.port=80 nginx

We added an Nginx container, and added a Label of traefik.port=80. Traefik will use the Docker metadata API exposed through the mounted unix socket to find container labels and bind the listener to port 80 that Nginx is listening on.

The Nginx container can be seen on the Traefik admin page listening on port 8080

Notice how the Rule is Host:nginx.docker.localhost. This is a combination of the container name we provided to Nginx with the –name argument. The docker.localhost part of the domain came from the –docker.domain=docker.localhost we gave to Traefik at startup.

Running a curl to the Docker host with a Host header of nginx.docker.localhost returns the Nginx welcome page.

$ curl -H “Host: nginx.docker.localhost” 192.168.33.10
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
….

Traefik is routing requests with this Host header to our Nginx container. We didn’t tell Traefik to do that, but it did (and that’s alright).

We can use other labels attached to the Nginx container to control how Traefik routes requests to it. Let’s try a path based rule.

$ docker run –name nginx -l traefik.port=80 -l traefik.frontend.rule=PathPrefix:/nginx nginx

Now all requests with the path prefix of /nginx will go to our Nginx container

$ curl 192.168.33.10/nginx/test
<html>
<head><title>404 Not Found</title></head>
….

We of course get a 404 since our nginx doesn’t have a /test resource, but we can see in the Nginx logs that the requests are coming through.

172.17.0.2 — — [10/Jan/2017:22:48:37 +0000] “GET /nginx/test HTTP/1.1” 404 169 “-” “curl/7.49.1” “192.168.33.1”

That was a quick proof-of-concept to show how quickly we got Traefik running as a discovery based routing service with Docker. By putting a Traefik container on every Docker host, we can dynamically setup routes to our containers running on those hosts. All we need to do now is identify our containers to the discovery service as the host IP address and the port that Traefik is listening on. The applications calling the services have to remember to include the path prefix in the request. Below is an architecture diagram showing Traefik set up on multiple Docker hosts.

Visit http://techblog.kenzan.com/ for more articles on containers, microservices, continuous delivery and more.