This post covers putting a Python script into a docker container and distributing it across a Mesos cluster. To skip to the end, here are the necessary files and steps.

Requirements

• 1 docker host capable of pulling images from docker hub
• (optional) 1 Mesos cluster with docker executors

The Script

The script below makes an HTTP request to Twitch TV’s stream API to extract the total number of live streams.

Save the content below as twitchcounts.py.

 1 import requests
 2 
 3 # Twitch TV 'streams' end point.
 4 BASE_URL = 'https://api.twitch.tv/kraken/streams'
 5 
 6 def crawl(url, params):
 7     r = requests.get(url, params=params, timeout=30)
 8 
 9     if r.status_code == 200:
10         js = r.json()
11         if '_total' in js:
12             print("There are {0} streamers doing it live.".format(js['_total']))
13 
14 
15 def main():
16     query_params = {
17         'limit': 1,
18         'stream_type': 'live'
19     }
20 
21     crawl(BASE_URL, query_params)
22 
23 
24 if __name__ == '__main__':
25     main()

Containerize It

In order for docker to create an image, it needs a Dockerfile. A Dockerfile contains a series of commands for docker to execute in order to assemble an image.

Save the contents below as Dockerfile in the same directory as twitchcounts.py.

1 FROM python:3-onbuild
2 
3 ## Run the python script
4 CMD [ "python", "./twitchcounts.py" ]

An Aside

The Dockerfile above uses python:3-onbuild as the base (starting) image, which is itself a Dockerfile!

1 FROM python:3.5
2 
3 RUN mkdir -p /usr/src/app
4 WORKDIR /usr/src/app
5 
6 ONBUILD COPY requirements.txt /usr/src/app/
7 ONBUILD RUN pip install --no-cache-dir -r requirements.txt
8 
9 ONBUILD COPY . /usr/src/app

Python Dependencies

The script begins with import requests, however, this library (requests) is not part of the base python installation. Executing the script as-is will result in an ImportError. The python:3-onbuild image will execute pip install -r requirements.txt and any dependencies listed inside the file will be installed into the image.

Save the line below as requirements.txt in the same directory as twitchcounts.py.

1 requests==2.8.1

Build It

Use the build command to build the new image. The tag argument (-t) allows naming the resulting image – without it the image can only be identified through a hash.

Run “docker build -t twitch-channels-app .” – the “.” references the local directory with twitchcounts.py.

 1 $ sudo docker build -t twitch-channels-app .
 2 Sending build context to Docker daemon 70.14 kB
 3 Sending build context to Docker daemon
 4 Step 0 : FROM python:3-onbuild
 5 # Executing 3 build triggers
 6 Trigger 0, COPY requirements.txt /usr/src/app/
 7 Step 0 : RUN pip install --no-cache-dir -r requirements.txt
 8  ---> Running in 947f5ed0406d
 9 Collecting requests==2.8.1 (from -r requirements.txt (line 1))
10   Downloading requests-2.8.1-py2.py3-none-any.whl (497kB)
11 Installing collected packages: requests
12 Successfully installed requests-2.8.1
13 Trigger 2, COPY . /usr/src/app
14 Step 0 : COPY . /usr/src/app
15  ---> 0333a991676d
16 Removing intermediate container e43e6fa4fcb9
17 Removing intermediate container 947f5ed0406d
18 Removing intermediate container 3f3747e2669d
19 Step 1 : CMD python ./twitchcounts.py
20  ---> Running in 9e36355a9d84
21  ---> 517db7e0fb8a
22 Removing intermediate container 9e36355a9d84
23 Successfully built 517db7e0fb8a

The images command will show docker images on the local system. twitch-count-app should be part of the listing.

Run docker images to view docker images on the current machine.

1 twitch-count$ sudo docker images
2 REPOSITORY            TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
3 twitch-count-app      latest              517db7e0fb8a        31 seconds ago      695 MB

Run It

Use the run command to run the new image and get the streamer count.

Run “docker run --rm --name my-running-app twitch-count-app” to run the docker container.

1 $ sudo docker run --rm --name my-running-app twitch-count-app
2 There are 32770 streamers doing it live.

The arguments are:

• --rm automatically removes the container when it exits; this prevents build-up of results when running docker ps -a
• --name my-running-app assigns a name to the container and prevents more than one from running at a time (with the same name)
• twitch-count-app is the image to execute, which was the tag (-t) used in the build command

Distribute It

The image is built, but only available on the host that did the building. There are two options:

1. Build the image on every docker host
2. Save the image and load it on every docker host

Since the first option was demonstrated above, this tackles saving and loading docker images.

Save

docker provides the save command, which outputs a docker image as a tar file. By default, it writes to STDOUT, but this can be changed with the output (-o) argument.

NOTE: The command below uses sudo, so the resulting tar file will be owned by root. Be sure to chown the file correctly for your setup.

Run “docker save -o twitch-count-app.tar twitch-count-app” to save the twitch-count-app image to a tar file.

1 $ sudo docker save -o twitch-count-app.tar twitch-count-app
2 $ ls -l
3 -rw-r--r-- 1 root root 716619264 Nov  7 11:58 twitch-count-app.tar
4 $ sudo chown $(id -un):$(id -gn) twitch-count-app.tar
5 $ ls -l
6 -rw-r--r-- 1 cdcd cdcd 716619264 Nov  7 11:58 twitch-count-app.tar

Load

docker provides the load command to ingest tar files that were created with the save command. Transfer the file from the section above to a target host and run the load command.

Similar to save, use the input (-i) argument to read from a file instead of STDIN.

1 $ sudo docker load -i twitch-count-app.tar
2 $ sudo docker images
3 REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
4 twitch-count-app    latest              517db7e0fb8a        18 minutes ago      695 MB

Conclusion

This post walked through creating a docker image specifically for a Python script and how to transfer that image to other docker hosts. With these basics, you can containerize Python applications and use them within a Mesos cluster.

The streamer counting script on Chronos:

A simple web service on Marathon: