Docker Backend

The Docker backend is user to spin up clusters on Docker containers, where each container is a DC/OS node.

Requirements

Docker

Docker version 17.06 or later must be installed.

Plenty of memory must be given to Docker. On Docker for Mac, this can be done from Docker > Preferences > Advanced. This backend has been tested with a four node cluster with 9 GB memory given to Docker.

IP Routing Set Up for Docker

On macOS, hosts cannot connect to containers IP addresses by default. Once the CLI is installed, run dcos-docker setup-mac-network.

ssh

The ssh command must be available.

Operating System

This tool has been tested on macOS with Docker for Mac and on Linux.

It has also been tested on Windows on Vagrant.

Windows

The only supported way to use the Docker backend on Windows is using Vagrant and VirtualBox.

vagrant plugin install vagrant-persistent-storage
  • Optionally install the Vagrant plugins to cache package downloads and keep guest additions updates:
vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-vbguest
  • Start Powershell and download the DC/OS E2E Vagrantfile to a directory containing a DC/OS installer file:
((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/dcos/dcos-e2e/master/vagrant/Vagrantfile')) | Set-Content -LiteralPath Vagrantfile
  • By default, the Vagrantfile installs DC/OS E2E from the most recent release at the time it is downloaded. To use a different release, or any Git reference, set the environment variable DCOS_E2E_REF:
$env:DCOS_E2E_REF = "master"
  • Start the virtual machine and login:
vagrant up
vagrant ssh

You can now run dcos-docker CLI commands or use the Python Library.

To connect to the cluster nodes from the Windows host (e.g. to use the DC/OS web interface), in PowerShell Run as Administrator, and add the Virtual Machine as a gateway:

route add 172.17.0.0 MASK 255.255.0.0 192.168.18.2

To shutdown, logout of the virtual machine shell, and destroy the virtual machine and disk:

vagrant destroy

The route will be removed on reboot. You can manually remove the route in PowerShell Run as Administrator using:

route delete 172.17.0.0

dcos-docker doctor

DC/OS E2E comes with the dcos-docker doctor command. Run this command to check your system for common causes of problems.

DC/OS Installation

Clusters created by the Docker backend only support installing DC/OS via install_dcos_from_path(). Nodes of Clusters created by the Docker backend do not distinguish between public_ip_address and private_ip_address.

Limitations

Docker does not represent a real DC/OS environment with complete accuracy. This section describes the currently known differences between the Docker backend and a real DC/OS environment.

SELinux

Tests inherit the host’s environment. Any tests that rely on SELinux being available require it be available on the host.

Storage

Docker does not support storage features expected in a real DC/OS environment.

Troubleshooting

Cleaning Up and Fixing “Out of Space” Errors

If a test is interrupted, it can leave behind containers, volumes and files. To remove these, run the following:

docker stop $(docker ps -a -q --filter="name=dcos-e2e")
docker rm --volumes $(docker ps -a -q --filter="name=dcos-e2e")
docker volume prune --force

If this repository is available, run make clean.

macOS File Sharing

On macOS /tmp is a symlink to /private/tmp. /tmp is used by the harness. Docker for Mac must be configured to allow /private to be bind mounted into Docker containers. This is the default. See Docker > Preferences > File Sharing.

Clock sync errors

On various platforms, the clock can get out of sync between the host machine and Docker containers. This is particularly problematic if using check_time: true in the DC/OS configuration. To work around this, run docker run --rm --privileged alpine hwclock -s.

Reference

class dcos_e2e.backends.Docker(workspace_dir=None, custom_container_mounts=None, custom_master_mounts=None, custom_agent_mounts=None, custom_public_agent_mounts=None, linux_distribution=<Distribution.CENTOS_7: 1>, docker_version=<DockerVersion.v1_13_1: 2>, storage_driver=None, docker_container_labels=None, docker_master_labels=None, docker_agent_labels=None, docker_public_agent_labels=None)

Create a configuration for a Docker cluster backend.

Parameters:
  • workspace_dir (Optional[Path]) – The directory in which large temporary files will be created. These files will be deleted at the end of a test run. This is equivalent to dir in tempfile.mkstemp().
  • custom_container_mounts (Optional[List[Mount]]) – Custom mounts add to all node containers. See mounts in Containers.run.
  • custom_master_mounts (Optional[List[Mount]]) – Custom mounts add to master node containers. See mounts in Containers.run.
  • custom_agent_mounts (Optional[List[Mount]]) – Custom mounts add to agent node containers. See mounts in Containers.run.
  • custom_public_agent_mounts (Optional[List[Mount]]) – Custom mounts add to public agent node containers. See mounts in Containers.run.
  • linux_distribution (Distribution) – The Linux distribution to boot DC/OS on.
  • docker_version (DockerVersion) – The Docker version to install on the cluster nodes.
  • storage_driver (Optional[DockerStorageDriver]) – The storage driver to use for Docker on the cluster nodes. By default, this is the host’s storage driver. If this is not one of aufs, overlay or overlay2, aufs is used.
  • docker_container_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster node containers. Akin to the dictionary option in Containers.run.
  • docker_master_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster master node containers. Akin to the dictionary option in Containers.run.
  • docker_agent_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster agent node containers. Akin to the dictionary option in Containers.run.
  • docker_public_agent_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster public agent node containers. Akin to the dictionary option in Containers.run.
workspace_dir

The directory in which large temporary files will be created. These files will be deleted at the end of a test run.

custom_container_mounts

Custom mounts add to all node containers. See mounts in Containers.run.

custom_master_mounts

Custom mounts add to master node containers. See mounts in Containers.run.

custom_agent_mounts

Custom mounts add to agent node containers. See mounts in Containers.run.

custom_public_agent_mounts

Custom mounts add to public agent node containers. See mounts in Containers.run.

linux_distribution

The Linux distribution to boot DC/OS on.

docker_version

The Docker version to install on the cluster nodes.

docker_storage_driver

The storage driver to use for Docker on the cluster nodes.

docker_container_labels

Docker labels to add to the cluster node containers. Akin to the dictionary option in Containers.run.

docker_master_labels

Docker labels to add to the cluster master node containers. Akin to the dictionary option in Containers.run.

docker_agent_labels

Docker labels to add to the cluster agent node containers. Akin to the dictionary option in Containers.run.

docker_public_agent_labels

Docker labels to add to the cluster public agent node containers. Akin to the dictionary option in Containers.run.