Docker Backend

The Docker backend is used to spin up clusters on Docker containers, where each container is a DC/OS node.

Requirements

Docker 17.06+

Docker version 17.06 or later must be installed.

Plenty of memory must be given to Docker. On Docker for Mac, this can be done from Docker > Preferences > Advanced. This backend has been tested with a four node cluster with 9 GB memory given to Docker.

IP Routing Set Up for Docker

On macOS, hosts cannot connect to containers IP addresses by default. This is required, for example, to access the web UI, to SSH to nodes and to use the DC/OS CLI.

Once the CLI is installed, run minidcos docker setup-mac-network to set up IP routing.

Without this, it is still possible to use some features. In the library, specify transport as dcos_e2e.node.Transport.DOCKER_EXEC. In the CLI, specify the --transport and --skip-http-checks options where available.

ssh

The ssh command must be available to use the SSH transport.

Operating System

This tool has been tested on macOS with Docker for Mac and on Linux.

It has also been tested on Windows on Vagrant.

Windows

The only supported way to use the Docker backend on Windows is using Vagrant and VirtualBox.

vagrant plugin install vagrant-persistent-storage

  • Optionally install the Vagrant plugins to cache package downloads and keep guest additions updates:

vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-vbguest

  • Start Powershell and download the DC/OS E2E Vagrantfile to a directory containing a DC/OS installer file:

((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/|github-owner|/|github-repository|/master/vagrant/Vagrantfile')) | Set-Content -LiteralPath Vagrantfile

  • By default, the Vagrantfile installs DC/OS E2E from the most recent release at the time it is downloaded. To use a different release, or any Git reference, set the environment variable DCOS_E2E_REF:

$env:DCOS_E2E_REF = "master"

  • Start the virtual machine and login:

vagrant up
vagrant ssh

You can now run minidcos docker commands or use the library.

To connect to the cluster nodes from the Windows host (e.g. to use the DC/OS web interface), in PowerShell Run as Administrator, and add the Virtual Machine as a gateway:

route add 172.17.0.0 MASK 255.255.0.0 192.168.18.2

To shutdown, logout of the virtual machine shell, and destroy the virtual machine and disk:

vagrant destroy

The route will be removed on reboot. You can manually remove the route in PowerShell Run as Administrator using:

route delete 172.17.0.0

doctor command

DC/OS E2E comes with the minidcos docker doctor command. Run this command to check your system for common causes of problems.

DC/OS Installation

Nodes of Clusters created by the Docker backend do not distinguish between public_ip_address and private_ip_address.

Limitations

Docker does not represent a real DC/OS environment with complete accuracy. This section describes the currently known differences between the Docker backend and a real DC/OS environment.

SELinux

Tests inherit the host’s environment. Any tests that rely on SELinux being available require it be available on the host.

Storage

Docker does not support storage features expected in a real DC/OS environment.

Reboot

DC/OS nodes cannot be rebooted. The cluster cannot survive a system reboot.

Resources UI

The DC/OS web UI Resources tab shows resources available to each DC/OS node. On a cluster with a Docker backend, resources are not specific to one node. That means, if the cluster has 16 GB memory available in total, the web UI will show that each node has 16 GB memory available.

Marathon-LB

Network configuration options vary by kernel version. If you see the following when installing Marathon-LB, change the Marathon-LB sysctl-params configuration value:

sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_reuse: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_syn_backlog: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_tw_buckets: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_orphans: No such file or directory

Remove the relevant sysctl-params values. This may leave:

net.ipv4.tcp_fin_timeout=30 net.core.somaxconn=10000

Troubleshooting

Cleaning Up and Fixing “Out of Space” Errors

If a test is interrupted, it can leave behind containers, volumes and files. To remove these, run the following:

 minidcos docker clean

macOS File Sharing

On macOS /tmp is a symlink to /private/tmp. /tmp is used by the harness. Docker for Mac must be configured to allow /private to be bind mounted into Docker containers. This is the default. See Docker > Preferences > File Sharing.

Clock sync errors

On various platforms, the clock can get out of sync between the host machine and Docker containers. This is particularly problematic if using check_time: true in the DC/OS configuration. To work around this, run docker run --rm --privileged alpine hwclock -s.

Reference

class dcos_e2e.backends.Docker(container_name_prefix='dcos-e2e', workspace_dir=None, custom_container_mounts=None, custom_master_mounts=None, custom_agent_mounts=None, custom_public_agent_mounts=None, linux_distribution=<Distribution.CENTOS_7: 1>, docker_version=<DockerVersion.v18_06_3_ce: 4>, storage_driver=None, docker_container_labels=None, docker_master_labels=None, docker_agent_labels=None, docker_public_agent_labels=None, transport=<Transport.DOCKER_EXEC: 2>, network=None, one_master_host_port_map=None, mount_sys_fs_cgroup=True)

Create a configuration for a Docker cluster backend.

Parameters

  • container_name_prefix (str) – The prefix that all container names will start with. This is useful, for example, for later finding all containers started with this backend.

  • workspace_dir (Optional[Path]) – The directory in which large temporary files will be created. These files will be deleted when the cluster is destroyed. This is equivalent to dir in tempfile.mkstemp().

  • custom_container_mounts (Optional[List[Mount]]) – Custom mounts add to all node containers. See mounts in Containers.run.

  • custom_master_mounts (Optional[List[Mount]]) – Custom mounts add to master node containers. See mounts in Containers.run.

  • custom_agent_mounts (Optional[List[Mount]]) – Custom mounts add to agent node containers. See mounts in Containers.run.

  • custom_public_agent_mounts (Optional[List[Mount]]) – Custom mounts add to public agent node containers. See mounts in Containers.run.

  • linux_distribution (Distribution) – The Linux distribution to boot DC/OS on.

  • docker_version (DockerVersion) – The Docker version to install on the cluster nodes.

  • storage_driver (Optional[DockerStorageDriver]) – The storage driver to use for Docker on the cluster nodes. By default, this is the host’s storage driver. If this is not one of aufs, overlay or overlay2, aufs is used.

  • docker_container_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster node containers. Akin to the dictionary option in Containers.run.

  • docker_master_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster master node containers. Akin to the dictionary option in Containers.run.

  • docker_agent_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster agent node containers. Akin to the dictionary option in Containers.run.

  • docker_public_agent_labels (Optional[Dict[str, str]]) – Docker labels to add to the cluster public agent node containers. Akin to the dictionary option in Containers.run.

  • transport (Transport) – The transport to use for communicating with nodes.

  • network (Optional[Network]) – The Docker network containers will be connected to. If no network is specified the docker0 bridge network is used. It may not be possible to SSH to containers on a custom network on macOS.

  • one_master_host_port_map (Optional[Dict[str, int]]) – The exposed host ports for one of the master nodes. This is useful on macOS on which the container IP is not directly accessible from the host. By exposing the host ports, the user can reach the services on the master node using the mapped host ports. The host port map will be applied to one master only if there are multiple master nodes. See ports in Containers.run. Currently, only Transmission Control Protocol is supported.

  • mount_sys_fs_cgroup (bool) – Whether to mount /sys/fs/cgroup from the host. This is required to run applications which require cgroup isolation.

default_user

A user which can be used to SSH into nodes.

workspace_dir

The directory in which large temporary files will be created. These files will be deleted at the end of a test run.

custom_container_mounts

Custom mounts add to all node containers. See mounts in Containers.run.

custom_master_mounts

Custom mounts add to master node containers. See mounts in Containers.run.

custom_agent_mounts

Custom mounts add to agent node containers. See mounts in Containers.run.

custom_public_agent_mounts

Custom mounts add to public agent node containers. See mounts in Containers.run.

linux_distribution

The Linux distribution to boot DC/OS on.

docker_version

The Docker version to install on the cluster nodes.

docker_storage_driver

The storage driver to use for Docker on the cluster nodes.

docker_container_labels

Docker labels to add to the cluster node containers. Akin to the dictionary option in Containers.run.

docker_master_labels

Docker labels to add to the cluster master node containers. Akin to the dictionary option in Containers.run.

docker_agent_labels

Docker labels to add to the cluster agent node containers. Akin to the dictionary option in Containers.run.

docker_public_agent_labels

Docker labels to add to the cluster public agent node containers. Akin to the dictionary option in Containers.run.

transport

The transport to use for communicating with nodes.

network

The Docker network containers will be connected to. If no network is specified the docker0 bridge network is used. It may not be possible to SSH to containers on a custom network on macOS.

one_master_host_port_map

The exposed host ports for one of the master nodes. This is useful on macOS on which the container IP is not directly accessible from the host. By exposing the host ports, the user can reach the services on the master node using the mapped host ports. The host port map will be applied to one master only if there are multiple master nodes. See ports in Containers.run. Currently, only Transmission Control Protocol is supported.

container_name_prefix

The prefix that all container names will start with. This is useful, for example, for later finding all containers started with this backend.

cgroup_mounts

Mounts to use for cgroups.