`dcos-docker` CLI¶

The dcos-docker CLI allows you to create, manage and destroy open source DC/OS and DC/OS Enterprise clusters on Docker nodes.

A typical CLI workflow for open source DC/OS may look like the following. Install the CLI, then create, manage and destroy a cluster:

# Fix issues shown by dcos-docker doctor
$ dcos-docker doctor
$ dcos-docker create /tmp/dcos_generate_config.sh --agents 0
default
$ dcos-docker wait
$ dcos-docker run --sync-dir /path/to/dcos/checkout pytest -k test_tls
...
$ dcos-docker destroy

Each of these and more are described in detail below.

Requirements
Installation
Creating a Cluster
- DC/OS Enterprise
“default” Cluster ID
Running commands on Cluster Nodes
Getting on to a Cluster Node
Destroying Clusters
Viewing Debug Information
Running Integration Tests
Viewing the Web UI
Using a Custom CA Certificate
Limitations
- SELinux
- Storage
CLI Reference

Requirements ¶

Docker ¶

Docker version 17.06 or later must be installed.

Plenty of memory must be given to Docker. On Docker for Mac, this can be done from Docker > Preferences > Advanced. This backend has been tested with a four node cluster with 9 GB memory given to Docker.

IP Routing Set Up for Docker ¶

On macOS, hosts cannot connect to containers IP addresses by default. This is required, for example, to access the web UI, to SSH to nodes and to use the DC/OS CLI.

Once the CLI is installed, run dcos-docker setup-mac-network to set up IP routing.

Without this, it is still possible to use some features. In the library, specify transport as dcos_e2e.node.Transport.DOCKER_EXEC. In the CLI, specify the --transport and --skip-http-checks options where available.

`ssh`¶

The ssh command must be available to use the SSH transport.

Operating System ¶

This tool has been tested on macOS with Docker for Mac and on Linux.

It has also been tested on Windows on Vagrant.

Windows ¶

The only supported way to use the Docker backend on Windows is using Vagrant and VirtualBox.

Ensure Virtualization and VT-X support is enabled in your PC’s BIOS. Disable Hyper-V virtualization. See https://www.howtogeek.com/213795/how-to-enable-intel-vt-x-in-your-computers-bios-or-uefi-firmware/.
Install VirtualBox and VirtualBox Extension Pack.
Install Vagrant.
Install the Vagrant plugin for persistent disks:

vagrant plugin install vagrant-persistent-storage

Optionally install the Vagrant plugins to cache package downloads and keep guest additions updates:

vagrant plugin install vagrant-cachier
vagrant plugin install vagrant-vbguest

Start Powershell and download the DC/OS E2E Vagrantfile to a directory containing a DC/OS installer file:

((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/dcos/dcos-e2e/master/vagrant/Vagrantfile')) | Set-Content -LiteralPath Vagrantfile

By default, the Vagrantfile installs DC/OS E2E from the most recent release at the time it is downloaded. To use a different release, or any Git reference, set the environment variable DCOS_E2E_REF:

$env:DCOS_E2E_REF = "master"

Start the virtual machine and login:

vagrant up
vagrant ssh

You can now run dcos-docker CLI commands or use the Python Library.

To connect to the cluster nodes from the Windows host (e.g. to use the DC/OS web interface), in PowerShell Run as Administrator, and add the Virtual Machine as a gateway:

route add 172.17.0.0 MASK 255.255.0.0 192.168.18.2

To shutdown, logout of the virtual machine shell, and destroy the virtual machine and disk:

vagrant destroy

The route will be removed on reboot. You can manually remove the route in PowerShell Run as Administrator using:

route delete 172.17.0.0

`dcos-docker doctor`¶

DC/OS E2E comes with the dcos-docker doctor command. Run this command to check your system for common causes of problems.

Installation ¶

The CLI can be installed with Homebrew on macOS or Linuxbrew on Linux. The library and CLI can be installed together with pip on any Linux and macOS. See “Operating System” requirements for instructions on using the CLI on Windows in a Vagrant VM.

CLI on macOS With Homebrew ¶

To install the CLI on macOS, install Homebrew.

Then install the latest stable version:

brew install https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

To upgrade from an older version, run the following command:

brew upgrade https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

Or the latest master:

Homebrew installs the dependencies for the latest released version and so installing master may not work.

brew install --HEAD https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

Run dcos-docker doctor to make sure that your system is ready to go:

$ dcos-docker doctor

CLI on Linux With Linuxbrew ¶

One way to install the CLI on Linux is with Linuxbrew. Install Linuxbrew, then install the latest stable version:

brew install https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

To upgrade from an older version, run the following command:

brew upgrade https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

Or the latest master:

Homebrew installs the dependencies for the latest released version and so installing master may not work.

brew install --HEAD https://raw.githubusercontent.com/dcos/dcos-e2e/master/dcose2e.rb

Run dcos-docker doctor to make sure that your system is ready to go:

$ dcos-docker doctor

Library and CLI with Python ¶

If the CLI has been installed with Homebrew, you do not need to install the library to use the CLI.

Requires Python 3.5.2+. To avoid interfering with your system’s Python, we recommend using a virtualenv.

Check the Python version:

python3 --version

On Fedora, install Python development requirements:

sudo dnf install -y git python3-devel

On Ubuntu, install Python development requirements:

apt install -y gcc python3-dev

Optionally replace master with a particular version of DC/OS E2E. The latest release is 2018.07.01.0. See available versions.

If you are not in a virtualenv, you may have to use sudo before the following command, or --user after install.

pip3 install --upgrade git+https://github.com/dcos/dcos-e2e.git@master

Run dcos-docker doctor to make sure that your system is ready to go for the Docker backend:

$ dcos-docker doctor

Creating a Cluster ¶

To create a cluster you first need to download a DC/OS release.

DC/OS Enterprise is also supported. Ask your sales representative for release artifacts.

Creating a cluster is possible with the dcos-docker create command. This command allows you to customize the cluster in many ways.

See the dcos-docker create reference for details on this command and its options.

The command returns when the DC/OS installation process has started. To wait until DC/OS has finished installing, use the the dcos-docker wait command.

To use this cluster, it is useful to find details using the the dcos-docker inspect command.

DC/OS Enterprise ¶

There are multiple DC/OS Enterprise-only features available in dcos-docker create.

The only extra requirement is to give a valid license key, for DC/OS 1.11+. See the dcos-docker create reference for details on how to provide a license key.

Ask your sales representative for DC/OS Enterprise release artifacts.

For, example, run the following to create a DC/OS Enterprise cluster in strict mode:

$ dcos-docker create /path/to/dcos_generate_config.ee.sh \
     --license-key /path/to/license.txt \
     --security-mode strict

The command returns when the DC/OS installation process has started. To wait until DC/OS has finished installing, use the dcos-docker wait command.

See the dcos-docker create reference for details on this command and its options.

“default” Cluster ID ¶

It can become tedious repeatedly typing the cluster ID, particularly if you only have one cluster. Any command which takes a cluster-id option defaults to using “default” if no cluster ID is given. This means that you can use dcos-docker wait with no arguments to wait for the default cluster.

Running commands on Cluster Nodes ¶

It is possible to run commands on a cluster node in multiple ways. These include using dcos-docker run, docker exec and ssh.

Running commands on a cluster node using dcos-docker run ¶

It is possible to run the following to run a command on an arbitrary master node.

$ dcos-docker run --cluster-id example systemctl list-units

See the dcos-docker run reference for more information on this command. In particular see the --node option to choose a particular node to run the command on.

Running commands on a cluster node using `docker exec`¶

Each cluster node is a Docker container. This means that you can use tools such as docker exec to run commands on nodes. To do this, first choose the container ID of a node. Use dcos-docker inspect to see all node container IDs.

Alternatively, use the --env flag to output commands to be evaluated as such:

$ eval $(dcos-docker inspect --cluster-id example --env)
$ docker exec -it $MASTER_0 systemctl list-units

Which environment variables are available depends on the size of your cluster.

Running commands on a cluster node using `ssh`¶

One SSH key allows access to all nodes in the cluster. See this SSH key’s path and the IP addresses of nodes using dcos-docker inspect. The available SSH user is root.

Getting on to a Cluster Node ¶

Sometimes it is useful to get onto a cluster node. To do this, you can use any of the ways of Running commands on Cluster Nodes.

For example, to use dcos-docker run to run bash to get on to an arbitrary master node:

$ dcos-docker run --cluster-id example bash

or, similarly, to use docker exec to get on to a specific node:

$ eval $(dcos-docker inspect --cluster-id example --env)
$ docker exec -it $MASTER_0 bash

See Running commands on Cluster Nodes for details on how to choose particular nodes.

Destroying Clusters ¶

There are two commands which can be used to destroy clusters. These are dcos-docker destroy and dcos-docker destroy-list.

Either destroy a cluster with dcos-docker destroy:

$ dcos-docker destroy
default
$ dcos-docker destroy pr_4033_strict
pr_4033_strict

or use dcos-docker destroy-list to destroy multiple clusters:

$ dcos-docker destroy-list pr_4033_strict pr_4019_permissive
pr_4033_strict
pr_4019_permissive

To destroy all clusters, run the following command:

$ dcos-docker destroy-list $(dcos-docker list)
pr_4033_strict
pr_4019_permissive

Viewing Debug Information ¶

The CLI is quiet by default. To see more information, use -v or -vv after dcos-docker.

Running Integration Tests ¶

The dcos-docker run command is useful for running integration tests.

To run integration tests which are developed in the a DC/OS checkout at /path/to/dcos, you can use the following workflow:

$ dcos-docker create /tmp/dcos_generate_config.ee.sh
$ dcos-docker wait
$ dcos-docker run --sync-dir /path/to/dcos/checkout pytest -k test_tls.py

There are multiple options and shortcuts for using these commands. See the dcos-docker run reference for more information on this command.

Viewing the Web UI ¶

To view the web UI of your cluster, use the dcos-docker web command. If you instead want to view the web UI URL of your cluster, use the dcos-docker inspect command.

Before viewing the UI, you may first need to configure your browser to trust your DC/OS CA, or choose to override the browser protection.

Using a Custom CA Certificate ¶

On DC/OS Enterprise clusters, it is possible to use a custom CA certificate. See the Custom CA certificate documentation for details. It is possible to use dcos-docker create to create a cluster with a custom CA certificate.

Create or obtain the necessary files:

dcos-ca-certificate.crt, dcos-ca-certificate-key.key, and dcos-ca-certificate-chain.crt.
Put the above-mentioned files, into a directory, e.g. /path/to/genconf/.
Create a file containing the “extra” configuration.

dcos-docker create takes an --extra-config option. This adds the contents of the specified YAML file to a minimal DC/OS configuration.

Create a file with the following contents:
```
ca_certificate_path: genconf/dcos-ca-certificate.crt
ca_certificate_key_path: genconf/dcos-ca-certificate-key.key
ca_certificate_chain_path: genconf/dcos-ca-certificate-chain.crt
```

Create a cluster.

dcos-docker create \
    /path/to/dcos_generate_config.ee.sh \
    --genconf-dir /path/to/genconf/ \
    --copy-to-master /path/to/genconf/dcos-ca-certificate-key.key:/var/lib/dcos/pki/tls/CA/private/custom_ca.key \
    --license-key /path/to/license.txt \
    --extra-config config.yml \

Verify that everything has worked.

See Verify installation for steps to verify that the DC/OS Enterprise cluster was installed properly with the custom CA certificate.

Limitations ¶

Docker does not represent a real DC/OS environment with complete accuracy. This section describes the currently known differences between the Docker backend and a real DC/OS environment.

SELinux ¶

Tests inherit the host’s environment. Any tests that rely on SELinux being available require it be available on the host.

Storage ¶

Docker does not support storage features expected in a real DC/OS environment.

CLI Reference ¶

dcos-docker ¶

Manage DC/OS clusters on Docker.

dcos-docker [OPTIONS] COMMAND [ARGS]...

Options

--version¶: Show the version and exit.

-v, --verbose¶

Commands

create: Create a DC/OS cluster.

destroy: Destroy a cluster.

destroy-list: Destroy clusters.

destroy-mac-network: Destroy containers created by “dcos-docker…

doctor: Diagnose common issues which stop DC/OS E2E…

inspect: Show cluster details.

list: List all clusters.

run: Run an arbitrary command on a node.

setup-mac-network: Set up a network to connect to nodes on…

sync: Sync files from a DC/OS checkout to master…

wait: Wait for DC/OS to start.

web: Open the browser at the web UI.

dcos-docker create ¶

Create a DC/OS cluster.

DC/OS Enterprise

DC/OS Enterprise clusters require different configuration variables to DC/OS OSS. For example, enterprise clusters require the following configuration parameters:

superuser_username, superuser_password_hash, fault_domain_enabled, license_key_contents

These can all be set in --extra-config. However, some defaults are provided for all but the license key.

The default superuser username is admin. The default superuser password is admin. The default fault_domain_enabled is false.

license_key_contents must be set for DC/OS Enterprise 1.11 and above. This is set to one of the following, in order:

* The license_key_contents set in --extra-config. * The contents of the path given with --license-key. * The contents of the path set in the DCOS_LICENSE_KEY_PATH environment variable.

If none of these are set, license_key_contents is not given.

dcos-docker create [OPTIONS] ARTIFACT

Options

--docker-version <docker_version>¶: The Docker version to install on the nodes. [default: 1.13.1]

--linux-distribution <linux_distribution>¶: The Linux distribution to use on the nodes. [default: centos-7]

--docker-storage-driver <docker_storage_driver>¶: The storage driver to use for Docker in Docker. By default this uses the host’s driver.

--masters <masters>¶: The number of master nodes. [default: 1]

--agents <agents>¶: The number of agent nodes. [default: 1]

--public-agents <public_agents>¶: The number of public agent nodes. [default: 1]

--extra-config <extra_config>¶: The path to a file including DC/OS configuration YAML. The contents of this file will be added to add to a default configuration.

--security-mode <security_mode>¶: The security mode to use for a DC/OS Enterprise cluster. This overrides any security mode set in --extra-config.

-c, --cluster-id <cluster_id>¶: A unique identifier for the cluster. Use the value “default” to use this cluster for other commands without specifying –cluster-id.

--license-key <license_key>¶: This is ignored if using open source DC/OS. If using DC/OS Enterprise, this defaults to the value of the DCOS_LICENSE_KEY_PATH environment variable.

--genconf-dir <genconf_dir>¶: Path to a directory that contains additional files for DC/OS installer. All files from this directory will be copied to the genconf directory before running DC/OS installer.

--copy-to-master <copy_to_master>¶: Files to copy to master nodes before installing DC/OS. This option can be given multiple times. Each option should be in the format /absolute/local/path:/remote/path.

--workspace-dir <workspace_dir>¶: Creating a cluster can use approximately 2 GB of temporary storage. Set this option to use a custom “workspace” for this temporary storage. See https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir for details on the temporary directory location if this option is not set.

--custom-volume <custom_volume>¶: Bind mount a volume on all cluster node containers. See https://docs.docker.com/engine/reference/run/#volume-shared-filesystems for the syntax to use.

--custom-master-volume <custom_master_volume>¶: Bind mount a volume on all cluster master node containers. See https://docs.docker.com/engine/reference/run/#volume-shared-filesystems for the syntax to use.

--custom-agent-volume <custom_agent_volume>¶: Bind mount a volume on all cluster agent node containers. See https://docs.docker.com/engine/reference/run/#volume-shared-filesystems for the syntax to use.

--custom-public-agent-volume <custom_public_agent_volume>¶: Bind mount a volume on all cluster public agent node containers. See https://docs.docker.com/engine/reference/run/#volume-shared-filesystems for the syntax to use.

--variant <variant>¶: Choose the DC/OS variant. If the variant does not match the variant of the given artifact, an error will occur. Using “auto” finds the variant from the artifact. Finding the variant from the artifact takes some time and so using another option is a performance optimization.

--wait-for-dcos¶: Wait for DC/OS after creating the cluster. This is equivalent to using “dcos-docker wait” after this command. “dcos-docker wait” has various options available and so may be more appropriate for your use case. If the chosen transport is “docker-exec”, this will skip HTTP checks and so the cluster may not be fully ready.

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Arguments

ARTIFACT¶: Required argument

Environment variables

DCOS_LICENSE_KEY_PATH: Provide a default for --license-key

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker list ¶

List all clusters.

dcos-docker list [OPTIONS]

dcos-docker wait ¶

Wait for DC/OS to start.

dcos-docker wait [OPTIONS]

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

--superuser-username <superuser_username>¶: The superuser username is needed only on DC/OS Enterprise clusters. By default, on a DC/OS Enterprise cluster, admin is used.

--superuser-password <superuser_password>¶: The superuser password is needed only on DC/OS Enterprise clusters. By default, on a DC/OS Enterprise cluster, admin is used.

--skip-http-checks¶: Do not wait for checks which require an HTTP connection to the cluster. If this flag is used, this command may return before DC/OS is fully ready. Use this flag in cases where an HTTP connection cannot be made to the cluster. For example this is useful on macOS without a VPN set up.

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Environment variables

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker run ¶

Run an arbitrary command on a node.

This command sets up the environment so that pytest can be run.

For example, run dcos-docker run --cluster-id 1231599 pytest -k test_tls.py.

Or, with sync: dcos-docker run --sync-dir . --cluster-id 1231599 pytest -k test_tls.py.

To use special characters such as single quotes in your command, wrap the whole command in double quotes.

dcos-docker run [OPTIONS] NODE_ARGS...

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

--dcos-login-uname <dcos_login_uname>¶: The username to set the DCOS_LOGIN_UNAME environment variable to.

--dcos-login-pw <dcos_login_pw>¶: The password to set the DCOS_LOGIN_PW environment variable to.

--sync-dir <sync_dir>¶: The path to a DC/OS checkout. Part of this checkout will be synced to all master nodes before the command is run.

--no-test-env¶: With this flag set, no environment variables are set and the command is run in the home directory.

--node <node>¶: A reference to a particular node to run the command on. This can be one of: The node’s IP address, the node’s Docker container name, the node’s Docker container ID, a reference in the format “<role>_<number>”. These details be seen with dcos_docker inspect.

--env <env>¶: Set environment variables in the format “<KEY>=<VALUE>”

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Arguments

NODE_ARGS¶: Required argument(s)

Environment variables

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker inspect ¶

Show cluster details.

To quickly get environment variables to use with Docker tooling, use the --env flag.

Run eval $(dcos-docker inspect <CLUSTER_ID> --env), then run docker exec -it $MASTER_0 to enter the first master, for example.

dcos-docker inspect [OPTIONS]

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

--env¶: Show details in an environment variable format to eval.

dcos-docker sync ¶

Sync files from a DC/OS checkout to master nodes.

This syncs integration test files and bootstrap files.

DCOS_CHECKOUT_DIR should be set to the path of clone of an open source DC/OS or DC/OS Enterprise repository.

By default the DCOS_CHECKOUT_DIR argument is set to the value of the DCOS_CHECKOUT_DIR environment variable.

If no DCOS_CHECKOUT_DIR is given, the current working directory is used.

dcos-docker sync [OPTIONS] [DCOS_CHECKOUT_DIR]

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Arguments

DCOS_CHECKOUT_DIR¶: Optional argument

Environment variables

DCOS_CHECKOUT_DIR: Provide a default for DCOS_CHECKOUT_DIR

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker destroy ¶

Destroy a cluster.

dcos-docker destroy [OPTIONS]

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Environment variables

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker destroy-list ¶

Destroy clusters.

To destroy all clusters, run dcos-docker destroy $(dcos-docker list).

dcos-docker destroy-list [OPTIONS] [CLUSTER_IDS]...

Options

--transport <transport>¶: The communication transport to use. On macOS the SSH transport requires IP routing to be set up. See “dcos-docker setup-mac-network”.It also requires the “ssh” command to be available. This can be provided by setting the DCOS_DOCKER_TRANSPORT environment variable. [default: ssh]

Arguments

CLUSTER_IDS¶: Optional argument(s)

Environment variables

DCOS_DOCKER_TRANSPORT: Provide a default for --transport

dcos-docker doctor ¶

Diagnose common issues which stop DC/OS E2E from working correctly.

dcos-docker doctor [OPTIONS]

dcos-docker web ¶

Open the browser at the web UI.

Note that the web UI may not be available at first. Consider using dcos-docker wait before running this command.

dcos-docker web [OPTIONS]

Options

-c, --cluster-id <cluster_id>¶: The ID of the cluster to use. [default: default]

dcos-docker setup-mac-network ¶

Set up a network to connect to nodes on macOS.

This creates an OpenVPN configuration file and describes how to use it.

dcos-docker setup-mac-network [OPTIONS]

Options

--force¶: Overwrite any files and destroy conflicting containers from previous uses of this command.

--configuration-dst <configuration_dst>¶: The location to create an OpenVPN configuration file. [default: ~/Documents/docker-for-mac.ovpn]

dcos-docker destroy-mac-network ¶

Destroy containers created by “dcos-docker setup-mac-network”.

dcos-docker destroy-mac-network [OPTIONS]

dcos-docker CLI¶

`dcos-docker` CLI¶