AWS Backend

The AWS backend is used to spin up clusters using EC2 instances on Amazon Web Services, where each instance is a DC/OS node.

Requirements

Amazon Web Services

An Amazon Web Services account with sufficient funds must be available.

The AWS credentials for the account must be present either in the environment as environment variables or in the default file system location under ~/.aws/credentials with a AWS profile in the environment referencing those credentials.

The Mesosphere internal AWS tool maws automatically stores account specific temporary AWS credentials in the default file system location and exports the corresponding profile into the environment. After logging in with maws clusters can be launched using the AWS backend.

For CI deployments long lived credentials are preferred. It is recommended to use the environment variables method for AWS credentials in that case.

The environment variables are set as follows:

export AWS_ACCESS_KEY_ID=<aws_access_key_id>
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>

The EC2 instances launched by the AWS backend will bring about costs in the order of 24 ct per instance, assuming the fixed cluster lifetime of two hours and m4.large EC2 instances.

ssh

The ssh command must be available.

Operating System

The AWS backend has been tested on macOS and on Linux.

It is not expected that it will work out of the box with Windows, see issue QUALITY-1771.

If your operating system is not supported, it may be possible to use Vagrant, or another Linux virtual machine.

DC/OS Installation

Clusters created by the AWS backend only support installing DC/OS via install_dcos_from_url().

This is because the installation method employs a bootstrap node that directly downloads the build_artifact from the specified URL.

Nodes of Clusters created by the AWS backend distinguish between public_ip_address and private_ip_address. The private_ip_address refers to the internal network of the AWS stack which is also used by DC/OS internally. The public_ip_address allows for reaching AWS EC2 instances from the outside e.g. from the dcos-e2e testing environment.

AWS Regions

When launching a cluster with Amazon Web Services there are a number of different regions to choose from where the cluster is launched using aws_region. It is recommended to use us-west-1 or us-west-2 to keep the cost low.

See the AWS Regions and Availability Zones for available regions.

Restricting access to the cluster

The AWS backend takes a parameter admin_location. This parameter restricts the access to the AWS stack from the outside to a particular IP address range. The default value '0.0.0.0/0' will allow accessing the cluster from anywhere. It is recommended to restrict the address range to a subnet including the public IP of the machine executing tests with the AWS backend. For example <external-ip>/24.

Accessing cluster nodes

SSH can be used to access cluster nodes for the purpose of debugging if workspace_dir is set. The AWS backend generates a SSH key file id_rsa in a cluster-specific sub-directory under the workspace_dir directory. The sub-directory is named after the unique cluster ID generated during cluster creation. The cluster ID is prefixed with dcos-e2e- and can be found through the DC/OS UI in the upper left corner or through the CCM UI when using maws with a Mesosphere AWS account. Adding this key to the ssh-agent or manually providing it via the -i flag after changing its file permissions to 400 will allow for connecting to the cluster via the ssh command. The SSH user depends on the linux_distribution given to the AWS backend. For CENTOS_7 that is centos.

It is important to keep in mind files in the given workspace_dir are temporary and are removed when the cluster is destroyed. If workspace_dir is unset the AWS backend will create a new temporary directory in an operating system specific location.

Cluster lifetime

The cluster lifetime is fixed at two hours.

If the cluster was launched with maws (Mesosphere temporary AWS credentials) the cluster can be controlled via CCM. This allows for extending the cluster lifetime and also for cleaning up the cluster if anything goes wrong.

EC2 instance types

By default the AWS backend launches m4.large instances for all DC/OS nodes.

It is possible to choose a different instance type through the aws_instance_type parameter.

See the AWS Instance types for available instance types.

Unsupported DC/OS versions

The AWS backend does currently not support DC/OS versions below 1.10. Adding support for DC/OS 1.9 is tracked in issue DCOS-21960.

Unsupported features

The AWS backend does currently not support the Cluster feature of copying files to the DC/OS installer by supplying files_to_copy_to_installer. The progress on this feature is tracked in issue DCOS-21894.

Troubleshooting

In case of an error during the DC/OS installation the journal from each node will be dumped and downloaded to the folder that the tests were executed in. The logs are prefixed with the installation phase that failed, preflight, deploy or postflight.

When using temporary credentials it is required to pay attention that the credentials are still valid or renewed when destroying a cluster. If the credentials are not valid anymore the AWS backend does not delete the public/private key pair generated during cluster creation. It is therefore recommended to periodically renew temporary AWS credentials when executing tests using the AWS backend.

In rare cases it might also happen that a AWS stack deployment fails with the message ROLLBACK_IN_PROGRESS. In that case at least one of the EC2 instances failed to come up. Starting a new cluster is the only option then.

Reference

class dcos_e2e.backends.AWS(aws_instance_type='m4.large', aws_region='us-west-2', admin_location='0.0.0.0/0', linux_distribution=<Distribution.CENTOS_7: 1>, workspace_dir=None, aws_key_pair=None)

Create a configuration for an AWS cluster backend.

Parameters:

• admin_location (str) – The IP address range from which the AWS nodes can be accessed.
• aws_instance_type (str) – The AWS instance type to use. See Instance types.
• aws_region (str) – The AWS location to create nodes in. See Regions and Availability Zones.
• linux_distribution (Distribution) – The Linux distribution to boot DC/OS on.
• workspace_dir (Optional[Path]) – The directory in which large temporary files will be created. These files will be deleted at the end of a test run. This is equivalent to dir in tempfile.mkstemp().
• aws_key_pair (Optional[Tuple[str, Path]]) – An optional tuple of (name, path) where the name is the identifier of an existing SSH public key on AWS KeyPairs and the path is the local path to the corresponding private key. The private key can then be used to connect to the cluster. If this is not given, a new key pair will be generated.

admin_location

The IP address range from which the AWS nodes can be accessed.

aws_instance_type

The AWS instance type to use. See Instance types.

aws_region

The AWS location to create nodes in. See Regions and Availability Zones.

linux_distribution

The Linux distribution to boot DC/OS on.

workspace_dir

The directory in which large temporary files will be created. These files will be deleted at the end of a test run.

aws_key_pair

An optional tuple of (name, path) where the name is the identifier of an existing SSH public key on AWS KeyPairs and the path is the local path to the corresponding private key. The private key can then be used to connect to the cluster.

Raises:NotImplementedError – In case an unsupported Linux distribution has been passed in at backend creation.