AWS Backend¶
The AWS backend is used to spin up clusters using EC2 instances on Amazon Web Services, where each instance is a DC/OS node.
Requirements¶
Amazon Web Services¶
An Amazon Web Services account with sufficient funds must be available.
The AWS credentials for the account must be present either in the environment as environment variables or in the default file system location under ~/.aws/credentials
with a AWS profile in the environment referencing those credentials.
The Mesosphere internal AWS tool maws automatically stores account specific temporary AWS credentials in the default file system location and exports the corresponding profile into the environment. After logging in with maws clusters can be launched using the AWS backend.
For CI deployments long lived credentials are preferred. It is recommended to use the environment variables method for AWS credentials in that case.
The environment variables are set as follows:
export AWS_ACCESS_KEY_ID=<aws_access_key_id>
export AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>
The EC2 instances launched by the AWS backend will bring about costs in the order of 24 ct per instance, assuming the fixed cluster lifetime of two hours and m4.large
EC2 instances.
ssh
¶
The ssh
command must be available.
Operating System¶
The AWS backend has been tested on macOS and on Linux.
It is not expected that it will work out of the box with Windows, see issue QUALITY-1771.
If your operating system is not supported, it may be possible to use Vagrant, or another Linux virtual machine.
DC/OS Installation¶
Cluster
s created by the AWS
backend only support installing DC/OS via install_dcos_from_url()
.
This is because the installation method employs a bootstrap node that directly downloads the build_artifact
from the specified URL.
Node
s of Cluster
s created by the AWS
backend distinguish between public_ip_address
and private_ip_address
.
The private_ip_address
refers to the internal network of the AWS stack which is also used by DC/OS internally.
The public_ip_address
allows for reaching AWS EC2 instances from the outside e.g. from the dcos-e2e
testing environment.
AWS Regions¶
When launching a cluster with Amazon Web Services there are a number of different regions to choose from where the cluster is launched using aws_region
.
It is recommended to use us-west-1
or us-west-2
to keep the cost low.
See the AWS Regions and Availability Zones for available regions.
Restricting access to the cluster¶
The AWS backend takes a parameter admin_location
.
This parameter restricts the access to the AWS stack from the outside to a particular IP address range.
The default value '0.0.0.0/0'
will allow accessing the cluster from anywhere.
It is recommended to restrict the address range to a subnet including the public IP of the machine executing tests with the AWS backend.
For example <external-ip>/24
.
Accessing cluster nodes¶
SSH can be used to access cluster nodes for the purpose of debugging if workspace_dir
is set.
The AWS backend generates a SSH key file id_rsa
in a cluster-specific sub-directory under the workspace_dir
directory. The sub-directory is named after the unique cluster ID generated during cluster creation. The cluster ID is prefixed with dcos-e2e-
and can be found through the DC/OS UI in the upper left corner or through the CCM UI when using maws with a Mesosphere AWS account.
Adding this key to the ssh-agent
or manually providing it via the -i
flag after changing its file permissions to 400
will allow for connecting to the cluster via the ssh
command.
The SSH user depends on the linux_distribution
given to the AWS
backend.
For CENTOS_7
that is centos
.
It is important to keep in mind files in the given workspace_dir
are temporary and are removed when the cluster is destroyed.
If workspace_dir
is unset the AWS
backend will create a new temporary directory in an operating system specific location.
Cluster lifetime¶
The cluster lifetime is fixed at two hours.
If the cluster was launched with maws (Mesosphere temporary AWS credentials) the cluster can be controlled via CCM. This allows for extending the cluster lifetime and also for cleaning up the cluster if anything goes wrong.
EC2 instance types¶
Currently the AWS backend launches m4.large
instances for all DC/OS nodes.
Unsupported DC/OS versions¶
The AWS backend does currently not support DC/OS versions below 1.10. Adding support for DC/OS 1.9 is tracked in issue DCOS-21960.
Unsupported features¶
The AWS backend does currently not support the Cluster
feature of copying files to the DC/OS installer by supplying files_to_copy_to_installer
.
The progress on this feature is tracked in issue DCOS-21894.
Troubleshooting¶
In case of an error during the DC/OS installation the journal from each node will be dumped and downloaded to the folder that the tests were executed in.
The logs are prefixed with the installation phase that failed, preflight
, deploy
or postflight
.
When using temporary credentials it is required to pay attention that the credentials are still valid or renewed when destroying a cluster. If the credentials are not valid anymore the AWS backend does not delete the public/private key pair generated during cluster creation. It is therefore recommended to periodically renew temporary AWS credentials when executing tests using the AWS backend.
In rare cases it might also happen that a AWS stack deployment fails with the message ROLLBACK_IN_PROGRESS
.
In that case at least one of the EC2 instances failed to come up. Starting a new cluster is the only option then.
Reference¶
-
class
dcos_e2e.backends.
AWS
(aws_region='us-west-2', admin_location='0.0.0.0/0', linux_distribution=<Distribution.CENTOS_7: 1>, workspace_dir=None, aws_key_pair=None)¶ Create a configuration for an AWS cluster backend.
Parameters: - admin_location (
str
) – The IP address range from which the AWS nodes can be accessed. - aws_region (
str
) – The AWS location to create nodes in. See Regions and Availability Zones. - linux_distribution (
Distribution
) – The Linux distribution to boot DC/OS on. - workspace_dir (
Optional
[Path
]) – The directory in which large temporary files will be created. These files will be deleted at the end of a test run. This is equivalent to dir intempfile.mkstemp()
. - aws_key_pair (
Optional
[Tuple
[str
,Path
]]) – An optional tuple of (name, path) where the name is the identifier of an existing SSH public key on AWS KeyPairs and the path is the local path to the corresponding private key. The private key can then be used to connect to the cluster. If this is not given, a new key pair will be generated.
-
admin_location
¶ The IP address range from which the AWS nodes can be accessed.
-
aws_region
¶ The AWS location to create nodes in. See Regions and Availability Zones.
-
linux_distribution
¶ The Linux distribution to boot DC/OS on.
-
workspace_dir
¶ The directory in which large temporary files will be created. These files will be deleted at the end of a test run.
-
aws_key_pair
¶ An optional tuple of (name, path) where the name is the identifier of an existing SSH public key on AWS KeyPairs and the path is the local path to the corresponding private key. The private key can then be used to connect to the cluster.
Raises: NotImplementedError
– In case an unsupported Linux distribution has been passed in at backend creation.- admin_location (