Deploy Docker Swarm on AWS EC2 via cloud-formation templates - Step 4 - Manager Instance
In this step we will configure and launch the Manager EC2 instance.
This post is part of a thread that includes these steps:
- Network Setup
- Storage
- Roles
- Manager Instance (this post)
- Worker Launch Template
- Worker Instances
- Docker Swarm
- Cleanup
Manager Instance (AWS EC2)
Start in the project directory:
cloud-formation Template
Create a folder ec2-manager and a ec2-manager.yml file in it.
Copy and paste this code into ec2-manager.yml:
Description: Docker Swarm Manager instance
Parameters:
KeyPair:
Type: AWS::EC2::KeyPair::KeyName
Description: Key pair that will be used to launch instances
SubnetId:
Type: AWS::EC2::Subnet::Id
Description: Subnet in the VPC where the instance will be launched
SecurityGroupId:
Type: AWS::EC2::SecurityGroup::Id
Description: Security group for the instance
InstanceProfile:
Type: String
Description: Instance profile to use for the instance
InstanceType:
Type: String
Default: c5.large
Description: Instance type to use for the instance
LatestAmiId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
HostedZoneId:
Type: AWS::Route53::HostedZone::Id
Description: ID of the Route53 HostedZone
HomeVolumeId:
Type: AWS::EC2::Volume::Id
Description: ID of the volume to be mounted as /home
Resources:
Instance:
Type: AWS::EC2::Instance
Properties:
KeyName: !Ref KeyPair
SubnetId: !Ref SubnetId
SecurityGroupIds:
- !Ref SecurityGroupId
IamInstanceProfile: !Ref InstanceProfile
InstanceType: !Ref InstanceType
ImageId: !Ref LatestAmiId
BlockDeviceMappings:
# Docker volume
- DeviceName: /dev/sdi
Ebs:
Encrypted: true
DeleteOnTermination: true
VolumeSize: 100
VolumeType: gp2
# categories:
- Key: Name
Value: manager
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -xe
# see: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
EC2_INSTANCE_ID=$(ec2-metadata --instance-id | awk '{print $2}')
EC2_REGION=$(ec2-metadata --availability-zone | awk '{print $2}' | sed 's/.$//')
## Timezone
# see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#change_time_zone
timedatectl set-timezone America/Los_Angeles
## DNS
PRIVATE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
DNS_NAME=$(aws ec2 describe-categories --filters "Name=resource-id,Values=$EC2_INSTANCE_ID" "Name=key,Values=Name" --region $EC2_REGION --output=text | cut -f5)
aws route53 change-resource-record-sets --hosted-zone-id ${HostedZoneId} --change-batch '{
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "'$DNS_NAME'.swift.internal.",
"Type": "A",
"TTL": 60,
"ResourceRecords": [
{
"Value": "'$PRIVATE_IP'"
}
]
}
}
]
}'
# Add the .swift.internal domain to the list of searchable domains
echo search swift.internal >> /etc/resolv.conf
# Amazon Linux specific hack to preserve the domain search config between reboots
echo 'prepend domain-search "swift.internal";' >> /etc/dhcp/dhclient.conf
## Hostname
# Change hostname to the DNS NAME, which in turn is the name tag of the instance
hostnamectl set-hostname $DNS_NAME.swift.internal
# Amazon EC2 specific hack to preserve hostname between reboots
echo 'preserve_hostname: true' >> /etc/cloud/cloud.cfg
## Attach the EBS volumes
# see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
# home
aws ec2 attach-volume --region $EC2_REGION --instance-id $EC2_INSTANCE_ID --volume-id ${HomeVolumeId} --device /dev/sdh
while [ ! -e /dev/sdh ]; do
echo Waiting for Home EBS volume to attach
sleep 30
done
# docker
# Docker volume is not shared. Each instance has its own Docker volume
## Format EBS volumes
# Check if formatted and if not, format using ext4
# home
device_fs_type=`file -sL /dev/sdh`
if [[ $device_fs_type != *"ext4"* ]]; then
mkfs --type ext4 /dev/sdh
fi
# docker
device_fs_type=`file -sL /dev/sdi`
if [[ $device_fs_type != *"ext4"* ]]; then
mkfs --type ext4 /dev/sdi
fi
## Mount EBS file systems
# home
mkdir -p /ebs/home
echo '/dev/sdh /ebs/home ext4 defaults,nofail 0 2' | tee -a /etc/fstab
# docker
mkdir -p /ebs/docker
echo '/dev/sdi /ebs/docker ext4 defaults,nofail 0 2' | tee -a /etc/fstab
mount --all
## Users
# add users
# runner
groupadd --gid 200000 runner
useradd --gid runner --uid 200000 runner
# worker
useradd --create-home --home-dir /ebs/home/worker worker
## Install Software
yum update -y
yum install docker git jq htop -y
## Docker config
#see: https://docs.docker.com/engine/security/userns-remap/
# - Use `/ebs/docker` as data-root (for containers and volumes)
# - Map container `root` user to host `runner` user
cat > /etc/docker/daemon.json <<EOF
{
"data-root": "/ebs/docker",
"userns-remap": "runner"
}
EOF
# additional config needed for the Docker user namespace mapping
touch /etc/subuid /etc/subgid
echo "runner:$(id -u runner):65536" | sudo tee -a /etc/subuid
echo "runner:$(id -g runner):65536" | sudo tee -a /etc/subgid
# Enable Docker to run at boot and start it
systemctl enable docker
systemctl start docker
# add users to the docker group
usermod --append --groups docker ec2-user
usermod --append --groups docker worker
## SSH
# enable ssh login for the worker account
user_home=/ebs/home/worker
if [ ! -f "$user_home/.ssh/id_rsa" ]; then
sudo -iu worker sh -c "ssh-keygen -t rsa -f $user_home/.ssh/id_rsa -q -P ''"
sudo -iu worker sh -c "cat $user_home/.ssh/id_rsa.pub > $user_home/.ssh/authorized_keys"
sudo -iu worker sh -c "chmod 700 $user_home/.ssh"
sudo -iu worker sh -c "chmod 600 $user_home/.ssh/authorized_keys"
fi
# download and install docker compose (optional)
# platform=$(uname -s)-$(uname -m)
# wget https://github.com/docker/compose/releases/latest/download/docker-compose-$platform
# mv docker-compose-$platform /usr/local/bin/docker-compose
# chmod -v +x /usr/local/bin/docker-compose
## Install AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
# signal that we are done
/opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource Instance
CreationPolicy:
ResourceSignal:
Timeout: PT15M
Outputs:
InstanceId:
Description: ID of the launched instance
Value: !Ref Instance
Scripts
Add a script deploy-ec2-manager.sh and paste this code in it:
#!/usr/bin/env bash
# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..
source config/names.sh
echo
echo "Deploying $stack_ec2_manager stack via cloud-formation:"
echo 'https://us-west-2.console.aws.amazon.com/cloudformation/home'
echo
subnet_id=$(aws ec2 describe-subnets --filters Name=tag:Name,Values=$subnet_pub_1 | jq -r '.Subnets[0].SubnetId')
security_group_id=$(aws ec2 describe-security-groups --filters Name=tag:Name,Values=$security_group_pub_1 | jq -r '.SecurityGroups[0].GroupId')
instance_profile=$(aws cloudformation describe-stacks --stack-name $stack_iam_manager | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "InstanceProfile") | .OutputValue')
hosted_zone_id=$(aws cloudformation describe-stacks --stack-name $stack_vpc | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "HostedZoneId") | .OutputValue')
# home volume is shared between manager and worker(s)
home_volume_id=$(aws cloudformation describe-stacks --stack-name $stack_ebs | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "HomeVolumeId") | .OutputValue')
set -x
aws cloudformation deploy \
--template-file ec2-manager/ec2-manager.yml \
--stack-name $stack_ec2_manager \
--parameter-overrides \
KeyPair=$ec2_key_pair \
SubnetId=$subnet_id \
SecurityGroupId=$security_group_id \
InstanceProfile=$instance_profile \
HostedZoneId=$hosted_zone_id \
HomeVolumeId=$home_volume_id
popd
Let's also add a clean up script rm-ec2-manager.sh:
#!/usr/bin/env bash
# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..
source config/names.sh
echo
echo "Destroying $stack_ec2_manager stack via cloud-formation:"
echo 'https://us-west-2.console.aws.amazon.com/cloudformation/home'
echo
set -x
aws cloudformation delete-stack \
--stack-name $stack_ec2_manager
aws cloudformation wait stack-delete-complete \
--stack-name $stack_ec2_manager
popd
Make the scripts executable:
Deploy
Finally let's run the "deploy" script to create the Manager instance:
You should see output similar to this:
Deploying swift-swarm-ec2-manager stack via cloud-formation:
https://us-west-2.console.aws.amazon.com/cloudformation/home
+ aws cloudformation deploy --template-file ec2-manager/ec2-manager.yml --stack-name swift-swarm-ec2-manager --parameter-overrides KeyPair=aws-ec2-key SubnetId=subnet-008f06da59b0f682a SecurityGroupId=sg-0668991ca731a2201 InstanceProfile=swift-swarm-iam-manager-InstanceProfile-qeqXudscgUtM HostedZoneId=Z07362313E0WMP6Y4DBYT HomeVolumeId=vol-08b4fb87713440e48
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - swift-swarm-ec2-manager
Login
Create a directory ssh and in it a script named ssh-manager.sh
Copy and paste this code in the ssh-manager.sh file:
#!/usr/bin/env bash
# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..
source config/names.sh
popd
ec2_key="~/.ssh/aws-ec2-key"
ec2_instance=$( aws cloudformation describe-stacks --stack-name $stack_ec2_manager | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "InstanceId") | .OutputValue' )
ec2_ip=$( aws ec2 describe-instances --instance-ids $ec2_instance | jq -r '.Reservations[0].Instances[0].PublicIpAddress' )
ssh -i $ec2_key ec2-user@$ec2_ip
You should be able to connect to the Manager machine via ssh now:
At this point your project structure should look like this:
.
├── config
│ └── names.sh
├── ebs
│ ├── deploy-ebs.sh
│ ├── ebs.yml
│ └── rm-ebs.sh
├── ec2-manager
│ ├── deploy-ec2-manager.sh
│ ├── ec2-manager.yml
│ └── rm-ec2-manager.sh
├── iam
│ ├── deploy-iam-manager.sh
│ ├── deploy-iam-worker.sh
│ ├── iam-manager.yml
│ ├── iam-worker.yml
│ ├── rm-iam-manager.sh
│ └── rm-iam-worker.sh
├── ssh
│ └── ssh-manager.sh
└── vpc
├── deploy-vpc.sh
├── rm-vpc.sh
└── vpc.yml
Congratulations!
We are done with Step 4. Manager Instance.
Next step is: Step 5. Worker Launch Template