Deploy Docker Swarm on AWS EC2 via cloud-formation templates - Step 4 - Manager Instance

In this step we will configure and launch the Manager EC2 instance.

This post is part of a thread that includes these steps:

1. Network Setup
2. Storage
3. Roles
4. Manager Instance (this post)
5. Worker Launch Template
6. Worker Instances
7. Docker Swarm
8. Cleanup

Manager Instance (AWS EC2)

Start in the project directory:

cd ~/swift-aws-ec2-swarm

cloud-formation Template

Create a folder ec2-manager and a ec2-manager.yml file in it.

mkdir -p ec2-manager
touch ec2-manager/ec2-manager.yml
nano ec2-manager/ec2-manager.yml

Copy and paste this code into ec2-manager.yml:

Description: Docker Swarm Manager instance

Parameters:
  KeyPair:
    Type: AWS::EC2::KeyPair::KeyName
    Description: Key pair that will be used to launch instances

  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: Subnet in the VPC where the instance will be launched

  SecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group for the instance

  InstanceProfile:
    Type: String
    Description: Instance profile to use for the instance

  InstanceType:
    Type: String
    Default: c5.large
    Description: Instance type to use for the instance

  LatestAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

  HostedZoneId:
    Type: AWS::Route53::HostedZone::Id
    Description: ID of the Route53 HostedZone

  HomeVolumeId:
    Type: AWS::EC2::Volume::Id
    Description: ID of the volume to be mounted as /home

Resources:
  Instance:
    Type: AWS::EC2::Instance
    Properties: 
      KeyName: !Ref KeyPair

      SubnetId: !Ref SubnetId
      SecurityGroupIds:
        - !Ref SecurityGroupId      

      IamInstanceProfile: !Ref InstanceProfile

      InstanceType: !Ref InstanceType

      ImageId: !Ref LatestAmiId

      BlockDeviceMappings: 
        # Docker volume
        - DeviceName: /dev/sdi
          Ebs: 
            Encrypted: true
            DeleteOnTermination: true
            VolumeSize: 100
            VolumeType: gp2          

      # categories:
        - Key: Name
          Value: manager

      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash -xe

            # see: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-log-user-data/
            exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

            EC2_INSTANCE_ID=$(ec2-metadata --instance-id | awk '{print $2}')
            EC2_REGION=$(ec2-metadata --availability-zone | awk '{print $2}' | sed 's/.$//')

            ## Timezone
            # see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#change_time_zone
            timedatectl set-timezone America/Los_Angeles

            ## DNS
            PRIVATE_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
            DNS_NAME=$(aws ec2 describe-categories --filters "Name=resource-id,Values=$EC2_INSTANCE_ID" "Name=key,Values=Name" --region $EC2_REGION --output=text | cut -f5)
            aws route53 change-resource-record-sets --hosted-zone-id ${HostedZoneId} --change-batch '{
              "Changes": [
                {
                  "Action": "UPSERT",
                  "ResourceRecordSet": {
                    "Name": "'$DNS_NAME'.swift.internal.",
                    "Type": "A",
                    "TTL": 60,
                    "ResourceRecords": [
                      {
                        "Value": "'$PRIVATE_IP'"
                      }
                    ]
                  }
                }
              ]
            }'

            # Add the .swift.internal domain to the list of searchable domains
            echo search swift.internal >> /etc/resolv.conf

            # Amazon Linux specific hack to preserve the domain search config between reboots
            echo 'prepend domain-search "swift.internal";' >> /etc/dhcp/dhclient.conf

            ## Hostname
            # Change hostname to the DNS NAME, which in turn is the name tag of the instance 
            hostnamectl set-hostname $DNS_NAME.swift.internal

            # Amazon EC2 specific hack to preserve hostname between reboots
            echo 'preserve_hostname: true' >> /etc/cloud/cloud.cfg


            ## Attach the EBS volumes
            # see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
            # home
            aws ec2 attach-volume --region $EC2_REGION --instance-id $EC2_INSTANCE_ID --volume-id ${HomeVolumeId} --device /dev/sdh
            while [ ! -e /dev/sdh ]; do 
              echo Waiting for Home EBS volume to attach
              sleep 30
            done

            # docker
            # Docker volume is not shared. Each instance has its own Docker volume 


            ## Format EBS volumes
            # Check if formatted and if not, format using ext4
            # home
            device_fs_type=`file -sL /dev/sdh`
            if [[ $device_fs_type != *"ext4"* ]]; then
                mkfs --type ext4 /dev/sdh
            fi

            # docker
            device_fs_type=`file -sL /dev/sdi`
            if [[ $device_fs_type != *"ext4"* ]]; then
                mkfs --type ext4 /dev/sdi
            fi


            ## Mount EBS file systems
            # home
            mkdir -p /ebs/home
            echo '/dev/sdh /ebs/home ext4 defaults,nofail 0 2' | tee -a /etc/fstab

            # docker
            mkdir -p /ebs/docker
            echo '/dev/sdi /ebs/docker ext4 defaults,nofail 0 2' | tee -a /etc/fstab

            mount --all


            ## Users
            # add users
            # runner
            groupadd --gid 200000 runner 
            useradd --gid runner --uid 200000  runner

            # worker
            useradd --create-home --home-dir /ebs/home/worker worker


            ## Install Software
            yum update -y
            yum install docker git jq htop -y


            ## Docker config
            #see: https://docs.docker.com/engine/security/userns-remap/

            # - Use `/ebs/docker` as data-root (for containers and volumes)
            # - Map container `root` user to host `runner` user             
            cat > /etc/docker/daemon.json <<EOF
            {
              "data-root": "/ebs/docker",
              "userns-remap": "runner"
            }            
            EOF

            # additional config needed for the Docker user namespace mapping
            touch /etc/subuid /etc/subgid
            echo "runner:$(id -u runner):65536" | sudo tee -a /etc/subuid
            echo "runner:$(id -g runner):65536" | sudo tee -a /etc/subgid

            # Enable Docker to run at boot and start it
            systemctl enable docker
            systemctl start docker

            # add users to the docker group
            usermod --append --groups docker ec2-user
            usermod --append --groups docker worker


            ## SSH

            # enable ssh login for the worker account
            user_home=/ebs/home/worker
            if [ ! -f "$user_home/.ssh/id_rsa" ]; then
              sudo -iu worker sh -c "ssh-keygen -t rsa -f $user_home/.ssh/id_rsa -q -P ''" 
              sudo -iu worker sh -c "cat $user_home/.ssh/id_rsa.pub > $user_home/.ssh/authorized_keys"
              sudo -iu worker sh -c "chmod 700 $user_home/.ssh"
              sudo -iu worker sh -c "chmod 600 $user_home/.ssh/authorized_keys"
            fi

            # download and install docker compose (optional)
            # platform=$(uname -s)-$(uname -m)
            # wget https://github.com/docker/compose/releases/latest/download/docker-compose-$platform 
            # mv docker-compose-$platform /usr/local/bin/docker-compose
            # chmod -v +x /usr/local/bin/docker-compose


            ## Install AWS CLI v2
            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
            unzip awscliv2.zip
            ./aws/install


            # signal that we are done
            /opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource Instance 
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M

Outputs:
  InstanceId:
    Description: ID of the launched instance
    Value: !Ref Instance

Scripts

Add a script deploy-ec2-manager.sh and paste this code in it:

#!/usr/bin/env bash

# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..

source config/names.sh

echo
echo "Deploying $stack_ec2_manager stack via cloud-formation:"
echo 'https://us-west-2.console.aws.amazon.com/cloudformation/home'
echo

subnet_id=$(aws ec2 describe-subnets --filters Name=tag:Name,Values=$subnet_pub_1 | jq -r '.Subnets[0].SubnetId')
security_group_id=$(aws ec2  describe-security-groups --filters Name=tag:Name,Values=$security_group_pub_1 | jq -r '.SecurityGroups[0].GroupId')

instance_profile=$(aws cloudformation describe-stacks --stack-name $stack_iam_manager | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "InstanceProfile") | .OutputValue')

hosted_zone_id=$(aws cloudformation describe-stacks --stack-name $stack_vpc | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "HostedZoneId") | .OutputValue')

# home volume is shared between manager and worker(s)
home_volume_id=$(aws cloudformation describe-stacks --stack-name $stack_ebs | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "HomeVolumeId") | .OutputValue')

set -x

aws cloudformation deploy \
    --template-file ec2-manager/ec2-manager.yml \
    --stack-name $stack_ec2_manager \
    --parameter-overrides \
        KeyPair=$ec2_key_pair \
        SubnetId=$subnet_id \
        SecurityGroupId=$security_group_id \
        InstanceProfile=$instance_profile \
        HostedZoneId=$hosted_zone_id \
        HomeVolumeId=$home_volume_id

popd

Let's also add a clean up script rm-ec2-manager.sh:

#!/usr/bin/env bash

# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..

source config/names.sh

echo
echo "Destroying $stack_ec2_manager stack via cloud-formation:"
echo 'https://us-west-2.console.aws.amazon.com/cloudformation/home'
echo

set -x

aws cloudformation delete-stack \
    --stack-name $stack_ec2_manager 

aws cloudformation wait stack-delete-complete \
    --stack-name $stack_ec2_manager

popd

Make the scripts executable:

chmod +x ec2-manager/deploy-ec2-manager.sh 
chmod +x ec2-manager/rm-ec2-manager.sh

Deploy

Finally let's run the "deploy" script to create the Manager instance:

./ec2-manager/deploy-ec2-manager.sh

You should see output similar to this:

Deploying swift-swarm-ec2-manager stack via cloud-formation:
https://us-west-2.console.aws.amazon.com/cloudformation/home

+ aws cloudformation deploy --template-file ec2-manager/ec2-manager.yml --stack-name swift-swarm-ec2-manager --parameter-overrides KeyPair=aws-ec2-key SubnetId=subnet-008f06da59b0f682a SecurityGroupId=sg-0668991ca731a2201 InstanceProfile=swift-swarm-iam-manager-InstanceProfile-qeqXudscgUtM HostedZoneId=Z07362313E0WMP6Y4DBYT HomeVolumeId=vol-08b4fb87713440e48

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - swift-swarm-ec2-manager

Login

Create a directory ssh and in it a script named ssh-manager.sh

mkdir -p ssh
touch ssh/ssh-manager.sh
chmod +x ssh/ssh-manager.sh
nano ssh/ssh-manager.sh

Copy and paste this code in the ssh-manager.sh file:

#!/usr/bin/env bash

# switch to parent directory
script_path=`dirname ${BASH_SOURCE[0]}`
pushd $script_path/..

source config/names.sh

popd

ec2_key="~/.ssh/aws-ec2-key"
ec2_instance=$( aws cloudformation describe-stacks --stack-name $stack_ec2_manager | jq -r '.Stacks[0].Outputs[] | select(.OutputKey == "InstanceId") | .OutputValue' )
ec2_ip=$( aws ec2 describe-instances --instance-ids $ec2_instance | jq -r '.Reservations[0].Instances[0].PublicIpAddress' )

ssh -i $ec2_key ec2-user@$ec2_ip

You should be able to connect to the Manager machine via ssh now:

./ssh/ssh-manager.sh

At this point your project structure should look like this:

.
├── config
│   └── names.sh
├── ebs
│   ├── deploy-ebs.sh
│   ├── ebs.yml
│   └── rm-ebs.sh
├── ec2-manager
│   ├── deploy-ec2-manager.sh
│   ├── ec2-manager.yml
│   └── rm-ec2-manager.sh
├── iam
│   ├── deploy-iam-manager.sh
│   ├── deploy-iam-worker.sh
│   ├── iam-manager.yml
│   ├── iam-worker.yml
│   ├── rm-iam-manager.sh
│   └── rm-iam-worker.sh
├── ssh
│   └── ssh-manager.sh
└── vpc
    ├── deploy-vpc.sh
    ├── rm-vpc.sh
    └── vpc.yml

Congratulations!

We are done with Step 4. Manager Instance.

Next step is: Step 5. Worker Launch Template