Here’s how you can use CloudFormation to have EC2, IAM, and S3 work together
This is the sixth article in our Infrastructure as Code blog series. Check out our other posts here:
-
Infrastructure as Code: 5 Reasons Why You Should Implement IaC Now
- 6 best practices to get the most out of IaC
-
15 Infrastructure as Code tools you can use to automate your deployments
-
What is AWS CloudFormation and how can it help your IaC efforts?
-
How AWS CloudFormation Works (and How to Create a Virtual Private Cloud with it)
-
How to create a Redshift stack with AWS CloudFormation
In our last article, we dug deep into how AWS CloudFormation works and provided an analysis of a VPC template we created.
Our next template example is that of SFTP Gateway, a product that we sell on the AWS Marketplace that makes it easy to transfer files via SFTP to Amazon S3. We have over 1000 customers using the product, so it’s a useful tool!
We’ll incorporate S3, EC2, IAM, Security Groups, and more to facilitate this file transfer.
Here’s the template in its entirety:
AWSTemplateFormatVersion: 2010-09-09 Mappings: RegionMap: ap-northeast-1: AMI: ami-0d1a9e6b ap-northeast-2: AMI: ami-0ca50362 ap-south-1: AMI: ami-989fd6f7 ap-southeast-1: AMI: ami-db7a25b8 ap-southeast-2: AMI: ami-8c14e0ee ca-central-1: AMI: ami-3d13a859 eu-central-1: AMI: ami-fd6fe192 eu-west-1: AMI: ami-e8922c91 eu-west-2: AMI: ami-09223c6d eu-west-3: AMI: ami-9563d4e8 sa-east-1: AMI: ami-eb4f0887 us-east-1: AMI: ami-c599febf us-east-2: AMI: ami-6a1c350f us-west-1: AMI: ami-99b983f9 us-west-2: AMI: ami-a7ca10df Resources: SFTPGatewayInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: commands: setup: command: /usr/local/bin/sftpgatewaysetup Properties: IamInstanceProfile: !Ref RootInstanceProfile ImageId: !FindInMap - RegionMap - !Ref AWS::Region - AMI InstanceType: !Ref EC2Type BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: VolumeSize: !Ref DiskVolumeSize VolumeType: gp2 KeyName: !Ref KeyPair SecurityGroupIds: - !Ref SFTPGatewaySG SubnetId: !Ref SubnetID Tags: - Key: Name Value: SFTPGateway Instance UserData: Fn::Base64: !Sub | #!/bin/bash yum update -y aws-cfn-bootstrap /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance \n --region ${AWS::Region} SFTPGatewayBucket: DeletionPolicy: Retain Type: AWS::S3::Bucket Properties: BucketName: !Sub sftpgateway-${SFTPGatewayInstance} DependsOn: - SFTPGatewayInstance RootInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref S3WritableRole S3WritableRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: ec2.amazonaws.com Action: sts:AssumeRole Path: / RolePolicies: Type: AWS::IAM::Policy DependsOn: - SFTPGatewayInstance Properties: PolicyName: SFTPGatewayInstancePolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: 's3:*' Resource: '*' Roles: - !Ref S3WritableRole SFTPGatewaySG: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: SFTPGateway Security Group VpcId: !Ref VPCIdName SecurityGroupIngress: - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0 IPAddress: Properties: Domain: vpc InstanceId: !Ref SFTPGatewayInstance Type: AWS::EC2::EIP Parameters: EC2Type: Description: SFTPGateway Instance Type Type: String Default: t2.micro AllowedValues: - t2.micro - t2.small - t2.medium - t2.large - m3.medium - m3.large - m3.xlarge - m4.large - m4.xlarge - c3.large - c3.xlarge - c4.large - c4.xlarge - r3.large - r3.xlarge KeyPair: Description: EC2 KeyPair Type: AWS::EC2::KeyPair::KeyName ConstraintDescription: Existing EC2 KeyPair. DiskVolumeSize: Default: 32 Description: Disk volume size in GB. Must be at least 32. ConstraintDescription: Must be a number greater or equal to 32 MinValue: 32 Type: Number VPCIdName: Description: Select the VPC to launch the SFTPGateway into Type: AWS::EC2::VPC::Id SubnetID: Description: Subnet ID Type: AWS::EC2::Subnet::Id Outputs: ElasticIP: Value: !Ref IPAddress Description: Elastic IP address
You can download the SFTP Gateway template here.
There’s a lot going on in the template, so we’ll just give a “brief” overview of what’s happening and point out some interesting syntax that you might use for your own projects.
Create an EC2 instance and S3 bucket
SFTP Gateway uses an EC2 server to upload files to S3. So we start off with an EC2 instance and S3 bucket:
Resources: SFTPGatewayInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: commands: setup: command: /usr/local/bin/sftpgatewaysetup Properties: IamInstanceProfile: !Ref RootInstanceProfile ImageId: !FindInMap - RegionMap - !Ref AWS::Region - AMI InstanceType: !Ref EC2Type BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: VolumeSize: !Ref DiskVolumeSize VolumeType: gp2 KeyName: !Ref KeyPair SecurityGroupIds: - !Ref SFTPGatewaySG SubnetId: !Ref SubnetID Tags: - Key: Name Value: SFTPGateway Instance UserData: Fn::Base64: !Sub | #!/bin/bash yum update -y aws-cfn-bootstrap /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region} SFTPGatewayBucket: DeletionPolicy: Retain Type: AWS::S3::Bucket Properties: BucketName: !Sub sftpgateway-${SFTPGatewayInstance} DependsOn: - SFTPGatewayInstance
Here are a few things worth noting:
- The EC2 instance has a Metadata section in addition to its properties. We’ll cover these in more detail below.
- The S3 bucket has a Deletion Policy of “Retain”. This means you keep the S3 bucket if you delete the CloudFormation stack.
- The S3 BucketName uses an intrinsic function called “!Sub”, which lets you do string interpolation. The syntax “${SFTPGatewayInstance}” gives you the EC2 instance ID, just like the “!Ref” function.
This is what we have so far:
Let’s take a closer look at the EC2 instance metadata and properties:
SFTPGatewayInstance: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: commands: setup: command: /usr/local/bin/sftpgatewaysetup Properties: IamInstanceProfile: !Ref RootInstanceProfile ImageId: !FindInMap - RegionMap - !Ref AWS::Region - AMI InstanceType: !Ref EC2Type BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: VolumeSize: !Ref DiskVolumeSize VolumeType: gp2 KeyName: !Ref KeyPair SecurityGroupIds: - !Ref SFTPGatewaySG SubnetId: !Ref SubnetID Tags: - Key: Name Value: SFTPGateway Instance UserData: Fn::Base64: !Sub | #!/bin/bash yum update -y aws-cfn-bootstrap /opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region}
There’s a lot going on here:
“CloudFormation::Init” – This is a powerful tool that lets you define config files and commands. In this case, we run a command called “sftpgatewaysetup” to initialize the software.
“ImageId” – This uses the “!FindInMap” intrinsic function that looks up an AMI ID based on the current region. The AMI mappings are located in the Mappings section of the CloudFormation template.
“InstanceType” – This refers to a parameter that we named “EC2Type” which gives you a drop-down list of common EC2 instance types.
“BlockDeviceMappings” – This sets the disk drive type to solid state (gp2). It also points to a parameter named “DiskVolumeSize” which allows the user to define disk size at stack creation.
“KeyName” – This refers to an SSH key that you use to log into the server.
“UserData” – This lets you run bash commands on server launch. In this case, we use “cfn-init” to read the “CloudFormation::Init” metadata we defined earlier.
A lot of the properties above reference parameters. Below is a snippet of the Parameters section of the template, which includes the “EC2Type”, “DiskVolumeSize”, and “KeyPair” parameters mentioned earlier:
. . . Parameters: EC2Type: Description: SFTPGateway Instance Type Type: String Default: t2.micro AllowedValues: - t2.micro - t2.small - t2.medium - t2.large - m3.medium - m3.large - m3.xlarge - m4.large - m4.xlarge - c3.large - c3.xlarge - c4.large - c4.xlarge - r3.large - r3.xlarge KeyPair: Description: EC2 KeyPair Type: AWS::EC2::KeyPair::KeyName ConstraintDescription: Existing EC2 KeyPair. DiskVolumeSize: Default: 32 Description: Disk volume size in GB. Must be at least 32. ConstraintDescription: Must be a number greater or equal to 32 MinValue: 32 Type: Number
Parameters let you pass dynamic values to make your template more flexible. Here are a few things to note:
- “AllowedValues” – This presents the user with a drop-down, so you don’t have to worry about form validation.
- “AWS::EC2::KeyPair::KeyName” – This is a special type that automatically presents the user with a list of key pairs in their AWS account.
- “MinValue” – This provides form validation that gives an error if the user puts in a value that is too small.
Want this in a handy eBook? Click here to download our 62-page Infrastructure as Code Handbook, which includes IaC benefits, best practices, tools, and analysis of three AWS CloudFormation scripts!
Create an IAM role
In order for our EC2 instance to access S3, we need to grant it permissions using an IAM role.
The architecture looks like this:
The diagram shown above simplifies what’s actually happening. If you look at the CloudFormation template, you’ll see that there’s more to it:
Resources: . . . RootInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref S3WritableRole S3WritableRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: ec2.amazonaws.com Action: sts:AssumeRole Path: / RolePolicies: Type: AWS::IAM::Policy DependsOn: - SFTPGatewayInstance Properties: PolicyName: SFTPGatewayInstancePolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: 's3:*' Resource: '*' Roles: - !Ref S3WritableRole
There are three resources involved when assigning permissions to an EC2 instance:
- “InstanceProfile” – You can’t assign an IAM role directly to the EC2 instance, but you can assign an instance profile, which passes role information to the EC2 instance.
- “IAM::Role” – The EC2 instance can assume a role and inherit any permissions from the role, via the instance profile.
- “IAM::Policy” – This contains the actual permissions. The policy is associated with the role.
Using an existing public subnet
The EC2 instance needs to be in a public subnet so that end users can access it via SFTP. This CloudFormation template doesn’t create this public subnet. Rather, you select an existing subnet and pass it as a parameter to the template.
This happens here:.
. . . Parameters: . . . VPCIdName: Description: Select the VPC to launch the SFTPGateway into Type: AWS::EC2::VPC::Id SubnetID: Description: Subnet ID Type: AWS::EC2::Subnet::Id
Here’s what’s going on:
- “AWS::EC2::Subnet::Id” – This is a special parameter type that lists existing subnets in your AWS account. The EC2 instance is provisioned in this subnet.
- “AWS::EC2::VPC::Id” – This is another special parameter type and it lists existing VPCs. In the next section, we will define a security group that gets provisioned in this VPC.
The architecture looks like this:
Incorporate Security Group settings
In order for SFTP users to access the server, we use a Security Group to expose port 22 for specific IP addresses.
Here’s the relevant code:
Resources: . . . SFTPGatewaySG: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: SFTPGateway Security Group VpcId: !Ref VPCIdName SecurityGroupIngress: - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 0.0.0.0/0
The security group has a property called “SecurityGroupIngress”, which accepts an array of rules. Here we have a single rule that allows all traffic (0.0.0.0/0) on TCP port 22.
Adding an elastic IP
Finally, we need to create a static IP so that our public IP address doesn’t change each time the server shuts down.
Resources: . . . IPAddress: Properties: Domain: vpc InstanceId: !Ref SFTPGatewayInstance Type: AWS::EC2::EIP
Here’s a brief explanation:
- “Domain” – This is set to “vpc”, since we’re using VPC instead of classic networking.
- “InstanceId” – This is the instance ID of the EC2 server that receives the IP address
We wind up with this final wonderful architecture:
Outputs section
The Outputs section lets you display concise information for easy access. Here, we show the public IP address to make it easier to connect for the first time.
Outputs: ElasticIP: Value: !Ref IPAddress Description: Elastic IP address
Conclusion
Now you can easily and securely upload your files to Amazon S3 via SFTP! Give the product a try by visiting the SFTP Gateway AWS Marketplace page.
You’ve also learned how to incorporate EC2, IAM, S3, Security Groups, and more to facilitate this file transfer.
We hope this has been helpful!
Check out other posts in our IaC series:
-
Infrastructure as Code: 5 Reasons Why You Should Implement IaC Now
- 6 best practices to get the most out of IaC
-
15 Infrastructure as Code tools you can use to automate your deployments
-
What is AWS CloudFormation and how can it help your IaC efforts?
-
How AWS CloudFormation Works (and How to Create a Virtual Private Cloud with it)
-
How to create a Redshift stack with AWS CloudFormation
Or you can download all of these articles together in one handy eBook by clicking the link below. Thanks for reading!