Accessing isolated network estate on AWS: Part 1 - Bastion Host

The classic Bastion host on EC2

Accessing isolated network estate on AWS: EC2 Bastion

Regardless of your profile, whether you're responsible for hands-on implementation or focused on the business aspects of a project, it's essential to have a solid understanding of how to best protect and secure your venture. This knowledge is important for everyone involved in the project, regardless of their technical expertise or business acumen.

I want to emphasize that for any project, regardless of its size, there will come a time when you need access to the underlying components of the system, such as databases, caches, pipelines and other processes. However, in an isolated network, such as an AWS VPC, this can be challenging, as opening up the network leaves it vulnerable to security threats and bad actors. As most systems with a non-trivial complexity are not hosted on SaaS services such as Netlify or Vercel as having access to the underlying infrastructure is essential for developmental, operational and support reasons.

In the following blog posts, we'll focus on how we can setup in a matter of minutes with just a few commands a variety of access patterns with varying levels of complexity and security guarantees with the help of AWS CDK (Cloud Development Kit) to provide secure access points to your private instances in a VPC, enabling you to manage your infrastructure safely and efficiently.

In brief, the access patterns that we will cover in this blog series will cover:

• EC2 Bastion Host

  • A special-purpose instance that acts as a proxy server for connecting to other instances in your VPC. It provides secure access using SSH or Remote Desktop Protocol (RDP). Simplest, most involved, requires instance hardening, monitoring, patching, further configuration to setup all additional processes & services (Terraform/Chef/Puppet for example), is a single point of failure.

• AWS Systems Manager Session Manager

  • Safer but more complex to configure, unless you use an AMI with it baked in ;). Main advantage compared to above options is that the instance does not require to be directly reachable from the internet, greatly reducing the threat profile. As long as there is a network path to AWS Systems Manager from your end and the target AWS EC2, the instance can function securely, providing an additional layer of protection to your infrastructure.

• ECS Fargate Service Bastion Host

  • Similar to an EC2 bastion host, but by leveraging ECS Fargate we can push the responsibility of base patching down to AWS, which reduces our overall threat surface to manage. By using this approach, we can ensure that our infrastructure is more secure, and we can focus our efforts on other critical areas. It is also potentially easier to replace and scale as most configuration can be contained in the task image Dockerfile.

• AWS Client VPN

  • Completely eliminates the need for a proxy bastion instance by granting the device we are VPN-ing in from access to the network within the VPC directly via the VPN tunnel. This not only simplifies the setup process but also eliminates the need to harden, monitor, patch or maintain a separate bastion host while minimizing the operational overhead for all parties involved.

---

For the remainder of this blog post we will focus on the first method of connecting, via an EC2 bastion host.

Note:

This is for demonstrative purposes, this blog posts bastion host design has no scalability or recovery mechanisms added for clarity and straightforwardness. For real workloads, please take into account the possibility that the EC2 instance may fail and needs replacing, look into EC2 Spot Fleet or the other posts in this blog series for alternatives.

Prerequisites:

Before we begin, make sure you have the following prerequisites installed and configured:

• AWS CLI [install guide]

• AWS CDK [install guide]

• Node.JS and NPM [install guide]

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

mkdir -p accessing-isolated-network/bastion-ec2
cd accessing-isolated-network/bastion-ec2
npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Terminal output

Step 2: Define the VPC Stack

Next, we need to define the VPC stack that our EC2 bastion host will be a part of. While this can all be done in one stack, it is best practice to separate the constructs as the layers of an onion, each layer building on top of the previous one, reducing potential blast radius for any change and reducing the number of resources that need to be touched for any one change.

Open up the lib folder and create a new file called vpc-stack.ts. Add the following code to create a new VPC with a public subnet:

GitHub Source

Step 3: Define the Bastion Stack

Now create a file named bastion-stack.ts with the following contents:

GitHub Source

Step 4: Define the App Stack

The glue to tie the two stacks together is:

GitHub Source

Which you can now run with

> AWS_PROFILE=<profile> npx cdk deploy --all

followed by the sequence of commands provided in the output of the command above, as can be seen below:

Terminal output

Step 5: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

Now while we could have used the classic process of uploading a long-lived SSH Key to AWS and referencing that when creating the bastion host, I chose for this example to highlight the newer approach using the AWS Instance Connect API which uploads using the “aws ec2-instance-connect send-ssh-public-key” to the EC2 Metadata service. This key is ephemeral and available for 60 seconds to perform the ssh connection after which it is deleted from AWS and you need to upload it once more. You can read more about the advantages and disadvantages of this method in my previous post

AWS EC2 Key Pairs versus EC2-Instance-Connect

Regardless of your preferred connection approach, gaining access to your AWS estate be it for development, break-glass emergency production access or anything in-between, it is good to know what your options are.

In the following post I will cover a similar approach using AWS Systems Manager Session Manager instead of conventional EC2 self-managed servers.

You can find the entire codebase for the above example in my GitHub repo here.