Apache Spark is a powerful distributed computing system that enables large-scale data processing. While most use cases involve deploying Spark on a cloud cluster, setting up a multi-node Spark cluster locally is invaluable for development, testing, and education. Docker Compose streamlines the process of configuring multiple Spark nodes, handling networking automatically. This eliminates the complexity of manual setups and allows you to focus on coding and testing your Spark applications. This blog post explains the steps to set up and run a multi-node Spark cluster locally within a DevContainer using Docker Compose.

When faced with any Big Data tasks such as MapReduce, Spark Streaming, MLlib (classification, regressions, clusterings, recommendations, feature extractions and others), Graph Processing (GraphX), SparkSQL and others, you usually have to jump through the hoops of accessing a remote Spark cluster to run your code that can range from installing Spark in Google Codelab to setting up an Elastic Map Reduce cluster on AWS. If you are in a sufficiently big company, you may also have internal clusters already running to connect to.

All of these require a continuous internet connection and a varying amount of setup steps. This Docker & DevContainer approach will show you a powerful alternative for rapid development cycles. In the below image you can see the overall architecture of what we will be doing here where on the developer computer we only have a thin client VSCode connecting to VSCode server within the DevContainer. This in turn communicates with the Spark Master node that deals with the inter-cluster comunication.

As we can see from the above diagram there are three ways to interface with the containers: VSCode itself, the Spark UI and the mounted volume from the local filesystem.

Note

We will not go into excruciating detail on how to setup a Spark cluster, as our purpose is to quickly get it setup and running locally as there are bigger fish to catch.

Note 2

While the target of this post is to run it all locally, we still want to keep it in line with the feature set existing on the live Spark cluster systems, as such, for the purposes of this post, we will assume the production systems use AWS EMR 7.5.0 and all our installed dependencies such as Python, Spark, Hadoop, Scala, Java JDK will try to adhere to the versions specified in the AWS EMR release notes.

Advantages

• Requires no more internet connection

  • You can go on a plane with this setup and keep progressing on big data tasks.

  • If you are travelling in low signal areas.

  • Significantly reduce development start-run-complete loops for your tasks as all data would be co-located on same SSD with your cluster.

• Canary release

  • You can use this setup to quickly test your pipelines work with a new major/minor release of your Spark cluster and dependnecies. It is a lot easier and faster to bump the version in the Dockerfile and run your pipelines than it is to try to deploy a new cluster on a vendor’s platform, configure it and retarget your existing DEV/QA/Production environments.

• Ease of setup

  • Using Docker Compose automates the configuration of multiple nodes. It simplifies the networking and avoids manual setup headaches.

• Consistency

  • DevContainers ensure that the development environment remains consistent across different machines, reducing the classic "works on my machine" problem.

• Cost-effectiveness

  • A local cluster eliminates the need for cloud resources, making it ideal for development and education purposes.

• Flexibility

  • You can customize Spark’s configurations to mimic production environments or test new features without affecting actual clusters.

Disadvantages

• Resource contraints

  • Running a multi-node cluster locally is resource-intensive and may not accurately represent the performance of real distributed environments. But for ironing out the logic this should not matter as much.

  • If leveraging only a local dataset, you are limited in the amount of data that can be run through your job, but for rapid development and experimentation, you would be leveraging a subset of your data regardless, so this should not be as big of an issue.

Implementation

The crux of the magic for this can be found in the docker-compose.yaml file with some shell scripts for setup of the various containers.

Prerequisites

1. Docker and Docker Compose installed.

2. Visual Studio Code with the DevContainers extension.

3. Basic familiarity with Docker and Spark.

Step 1: devcontainer.json

The main definition file for any DevContainer in VSCode is the .devcontainer/devcontainer.json file which specifies all of the configuration aspects for your setup.

We will use a more advanced, “escape hatch” approach leveraging Docker Compose for this task, as such we use the “dockerComposeFile” and the “service” properties in particular to provide more advanced container configuration options.

The rest of the “customizations” category is aimed at configuring VSCode to a “batteries included” approach for Python Spark development. Adding Scala/Java plugins and configurations would be needed to get it running for both languages.

Step 2: Docker Compose YAML

As mentioned above, we are delegating to Docker Compose the exact DevContainer configuration, which allows us to setup not only the container in which VSCode will configure it’s remote server but also auxiliary containers to augment our development experience, such as a 3-node Spark cluster with a master and two slaves.

Step 3: Dockerfile

The third element of the trio of core files for this exercise is the multi-stage Dockerfile which contains all of the common and shared setup logic for the development and Spark containers.

Step 4: PySpark test harness

Now we need some simple code that can be submitted as a Spark job to verify that all the wiring is done properly, cluster communication is working and that the execution completes successfully.

Step 5: VSCode Task run configuration

In order to easily run the code without having to copy paste from anywhere or remember “Yet-Another-Command”, we can create task run configurations to be invoked with “SHIFT + ⌘ + P” > “Tasks: Run Task”

Step 6: Execute PySpark job

We can now run the above task and receive the following successful output:

It has read the public transport CSV file downloaded from here and placed in “/workspace/data/B63.csv” and outputted the top values from the DataFrame in the console, as expected.

We can also go to http://localhost:9090 to view the SparkUI and see the app “bustime-app” executed successfully.

Step 7: Use Jupyter Notebook (Optional)

As any data scientist will know, running an entire end-to-end pipeline may take a very long time and as our purpose here is to have as fast as possible development & feedback cycles, we can create a .ipynb file, connect it to the installed Python kernel and run individual jobs from there.

We can also confirm using the Spark UI that the job has been submitted and executed properly on the cluster by verifying the existence of the “SparkConnectivityTest” job in the list.

Conclusion

Setting up a local multi-node Spark cluster using Docker Compose and DevContainers provides a practical environment for development and testing. While it’s resource-limited compared to production, it’s an excellent tool for learning and debugging. By following the steps in this guide, you can create a Spark cluster that’s easy to manage, reproducible, and flexible.

Experiment with different configurations and integrate additional tools to enrich your development experience. The combination of Docker Compose and DevContainers ensures your environment is consistent, cost-effective, and aligned with best practices. While we skipped over some of the tertiary files part of this setup, you can find the entire repository for this post here.

Due to a variety of reasons, we can find ourselves wanting to do a 0 downtime migration from a legacy URL or domain (legacydomain.com) to a new one (newdomain.com) without any clients noticing the change and in this post we will do just that. As the legacy environment is to be decommissioned ASAP while the legacy URL’s are still to be used by clients for the foreseeable months due to low change velocity at their end, we will need to do a few extra steps in the migration.

We will do this while touching upon several important topics that might cause unexpected gotcha’s to happen during your digital transformation which could hamper your smooth transition if not careful:

• How to redirect website URL’s where the first request is always a HTTP GET.

• How to redirect API URL’s where requests are isolated and can be any HTTP verb.

• Gotcha’s with Authentication headers and redirects.

• HTTP 30x redirect codes and details of each.

• Move all traffic over from legacy environment to new location and completely shut off the legacy environment, effectively serving all traffic, both traffic going to the old domain and traffic going to the new domain, from the new environment.

In essence, we will go from this where users reach your legacy.com domain

To this where users access your legacy.com domain but end up accessing new.com:

As we can see, we must go in the DNS registrar and update our entry for legacy.com to point to the same A/CNAME as the new.com entry so we can handle both legacy redirects and proper application traffic from the same load balancer, removing entirely any traffic or dependency to the old environment, allowing us to decommission the entire infrastructure of legacy while still serving legacy URL traffic.

1. Redirect website applications

To first complete the low hanging fruits of this migration, we configure the AWS Application Load Balancer HTTPS Listener to redirect based on Host Header from legacy.com domain to new.com domain. As AWS ALB only supports HTTP 301 & 302 redirect, we will use the 301 code, as that is Permanent Redirect, allowing browsers to cache the resolution, compared to 302, which is a Temporary Redirect. Note you can make 302 cacheable as well using Cache-Control or Expires response headers.

The sequence diagram for the above actions would be as follows:

2. Redirect API applications

With the less than 5 minutes that it takes for the above, armed with enthusiasm and confidence in solving the entire task in less than 10 minutes, we go ahead and attempt the same exact Host Header redirect on the ALB.

Once we attempt to access a fictional URL at an API hosted on https://api.legacy.com/api/v1/data on an API server that only serves POST requests there, we will encounter a 404 Not Found error.

The sequence of events of this would look as follows:

The first time I tried the API and saw the 404 Not Found error I was confused as to what was happening, why the 404 given that I know my API has POST, I checked the Listener rules which were correct. Upon investigating the API logs I saw the request reaching the new.com domain was a HTTP GET which led me down the rabbit hole to investigate HTTP Redirect codes in detail, as the knowledge up to then of 30x codes being all redirects did not suffice any longer. This research spike led me to discover the following distilled information:

The original HTTP 1.0 spec did not have 307 Temporary Redirect and 308 Permanent Redirect, as these roles were meant to be filled by 301 Moved Permanently and 302 Found.

However, as most clients changed the HTTP request method from POST to GET for 301 and 302 redirect responses, despite the HTTP specification not allowing the clients to do so. This behavior necessitated the introduction of the stricter 307 Temporary Redirect and 308 Permanent Redirect status codes in the HTTP/1.1 update.

I then did a quick and easy test to confirm using curl with verbose logging to see the steps taken by the client:

As we can see in the above verbose curl logging statement, our request get altered based on the 301 code.

Therefore, we can easily fix the 404 Not Found by leveraging 308 response code to achieve the following sequence of events:

Finding the cause and solution to the redirect is not the end of our troubles as the AWS ALB only supports HTTP 301 and HTTP 302 as per the documentation.

For various reasons AWS has elected to not permit more fine grained redirect codes effectively requiring us to roll out our own load balancer to satisfy all cases or to simply host a minimum .25 CPU Fargate Nginx container to perform the HTTP 308 redirects.

For which we will need a minimal Nginx Dockerfile to load this custom configuration from the environment variables, perform base64 decoding on it and place it at the expected location so we can have the same base image running on all environments:

With the following docker-entrypoint.sh:

With all of this new routing in place, we test our verbose curl once more, and we can see a successful response.

We can notice that while now we stopped getting 404 , we are now getting 403, which is expected, as this is an authenticated endpoint that is expecting an Authorization header. We easily add it to the request but we still notice that the redirected request has no Authorization header:

This prompted a new research spike to investigate the cause of it, which resulted in the conclusion that due to security concerns when doing a redirect from one domain to another which could be exploited for malicious purposes, clients drop the Authorization header in HTTP 30x requests.

In order to work with this security limitation, we have to drop the “redirect” concept entirely as it is unfit for purpose, we must perform a seamless transition while not adding in security vulnerabilities therefore our Nginx server is taking on a greater role, that of a forwarding proxy from legacy URL’s to the new URL’s with the following configuration:

Now the Nginx server acts as a proxy from the old to the new, acting as a bridge linking the two worlds.

3. Conclusion

In conclusion, leveraging AWS Application Load Balancer for URL redirects offers a powerful solution for seamlessly transitioning users from legacy URLs to new URLs, mainly if it for website servers only. While ALB's built-in HTTP 301/302 redirects are ideal and suffice for websites, additional considerations are needed for APIs, where custom Nginx servers can ensure the correct HTTP actions are maintained post-redirect along with the correct, complete header set. By following these guidelines, you can effectively manage URL redirections and enhance user experience across your applications with seamless migrations.

As engineers we constantly strive to automate and simplify the dull and repetitive tasks as much as possible, especially the laborious and long-winded ones that are more of an annoyance than anything else. And as I rollout many small proof of concept projects both for myself and for clients, it becomes necessary for my mental sanity to automate even those steps that for most is a once in a blue moon nuisance, such as the configuration of a self-signed certificate based AWS VPC Client VPN setup.

Advantages

• Running “npx cdk deploy” is truly all that is needed to get the entire environment deployed.

• No need to constantly reference documentation for all the commands needed to rollout a self-signed Certificate Authority (CA), client certificates, upload them to AWS Certificate Manager (ACM), copy the newly imported certificate ID’s into my CDK code and deploy.

• Becomes a blueprint for all projects.

The overall architecture of what we will be building is identical to what we covered in this post where I highlighted the benefits of the AWS Client VPN in more detail:

The architectural diagram is nearly identical to the one in the above post, with the added focus on the appearance of the S3 bucket for storing the raw certificate artifacts created during the first run of our VPC stack and the ACM being used by the Client VPN for the server certificate.

While the diagram bears significant similarities to the previous post, the CDK code has several modifications.

Step 1: The S3 Stack

In order to deterministically assess during each run, regardless of the machine the command is being run on, that we have the certificates generated, we need to leverage a common shared file store, such as S3:

Step 2: The VPC Stack

This bears significant semblance to the previous stack with minor cosmetic changes:

Step 3: The VPN Stack

This is where the whole magic happens, we leverage the fact that while the CloudFormation JSON/YAML that CDK synthesizes our code into is declarative, the TypeScript that generates the declarative code is imperative, so we can add some if statements to assess with the help of the aws-sdk for our language of choice, that the conditions hold true (the certificates exist), otherwise we create the certificates, upload them to the S3 bucket and create the SSM parameters, so that the AWS CDK code can find the parameters when the stack is applied.

Step 4: The Application Stack

Now the final element to make this all work together is the entry point for the application, which allows us to guard for the very first deployment against the synthesis of the VPNStack.

Although we specify the name of the stack to be deployed, there is no way to control through CDK that only the specified stack gets synthesized as well, only way to achieve that would be by creating separate entry points for each stack and invoking them individually. As such, I am adding a latch that is by default open, allowing the creation of the VPN stack as well, but for the first run, it will be set to false, so that the S3 bucket and the VPC are created first, ensuring the VPC.fromLookup will succeed along with the SSM Parameter.valueFromLookup.

Step 5: The certificate creation script

By reading the AWS documentation found here, I have created the following shell script to automate the process, allowing the above CDK code to create the certificates needed for the mutual authentication automatically:

With this final piece of the lego, we can now see how the VPNStack generateCertificates method is invoking the shell script, and capturing the output, filtering for the “SERVER_CERT:” and “CLIENT_CERT:” rows to obtain the ACM certificate ARN’s that are subsequently stored within SSM.

Step 6: The two step deployment

As mentioned previously, we need to deploy the stack twice, once to create the required S3 & VPC resources and a second time to create the Client VPN leveraging the dependencies:

npx cdk deploy --all -c deployVpn=false

This first run, with deployVpn=false prevents the VPNStack from being synthesized, allowing the dependencies to be created first, followed by a second run with:

npx cdk deploy --all

At this point we are finally complete, with a fully functional VPN service, for which we can download the Client VPN configuration, along with the certificates we now have in the “certificates” folder can be used to connect to the VPC network.

Conclusion

In conclusion, the journey through automating the setup of a self-signed certificate-based AWS VPC Client VPN has not only been a testament to the power of automation but also a deep dive into the intricacies of cloud infrastructure management.

The advantages of this approach are clear: from the ease of deployment with a single command to the creation of a reusable blueprint that can be adapted across various projects, the methodology outlined in this post offers a robust framework for managing VPN configurations.

As engineers, our goal is to tackle challenges head-on, leveraging technology to our advantage. This blog post serves as a blueprint for those looking to automate their AWS VPC Client VPN setup, offering a comprehensive guide that marries the imperative power of scripting with the declarative nature of infrastructure as code. By embracing these practices, we not only streamline our workflows but also pave the way for more innovative and efficient solutions in the future.

I hope this post inspires you to explore the possibilities of automation in your own projects and that it encourages a mindset of continuous improvement and innovation.

As always, you can find the full source code for this project on Github here.

In this post I will showcase a small utility that has proven useful across many projects to quickly gain interactive shell to secure isolated environments that are heavily restricted and monitored with a strong emphasis on having all access audited and tightly controlled through a central authentication and authorization tool, which for these cases, I have steered towards AWS Identity and Access Management (IAM).

In a previous post I have went over the architecture design and how to implement an isolated network that can be accessed via AWS ECS Exec, so I will refer you to this post an in-depth dive on how to roll it out:

Given the above foundational work done, I will now cover a useful utility script which with the help of a consistent and predictable naming convention, allows all authorized developers and support personnel to quickly access any service’s container across any environment.

Without further ado, the script:

#!/bin/bash

set -e

# environment is first argument passed to script
ENV=$1

# service is second argument passed to script
SERVICE=$2

# container name is third argument passed to script
CONTAINER_NAME="${3:-$(echo $SERVICE | tr 'A-Z-' 'a-z_')}"

# region for the service
REGION=${4:-"eu-west-1"}

usage_string="Usage: ./ecs-exec.sh <env> <service> [<container_name>:lower(service)] [<region>:eu-west-1]"

# check if ENV is passed
if [ -z "$ENV" ]; then
  echo "environment not passed in."
  echo $usage_string
  exit 1
fi

# check if SERVICE is passed
if [ -z "$SERVICE" ]; then
  echo "service target is not passed in."
  echo $usage_string
  exit 1
fi

# Check if required commands are available
for cmd in aws; do
  if ! command -v "$cmd" > /dev/null; then
    echo "Error: $cmd is not installed." >&2
    exit 1
  fi
done

CLUSTER_NAME="EngineerMindscape-ECS-$ENV"
SERVICE_NAME="EngineerMindscape-$SERVICE-$ENV"

echo "Cluster Name: $CLUSTER_NAME"
echo "Service Name: $SERVICE_NAME"
echo "Container Name: $CONTAINER_NAME"

TASK_ARN=$(aws ecs list-tasks --region $REGION --cluster $CLUSTER_NAME --service-name $SERVICE_NAME --query 'taskArns[0]' --output text --no-cli-pager)
echo "Task ARN: $TASK_ARN"

aws ecs execute-command --region $REGION --cluster $CLUSTER_NAME --task $TASK_ARN --container $CONTAINER_NAME --command '/bin/sh' --interactive

We will now discuss this script in the context of a theoretical service named EngineerMindscape-EFS-Util-DEV.

Option 1: Basic Example with all Defaults

In it’s simplest form, leveraging all defaults, the script can simply be invoked as such:

./ecs-exec.sh DEV EFS-Util

where

• DEV: The environment we are targeting, could be any existing environment such as DEV, QA, UAT, INT, PROD, etc.

• EFS-Util: This is the name of the ECS Service we are targeting. Note that the container name must follow a strict naming convention, namely, it must be the same as the service, all lowercase, with “-” replaced by “_”. As you may have noticed, this only works if there is only one, default container. If you have more containers, depending on your needs, the standardization can be extended to include sidecar containers as required. For the scope of this demo, we will keep it to one container, but have mentioned the possibility of one or more sidecar containers nevertheless as part of the script optional parameters for flexibility and extensibility.

Option 2: Sidecar container present

In this slightly more advanced example, we have a sidecar container, for example a Stackdriver container, sending data to a custom Prometheus target:

./ecs-exec.sh DEV EFS-Util stackdriver

where the first two arguments are the same as before and

• stackdriver: Exact name of the target container to connect to. This parameter can also be the name of the default, essential container within the service if it does not adhere to the expected naming convention due to any reason.

Option 3: Sidecar container present in another region

In this scenario, there is a sidecar container in a service located in another region. Of course the script can be expanded to also make the cluster name configurable to accomodate multiple clusters in the same region or in different ones, but for now, le’ts focus on the region:

./ecs-exec.sh DEV EFS-Util stackdriver us-east-1

where the first three arguments are the same as before and

• us-east-1: Exact name of the AWS region to connect to.

As an example of how the terminal session might look like while using this:

If you have read my other post, namely

You will find an example use case where this script is used. Within the above posts repository, it can be found here.

Conclusion

In this brief post, we explored a powerful utility script that streamlines access to containers within secure, isolated AWS environments. Emphasizing stringent access controls and auditing through AWS Identity and Access Management (IAM), this tool is a testament to the importance of security and efficiency in modern cloud infrastructure management. By leveraging AWS ECS Exec, the script enables authorized developers and support staff to swiftly connect to any service container, across various environments, with minimal hassle.

Whether dealing with a primary service container or navigating through sidecar containers, possibly even across different regions, this utility facilitates essential interactive shell access, thereby enhancing operational flexibility and responsiveness.

Through practical examples, from basic usage to more complex scenarios involving sidecar containers and different AWS regions, the post demonstrates the script's versatility. This utility not only exemplifies the power of standardized naming conventions but also underscores the critical role of accessible, yet secure, container management in today's cloud-centric landscapes.

In this digital age we live in, where data is the most valuable currency, safeguarding your files and ensuring access in times of crisis is paramount. AWS Elastic File System (EFS) offers organizations a scalable, cloud-native file storage solution that integrates seamlessly with AWS cloud services. However, even with the most advanced systems, the risk of access disruptions due to technical failures or security breaches remains a tangible threat. This brings us to the concept of a "breakglass" scenario—a situation where standard access protocols are ineffective or compromised, necessitating an alternative method to gain immediate, secure access to your data. Enter the breakglass EFS Fargate Docker container, a specialized emergency investigatory tool that enables you to quickly access typically locked down and segregated views of a larger filesystem. This blog post delves into the why and how of setting up such a container, ensuring that when the unexpected occurs, you're ready—not just to react, but to proactively inspect and remediate any issues affecting your vital data.

The Why: The Importance of a Breakglass Container in Crisis Management

Imagine this: Your primary access controls have failed, or a configuration mishap has left your EFS data unreachable through standard means. The clock is ticking, operational downtime is costing you, and the pressure is mounting. This is where a breakglass EFS Fargate Docker container comes into play. It's your emergency access point, a pre-configured, secure backdoor that allows you to bypass normal access protocols safely and access your raw data as it is directly stored on the filesystem. By spinning up this ephemeral container which has the root of your EFS filesystem mounted, you can quickly investigate and remediate any inconsistent data that is bringing down your application.

Advantages:

• Immediate Access: When every second counts, having a container ready to deploy ensures you can access your data without the delay of configuring access from scratch. This can be the difference between adhering to your RTO (Recovery Time Objective) and still trying to gain access while it sails past you.

• Secure Yet Flexible: Designed with the principle of least privilege in mind, it provides just enough access to manage the crisis without exposing your system to further risk. The task does not exist, only the task definition, so it is running only when it is truly needed.

• Valuable Investigation Tool: This tool can be used not only for the worst case scenario, it can also be used for pure investigate purposes.

Subscribe now

The What: A Step-by-Step Guide to Building a Breakglass Container

In order to properly showcase this concept, I will showcase the EFS Breakglass container service within a minimally configured AWS VPC with an EFS filesystem. In order to apply this into your own ecosystem you only need to copy the AWS ECS EFS Util Service. The Breakglass service will have a desired count of 0 so there are no running containers and if access is needed, it can be set to 1. Once a container boots and stabilizez, we can leverage AWS ECS Exec, which I already covered in a previous post here, to land with an interactive shell within the container with the root of the EFS mounted in /mnt/efs .

The high level architectural diagram of what we are going to build is as follows:

Without further ado, let’s start building it.

Prerequisites

Before we begin, make sure you have the following prerequisites installed and configured:

• AWS CLI [install guide]

• AWS CDK [install guide]

• Node.JS and NPM [install guide]

• Java [install guide]

Although the IaC will be written in AWS CDK using Java, the fastest and easiest way to bootstrap the project is by leveraging the cdk CLI directly from NPM.

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in your preferred language, for this series, I will go with Java. Let’s open up a terminal and run the following commands to get started:

mkdir -p efs-escape-hatch
cd efs-escape-hatch
npx cdk init app --language=java

This will create a new CDK app in Java with the following structure:

Step 2: Define the VPC Stack

In this simple skeleton configuration, the VPC is simple and straightforward. While we could use isolated subnets with no egress, beyond the VPC Endpoints for ECR (to pull containers images), S3 (ECR stores container image data on AWS S3 owned buckets) and SSM ( we will use SSM Exec to obtain an interactive terminal into the Fargate container, more on that later), in order to keep this as simple as possible, we will use the generic public & private with egress subnet configuration.

Step 3: Define the ECS Stack

As we aim to have a modular design, in which the ECS stack may very well be maintained by a completely different team, we will define the ECS cluster in its own dedicated stack, with ECS execute command enabled, in a straightforward manner as follows:

Step 4: Define the EFS Stack

As the purpose of this exercise is to access an EFS file system, we will create a simple and straightforward one now. The only added complexity for the purpose of the demo is to initialize the EFS file system with some folders, in this case, we will use this EFS system for a MySQL / RabbitMQ cluster combo, where both clusters share the same EFS and we are using AWS CDK Custom Resources to trigger a Lambda that will mount the EFS via an Access Point and run a simple inline NodeJS snippet to create the /mysql/data and /rabbitmq/data directories.

Step 5: Define the EFS Breakglass Stack

And now the crux of this article, the service that is by default in cold standby, with 0 active containers, waiting for a signal to be started and mount the EFS filesystem, to which we can connect via AWS ECS Exec to land with a shell directly in the container.

Step 6: Define the ECS Exec utility script

The final setup step needed now is to configure a lightweight shell script that will allow us to quickly leverage ECS Exec and land within an interactive shell inside the busybox container. This script is done in a such a way that it can be reused to land in any container that respects the container naming convention across any environment name although for this demo purpose, we only have the DEMO environment.

As we can see in the above script, we can reap many benefits from adhering to a standardized and predictable naming format across the resources on our entire estate.

Step 7: Defining the CDK entrypoint

Final step in the AWS CDK setup is to define the entry class for the entire application as follows:

Step 8: Deploying the stacks

We can deploy all of the above stacks in one go with the following command:

npx cdk deploy —all -c skipDependencies false

And now we can finally use ECS Exec and start browsing our EFS file system at will with the following:

./ecs-exec.sh DEMO EFS-Util

Step 9: Cleanup

Now that we have completed this exercise, we can delete all the resources to ensure no nefarious bill hits us at the end of the month !

./ecs-exec.sh DEMO EFS-Util

Conclusion: A Stitch in Time

The adage "a stitch in time saves nine" holds particularly true in the realm of IT and data management. You're not just setting up a technical solution; you're laying the foundation for countering the unexpected.

In conclusion, the implementation of a breakglass EFS Fargate Docker container is not merely a technical exercise; it is a critical step towards ensuring operational resilience in the face of unforeseen challenges. By preparing for emergency access scenarios, organizations can significantly reduce the impact of access disruptions, safeguard their data integrity, and maintain business continuity with confidence.

It helps you stick to and outperform your RTO while keeping calm and collected. In a world where the unexpected is the only certainty, such preparedness is not just beneficial; it's essential for business continuity and client confidence.

Let this guide serve as a reminder of the power of preparation and the critical role of emergency access solutions in building a robust, resilient IT infrastructure. I encourage you to take the necessary steps to implement this breakglass scenario solution within your organization. By doing so, you're not just protecting your data; you're safeguarding the future of your business.

Remember, in the world of IT, the unexpected is the only certainty. Equip your organization with the tools and strategies to navigate these uncertainties with confidence. Operational resilience is not just about surviving the storm; it's about thriving in the aftermath. Prepare today to ensure your organization's resilience for tomorrow.

As always, you can find the full source for this project on my Github repository here.

Subscribe now

In this post I will share with you the steps I go through for every new project, be it a quick experimentation or professional greenfield project to quickly rollout a basic skeleton for a web front-end, API back-end and AWS CDK (in this example, could also be Azure ARM, Terraform, Pulumi) IaC (Infrastructure-as-Code). The same applies for any permutation of the above with any number of services.

While it is possible to create a shared code monorepo manually, it would not be complete in 5 minutes and would entail undifferentiated heavy lifting on our part. To avoid such inneficiencies, we will use Nx for this (could also use Yarn workspaces if scope is limited and small) as it provides plentiful of plugins for most of the frequently used toolings in the wild (you can use this as a measuring stick for the popularity of some tools, I have discovered some new ones browsing the Nx plugin list).

This is the same process I started with when creating my CV, as detailed here and here, so as you can see, this is another powerful tool in your belt, that can be used for the simplest to the most complex setups.

The Nx CLI is a robust tool that offers a wide range of options and follows specific best practices. To guarantee that our setup remains consistent over time—whether the commands are executed today, next week, or a year from now—we will specify a few parameters. This ensures our project configuration remains uniform, regardless of when the commands are run.

This post is divided into two primary sections:

• Project bootstrapping

  • This involves setting up the initial structure for web, API, and Infrastructure-as-Code (IaC) applications, along with a shared library that these applications will use.

• Coding

  • This section covers adding minimal code to enhance the modules with essential functionality.

In order to better visualize the final relation between the modules, I have ran

npx nx graph

to view in the browser the dependency relationships between the projects, which looks as follows:

Note

The diagram mentioned earlier represents a compile-time dependency graph, created using the graph command. This graph shows the dependencies that are determined during the compilation process. If we were to create a graph illustrating runtime dependencies, it would include an additional arrow pointing from the web to the API. This arrow signifies that the ReactJS front end makes calls to the API to fetch data. This data fetching is facilitated by code that is compiled into the application's binary, using functions imported from the shared library. The concept and its implementation will be more comprehensible when we delve into the code in section 2.

The architectural view of the deployed solution will look as follows:

1. Project bootstrapping

This section is mainly in the CLI so the feedback loop will be fast.

1.1. Create Nx Workspace

The documentation of the command can be found here.

npx create-nx-workspace@latest --preset empty --nxCloud skip --packageManager yarn --workspaceType integrated engineeringmindscape

where the parameters are as follows:

• --preset none

  • Creates an empty monorepo project with no dependencies. We will manually populate only what we need in the next steps.

• --nxCloud skip

  • No need for cloud build caching for this demo app but it is nice to have otherwise.

• --packageManager yarn

  • Personal prefference to use yarn but npm, pnpm are also available.

• --workspaceType integrated

  • As I want to highlight the power of the integrated monorepo, we need to ensure we can refer to libs cleanly :).

• engineeringmindscape

  • This will be the name of the overall monorepo project and the namespace that will be used for the libraries.

The output of the above command looks as follows:

Now we can enter the newly created monorepo directory:

cd engineeringmindscape

The file structure of the generated base monorepo looks as follows:

1.2. Install Nx plugins

Now we can add as dev dependency the Nx plugins for NodeJS and AWS CDK and the needed CDK construct plugin.

yarn add -D @nx/react @nx/node @nx-iac/aws-cdk @aws-solutions-constructs/aws-apigateway-lambda

• @nx/react

  • This is the plugin that we will use to generate the web application. It has several generators such as hooks, stories, redux, components, etc but we will only be using the application one for now.

• @nx/node

  • We will be using this for the API backend. It also has several generators from which we will also be using the application generator.

• @nx-iac/aws-cdk

  • This is a custom plugin that will bootstrap our AWS CDK IaC application from which we will need the app generator.

• @aws-solutions-constructs/aws-apigateway-lambda

  • AWS provided L3 construct for CDK to generate all the resources required for a API Gateway fronted Lambda with minimal code.

1.3. Create the ReactJS Web App

Let us begin bootstrapping the first application of our stack, the ReactJS website with the Vite bundler:

npx nx g @nx/react:application --name web --directory apps/web --bundler vite --style css --routing false --e2eTestRunner playwright --minimal true --projectNameAndRootFormat as-provided

• --name web

  • The name of the ReactJS web application.

• --bundler vite

  • The bundler to use for the ReactJS project, vite or webpack is available.

• --style css

  • The stylesheet system to use for the web app, we are keeping it lightweight as possible and not including SASS/LESS/etc.

• —routing false

  • For this basic scenario I do not want to have routing configured as it will be literally a single page SPA(Single Page Application)

• --e2eTestRunner playwright

  • Each web project gets a side E2E testing project, with options for cypress and playwright. We are going with playwright.

• —minimal true

  • No need for separate test files, let’s keep it light.

• --projectNameAndRootFormat as-provided

  • To keep the structure specifically as I have provided in the directory parameter.

After running the above command, the following output and files have been created:

1.4. Create the NodeJS API App

We can now leverage the previously installed @nx/node plugin to generate our API application:

npx nx g @nx/node:application --name api --bundler esbuild --framework none --projectNameAndRootFormat as-provided --directory apps/api

• --name api

  • Name of the application and folder where it will be placed in.

• --bundler esbuild

  • The bundler to use for the NodeJS project, esbuild or webpack is available.

• --framework none

  • As we will have a basic Request-Response Lambda, we will simply return a canned response with no need for Express/Fastify/Koa.

• --directory apps/api

  • We specify concretely the folder to place the new application in.

• --projectNameAndRootFormat as-provided

  • To keep the structure specifically as I have provided in the above parameter.

The documentation for the command can be found here. After running the above command, this is what we get in the terminal:

1.4. Create the shared NodeJS library

In order to fully showcase the power of this monorepo setup, we will create a library that will contain shared definitions, such as the name of the project, that will be used across all of the applications in a visible manner to the end user. For now we will simply bootstrap the library as follows leveraging the same @nx/node plugin:

npx nx g @nx/node:library --name shared --directory libs/shared --projectNameAndRootFormat as-provided --testEnvironment node

• --name shared

  • The name of the library

• --directory libs/shared

  • The directory where to place the new library specifically.

• --projectNameAndRootFormat as-provided

  • As before, this tells Nx that I want to specifically have the folder structure I provided, to ensure consistency throughout time.

• --testEnvironment node

  • The environment for the testing aspect of the library, node or jsdom are available options.

The documentation for generating the library command can be found here. After running the above command, we can see the following in the CLI:

1.5. Create the AWS CDK IaC App

As the final bootstrap step, we generate a base AWS CDK application using the second

installed plugin @nx-iac/aws-cdk:

npx nx g @nx-iac/aws-cdk:app --name iac --directory apps/iac --projectNameAndRootFormat as-provided

• --name iac

  • The name of the application

• --directory apps/iac

  • The directory where we are going to place the AWS CDK code.

• --projectNameAndRootFormat as-provided

  • We ensure the layout will be as specified in the above directory parameter.

The documentation for this custom Nx plugin can be found here. The output of running the above command is as follows:

1.6. Full monorepo skeleton

Now after running the above bootstrap commands for the web, api and IaC apps as well as for the shared library, with the 18.0.2 version of Nx we get the following files:

2. Coding

Starting from the foundational elements and moving upward, we will enhance the initially bootstrapped files with necessary modifications to showcase the power and speed of our setup.

2.1. Shared Library

In shared/src/lib/shared.ts we add the following:

export const PROJECT_NAME = "EngineeringMindscape";

2.2. API App

In apps/api/src/main.ts already we can start leveraging the shared code above:

// this is the imported symbol from the shared library
import { PROJECT_NAME } from '@engineeringmindscape/shared';

export const handler = (event: unknown, context: unknown, callback: (err: null, resp: Record<string, number|string|Record<string, string>>) => void) => {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin": "*",
    },
    body: JSON.stringify({
      api: PROJECT_NAME,
    }),
  };

  callback(null, response);
};

We also need to modify in apps/api/project.json

“bundle”: true

as this will ensure the entire code is bundled in a single output file to be used by Lambda.

2.3. Web App

In apps/web/src/app/app.tsx now we also leverage the shared library directly as well as call the API to receive a JSON payload that will contain the shared content, both of which we will display on the page:

// eslint-disable-next-line @typescript-eslint/no-unused-vars
import styles from './app.module.css';
// this is the imported symbol from the shared library
import { PROJECT_NAME } from '@engineeringmindscape/shared';
import { useEffect, useState } from 'react';

export function App() {
  const [data, setData] = useState(null);
  useEffect(() => {
    // this is the environment variable from the .env file, 
    // which we will populate with the API URL after first deployment
    // as that is when we will know the URL of the API
    fetch(`${import.meta.env.VITE_API_URL}`)
      .then(response => response.json())
      .then(data => setData(data));
  }, []);

  return (
    <div>
      {PROJECT_NAME}
      {data && <div>{JSON.stringify(data)}</div>}
    </div>
  );
}

export default App;

Note

The VITE_API_URL environment variable will be populated with the API's URL after its initial deployment, as that's when the API Gateway's URL becomes known. Ideally, to manage dependencies between stacks, the web application's build and deployment would follow the API's creation. However, prioritizing speed and simplicity, we'll perform the CDK deployment process twice, which will be detailed in section 2.5.

2.4 IaC App

In apps/iac/cdk/IacApp.ts we update the code of the entrypoint for the CDK app as follows:

import * as cdk from 'aws-cdk-lib';
import { SampleStack } from './stacks/SampleStack';
// this is the imported symbol from the shared library
import {PROJECT_NAME} from '@engineeringmindscape/shared';

const app = new cdk.App();
new SampleStack(app, PROJECT_NAME, {
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION,
  },
});

As well we need to edit the apps/iac/cdk/stacks/SampleStack.ts to actually deploy the appropriate API Gateway, Lambda, Roles, CloudWatch Log Groups, S3 Bucket and bucket deployment Lambda:

// this is the imported symbol from the shared library
import {PROJECT_NAME} from '@engineeringmindscape/shared';
import { CfnOutput, RemovalPolicy, Stack, StackProps } from 'aws-cdk-lib';
import { ApiGatewayToLambda } from '@aws-solutions-constructs/aws-apigateway-lambda';
import { Construct } from 'constructs';
import * as api from 'aws-cdk-lib/aws-apigateway';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import { BucketDeployment, Source } from 'aws-cdk-lib/aws-s3-deployment';

export class SampleStack extends Stack {
  constructor(construct: Construct, id: string, props?: StackProps) {
    super(construct, id, props);

    // AWS provided L3 constructs to deploy API Gateway -> Lambda configuration
    new ApiGatewayToLambda(this, `${PROJECT_NAME}-API-GW-Lambda`, {
      lambdaFunctionProps: {
        functionName: `${PROJECT_NAME}-API-GW-Lambda`,
        runtime: lambda.Runtime.NODEJS_20_X,
        handler: 'main.handler',
        code: lambda.Code.fromAsset(`${__dirname}/../../../../dist/apps/api`),
      },
      apiGatewayProps: {
        restApiName: `${PROJECT_NAME}-API-GW`,
        defaultCorsPreflightOptions: {
          allowOrigins: api.Cors.ALL_ORIGINS,
          allowMethods: api.Cors.ALL_METHODS
        },
        defaultMethodOptions: {
          authorizationType: 'NONE'
        }
      },
    });

    const s3Bucket = new s3.Bucket(this, `${PROJECT_NAME}-S3-Bucket`, {
      removalPolicy: RemovalPolicy.DESTROY,
      bucketName: `${PROJECT_NAME.toLowerCase()}-monorepo-demo`,
      publicReadAccess: true,
      autoDeleteObjects: true,
      websiteIndexDocument: 'index.html',
      websiteErrorDocument: 'index.html',
      // required due to https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/
      objectOwnership: s3.ObjectOwnership.OBJECT_WRITER,
      blockPublicAccess: new s3.BlockPublicAccess({
        blockPublicAcls: false,
        ignorePublicAcls: false,
        blockPublicPolicy: false,
        restrictPublicBuckets: false,
      }),
    });

    // deploy web app to s3 bucket
    new BucketDeployment(this, `${PROJECT_NAME}-S3-Deployment`, {
      sources: [Source.asset(`${__dirname}/../../../../dist/apps/web`)],
      destinationBucket: s3Bucket,
      metadata: { ForceRedeployment: Date.now().toString() }
    });

    // output the S3 bucket URL
    new CfnOutput(this, `${PROJECT_NAME}-S3-Url-Output`, {
      value: s3Bucket.bucketWebsiteUrl,
    });
  }
}

The above CDK stack while not production ready is very short and readable achieving the swiftness and conciseness criteria which are of utmost importance for this current endeavour.

2.5. Building and Deploying

Now that we have all the code in place, all that is left is to build and deploy:

nx build api

nx build web

Now to deploy it all (I assume you have an AWS account that is bootstrapped already, otherwise you also need to run the @nx-iac/aws-cdk:bootstrap command):

nx deploy iac

After deploying the whole stack, terminal will output something similar to

Now as mentioned during the web app editing done in section 2.4, now that we have the Outputs from the stack, we can use the APIGWLambdaRestApiEndpoint value as follows:

echo “<APIGWLambdaRestApiEndpoint value>“ > .env

nx build web

nx deploy iac

Now we can finally access the second Outputs value of S3UrlOutput to view our final result:

When accessing the S3 website URL we are greeted with a plain page with the first row being the shared library PROJECT_NAME imported into the web project and build within the distributable.

The second row is the API Gateway Lambda response which also contains within the build distributable the shared library value, which we fetched and are displaying directly in the page.

Finally, the AWS CDK CloudFormation stack itself uses the value in the name of the Lambda, Stack and even in the S3 bucket name, which is visible client side as well by looking at the URL in the browser above.

Conclusion

In conclusion, the journey from initializing a monorepo with Nx to deploying a fully functional web application stack on AWS demonstrates the power and efficiency of modern development tools and practices. By leveraging Nx for project scaffolding, we benefit from a streamlined development process that integrates seamlessly with various technologies, including React for the front end, Node.js for the API backend, and AWS CDK for infrastructure-as-code management. This approach not only accelerates the setup phase but also ensures consistency and scalability across the entire project lifecycle.

The shared library concept further exemplifies the advantages of a monorepo setup, facilitating code reuse and maintaining consistency across different parts of the application. By centralizing shared definitions and functionalities, developers can ensure that changes in one area are automatically propagated throughout the project, reducing the risk of discrepancies and bugs.

Deploying the application stack using AWS CDK showcases the practical benefits of infrastructure-as-code, allowing for reproducible and predictable infrastructure provisioning. This method simplifies the deployment process, making it easier to manage and scale cloud resources efficiently. The integration of AWS solutions constructs further streamlines the creation of cloud resources, enabling developers to focus on building the application logic rather than managing infrastructure.

This post has outlined a comprehensive yet straightforward approach to kickstarting a web application project, from setup to deployment. By embracing these modern development practices and tools, developers can significantly reduce the time and effort required to bring their projects to life. Whether you're working on a pet project or a professional greenfield project, the methodology described here provides a solid foundation for developing robust, scalable, and maintainable web applications.

As always, the entire repository code for this solution can be found here.

The problem

In a recent project in which I undertook the modernization of a legacy monolithic Java application running on EC2 to modern “cattle not pets” in cloud native realm. If you are not familiar with the term, it was coined originally by Microsoft engineer Bill Baker regarding SQL deployments where

Pets

Servers get unique names and special treatment. When they’re “sick”, they’re carefully nursed back to health, often with a significant time and financial investment.

while

Cattle

Individual servers are part of an identical group. Numbers, not names, identify them, and they receive no special treatment. When something goes wrong, the server is replaced, not repaired in place.

After a laborious process that is out of scope for this topic, I finally reached the point of attempting to deploy the service on ECS, only to be welcomed by an error related to the task definition being limited to 65536 bytes.

This was quite the daunting error to encounter at the time as the Java application has gathered nearly a thousand environment variables in the past two decades and overall, due to them, the task definition is over the hard limit of the service. After a brief moment of dread, I set to apply my engineering mindset and find a workaround to this new limitation.

After a brief investigation I concluded that as the environment variables have all been extracted from a multitude of Spring properties files and injected via Terraform(conversion done via a script of course), the total storage required for the content and all of the final resulting JSON with all it’s metadata needed for the Task Definition exceeded the 65536 byte hard limit.

The basic solution

The solution I settled on was to coalesce the property files into fewer, functionality focused, and as part of the Terrafrom deployment, to leverage filebase64 to encode the entire file into a single string that can be stored and keep the Task Definition within the limits. Alternatively, we can also leverage Terraform data.local_file as the .zip does not exist at the initial terraform run point, which would result in an error.

As the containers all have a custom docker-entrypoint.sh file, part of the boot sequence before exec-ing the main application, as part of the shell script, I take the environment variables, decode and place them in files based on the environment variable name, which follows a preset naming convention. In this way an environment variable named FILE_HYSTRIX_PROPERTIES would have the FILE_ prefix dropped, the remainder converted to lower case and “_” replacted with “.” to yield hystrix.properties file with contents base64 decoded.

The shell script to achieve this would could look as follows:

Now having done this we can be in one of the two following scenarions:

1. Deployment successful, but brittle as we are already near the hard limit of the service and adding an additional trivial amount of variables or simply updating some of them to lengthier values could result in the definition exceeding the limit.

2. Deployment failed, as we are still over the limit even with these optimizations.

The improved solution

As the properties files are just text files, we can obtain further significant storage gains by also doing a compression of the file before performing the filebase64 / data.local_file.zip_file.content_base64. As such, for each of our files we can do

zip hystrix.zip hystrix.properties

and reference the “.zip” instead of the raw file in the filebase64 step.

This can yield us significant storage gains as below properties file zip showcases:

In Terraform these steps would look like the following:

This allows us to continue having the raw text files available for editing as opposed to storing them in encoded, zipped format and having to perform

1. base64 decode

2. unzip

3. edit property

4. zip

5. base64 encode

As we now also have zip files placed in the environment variables, we also need to expand our docker-entrypoint.sh to handle zipped base64 encoded variables.

The updated scripts looks as follows:

These encoded variables can then be referenced in the environment section of the task definition. If you are using the Terraform module for ECS, the code would look as below:

And as our container has the docker-entrypoint.sh as showcased above, the file will be decoded, unzipped and placed in the desired directory to be consumed by the application.

The optimal solution

While the above steps will significantly reduce the task definition size and most likely suffice for must usecases, if you have several large property files, you will still risk reaching the hard limit quickly, as the entirety of the data is still located within the task definition. The best approach would be to hold the data in a different “container” somewhere on AWS and only hold references to it within the definition specification.

It comes naturally then that the best location to hold these encoded variables would be within SSM Parameter Store(Systems Manager Parameter Store) as an Standard(4KB) or Advanced(8KB) parameter or within Secrets Manager(10KB).

The added benefit of this move is that we increase our security posture as we can apply more stringent IAM permissions on the parameters and secrets, denying read/write permission to them for developers while still allowing view/edit of the task definition itself.

Not only that but the main benefit of this approach is that if we were to have 9 theoretical variables of 8KB in size, the improved solution above would run into the ECS service limit of 65535 bytes. Moving them into SSM as an “Advanced” parameter, we would circumvent that limitation and be able to store countless more variables.

Immense gains for a very small change in the Terraform script from above:

Conclusion

In navigating the complexities of modernizing a legacy Java application for deployment on AWS ECS, the journey from encountering a daunting task definition limit to innovating a viable solution underscores the essence of cloud engineering: adaptability and problem-solving. The initial workaround, involving encoding and compressing environment variables, showcased creative engineering to fit within stringent constraints. However, the true breakthrough came with the optimization of utilizing AWS SSM Parameter Store and Secrets Manager to externalize configuration data, demonstrating a strategic pivot towards leveraging cloud-native services for scalability and efficiency.

This experience not only illustrates the technical challenges inherent in migrating to cloud-native architectures but also highlights the importance of embracing cloud services to enhance application deployment and management. By moving from a direct embedding of environment variables in the task definition to storing them in AWS's managed services, we not only circumvent the limitations of ECS but also gain the benefits of security, manageability, and scalability provided by AWS.

Such a journey is a testament to the evolving nature of cloud computing, where limitations are not roadblocks but rather catalysts for innovation. It emphasizes the need for continuous learning, experimentation, and adoption of cloud-native solutions to overcome challenges. As organizations continue to modernize their applications, this example serves as a reminder of the power of cloud services to transform application deployment strategies, ensuring they are both scalable and resilient in the face of changing technical landscapes.

In conclusion, the path from a constrained legacy application to a streamlined, cloud-native deployment is fraught with challenges. Yet, it is precisely these challenges that drive the innovation and strategic thinking necessary to leverage the full potential of cloud computing. By adopting a mindset that views limitations as opportunities for optimization, organizations can navigate the complexities of modernization with confidence, ensuring their applications are not only compatible with the cloud but are also positioned to take full advantage of its capabilities.

The example codebase for this can be found here.

It contains a minimal example ECS cluster with a task definition of a custom image built on top of alpine. It copies the custom docker-entrypoint.sh and sets the command that will be executed after the scripts completion, which in our case will be to print the contents of the /tmp/hystrix.properties, which is where we have CONFIG_DIR pointig to (/tmp) and where our local hystrix.properties should be placed.

You can run it via

terraform init && terraform apply -auto-approve

Once it completes, we can check the logs which outputs the /tmp/hystrix.properties from the container by running

aws logs tail /aws/ecs/EngineeringMindscape-Task-Limits-DEV/example

which should output the following:

This is the second part in the series covering my beautifully engineered CV where we will cover the DevOps aspects of the project, namely, the creation of the AWS infrastructure and the CI/CD setup to bind it all together seamlessly.

In the first part, which you can read here, we covered the bootstrapping of the Nx monorepo, the creation of the React CV website and of the Lambda Exporter that leverages headless Chromium orchestrated by Puppeteer to save a PDF of the CV.

We will be now be using AWS CDK(AWS Cloud Development Kit - a modern Java,NodeJS,Go, .NET interface that gets synthesized to CloudFormation JSON templates and applied as CloudFormation stacks) for the IaC aspects as it allows us to benefit from having the front-end, back-end and IaC in one language, sharing configuration and symbols without having to deal with duplicate code.

Subscribe now

1. Infrastructure

While having it all run locally is great, unfortunately it is not visible to the rest of the world, so we need to deploy it somewhere the world can access it, which for us is on AWS S3 with a CloudFront CDN(Content Delivery Network).

There are 3 main component stacks for this setup:

• ECR

  • Simple and straightforward stack that creates the Elastic Container Registry that will hold our Lambda Docker Runtime image and that the Lambda service will pull from.

As I am not expecting any advanced and complicated use-case scenarios, I am keeping it simple, with only latest tag and lifecycle rule to only have one image thus reducing costs.

• Lambda Exporter function

  • Lightweight stack centered around a DockerImageFunction, which is the L3 AWS CDK construct (higher level CDK language object providing more error checking, configurations and resource interconnection) equivalent of a CloudFormation AWS::Lambda::Function with code pointing to the above ECR image registry, with versioning and aliasing alongside a role with minimal permissions configuration.

• React Web

  • This is more involved compared to the other two as it also deals with ACM certificates, S3 bucket with it’s policies, CDN, WAF, S3 deployment of website bundle and of PDF via CloudFormation Custom Resource.

  • On each run of the WebStack deployment, the web bundle is deployed via the BucketDeployment construct, the CustomResource is triggered which runs the Lambda to export the latest deployed website bundle as a PDF to be readily available for download for anybody browsing the deployed website.

These can be run directly with the cdk CLI(Command Line Interface) with a few extra parameters being required due to the monorepo context resulting in the below mouthful:

cdk synth -q -a 'npx ts-node --prefer-ts-exts -P apps/infra/tsconfig.app.json -r tsconfig-paths/register -r dotenv/config apps/infra/src/bin/<ECR|Exporter|Web>.ts

or by creating utility Nx scripts to bring it down to a more memorable

nx deploy infra

which seems more memorable and easier to type.

This of course has the precursor dependency of having to obtain a domain, have a hosted zone for it AWS Route53. The domain can be either directly purchased there or with a custom Hosted Zone to which you direct the NS in your registrar of choice.

2. GitHub Actions CI/CD

The fourth and final step in the story, which enables us to edit our CV’s JSON files, run git commit, git push and have the linting, testing, building, deploying, semversioning and releasing done automatically is by setting up our GitHub Actions pipelines. We will have two pipelines, one triggered by a pull request which only does the evaluation of the validity of our commit and another for merges on main, which does the full suite of evaluation, building & deploying.

2.1 Pull Requests

The step names are self-explanatory and all adhere to the single responsibility principle, they do only one thing, be it setup a a dependency or running a process.

This verify ensures any new pull request has passed linting, testing, building of web, exporter, exporter Docker image and of all the AWS CDK stacks by chaining the builds from the root repository

yarn build

to

nx run-many --all --target=build

Same principle applies to lint and test.

Note

Although we do not do any deploying in this step, we have here the 🔑 AWS Credentials step as the CDK build does a synthesis of the CloudFormation stacks, which requires authenticated access to the target account.

The 🧰 Setup QEMU and 🧰 Setup Docker BuildX are needed to be able to run docker build via the docker/build-push-action@v5 action.

2.2 Merging to main

Merging to main pipeline has the same verify job (you can get code on a branch through other means besides that of a pull requests :) ) with an added deploy job, which has the same setup steps as the verify job, with the added deployment steps of:

As I have ensured all the steps can be run from local first before I spent time in setting up any pipeline, it makes sense that most of the content of the pipeline definition is simply delegating the work to the appropriate Yarn and Nx scripts that trigger AWS CDK CloudFormation synthesis with the addition of injecting the appropriate environment variables.

The cherry on top of the cake is the 🚀 Release step which runs

semantic-release

that reads all of the commits from the last tag to HEAD and based on conventional commits message formatting increments MAJOR.MINOR.PATCH version appropriately.

To ensure the commit messages always follow a pattern that semantic-release can parse, we use husky to setup git commit-msg hook to run

with cz-conventional-changelog plugin. In this manner we can be sure that we never push a commit that does not adhere to the standard.

Conclusion

Over the course of this two-part series, we've embarked on a comprehensive journey through the development and deployment of a modern, dynamic CV. From the initial bootstrapping of the Nx monorepo and crafting a React-based CV in part one, to the intricate DevOps orchestration involving AWS infrastructure and CI/CD pipelines in part two, this project has been a testament to the power of combining cutting-edge web development with robust DevOps practices.

Key Takeaways

• Streamlined Development: By leveraging tools like AWS CDK and GitHub Actions, we've demonstrated how complex infrastructure and deployment processes can be streamlined, making them more efficient and less prone to error.

• Rapid Deployment and Scalability: The use of Docker, CloudFront CDN, and serverless technologies ensures that our applications are not only rapidly deployable but also scalable to meet varying demands.

• Automated Workflows: The integration of CI/CD pipelines exemplifies the automation of testing, building, and deployment processes, significantly reducing manual effort and enhancing productivity.

The methodologies and technologies employed in this project are not confined to personal CV development. They can be seamlessly adapted to a wide range of applications, from e-commerce websites to complex enterprise-level solutions. The principles of automation, scalability, and efficient deployment are universally applicable, offering a blueprint for future projects that demand high availability, rapid scalability, and continuous integration/delivery.

I'm curious to hear about your experiences. Have you undertaken a project that combines web development, backend event driven processes with DevOps in a similar manner? What challenges did you face, and how did you overcome them? Additionally, I welcome any suggestions or insights you might have. Perhaps there's an aspect of DevOps or a particular technology you've found particularly useful or challenging? Let's start a conversation in the comments below and learn from each other's experiences in this ever-evolving field of web development, backend development and DevOps.

You can find the entire code for this series here and the final product here.

Subscribe now

In this two-part series, I'll detail how I transformed my CV into a dynamic, automated, and easily accessible digital asset. Usually when you think of getting your CV you have to go to Drive, iCloud, Dropbox or Office 365 to export the latest PDF. Additionally, while writing the CV, you not only have to wrangle with the content but also with the layout, having to deal with every indentation block, image alignment and column header.

A popular solution for the second option is to use tools such as LaTeX, a tool which is used extensively in the academic world for this exact purpose. But even so, the first issue is still present, your CV is not readily available to be handed over.

To fix the above pain points and more, as an engineer, I have done what engineers do and I went and engineered a solution to my problem and here I will outline the steps I have gone through to attain this goal. At the end of it, the CV is be versioned, linted, tested, has IaC, is CDN hosted, has CI/CD and is available for download at a click of button. The end result is available at cv.crisboarna.com.

The steps I have to follow now to update my CV are as follows:

1. Open .json file containing relevant section of my CV.

2. Perform relevant edits.

3. git commit

4. git push

5. ???

6. Profit!

The CI/CD pipeline takes cares of the rest and the website cv.crisboarna.com and downloadable PDF are available within 5 minutes with no further input from me. I even get an email to notify me of the latest version being published.

To obtain the end result I am using Nx monorepo with 3 apps. As there are several components that all share common symbols, such as the name of the file and all services are written in the same language, I am leveraging a powerful tool, Nx, to bootstrap and increase my development agility. Nx is similar to Lerna and Yarn workspaces in that it handles the wiring between microservices but excels beyond the competition in DevEx and bootstrapping capabilities for different use cases. There are 4 main services that coalesce to achieve the end result hosted at the above link:

• @nx/react CV website

  • This is the actual CV, written in React

• Exporter Lambda Function

  • This is the CloudFormation Custom Resource Lambda leveraging Docker Custom Runtime with Puppeteer driven Chromium headless browser for PDF exporting

• Infrastructure code

  • AWS CDK NodeJS IaC(Infrastructure-as-Code) for ECR(Docker image hosting), Lambda(above exporter), S3 bucket(website distributable hosting), CloudFront CDN(web front to serve files)

• GitHub Actions

  • Glue that binds all of the above together to lint, test, build, deploy on every pull request and merge to main.

The first two development focused points are the subject of this post and the last two DevOps focused points are the subject of the next post.

I have done this many years ago but after revisiting and updating all of the dependencies, I have run into a couple of issues that prompted a more involved refactor, the result of which is detailed here.

The Functions-as-a-Service (FaaS) offerings have been a godsend that have significantly reduced the boilerplate needed to run one-time applications in an event-driven context but sometimes you need an escape hatch to perform more advanced actions.

One such scenario is when you want to have Puppeteer orchestrate a headless Chromium browser in order to perform a PDF export of a specific web page. I have done this historically using the chrome-aws-lambda NPM package that contains a pre-built binary of Chromium leveraging default AWS provided NodeJS Lambda runtimes. Unfortunately this package has since seemingly been abandoned with no support for versions of NodeJS >=16 combined with AWS deprecating the AWS Lambda NodeJS 14 runtime, the impetus has been given to search for alternative solutions.

While there now exists @sparticuz/chromium to carry on the torch of this, with a well defined lockstep with latest Chromium Testing versions, the local testing solution suggested does not an ensure apple-to-apples comparison between the local and Lambda results which resulted in PDF’s that looked vastly different between local and Lambda.

Another issue is the above mentioned AWS deprecation of runtime versions which would manifest itself in the future for NodeJS 20.x as it has now for NodeJS 14.x at an inappropriate moment.

As such given the above, I have set to leverage the provided “escape hatch” provided by AWS and create my own custom AWS Lambda Runtime Docker image.

1. CV Website

This is a standard @nx/react bootstraped website with TypeScript and CSS modules and Webpack bundling. The content of my CV is stored within JSON files for each section and in similar vein to LaTeX, I just write the content and React deals with laying it out on the page.

There are some particularities of course as I have a 2 page A4 CV, there is special care needed at the page break so it does not cut the text midway.

The web application code can be found here. You can also run it locally or via Docker:

nx serve web

docker compose up web

2. Exporter Lambda Function

This is the Lambda that runs a headless Chromium browser using Puppeteer to load the web application and create a PDF export of it. It passes query parameters that hide the top Download and GitHub buttons for the PDF to only contain the CV itself.

The code itself is straightforward but custom enough to not be able to directly leverage a Puppeteer Docker image.

This code is built leveraging esbuild which creates similarly to webpack a bundle of the code and all third party dependnecies into a single main.js as this will be shipped in the Docker image for Lambda to run, so it does not need any splitting.

Due to the @sparticuz/chromium dependency containing binary elements, it is ignored from the bundle and installed directly within the container.

At this point the Web and Exporter services can come together locally with the help of a simple docker-compose.yaml to be able to view the end result CV PDF saved on the repository root.

Troubleshooting

A particular stumbling block that I encountered due to being on an ARM64 CPU running Chromium:

qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

Upon investigating I realized that only now is the Chrome team starting to roll out ARM builds for Chrome and the container having Linux ARM64 architecture would error out elusively with the above message which can easily be solved by adding

“platform: linux/amd64” to your compose file.

If you are to run it now, you will get a new error, namely:

qemu: uncaught target signal 11 (Segmentation fault) - core dumped

which is even more elusive than the last. To cut the store short, the solution to this is to enable

Docker Desktop > Features in development > Use Rosetta for x86/amd64 emulation on Apple Silicon.

These last few items only apply if you are on a M-Series Mac of course.

Now we are ready to start the web and the exporter services up and view our publishable CV website and downloadable CV PDF locally through the following 3 commands in 3 separate terminals:

nx serve:docker web

nx serve:docker exporter

curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

The first two commands start the services and the last one invokes the Lambda runtime with an empty payload to trigger our handler which will access the CV website within the Docker network on ‘http://cv_web:4200’, as that is the configured hostname we have given the React service container and we have configured webpack for ‘0.0.0.0’ to listen on all adapters and accept connections from ‘all’.

Conclusion

I have described above how we can leverage the power of Nx monorepos to manage React & NodeJS applications and used Docker to create a custom AWS Lambda Runtime image. We also encountered and resolved issues related to running Chromium on an ARM64 CPU.

In the next part of this blog post series, we will delve into the details of setting up the Infrastructure-as-Code (IaC) using AWS CDK NodeJS and how I used GitHub Actions to automate the process of linting, testing, building, and deploying our applications. We will also discuss how I managed to host our Docker image on ECR, set up our Lambda function, and host our website distribution on an S3 bucket with CloudFront CDN.

This is a clean and beautiful approach to solving the discoverability issue of our CV while applying true and tested techniques for linting, testing, versioning and showcasing our abilities.

Please share your suggestions and opinions on my approach to solving this, I am curious to see what ideas other have and how this can be improved upon !

Given that we are living in the 2020’s, having to install various small utility packages in order to interact with applications and tooling software is quite old fashioned, error prone and non-reproducable.

This has been solved historically with writing documentation that may go out of date / sync with the code or small scripts that achieve the goal but leave artifacts laying around. Not to mention that with the advent of

One such scenario I have encountered recently is when trying to create a custom AWS Lambda Runtime using an custom base image that requires installing aws-lambda-ric and testing it out locally.

Based on the above, on AWS’s recommendation, we need to download and make available the executable of the aws-lambda-rie, an emulator that allows us to invoke via curl the Lambda.

Now the trick is to create a lightweight Docker container with the executable, under the same architecture as your Lambda:

The above will then be leveraged in a local docker-compose.yaml as follows:

The most interesting part to see is that in the above Dockerfile we unpackage and set the executable in /rie-bin, which is a mounted Docker volume on the same path in the docker-compose.yaml. This is then shared at the same time with the Lambda container on the same path, effectively sharing the executable across the containers.

The Lambda container is based on the AWS example here:

Which contains a simple NodeJS Lambda handler:

And it all comes together for a ‘no straggler left behind’ approach as follows:

Now from a separate terminal we can use curl to invoke the function:

As we can see we invoked the function on the AWS Lambda standard path and got the response returned from the Lambda once it completes execution. In Terminal 1 where the Lambda is running, we can see the same output we see in CloudWatch Logs when viewing the output:

This is a brief yet powerful example on how to configure sidecars for any dependencies that you require

The full code for having a AWS Lambda Runtime Interface Emulator Sidecar with a custom Lambda Docker runtime can be found here.

While this example is entirely in NodeJS, there is nothing stoping you from having the Lambda container use Go, Rust or any other language with custom configured dependencies.

In conclusion, the use of Docker containers and AWS Lambda Runtime Interface Emulator provides a robust and efficient solution for managing dependencies and testing out custom AWS Lambda Runtimes locally.

This approach not only ensures reproducibility but also eliminates the need for installing multiple utility packages, thereby reducing potential errors. The flexibility of this method allows for the use of various programming languages, making it a versatile tool for any developer's toolkit.

Remember, the key to this process is the effective sharing of the executable across containers using Docker volumes. As we continue to innovate and streamline our development processes, such practices pave the way for more efficient and error-free coding environments.

As a closing to this series of post related to accessing isolated networks inside AWS, I will cover the approach of using the AWS Client VPN. This is a managed client-based VPN service that enables you to securely access your AWS resources and your on-premises network via a managed VPN link.

The overall advantage of this solution is that it greatly simplifies the network and resources required to achieve the audited and authenticated access to the system.

How it works

This solution leverages a mutual authentication OpenVPN based link using certificates that you provide along with the possibility to setup a self-service certificate vending dashboard for users if you have Active Directory/SAML based auth.

Use Cases

Advantages

• Consistent Access: Utilizes a consistent method of access that is public and widely used, regardless of the underlying service. This is in contrast to services like ECS Exec and Instance Connect, which are specific to certain AWS services.

• Reduced overhead: The VPN endpoint is AWS managed and only some ENI are placed within your target subnets, removing any management need.

• Self-Service Certificates: A management portal can be configured to enable self-service for users to obtain the connection credentials, further removing operational steps.

• Scalable: Scales based on number of users and traffic thus reducing situations where corporate VPN is slow or fails due to load.

• Security: This uses OpenVPN, which is an industry-standard VPN protocol, that provides a high level of security. EC2 instances in comparison use SSH keys, which can be vulnerable if keys are not managed properly, although Instance Connect mitigates this issue.

• Centralized Control: Centrally managed access to your AWS resources from a single access endpoint. This is of similar vein to the ECS Exec and Instance Connect, where you can limit the user via IAM by granting/blocking the core permission for the action. It also allows live monitoring and termination of active connections.

• No VPC endpoints: Allows the creation of an isolated network with no Internet Gateway, without requiring VPC endpoints.

Disadvantages

• Cost: Client VPN can be more expensive than other solutions, especially for large numbers of users or heavy data usage. This is ideal if you are mainly looking for access for a certain number of users on an ad-hoc basis such as developers / QA accessing the environment for investigations or testing.

• Complexity: Setting up Client VPN can be complex, especially if you're not familiar with VPNs or networking. But hopefully this will cover the basics of it ;)

• Limited Protocol Support: Client VPN only supports the OpenVPN protocol. If you need to use a different protocol, you'll need to look at other solutions.

Example

As the VPN does not require any further resources to be functional and is completely self-contained with just the VPC and VPN constructs, these are the only ones showcased below. As once the VPN connection is established, all resources contained within the private network that have a reachable route defined in the route tables of the subnets where the VPN endpoints are placed, are reachable from the local PC.

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in your preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

mkdir -p accessing-isolated-network/bastion-vpn
cd accessing-isolated-network-access/bastion-vpn
npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Step 2: Define the VPC Stack

Next, we define the VPC stack where the Client VPN Elastic Network Interface will be deployed.

Open up the lib folder and create a new file called vpc-stack.ts. We will only need private isolated subnets.

Step 3: Define the VPN Stack

We now create the VPN setup for the service. Open up the lib folder and create a new file called vpn-stack.ts.

As we are taking the mutual authentication path, we need to generate a base pair of certificates for the server and the client. This process is not as hands-off as the Active Directory / federated path with self-service portal, but it achieves the same target goal. The steps to generate the certificate pair is eloquently described here. Once you have completed the linked steps, you will have a pair of certificates in ACM that we need to pass the id’s of here under `vpcCert`.

They can be found in ACM here, assuming you followed the pattern of vpn.<domain>.com for server certificate and <user>.vpn.<domain>.com for client certificate.

Step 4: Define the App Stack

Deploying this stack is done via the main of:

Step 5: Deploy app

The cleanest way to pass the newly created certificates is via the CDK context construct directly from the terminal.

> npx cdk deploy --profile=<profile> --all -c serverCert=<cert_id> -c clientCert=<cert_id>

This will take a few minutes as the deployment of the VPN endpoints and the authorizations are not instantaneous.

Once deployment is complete, you need to download the client configuration for the VPN application, which can be found in AWS Console > VPC > Client VPN endpoints.

Once that is downloaded, and you have installed the VPN client, you can open the application,

File > Manage Profiles > Add Profile

Select the downloaded file, give it a name, confirm, confirm.

Now you are ready to click on Connect and be welcomed by this view:

Step 6: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

Conclusion

The strategic implementation of a managed VPN solution, with dynamic scaling based on usage, self-service capabilities is an invaluable asset in our technological toolkit that paves the way for a fully automated, secure and audited access pattern into cloud and on-premises networks.

This solution, characterized by its ability to maintain complete isolation from the internet while retaining global accessibility to all of the resources present within the private network that have a reachable route from the subnet where the VPN endpoints are placed, represents a significant advancement in our infrastructure capabilities.

In conjunction with the AWS SSM Session Manager, especially that of ECS Exec, offer a comprehensive approach to managing scenarios where both ad-hoc and continuous access to components of applications running on AWS is required.

As always, I invite you to explore the complete codebase for the aforementioned example, available in my GitHub repository, linked here.

I hope this 4 part series has proved useful and helped you in making a decision regarding the route you want to take in accessing your isolated networks on AWS. If you know of any other practical or esoteric and interesting solutions to achieve the goal of connecting to isolated networks, do comment or send me a message on it, I am keen to learn !

We will continue where we left off in the previous post here, by talking about another, more advanced way to reach your private resources within your AWS network estate.

Do note that this approach does not require a host exposed in a public subnet which means the need for hardening, auditing and strict compliance measures that you would have to take with normal bastions does not apply as stringently to this method as it is significantly harder for bad actors to use it as a spring board to wreck chaos on your platform. This is the second post where we cover the methods that do not require exposed network connectivity which will greatly reduce the security exposure of your solution, especially if it has no web or API component. The first method not requiring any internet access for accessing the bastion can be found here:

Advantages

• No servers to manage.

  • No EC2, no EBS, no instance profile, AMI, lifecycle resources to worry about and manage manually yourself. While it is not a true “serverless” concept, as you still have potentially long-running processes, this pushes the undifferentiated heavy lifting across the shared responsibility model into AWS’s court, releasing you from all of this operational burden.

• Easier to compose the final image.

  • There are significantly more individuals that know how to write Dockerfile images than Packer, Chef or Ansible. While it is recommended for somebody with experience in creating hardened images, especially ones that are publicly available and configuring them well, this lowers the bar, allowing more hands to participate and quickly iterate.

    1. This also reduces the complexity of building a golden image due to the simplified build process.

• Reduces net total resource count to be managed.

  • Especially if you are using AWS CDK, there are L3 constructs that allow the creation of the entire resource stack with less than 50 lines of code.

• Cost savings.

  • With AWS ECS Fargate, you only pay for the resources you use. Fargate allows you to specify the amount of CPU and memory required for your container, and you are only billed for the amount of resources that your container uses, rounded up to the nearest second.

    This means that if you only need a bastion host for a short period, such as during an incident response, you can launch a Fargate task with a smaller instance size, and only pay for the exact amount of resources that you use.

Disadvantages

• More configuration needed.

  • If we are comparing the basic resources and configuration items needed to get a EC2 running compared to a Fargate task, then the Fargate route is a much more involved and knowledge intensive process.

  • This difference is offset once we start discussing scalability with auto scaling, cost savings with Spot instances, load distribution with load balancers of course.

  Thanks for reading Engineer Mindscape! Subscribe for free to receive new posts and support my work.

Note

When discussing this with colleagues it has been brought up that boot time is longer for the Fargate option which I see as being more nuanced as opposed to a clear cut disadvantage.

As with any architectural design, the presence of our dear friend ‘it depends’ is felt due to the variety of elements that can affect the total boot time that we need to take into account:

Fargate

1. Task definition size

   1. The size of the task definition can impact boot time. Larger task definitions may take longer to load into memory, which can increase the time it takes for the task to start.

2. Container image size

   1. The size of the container image can also impact boot time. Larger images take longer to download and extract, which can delay the start of the container.

3. Resource allocation

   1. If your task requires a large amount of CPU or memory, it may take longer to provision the resources needed to start the task.

EC2

1. Instance type

   1. The instance type you choose can impact boot time. Larger instances may take longer to provision and start up.

2. AMI size

   1. The size of the AMI can also impact boot time. Larger AMIs may take longer to load into memory and start up.

3. Boot volume type

   1. The type of boot volume you choose can also impact boot time. For example, an instance with an EBS-backed boot volume may take longer to start than an instance with an instance-store backed boot volume.

4. User data scripts

   1. If you are running user data scripts on your instance, they can impact boot time. Long-running scripts or scripts that perform complex tasks can delay the instance startup.

Of particular note in the factors above is the juxtaposition of the resource allocation versus the instance type which, while both are configurable, if the requested instance type pool is exhausted, your server request is not getting processed. Whereas for Fargate, as long as you have not requested the higher end 8/16 vCPU, your request will be processed in a timely manner.

Example

Note

This is an optimized implementation that references an image from ECR and uses VPC Endpoints to completely lock down the bastion network and disallow any WAN access to avoid possibility of exfiltration of data.

While we are optimizing for simplicity here, there is still a lot of “supporting” implementation needed, to achieve the locked down environment in ECS Fargate is the focus of my other blog post:

Let’s now go through the code needed to launch a simple bastion host on AWS ECS Fargate using AWS CDK.

Do note that I will not be using the ‘@aws-cdk/aws-ecs-patterns’ QueueProcessingFargateService or ScheduledFargateTask L3 construct as I want to keep it as simple and straightforward as possible to digest.

Of course you should strive to leverage the full suite of tools offered by AWS for scalability and resiliency, but for our use case, conciseness is of greater value and I have mentioned the L3 construct above as a guidance towards what you should be looking for when trying to scale up this approach for your use case.

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

> mkdir -p accessing-isolated-network/bastion-fargate

> cd accessing-isolated-network-access/bastion-fargate

> npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Step 2: Define the VPC Stack

Next, we define the VPC stack where Fargate will place the container ENI.

Open up the lib folder and create a new file called vpc-stack.ts. Add the following code to create a new VPC with a private subnet and the required 6 VPC Endpoints:

• ECR-API

• ECR-Docker

• S3

• SSM

• SSM-Messages

• EC2-Messages

• Logs

The ECR-API, ECR-Docker & S3 endpoints are needed for pulling the image, SSM, SSM-Messages and EC2-Messages are needed for remote accessing while Logs & S3 is needed to send audit data to target system.

Step 3: Define the Fargate Bastion Stack

Now create a file named fargate-stack.ts with the below snippets pertaining to the setup of ECS & ECS Exec of content. The full code can be found linked in the caption and at the end in GitHub. The explanation of the locked down ECS Fargate foundation setup is linked above in my other blog post.

Here we are defining the Fargate Service that will launch the container in the private subnets and will communicate only with the required AWS services via the VPC Endpoints.

Step 4: Define the App Stack

Deploying this stack is done via the main of:

Step 5: Deploy app

> AWS_PROFILE=<profile> npx cdk deploy --all

Do note that in-between the “VpcStack” and “BastionFargateStack” ideally you should also populate the ECR repository with the image from DockerHub as we do not have any WAN access to pull it from there.

As such you should run the following commands in the terminal

> export REGION=<region> && export ACCOUNT_ID=<account_id>

> docker pull amazonlinux:2023.0.20230607.0

> docker tag amazonlinux:2023.0.20230607.0  $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/bastion-fargate:latest

> aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com

> docker push $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/bastion-fargate:latest

After the ECR repository is populated with the "amazonlinux” image, we can proceed with the second stack deployment:

All commands in this terminal session will be recorded and logged in Cloudwatch as configured.

Any command you enter in the terminal will be logged to the target log group. You can send logs to CloudWatch and/or S3 for future data mining and monitoring.

This is a clean and simple way to ensure that any “break-glass” done that reaches into an application container or a bastion jump host is logged, audited and alarms can be triggered based on this data stream.

Step 6: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

In the following post I will cover a similar approach using AWS Client VPN which removes any running server requirement and links your local machine to the AWS private VPC network representing the smallest footprint and management overhead option.

Conclusion

The strategic implementation of a serverless bastion host, capable of dynamic scaling to optimize cost efficiency that leverages Spot instances to potentially reduce operational expenses by up to 70%, is an invaluable asset in our technological toolkit.

This solution, characterized by its definition in user-friendly Dockerfiles and its unique ability to maintain complete isolation from the internet while retaining global accessibility, represents a significant advancement in our infrastructure capabilities.

In conjunction with the AWS Client VPN, which will be the focus of my next post, these two solution offer a comprehensive approach to managing scenarios where ad-hoc access to components of applications running on AWS is required.

For a deeper understanding and practical application, I invite you to explore the complete codebase for the aforementioned example, available in my GitHub repository, linked here.

In a typical setup, compute nodes such as AWS Fargate tasks within an Amazon ECS cluster require internet access to pull container images, push logs, and other tasks. However, in certain scenarios, you might want to restrict internet access due to security or compliance requirements. This is where a Fargate ECS cluster with no internet connectivity comes into play.

In this post, we leverage VPC endpoints, which allow private connectivity between your VPC and supported AWS services. This means your Fargate tasks can communicate with necessary AWS services without requiring access to the public internet to the point where there is no network traffic that leaves your private VPC network beyond reaching out to integrations via PrivateLink endpoints.

Use Cases

1. Regulated Industries: For industries like finance (PCI DSS) or healthcare (HIPAA), where data security is paramount, this setup provides an additional layer of security by eliminating unnecessary internet access.

2. Data-Intensive Applications: For applications that require high-throughput, low-latency access to AWS services, using VPC endpoints can provide performance benefits.

3. Multi-Tier Applications: For applications with strict network segmentation requirements, you can use this setup to enforce network boundaries between tiers.

Advantages

• Enhanced Security:

  • By eliminating the need for internet access, you reduce the attack surface that bad actors can leverage. This setup is particularly beneficial for sensitive workloads that require stringent security controls such as PCI DSS compliant systems.

• Network Performance:

  • VPC endpoints enable private connectivity to AWS services, which can lead to lower latency and higher throughput compared to internet-based connections as they do not leave the AWS backbone network and follow the shortest possible path to target servers.

• Cost Optimization:

  • Data transfer costs for VPC endpoints within the same region are typically lower than data transfer costs over the public internet.

Disadvantages

• Complexity:

  • This of course comes with the added complexity of setting up more resources that are more laborious and knowledge intensive resulting in a more complex network architecture.

• Service Limitations:

  • Not all AWS services support VPC endpoints. You'll need to ensure that the services your tasks depend on can be accessed via VPC endpoints for this to be a viable approach for you.

• Cost:

  • While you can save on data transfer costs, there are costs associated with using VPC endpoints, unless you are only using S3/DynamoDB which will can add a pretty penny to your bill.

Example

In this example we will create a simple ECS Fargate service that is auto-scaled across AZ, which pulls it’s image from a private ECR repository and logs data to CloudWatch logs, without any network connectivity.

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

> mkdir -p isolated-network-workloads/fargate

> cd isolated-network-workloads/fargate

> npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Step 2: Define the VPC Stack

Next, we define the VPC stack where Fargate will place the container ENI.

Open up the lib folder and create a new file called vpc-stack.ts. Add the following code to create a new VPC with a private subnet and the required 4 VPC Endpoints:

• ECR-API

• ECR-Docker

• S3

• Logs

The ECR-API, ECR-Docker & S3 endpoints are needed for pulling the image and Logs is needed to send logs to CloudWatch.

Step 3: Define the Fargate Workload Stack

In this stack we will create the log group, security group, security group rules, cluster and service definition.

Create a file named fargate-stack.ts with the following content:

Step 4: Define the App Stack

Deploying this stack is done via the main of:

Step 5: Deploy app

> AWS_PROFILE=<profile> npx cdk deploy --all

Do note that in-between the “VpcStack” and “FargateStack” ideally you should also populate the ECR repository with the image from DockerHub as we do not have any WAN access to pull it from there.

As such you should run the following commands in the terminal

> export REGION=<region> && export ACCOUNT_ID=<account_id>

> docker pull amazonlinux:2023.0.20230607.0

> docker tag amazonlinux:2023.0.20230607.0  $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/workload-service:latest

> aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com

> docker push $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/workload-service:latest

After the ECR repository is populated with the "amazonlinux” image, we can proceed with the second stack deployment:

Now we can see in the ECS console the running task processing our workload.

And this task is running completely disconnected from the internet on a compute cluster that can easily be configured to scale, react to inputs and process varying payloads in isolation.

Step 6: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

Conclusion

Running a Fargate ECS cluster with no internet connectivity is a powerful strategy for enhancing security and performance. However, it's not a one-size-fits-all solution. As with any architectural decision, it's crucial to weigh the benefits against the potential drawbacks and costs.

Remember, the goal is not to apply every possible security measure, but to apply the right measures for your specific use case. As always, keep your business requirements, regulatory landscape, and risk tolerance in mind when making these decisions.

I hope this deep dive has provided you with valuable insights into this advanced networking setup. As always, I'm here to answer any questions you might have. Let's continue to push the boundaries of what's possible with AWS!

You can find the entire codebase for the above example in my GitHub repo here.

In this post we will talk about creating a bastion host that can be used as a jump box but with an added twist, it is not reachable and has no access to the internet. We will be accessing it solely through the AWS Systems Manager Session Manager.

Let’s get directly into the advantages:

Advantages

• Reduced operational complexity

  • No more need to manage SSH keys and their lifecycle. No requirement to add/remove people from SSH agent when they join/leave. No need to worry about them being compromised.

• Enhanced Security

  • Leverages AWS Identity and Access Management (IAM) for access control. To start a session with an instance, a user needs IAM permissions. This model integrates well with your existing policies and procedures for IAM.

  • Without internet connectivity, our Bastion Host is insulated from potential external attacks. The access via Session Manager further fortifies its security, as it negates the need for managing SSH keys.

  • The security footprint is smaller as there is no WAN port that is continously scanned with failed login attempts spamming the server logs.

• Audit and Compliance

  • AWS Systems Manager Session Manager logs all session activity, making it easier to maintain an audit trail and comply with governance and regulatory requirements. This means every connection and command is audited and logged in CloudWatch/S3/CloudTrail as configured by you.

• Cost-Effective

  • Since there is no need for an Elastic IP or an Load Balancer, we save on costs associated with them, which is non-trivial given the usual 24/7 running of these resources. As you can see in the above architectural diagram, besides the VPC Endpoints and the EC2 instance, there are no other resources needed for a MVP. Ideally for more advanced, stable setups you would use a Spot Fleet to ensure the livelyness of the bastion host.

Disadvantages

• Dependency

  • This relies exclusively on AWS Systems Manager, making us susceptible to potential service disruptions when there are issues on AWS with the service.

• Limited Network Testing

  • Lack of internet connectivity may limit certain network diagnostic capabilities. But if this is needed on an ad-hoc basis, it can be added in and removed once done.

How it works

In order to get this working, we need any Linux distribution that supports the SSM Session Manager agent to be installed. Alternatively, you can use a more recent AMI that has the agent already baked in such as a recent ECS-Optimized AWS managed AMI.

The way Session Manager works is by establishing a secure, bidirectional, and authenticated channel for communications between our terminal and the managed instance, without the need for an open inbound port or to maintain SSH keys via the agent.

The SSM Agent process runs on the EC2 instances and communicates with the Systems Manager service. When a Session Manager session is initiated, the SSM Agent establishes a WebSocket connection with the Systems Manager service, facilitated by the Amazon Message Delivery Service. The WebSocket connection acts as a conduit for command-and-control (C2) interactions, enabling you to run commands, scripts, or even PowerShell cmdlets interactively on the instance.

All the data transmitted during a Session Manager session is encrypted using Transport Layer Security (TLS) 1.2. The communications traverse through the Amazon network backbone, eliminating exposure to the public internet.

You need to install the Session Manager plugin on your local machine so the AWS CLI can leverage it when setting up the connection. The installation steps can be found here.

Example

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

mkdir -p accessing-isolated-network/bastion-ssm
cd accessing-isolated-network-access/bastion-ssm
npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Step 2: Define the VPC Stack

Next, we define the VPC stack where the EC2 instance will be deployed.

Open up the lib folder and create a new file called vpc-stack.ts. We will only need a single public subnet. Of course for actual use-cases you will have more subnets and ideally many of them private ;) but for simplicity we will only focus on the topic at hand.

As you can see we create VPC Endpoint for SSM, SSM-Messages and EC2-Messages. This is required as the SSM Session Manager requires to communicate with all 3 endpoints in order for the connection to function properly.

Step 3: Define the Bastion EC2 Stack

We now create the standard EC2 instance running Amazon Linux 3 (see supported operating systems here.)

Note

Depending on the configurations done to your account, the following error may occur:

To fix this you need to go and disable Systems Manager > Session Manager > Preferences > Edit > “Enable Run As support for Linux instances” in the AWS Console for the target account.

Step 4: Define the App Stack

Deploying this stack is done via the main of:

Step 5: Deploy app

> AWS_PROFILE=<profile> npx cdk deploy --all

Now any command in this terminal session will be recorded and logged in S3/Cloudwatch if configured in Systems Manager > Session Manager > Preferences > Edit > CloudWatch/S3 logging.

Step 6: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

In the following post we will use a more advanced, “serverless” offering from AWS where we will leverage Docker containers for our bastion operating system and ECS Fargate for hosting and running. More details regarding the benefits for this feature in the next post.

You can find the entire codebase for the above example in my GitHub repo here.

Regardless of your profile, whether you're responsible for hands-on implementation or focused on the business aspects of a project, it's essential to have a solid understanding of how to best protect and secure your venture. This knowledge is important for everyone involved in the project, regardless of their technical expertise or business acumen.

I want to emphasize that for any project, regardless of its size, there will come a time when you need access to the underlying components of the system, such as databases, caches, pipelines and other processes. However, in an isolated network, such as an AWS VPC, this can be challenging, as opening up the network leaves it vulnerable to security threats and bad actors. As most systems with a non-trivial complexity are not hosted on SaaS services such as Netlify or Vercel as having access to the underlying infrastructure is essential for developmental, operational and support reasons.

In the following blog posts, we'll focus on how we can setup in a matter of minutes with just a few commands a variety of access patterns with varying levels of complexity and security guarantees with the help of AWS CDK (Cloud Development Kit) to provide secure access points to your private instances in a VPC, enabling you to manage your infrastructure safely and efficiently.

In brief, the access patterns that we will cover in this blog series will cover:

• EC2 Bastion Host

  • A special-purpose instance that acts as a proxy server for connecting to other instances in your VPC. It provides secure access using SSH or Remote Desktop Protocol (RDP). Simplest, most involved, requires instance hardening, monitoring, patching, further configuration to setup all additional processes & services (Terraform/Chef/Puppet for example), is a single point of failure.

• AWS Systems Manager Session Manager

  • Safer but more complex to configure, unless you use an AMI with it baked in ;). Main advantage compared to above options is that the instance does not require to be directly reachable from the internet, greatly reducing the threat profile. As long as there is a network path to AWS Systems Manager from your end and the target AWS EC2, the instance can function securely, providing an additional layer of protection to your infrastructure.

• ECS Fargate Service Bastion Host

  • Similar to an EC2 bastion host, but by leveraging ECS Fargate we can push the responsibility of base patching down to AWS, which reduces our overall threat surface to manage. By using this approach, we can ensure that our infrastructure is more secure, and we can focus our efforts on other critical areas. It is also potentially easier to replace and scale as most configuration can be contained in the task image Dockerfile.

• AWS Client VPN

  • Completely eliminates the need for a proxy bastion instance by granting the device we are VPN-ing in from access to the network within the VPC directly via the VPN tunnel. This not only simplifies the setup process but also eliminates the need to harden, monitor, patch or maintain a separate bastion host while minimizing the operational overhead for all parties involved.

For the remainder of this blog post we will focus on the first method of connecting, via an EC2 bastion host.

Note:

This is for demonstrative purposes, this blog posts bastion host design has no scalability or recovery mechanisms added for clarity and straightforwardness. For real workloads, please take into account the possibility that the EC2 instance may fail and needs replacing, look into EC2 Spot Fleet or the other posts in this blog series for alternatives.

Prerequisites:

Before we begin, make sure you have the following prerequisites installed and configured:

• AWS CLI [install guide]

• AWS CDK [install guide]

• Node.JS and NPM [install guide]

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

mkdir -p accessing-isolated-network/bastion-ec2
cd accessing-isolated-network/bastion-ec2
npx cdk init app --language=typescript

This will create a new CDK app in TypeScript with the following structure

Step 2: Define the VPC Stack

Next, we need to define the VPC stack that our EC2 bastion host will be a part of. While this can all be done in one stack, it is best practice to separate the constructs as the layers of an onion, each layer building on top of the previous one, reducing potential blast radius for any change and reducing the number of resources that need to be touched for any one change.

Open up the lib folder and create a new file called vpc-stack.ts. Add the following code to create a new VPC with a public subnet:

Step 3: Define the Bastion Stack

Now create a file named bastion-stack.ts with the following contents:

Step 4: Define the App Stack

The glue to tie the two stacks together is:

Which you can now run with

> AWS_PROFILE=<profile> npx cdk deploy --all

followed by the sequence of commands provided in the output of the command above, as can be seen below:

Step 5: Cleanup

Now that we have created all the needed resources and have gained access within the estate, time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

Now while we could have used the classic process of uploading a long-lived SSH Key to AWS and referencing that when creating the bastion host, I chose for this example to highlight the newer approach using the AWS Instance Connect API which uploads using the “aws ec2-instance-connect send-ssh-public-key” to the EC2 Metadata service. This key is ephemeral and available for 60 seconds to perform the ssh connection after which it is deleted from AWS and you need to upload it once more. You can read more about the advantages and disadvantages of this method in my previous post

Engineer Mindscape

AWS EC2 Key Pairs versus EC2-Instance-Connect

When it comes to managing secure shell (SSH) access to Amazon Elastic Compute Cloud (EC2) instances, there are two primary methods: AWS EC2 Key Pairs SSH AWS Instance Connect. Both methods provide secure and convenient ways to access EC2 instances, but they have different pros and cons that make them suitable for different use c…

Read more

3 years ago · Cristian Boarna

Regardless of your preferred connection approach, gaining access to your AWS estate be it for development, break-glass emergency production access or anything in-between, it is good to know what your options are.

In the following post I will cover a similar approach using AWS Systems Manager Session Manager instead of conventional EC2 self-managed servers.

You can find the entire codebase for the above example in my GitHub repo here.

When it comes to managing secure shell (SSH) access to Amazon Elastic Compute Cloud (EC2) instances, there are two primary methods:

• AWS EC2 Key Pairs SSH

• AWS Instance Connect.

Both methods provide secure and convenient ways to access EC2 instances, but they have different pros and cons that make them suitable for different use cases. In this blog post, I'll briefly explain, compare and contrast between the two options highlighting which may suit your needs best.

EC2 Key Pairs

AWS EC2 Key Pairs SSH is the conventional method of SSH access to servers from times immemorial. With this method, users generate a key pair consisting of a public key that is uploaded to AWS, and a private key that must be kept safe, in a Key Management tool for example.

How it works

You either upload the SSH key to AWS as a Key Pair and reference it during instance creation for AWS to set it up for you or you leverage the User data property at AMI boot (if plain EC2) to download and configure it yourself.

If you are using ECS Fargate / Kubernetes, the process varies according to preferences.

Pros:

• Wide usage and familiarity among users spanning back decades meaning there are many resources available to help with key pair management.

• Plentiful of SSH clients and tooling around it.

Cons:

• Keys must be stored securely

• Distribution to authorized users is an operationally cumbersome process

• A compromised key requires updating all associated instances which can be quite an involved and lengthy process

• Keys must be uploaded into AWS EC2 Key Pairs before starting an instance

Example

Step 1: Initialize the CDK App

First, we need to initialize a new CDK app in our preferred language, which for me is currently TypeScript. Let’s open up a terminal and run the following commands to get started:

mkdir -p ec2-access-ssh
cd ec2-access-ssh
npx cdk init app --language=typescript

Step 2: Define the VPC Stack

Next, we need to define the VPC stack that our EC2 server will be a part of. While this can all be done in one stack, it is best practice to separate the constructs as the layers of an onion, each layer building on top of the previous one, reducing potential blast radius for any change and reducing the number of resources that need to be touched for any one change (blast radius for lower layers is greater than that of higher layers, but you still get reduces deployment time).

Open up the lib folder and create a new file called vpc-stack.ts. Add the following code to create a new VPC with a public subnet:

Step 3: Define the EC2 Stack

Now create a file named ec2-stack.ts with the following contents:

Note

The SSH key found in the example source code has been created using

> ssh-keygen -t rsa -f ec2_rsa_key

and the key is committed into source to have a fully running example. You should use the command above to generate your own secure key and keep it safe. If you are within an enterprise organization, there surely are established processes for keeping the key safe.

Step 4: Define the App Stack

Which you can now run with

> npx cdk deploy --all

followed by the ssh command provided in the output of the command above, that is contained in the Ec2Stack class, as can be seen below:

Step 5: Cleanup

Now that we have created all the needed resources and have gained SSH access, it is time to delete all the resources to avoid accruing unwanted charges to our account:

> npx cdk destroy --all

EC2 Instance Connect

AWS Instance Connect provides a more streamlined approach to SSH access management. With Instance Connect, you don't need to manage SSH keys at all or configure the SSH daemon yourself. Instead you can use the AWS Management Console, AWS CLI, or SDKs to initiate a one-time use SSH connection to an instance using the Instance Connect client installed on the server. The client comes already installed on AWS AMI’s based on

• Amazon Linux 2 2.0.20190618 or later

• Ubuntu 20.04 or later

How it works

This method relies on you creating an ephemeral 60 seconds short-lived SSH key and pushing it to the EC2 Metadata Service.

As the Instance Connect client is registered in the SSH daemon config file located in /etc/ssh/sshd_config for the:

• AuthorizedKeysCommand

• AuthorizedKeysCommandUser

hooks, when you initiate a SSH connection it will leverage the EC2 Metadata service to verify your ephemeral key.

To achieve this simplified secure connection you can either use:

• mssh

  • This is the AWS provided Instance Connect CLI utility that automatically generates the SSH key and publishes it to the EC2 Metadata service to be used for the next 60 seconds abstracting away the entire process from you. Install process can be found here.

• aws cli

  • Use the standard aws cli to perform the connection. This is a two step process as you first need to create a key to be pushed. Herein lies part of the abstraction performed by the mssh option above.

How to do it

In order to limit who can connect to a specific instance, here is an example IAM policy that would grant access to exampleuser to a specific instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2-instance-connect:SendSSHPublicKey",
            "Resource": "arn:aws:ec2:eu-west-1:111122223333:instance/i-0123456789abcdefg",
            "Condition": {
                "StringEquals": {
                    "ec2:osuser": "exampleuser"
                }
            }
        }
    ]
}

In order to connect, you can either use

mssh <instance-id>

or

ssh-keygen -t rsa -f ephemeral_key

aws ec2-instance-connect send-ssh-public-key \
    --region <region> \
    --availability-zone <az> \
    --instance-id <instance-id> \
    --instance-os-user <ec2-user/ubuntu> \
    --ssh-public-key file://ephemeral_key.pub

ssh -o "IdentitiesOnly=yes" -i ephemeral_key <user>@<instance-dns>

Pros:

• Removes need for a Key Management infrastructure and run-book process for creating, distributing, maintaining, and rotating.

• User does not need to download key to their local machine and removes risk of it being extracted if user machine is compromised.

• Granular RBAC to instances is pushed into the already existing IAM role distribution policy of the company. To provide access you need to grant

  • ec2-instance-connect:SendSSHPublicKey

  • ec2:DescribeInstances

• Generates audit trail in CloudTrail for each key push performed

• You do not need to contact somebody to see which key is compatible, you generate it on the spot

Cons:

• Requires additional software package to be installed on servers unless running AMI which already contains it. This may entail generating a new AMI golden image through Packer or whichever may be the company’s standard golden image generating flow.

• Users need to be educated on this new process flow ( should be straightforward though).

• Vendor specific access flow

Example

In order to get a basic EC2 configured for access, you can follow the example code from the EC2 Key Pairs section above, with the following to the ec2-stack.ts file so we do not have any SSH key loaded by AWS into the instance at boot time:

Once you have the EC2 instance running, you can use the commands in the ‘How to do it’ section above to connect, conveniently outputted by the deploy command:

Note

This assumes you are using a recent, compatible AMI that has the EC2 Instance Connect agent already installed. For images that do not contain it, you can find the installation steps here.

Conclusion

Ultimately, the decision between EC2 key pairs and AWS Instance Connect will depend on your specific use case and requirements. It is up to you to decide what best fits your organization’s needs but if your conditions allow, the standardized RBAC and greatly simplified key maintenance and connection process can help in keeping a team nimble and productive.