AWS Networking
How To Build A Highly Available AWS Direct Connect Architecture
A reference HA architecture with AWS networking services and components, without single points of failure, and other considerations
In this article, we’ll look into how we can build a highly available (HA) AWS Networking solution connecting AWS and On-Premise data center using Direct Connect. We’ll also look into how other AWS services like Direct Connect Gateway and Transit Gateway can help simplify the overall architecture and its management.
Although I am assuming that you have a fair understanding of the AWS networking services we’ll do a quick and dirty recap before we look into the architecture.
A Quick Detour -
It is important to have an understanding of AWS accounts and how various account structures can help to build a scalable solution. We’ll touch upon a few aspects as part of this article. But, I would refrain from providing detailed explanations and would like to keep it out of the scope of this discussion. I highly recommend exploring in detail the importance of various strategies of multiple AWS account structures and how it can help in building a secure, scalable, and cost-efficient AWS foundation.
So, Here Are The Simplified Definitions Of The AWS Networking Services Relevant To This Article —
Amazon VPC
Amazon Virtual Private Cloud (VPC) is a fundamental building block of AWS networking. It lets you create a private network within AWS to provision resources. VPCs can have public and/or private subnets.
VPCs and Subnets are associated with various networking services and components like Route Tables, Network ACLs, NAT Gateways, Internet Gateways, Security Groups.
AWS Direct Connect (DX)
Direct Connect enables establishing a dedicated private network connection from on-premises to the AWS cloud. It can provide more reliable network connectivity than the public internet, reduce network costs, and increase bandwidth throughput as well.
AWS Site-to-Site VPN
In this context, a VPN connection would be between the on-premises network and AWS VPC. Here, we would be looking into VPN as a failover for Direct Connect instead of the primary connection.
AWS Direct Connect Gateway (DXGW)
DXGW is a grouping of Virtual Private Gateways (VGW) and Private Virtual Interfaces (Private VIF) that belong to the same AWS account. It allows connecting multiple Transit Gateways (via Transit Virtual Interface) or VPCs (via VGWs) in the same or different regions to a Direct Connect connection (via Private VIF).
Basically, it simplifies the architecture by allowing a single DX location to connect to multiple AWS regions. So, it reduces management complexity as well.
Explore https://docs.aws.amazon.com/directconnect/latest/UserGuide/WorkingWithVirtualInterfaces.html and https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-gateways.html to learn more on Virtual Interfaces and Gateways.
AWS Transit Gateway (TGW)
TGW enables us to interconnect multiple VPCs, or connect to a DXGW or VPN. TGW simplifies VPC to VPC interconnectivity by eliminating the need for VPC peering and providing a central management hub for connectivity policies. This hub-spoke connectivity model makes it easy to scale out the network.
A High Level Reference Architecture
I have considered the below as a reference:
2 on-premise data centers which will be connected to AWS cloud.
2 AWS accounts — 1 business account (Account A). And, a Network Account hosting the networking services.
It is not necessary to set up the accounts this way, but going by the multi-account strategy I have kept the Direct Connect related networking resources in a dedicated account for scalability and operational efficiency reasons. There are multiple approaches to build highly available, reliable, and redundant connectivity between AWS and on-premise data centers.
Let us look at the different sections of the architecture to understand the purpose of each of the components and services.
To simplify, let us assume that the existing data centers are already connected and that section is not relevant in this context of the Direct Connect connectivity to AWS.
1. Primary Data Center To AWS Connectivity
The primary data center is connected via a dual-pipe Direct Connect (DX) connection to a Direct Connect Gateway (DXGW) on AWS in the Network account. A dual connection is recommended for higher resiliency with connections terminating on separate devices. This would provide resiliency for device failures. For cross-location resiliency, it is recommended to have dual pipes across locations as well. We would look into it more in the later sections.
Set up the connection via AWS Direct Connect Gateway to simplify the Direct Connect connectivity setup, which enables easier configuration across AWS Transit Gateways and VPCs.
2. Primary Connection via Direct Connect Gateway
DXGW attached to the primary DX connection can, in turn, be attached to 3 TGWs (3 is the AWS hard limit at the time of writing) across AWS regions.
In this case, 2 TGWs (1 Prod and 1 Non-Prod) are in the same region, whereas the 3rd TGW (Prod) is in another region. This can ensure that Prod traffic still has a route to the applications hosted in another region. This can be critical for applications requiring near ~100% uptime. Yes, I said ‘near ~100%’ because achieving a 100% uptime can not only be highly challenging but may not be cost-efficient or practically required as well.
3. Cross Region Transit Gateway Peering
Configure AWS Transit Gateways to simplify VPC connectivity and traffic routing across VPCs and Direct Connect Gateways. Transit Gateways can also be peered across regions. This enables to extend the connectivity across multiple AWS regions and build a global network. This traffic is secure as it stays within the AWS global network infrastructure and is encrypted as well.
Together, DXGW and TGW can simplify the connectivity configuration between on-prem and AWS (across accounts and regions).
Note that at the time of writing, AWS TGW peering attachments support static routing only. Carefully consider all the limits and other factors specific to your AWS foundation, security, and routing requirements.
4. Prod vs Non-Prod TGW connectivity
Separate Transit Gateways can be used to connect to separate Prod and Non-Prod VPCs across AWS accounts.
It is not necessary to have multiple TGWs across environments like Prod vs Non-Prod. I have kept it that way to show an approach where it can help in traffic monitoring across prod vs non-prod environments, or the ability to simulate routing changes in non-prod environments before implementing similar changes in production.
5. Secondary Data Center To AWS Connectivity
Depending on the Direct Connect HA requirements, secondary data centers can be connected to alternate direct connect locations. In this case, the second Direct Connect is connected to us-east-2.
Note that this secondary connection can be optional (gray section) in case AWS connections are primarily required from the primary data center only. The secondary data center should still be able to reach AWS via the existing link connecting the data centers.
This secondary Direct Connect can increase the resiliency in case of location failures. A dual-pipe secondary DX like the primary connection can provide the highest level of resiliency across location and device failures. But, this comes at a higher cost which needs to be considered as well.
6. Secondary Connection Direct Connect Gateway
Even the secondary DX should be connected via DXGW, similar to the primary connection for operational efficiency reasons.
This DXGW should be associated with the TGWs in the connected region. Region us-east-2 and the Prod and Non-Prod TGWs in this case. This allows for Prod traffic to reach us-east-1 in case of us-east-2 regional failure.
7. Site-to-Site VPN Failover
In case a single location Direct Connect meets the HA connectivity requirements, backup site-to-site VPN should be set up to meet the failover requirements.
Having a failover VPN can be a less expensive solution compared to the multiple DX connections, but it comes at the cost of lower network reliability and bandwidth throughput. If consistent network performance is required even during failover periods then it would be better to opt for multiple DX connections to meet the requirements.
Other Considerations
When designing any critical AWS networking solution various factors need to be carefully considered before embarking on it.
Here are some basic areas to start with —
- Network bandwidth throughput requirements.
- Highest level of RTO/RPO requirements for applications that would be hosted. This can be a deciding factor in the overall design due to cost vs availability factors.
- Network segmentation.
- Access control security — Users vs. AWS vs. Network vs. Security engineers etc.
- Data-in-transit security.
- Operational efficiency and management overhead.
- Monitoring and Auditing requirements.
- Future network expansion plans.
- AWS service soft and hard limits for each of the services that you are planning to use.
Conclusion
I hope this article gave you some directions on building a highly available AWS Direct Connect architecture. Using DXGW and TGW helps simplify the architecture while keeping the costs and operational overhead low.
This particular reference architecture is designed to achieve maximum resiliency; simpler architectures can be designed which are good enough to meet lower availability requirements at a lower cost. For instance, as mentioned in the above sections, using a VPN as a failover route instead of multiple Direct Connect connections.
Important References
Follow me on LinkedIn, Twitter.
Subscribe on Medium for more such posts.
