Network Design: Developer Network

ACM.72 Network architecture for developer access to GitHub and AWS

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Part of my series on Automating Cybersecurity Metrics. The Code.

🔒 Related Stories: Network Security | GitHub Security

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the last post we covered some networking defaults in AWS.

If you’ve been following along you know that I’ve been setting up components to run batch jobs on AWS, but not only run them — we want to think through how we will securely deploy and operate them. That entails a lot more than simply writing some code, sticking it in a container in a registry and running it in AWS Batch.

I’m to the point where I need to stop and think about the network architecture a bit more. Mind you this scenario might need to be altered for use in a large organization with additional network constructs but it should be pretty easy to modify.

Diagrams at the end of the post and added to the overall architecture post I’m updating as I go if you want to skip the reasoning below and get to what I think I will implement, if I have time. Subject to change, as new discoveries come to light.

Connecting to AWS

First I need to think about how I am connecting from my remote location to AWS. There are a few services I can consider using or my particular use case and architecture.

Site to Site VPN

I’m connecting to my AWS account remotely develop and kick off batch jobs. I also manage my AWS account. I’m connecting from a remote office. I don’t have a Direct Connect because the traffic is not coming from a data center and it’s not a large downtown office building. We don’t really have those in Savannah and I hope it stays that way. :-)

I’ll probably want to use a site to site VPN to encrypt traffic heading for AWS if I want a fully encrypted tunnel that protects ALL traffic between the two networks. My local device needs to authenticate to connect the VPN.

I can use a product like those from Netgate that include PFSense and an AWS module (and I’m not selling or recommending that product I just know it has that capability and is easy to use).

Note that the above is not to be confused with pfSense plus for Amazon AWS which is a version of pfSense that runs on AWS. You would use this if you wanted to deploy your own firewall in the cloud to use within AWS or deploy it as a want to manage client VPN connections to your AWS network, for example:

I’ll need to be mindful of the cost of using a site-to-site VPN. AWS has some pricing scenarios here. It’s going to be a minimum of $36 per month for one VPN connection and double that if you want failover. Then you’ll have some data transfer costs.

Note that with this option, anything on my local network could potentially get to my AWS account over that VPN tunnel. If my iRobot goes rogue and gets a mind of it’s own it might try to hop onto that connection and get to my AWS account? Of course, it would need to be able to bypass any other security controls I have in place such as authentication. Since Amazon recently acquired iRobot hopefully that’s not going to happen as they have appropriate security in place to prevent it, but I have a myriad of IoT devices that all want to get on my network. It seems like you can’t even buy things that don’t have a network connection these days.

Local Network Security

To prevent rogue devices from connecting to my AWS Network I can also set up segregation on my local network. This is another one of my blog posts series in progress which I hope to get back to sooner than later:

Client VPN

Next up I could use the AWS Client VPN. The client VPN allows me to install software on my laptop that I then use to connect to the AWS Client VPN endpoint on AWS.

The AWS Client VPN uses SSL/TLS to communicate with AWS resources. Traffic will be authenticated and encrypted.

So what happens if you are send data via UDP? AWS supports that now:

You’ll need to test other protocols if you need them to see if you can connect and if they are supported, otherwise you might not be able to connect at all or potentially worse — your traffic might bypass the VPN and traverse the network unauthenticated and unencrypted. You always need to know what is and is not going through or bypassing your network security controls.

You can find the client software that you would install on a local laptop here:

As always we want to be aware of the cost. You could consider the cost of this service versus setting up your own VPN on an EC2 instance and having your users connect to that instead. Of course, you have to manage the availability and performance of your client VPN in that case vs. having AWS manage it for you.

I’ll probably save the client VPN configuration for when I get back to my networking series but be aware of this option. If we use it then we can set up networking rules to only allow access to certain resources and networks from our client VPN.

I probably won’t cover the client VPN in this series but it’s something to think about in the overall network architecture.

Bastion Host and Developer VMs in the cloud

I always do my development on a development machine in the cloud. That way I only have to install minimal tools on my local laptop. I can easily rebuild and upgrade all the tools on my developer VM. This method only requires that I open up either port 22 or port 3389 to the cloud depending on which protocol I’m using to connect and in my case, I can restrict it to my own IP address. I’m going to explore that concept in a few posts, along with a closer look at VPNs eventually if I have time.

A true bastion host would be a very locked down and restricted host that would disallow potentially installing malware on it that could get into the rest of your network. You would first connect to the bastion host. Then you would connect to the rest of the network from that point. A developer machine is a bit different because likely developers will likely be logging in and installing some software on the host they are using for development. In a production environment I would (and did) prevent that on a true bastion host.

It is important to really lock down (harden) a bastion host so that any attackers that try to access it cannot, and even if they do somehow they cannot install malware on the host that can escalate privileges, scanners, or act as a proxy to reach other parts of the network to which the bastion host has access. The security team will also have very sensitive monitoring on the bastion host to get alerts in the event it may have been compromised.

Deployment Network

For the moment, deployments are carried out from the developer host via CloudFormation scripts. That will change later as you’ll see, but for now we have to start with what we have.

AWS CloudFormation supports Private Link.

Although the developer machine is not on the Internet, ideally our requests to AWS CloudFormation will stay on a private network. We can see if we can make that happen. We may need to set up VPC endpoints for some other AWS services as well.

Github Connections

I remember when I did have to do that when I colocated servers at a data center. I hated it. I said to myself, “If someone could just host all this stuff for me so I could focus on programming that would be great.” That was long before AWS or GitHub existed. Those services are my wish come true.

Ideally, I like to host internal repositories when I can but I’m torn on hosting all my code in the same cloud where I’m deploying it. Additionally, I don’t really personally have time to manage a source control system. The decision for a large company hosting sensitive data such as credit card who has the staff and resources to manage an internal source control repository probably should be and will be different.

So how can we connect GitHub and AWS in my scenario? It would be nice if GitHub had an option to support AWS Private Link but I don’t see one. There are a few things we can do, however.

One is that we can connect with SSH which encrypts requests to GitHub in transit. The one thing we need to consider is where we store the SSH key. We probably don’t want that sitting around on an EC2 instance.

We can also lock down GitHub to only allow access from specific IP addresses with an Enterprise account.

If you want to restrict access from a particular network to GitHub we can get a list of GitHub IP addresses.

You can find other GitHub security option here as this is not an exhaustive list but it provides a few network-related security controls we can use to restrict access to our GitHub repository.

Fixed IP Addresses (Elastic IP Address) for firewall rules

One of the interesting things about working in cloud environments is that IP addresses change frequently. When you stop and start an EC2 instance, the public IP will likely change. If you are basing your firewall rules on those public IP addresses, you are going to have problems.

Luckily if you are using private subnets you can base rules on your private IP ranges assigned to the subnets within AWS. But if we are trying to restrict network connections to our GitHub account? It can’t see the private IPs. That’s not going to help us.

In order to connect to GitHub from a fixed IP address I can install an Elastic IP address to make those connections. To be safe I will probably want to create some backup IP addresses in a separate account or region as well so I don’t get locked out of my GitHub account. I’ll probably want to make sure that those backup IP addresses cannot be accessed by the credentials I use on a day to day basis to manage my AWS account and work on developing resources.

I can assign my fixed IP address to a specific host. A larger organization might reserve a range of contiguous IP addresses instead of one at a time. It looks like that is possible with a request to AWS support:

Why is that range important? If your EIPs are not contiguous you would need to create a separate IP address for each one. If they are contiguous you can hopefully reference one CIDR block instead (but that depends based on the start an end IP address and some math which I’m not getting into here.)

Organizations may also be able to bring their own IP addresses to AWS:

Then you can limit network rules to your own IP range.

Network Diagrams for our Developer Network

For now it’s just me in this network so I can set up something like the following with an EIP associated with my developer VMs. We’ll need to access the developer VMs from the remote office and make outbound connections to GitHub (and probably others but we’ll start with GitHub)

But EIPs cost money and are limited, so eventually a larger organization will likely want to set up a Client VPN (not to be confused with the Site to Site VPN in the digram above) associated with a specific CIDR block and have developers connect to that. Then lock down Github to that Client VPN IP range.

When I wrote this post about why you might want a VPN, it was not the only reason you might want a VPN. Also after this point Google started trying to make everyone use DNS over HTTPS which can cause more security problems than it solves. But that is a topic for another day.

The primary reason I first used a VPN when I ran my own e-commerce company was to create a private network I could access from anywhere — like from a Starbucks or the side of the road when I driving from San Diego to Seattle and I got an alert that there was a problem with one of the sites I was hosting. Ah…those were the days. I set that up after my first data breach which helped me understand the importance of network security.

Having remote workers first connect to a VPN before accessing network resources can simplify your network design. You can also carefully lock down and monitor your VPN since it is the entry point to the network. Do not depend on a VPN alone, since there are recently been many vulnerabilities in VPN products. It is part of a defense in depth strategy.

As mentioned there’s no option to use GitHub with AWS PrivateLink at the time of this writing. It looks like GitLab is exploring an option that offers AWS Private Link, which would be something to check out later potentially.

Organizations may also opt to deploy source control systems locally within their own network if they have the resources to securely support it.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab