Configuring SSH to Protect Against the Terrapin Attack on AWS EC2 Amazon Linux Instances

Patch OpenSSH — AND — Disallow insecure connections by removing them from your configuration

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🔒 Related Stories: Encryption | Pentesting | Cyber Attacks

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A recent vulnerability in SSH means that many servers allowing users to connect using certain encryption algorithms are vulnerable to attack.

TL;DR 

Edit this file:
sudo vi /etc/crypto-policies/back-ends/opensshserver.config

If you want to know how I figured that out, read on.

On other instances, the Cipher node is missing in the configuration.

Ciphers [email protected]

Consider your options (GCM or CTR + HMAC) and create a backup before making changes as explained below.

By the way — there were two vulnerabilities announced in SSH recently. There’s a separate SSH issue I covered in this post which has to do with the keys you use to authenticate to SSH:

I may provide an update for the above later as well.

About the Terrapin Attack

I first read about the Terrapin Attack here:

And see that many people still have not updated their configuration to protect against this attack:

How does it work?

Here’s what happens at a high level. A person wants to connect with SSH so they run a command to connect to the server.

The client machine and the server machine perform what is called a “handshake” to determine which encryption algorithms can be used to encrypt the communications between the two hosts.

The attackers manipulate the sequence numbers to reduce the integrity of the connection. Sequence numbers are part of networking protocols and I wrote more about those in my other posts. I didn’t really get into the nitty gritty of sequence numbers yet — if I ever get around to it — but you get an introduction to protocols here and how they work and then you can go dig into the TCP spec to learn more about sequence numbers if you want.

For this post, I am concerned with preventing the problem more than getting into exactly how it works as that is a distraction from securing your servers at this point.

The above article says this attack is relevant:

particularly when specific encryption modes like ChaCha20-Poly1305 or CBC with Encrypt-then-MAC are used.

• ChaCha20-Poly1305
• CBC with Encrypt-Then-Mac

I’m not sure if “particularly” means other versions are still insecure as well but less so.

Patching OpenSSH — may help to a point

There’s one thing you can do to try to defend against this attack and that would be to update every instance of OpenSSH in your environment. There’s a patch out for this attack int the latest version as described here:

This note is curious:

OpenSSH 9.6 addresses this protocol weakness through a new “strict KEX” protocol extension that will be automatically enabled when both the client and server support it. This extension makes two changes to the SSH transport protocol to improve the integrity of the initial key exchange.

That means both your client and server need to be updated. Currently my Mac with all the latest Apple updates shows an outdated version of OpenSSH. I can see that when I connect to a host using verbose mode as follows:

ssh -vvv -i localhost

Note that I just stuck localhost in there to get some output. I’m not running an SSH server on my laptop!

I can also see where my local configuration file exists. But in this post, I’m focusing on the server side of the SSH configuration.

Although you can fix the problem in OpenSSH with a patch and the update downplays the dangers of this attack, I’m not feeling too sure about those comments. Here’s a description of the attack:

The attack can be performed in practice, allowing an attacker to downgrade the connection’s security by truncating the extension negotiation message (RFC8308) from the transcript. The truncation can lead to using less secure client authentication algorithms and deactivating specific countermeasures against keystroke timing attacks in OpenSSH 9.5.

So we’re not only worried about timing. We’re worried about downgrading to a less secure algorithm that can potentially then allow an attacker to obtain a MITM attack and read the information in transit as it traverses the Internet.

Reading on…

For example, we found several weaknesses in the AsyncSSH servers’ state machine, allowing an attacker to sign a victim’s client into another account without the victim noticing. Hence, it will enable strong phishing attacks and may grant the attacker Man-in-the-Middle (MitM) capabilities within the encrypted session.

That doesn’t sound like the description above which simply says, “oh some packets will just be missing.”

Removing encryption ciphers that could lead to a downgrade attack

Perhaps I am misunderstanding something, but just to be sure my connections cannot be downgraded, I can remove the ability to connect with insecure algorithms altogether from my server. That also helps prevent insecure clients from having their connections downgraded (though you should still update the client software in case they connect to some other server besides yours.)

Here’s where the fun begins. I haven’t looked at a Linux SSH configuration in a bit and forgot some things. Also — I wanted to verify exactly what was running on my server and that my changes took effect, so I took some additional steps that you won’t have to do to sort this all out. You can skip to the end to get the steps to fix the problem if you want. But here are some caveats to make sure you are only running what you think you are running and you can verify you are changing the correct files.

Checking which ciphers are available

I started with my AWS EC2 Amazon Linux instance where I have SSH configured and nmap installed.

sudo yum install nmap

I ran this command:

sudo nmap --script ssh2-enum-algos localhost

I got back the following on this particular host:

What are these things? Well we can get a clue from this documentation even though it says “legacy”:

• KexAlgorithms: the key exchange methods that are used to generate per-connection keys
• HostkeyAlgorithms: the public key algorithms accepted for an SSH server to authenticate itself to an SSH client
• Ciphers: the ciphers to encrypt the connection
• MACs: the message authentication codes used to detect traffic modification

Honestly, I never did find good information on all this configuration — I basically had to reverse engineer it. That Ciphers description is relevant but on AWS Amazon Linux we’ll have to dig around a bit to find it. At least on this particular machine I’m using which may have an outdated configuration. More on that in future posts.

Note that the chacha-poly1305 algorithm mentioned in the vulnerability above is in the list of algorithms configured on this particular SSH server.

In addition to using nmap I later discovered this command for checking the ciphers on the server (I threw in “localhost” just to force a response):

ssh -T localhost

The other thing you can do is check what algorithm is actually agreed upon in the handshake between the client and server by using the verbose option when you make a connection like this:

ssh -vvv -i <your key> <user>@<yourhost>

Doing that I can still see the available ciphers including the one I don’t want to allow:

I can also see that dancing algorithm is the one actually being used:

I say dancing 💃🕺algorithm because I can’t help but think of my ballroom and salsa dancing lessons every time I read chacha 😆. Sorry.

What SSH server is my system running?

Well, out of curiosity, I took a few steps to see what is running SSH on my system. These steps are probably completely unnecessary but if you want to verify exactly what is running as I like to do, you can also do the following.

If I run this command:

sudo lsof -i

I can see the following pid (2243) is running ssh.

sshd       2243            root    5u  IPv4  17072      0t0  TCP *:ssh (LISTEN)
sshd       2243            root    7u  IPv6  17074      0t0  TCP *:ssh (LISTEN)

If I run this command I can see what program is running that service:

sudo ps -aux | grep 2243

/usr/sbin/sshd

root        2243  0.0  0.2  29120  8552 ?        Ss   Jan05   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Note that initially I ran the man command for ssh instead of sshd. It is somewhat confusing that both these instances still exist on this server. Perhaps they are removed in a later iteration but just remember to look at ssh with a d at the end not ssh if that’s what your system is running.

But what is interesting is that by doing so I discovered a few things.

I used the man (manual) ssh command to get the help for the ssh service.

man ssh

If I peruse the documentation I can see that there’s information about Ciphers here which seem to be related to our encryption algorithms in the list above:

 -c cipher_spec
      Selects the cipher specification for encrypting the session.  cipher_spec is a comma-separated list of ciphers listed in order
      of preference.  See the Ciphers keyword in ssh_config(5) for more information.

Unfortunately this man information doesn’t tell you what the default ciphers are, but I looked at a different flavor of Linux and the defaults included the unwanted ChaCha20-Poly1305 cipher per some OpenSSH information on Ubuntu. It appears that AWS may be changing defaults for various crypto algorithms in Amazon Linux 2023 so your best option is to verify using the above commands what is actually running and available.

The above documentation says the configuration file for ssh is here:

/etc/ssh/ssh_config

But remember that we are interested in sshd.

If I run the man command for sshd it tells me that the configuration file is at /etc/ssh/sshd_config by default and can be overridden with -f.

It’s a good idea to verify what is running on your system. If someone ran SSH with -f you might be updating the wrong configuration file if the default is not in use.

I can see in the command above that ran my SSH server that the sshd service is running with the -D option but not the -f option so the default configuration file should be in use.

Now if you look at the configuration file for the ssh (no d) service you’ll see this line for Ciphers is commented out. You’d think that might be in the sshd service as well but it’s not.

#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc

Also notice that the whole Host section is commented out.

If you scroll down you will notice the following comment and configuration:

Well, I initially made a mistake and went down a rabbit hole looking at those files. But keep in mind that the configuration file may include other configuration files. So if the same setting is in both configuration files, which one takes precedence? That will be important to understand. It depends which file is read first and whether the software that reads it uses the first or last value it reads.

Let’s skip over to the sshd configuration file since that’s the one we care about.

sudo cat /etc/ssh/sshd_config

You’ll find a similar included at the top of that file.

# To modify the system-wide sshd configuration, create a  *.conf  file under
#  /etc/ssh/sshd_config.d/  which will be automatically included below
Include /etc/ssh/sshd_config.d/*.conf

So what I did was jump through a series of hoops editing different files until I figured out which one I actually needed to modify to get the results I wanted.

Create a backup image of your EC2 instance

Before you make these changes you might want to make an image of your EC2 instance in case something goes wrong.

Check the box next to your EC2 instance in the list and choose these options:

Actions > Image and templates > Create image

If something goes wrong you can deploy an image using your backup Amazon Machine Image (AMI):

Mind the cost when you start creating backups here due to the images and EBS volumes. But this is a good way to make sure you can get back to your image in the prior state should something go wrong.

Wait for the pending image creation to complete before proceeding:

Note that when you refresh this page it reorders the AMIs for some odd reason so your pending item may not be at the top of the list anymore.

OK once that is complete, reconnect to your linux host and return to the configuration files above and make some modifications.

Which encryption algorithms should we allow?

Recall that our server allows clients to connect with these encryption algorithms for encryption of data in transit:

Well, I know I want to get rid of the cha-cha-cha algorithm. 💃

But what’s the difference between CTR and GCM and which one is recommended?

As noted before I tend to rely on NIST for encryption recommendations.

Well, the documentation from NIST is not exactly crystal clear.

It appears this document covers block cipher modes.

The latest version of this document was written in 2016:

Well, OK, that’s like 8 years ago.

Here are some more recent posts on block cipher modes:

The latest one proposes to change the guidance for GCM which is used in the latest version of TLS.

There’s a lot of details in just how these modes work and construct the necessary bits that secure your encryption that I’m not getting into here — like initialization vectors (IVs) that are commonly a source of attack.

But let’s consider that since GCM is the latest one used in TLS 1.3 and because it uses authentication as well as encryption. So I am presuming that’s the one we want since it’s in TLS 1.3 and it’s supposed to help with maintaining the integrity of the encrypted connection.

But how do we know that our implementation on our host abides by the above guidance? GCM adds authentication to encryption but how do we know we’re leveraging encryption even if we set that mode. Is it implicit in how it works or something else that needs to be correctly configured? So many questions arise if you really understand how this all works.

Well we don’t know all that at this point unless we test it out and inspect it but I’ll save that for a later post. But let’s see if we can change our configuration to limit the use of the GCM algorithm. I did see a post that said CTR with HMAC may be more widely supported, so that’s why I told you to make an image of your instance first — in case there’s something that needs to connect that can only use CTR with HMAC.

Figuring out which configuration file to modify and how

One thing I did was look in the cloud-init.log to figure out what it was doing on startup and which files it was reading.

/var/log/cloud-init.log

I ran this command:

cat cloud-init.log | grep ssh

I can see the system is reading this file in those logs (notice the d at the end of ssh):

That’s the file the sshd man info above says is the default configuration file for sshd so that’s a good sign.

What’s in that file?

sudo cat /etc/ssh/sshd_config

This file says it is the system wide file:

OK let’s look at the man command for sshd and try to figure out the changes to make to only allow the ciphers we want.

man sshd

Not sure if it’s just me but I’m not immediately seeing the cipher configuration. Let’s just try to edit the file.

sudo vi /etc/ssh/sshd_config

Well, I don’t see any ciphers in that file either.

Let’s see if we can find the changes we need to make in the AWS documentation.

Well my first attempt led to this page:

That’s interesting. There’s some kind of policy on this page but where is it exactly on my host?

Oh there’s that Q icon. I wrote about Q here:

Let’s try it:

Locate the Ciphers parameter. Mmm hhhmmmm. I didn’t see that parameter as explained above.

Well let’s just make sure. I’ll cat and grep for it:

sudo cat /etc/ssh/sshd_config | grep -i ciphers

I did miss something but it’s only a comment:

Is that answer referring to outdated documentation? That is another thing that AI will have a problem distinguishing between since AWS has changed their operating system and defaults over time.

If I search for Amazon Linux 2023 specifically it wants me to enable legacy settings and is recommending an insecure crypto configuration. Clearly this is not helping.

Well the above documentation with policies seems to be related to access to storage services using the AWS transfer family and is not specific to AWS EC2. Or is it? I can’t really tell.

Most of the time I just figure things out by reverse-engineering them.

So here’s what I’m going to do. I’m going to insert the missing configuration key.

Then I restart my linux server.

sudo service sshd restart

That doesn’t fix it, nor does with a reboot.

I also tried capitalizing the C in ciphers but that didn’t work either.

Let’s take a look at the included configuration file and see if we can edit that:

sudo vi /etc/crypto-policies/back-ends/opensshserver.config

I remove all the ciphers on the first line except this one:

I re-run my nmap command. Finally!:

(I left a bunch of things out that I did that were essentially going down the wrong path.)

Now let’s see if I can still connect. Exit the existing connection:

exit

Reconnect with the verbose command. Yes, I can.

I am now seeing the correct connection type when I look at the details of the connection using the verbose ssh connection command I used above:

Now here’s another interesting tidbit. Remember that -T command above? I ran it on the server and got this:

So apparently the SSH client on my particular AWS EC2 Amazon Linux instance (not my Mac laptop) doesn’t support that SSH mode I’m specifying above whereas my Mac laptop does. Intriguing…but something to explore further on another day because I already spent far too long on this.

Now although adding the “Ciphers” node in the config file didn’t work on this particular server, it actually did on another AWS EC2 instance where the sshd file had no includes in it and no Cipher node.

I edited this file:

sudo vi /etc/ssh/sshd_config

I added this node:

Ciphers aes256-gcm@openssh.com

I restarted the SSH server:

sudo service sshd restart

I verified that my client connected with the expected cipher.

The other weird thing I noticed while testing that last server is that if you remove the line for EC2 connect, the SSH server won’t restart. Huh? What if you don’t want to allow that option for some reason? AWS should fix that.

Summary of steps to take to protect against this attack

In the case of my particular AWS EC2 instance, I had to figure out what the configuration file was for my currently running SSH server.

From there I figured out that the following include file controls which encryption algorithms are offered by once server.

sudo vi /etc/crypto-policies/back-ends/opensshserver.config

On another I simply added the missing Cipher directive to the configuration file:

#edit sudo vi /etc/ssh/sshd_config
#add this line:

Ciphers aes256-gcm@openssh.com

I modified the appropriate file, restarted my SSH server, and then tested my configuration using nmap, adding -T into the ssh command on the server, and the verbose option (-vvv) in my ssh connection on the client as detailed above. I was able to verify that the server was only offering the desired encryption ciphers after making that change.

A few other things you can also do:

You could also take similar steps to those shown above to only allow specific algorithms on the client side as well.

Update your client and server software as mentioned.

Your operating system provider should update your system to use the latest version of OpenSSH if it provides that as part of the operating system.

Hopefully on Amazon Linux this will do it:

sudo yum update

If you can’t automatically get the latest update from your operating system update method, get the latest OpenSSH version as explained here:

Also, we could spend a lot more time digging into the nitty gritty aspects of the algorithm we are using and how it’s actually working which is something I can do on penetration tests when I’m not bogged down with all the simple things like cross-site scripting and basic misconfigurations — and if the SSH connection is exposed to the Internet and my test servers. 😁 If it’s not, there’s less chance of a man-in-the-middle attack.

Happy SSH-ing!

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

The best way to support this blog is to sign up for the email list and clap for stories you like. If you are interested in IANS Decision Support services so you can schedule security consulting calls with myself and other IANS faculty, please reach out on LinkedIn via the link below. Thank you!

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
Author: Cybersecurity for Executives in the Age of Cloud
Presentations: Presentations by Teri Radichel
Recognition: SANS Difference Makers Award, AWS Security Hero, IANS Faculty
Certifications: SANS
Education: BA Business, Master of Software Engineering, Master of Infosec
Company: Cloud Penetration Tests, Assessments, Training ~ 2nd Sight Lab

Like this story? Use the options below to help me write more!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
❤️ Clap
❤️ Referrals
❤️ Medium: Teri Radichel
❤️ Email List: Teri Radichel
❤️ Twitter: @teriradichel
❤️ Mastodon: @[email protected]
❤️ Facebook: 2nd Sight Lab
❤️ Threads: @teriradichel
❤️ Bluesky: @teriradichel
❤️ YouTube: @2ndsightlab
❤️ Buy a Book: Teri Radichel on Amazon
❤️ Request a penetration test, assessment, or training
 via LinkedIn: Teri Radichel 
❤️ Schedule a consulting call with me through IANS Research

My Cybersecurity Book: Cybersecurity for Executives in the Age of Cloud