Configuring SSH to Protect Against the Terrapin Attack on AWS EC2 Amazon Linux Instances
Patch OpenSSH — AND — Disallow insecure connections by removing them from your configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Related Stories: Encryption | Pentesting | Cyber Attacks
💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A recent vulnerability in SSH means that many servers allowing users to connect using certain encryption algorithms are vulnerable to attack.
TL;DR
Edit this file:
sudo vi /etc/crypto-policies/back-ends/opensshserver.config
If you want to know how I figured that out, read on.
On other instances, the Cipher node is missing in the configuration.
Ciphers [email protected]
Consider your options (GCM or CTR + HMAC) and create a backup before making changes as explained below.
By the way — there were two vulnerabilities announced in SSH recently. There’s a separate SSH issue I covered in this post which has to do with the keys you use to authenticate to SSH:
I may provide an update for the above later as well.
About the Terrapin Attack
I first read about the Terrapin Attack here:
And see that many people still have not updated their configuration to protect against this attack:
How does it work?
Here’s what happens at a high level. A person wants to connect with SSH so they run a command to connect to the server.
The client machine and the server machine perform what is called a “handshake” to determine which encryption algorithms can be used to encrypt the communications between the two hosts.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*-ut4nc755wyxNpgiCw6_Mw.png)
The attackers manipulate the sequence numbers to reduce the integrity of the connection. Sequence numbers are part of networking protocols and I wrote more about those in my other posts. I didn’t really get into the nitty gritty of sequence numbers yet — if I ever get around to it — but you get an introduction to protocols here and how they work and then you can go dig into the TCP spec to learn more about sequence numbers if you want.
For this post, I am concerned with preventing the problem more than getting into exactly how it works as that is a distraction from securing your servers at this point.
The above article says this attack is relevant:
particularly when specific encryption modes like ChaCha20-Poly1305 or CBC with Encrypt-then-MAC are used.
- ChaCha20-Poly1305
- CBC with Encrypt-Then-Mac
I’m not sure if “particularly” means other versions are still insecure as well but less so.
Patching OpenSSH — may help to a point
There’s one thing you can do to try to defend against this attack and that would be to update every instance of OpenSSH in your environment. There’s a patch out for this attack int the latest version as described here:
This note is curious:
OpenSSH 9.6 addresses this protocol weakness through a new “strict KEX” protocol extension that will be automatically enabled when both the client and server support it. This extension makes two changes to the SSH transport protocol to improve the integrity of the initial key exchange.
That means both your client and server need to be updated. Currently my Mac with all the latest Apple updates shows an outdated version of OpenSSH. I can see that when I connect to a host using verbose mode as follows:
ssh -vvv -i localhost
Note that I just stuck localhost in there to get some output. I’m not running an SSH server on my laptop!
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*6sL1xsiABvMQKrHKa-HAMA.png)
I can also see where my local configuration file exists. But in this post, I’m focusing on the server side of the SSH configuration.
Although you can fix the problem in OpenSSH with a patch and the update downplays the dangers of this attack, I’m not feeling too sure about those comments. Here’s a description of the attack:
The attack can be performed in practice, allowing an attacker to downgrade the connection’s security by truncating the extension negotiation message (RFC8308) from the transcript. The truncation can lead to using less secure client authentication algorithms and deactivating specific countermeasures against keystroke timing attacks in OpenSSH 9.5.
So we’re not only worried about timing. We’re worried about downgrading to a less secure algorithm that can potentially then allow an attacker to obtain a MITM attack and read the information in transit as it traverses the Internet.
Reading on…
For example, we found several weaknesses in the AsyncSSH servers’ state machine, allowing an attacker to sign a victim’s client into another account without the victim noticing. Hence, it will enable strong phishing attacks and may grant the attacker Man-in-the-Middle (MitM) capabilities within the encrypted session.
That doesn’t sound like the description above which simply says, “oh some packets will just be missing.”
Removing encryption ciphers that could lead to a downgrade attack
Perhaps I am misunderstanding something, but just to be sure my connections cannot be downgraded, I can remove the ability to connect with insecure algorithms altogether from my server. That also helps prevent insecure clients from having their connections downgraded (though you should still update the client software in case they connect to some other server besides yours.)
Here’s where the fun begins. I haven’t looked at a Linux SSH configuration in a bit and forgot some things. Also — I wanted to verify exactly what was running on my server and that my changes took effect, so I took some additional steps that you won’t have to do to sort this all out. You can skip to the end to get the steps to fix the problem if you want. But here are some caveats to make sure you are only running what you think you are running and you can verify you are changing the correct files.
Checking which ciphers are available
I started with my AWS EC2 Amazon Linux instance where I have SSH configured and nmap installed.
sudo yum install nmap
I ran this command:
sudo nmap --script ssh2-enum-algos localhost
I got back the following on this particular host:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*o860tvUzUgk_aJLzkUktNA.png)
What are these things? Well we can get a clue from this documentation even though it says “legacy”:
KexAlgorithms
: the key exchange methods that are used to generate per-connection keysHostkeyAlgorithms
: the public key algorithms accepted for an SSH server to authenticate itself to an SSH clientCiphers
: the ciphers to encrypt the connectionMACs
: the message authentication codes used to detect traffic modification
Honestly, I never did find good information on all this configuration — I basically had to reverse engineer it. That Ciphers description is relevant but on AWS Amazon Linux we’ll have to dig around a bit to find it. At least on this particular machine I’m using which may have an outdated configuration. More on that in future posts.
Note that the chacha-poly1305 algorithm mentioned in the vulnerability above is in the list of algorithms configured on this particular SSH server.
In addition to using nmap I later discovered this command for checking the ciphers on the server (I threw in “localhost” just to force a response):
ssh -T localhost
The other thing you can do is check what algorithm is actually agreed upon in the handshake between the client and server by using the verbose option when you make a connection like this:
ssh -vvv -i <your key> <user>@<yourhost>
Doing that I can still see the available ciphers including the one I don’t want to allow:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*KmMcw55o1Tn-jQ9S9JPzzw.png)
I can also see that dancing algorithm is the one actually being used:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*QxaQ0XxIQKh8BOIZzaUYAQ.png)
I say dancing 💃🕺algorithm because I can’t help but think of my ballroom and salsa dancing lessons every time I read chacha 😆. Sorry.
What SSH server is my system running?
Well, out of curiosity, I took a few steps to see what is running SSH on my system. These steps are probably completely unnecessary but if you want to verify exactly what is running as I like to do, you can also do the following.
If I run this command:
sudo lsof -i
I can see the following pid (2243) is running ssh.
sshd 2243 root 5u IPv4 17072 0t0 TCP *:ssh (LISTEN)
sshd 2243 root 7u IPv6 17074 0t0 TCP *:ssh (LISTEN)
If I run this command I can see what program is running that service:
sudo ps -aux | grep 2243
/usr/sbin/sshd
root 2243 0.0 0.2 29120 8552 ? Ss Jan05 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
Note that initially I ran the man command for ssh instead of sshd. It is somewhat confusing that both these instances still exist on this server. Perhaps they are removed in a later iteration but just remember to look at ssh with a d at the end not ssh if that’s what your system is running.
But what is interesting is that by doing so I discovered a few things.
I used the man (manual) ssh command to get the help for the ssh service.
man ssh
If I peruse the documentation I can see that there’s information about Ciphers here which seem to be related to our encryption algorithms in the list above:
-c cipher_spec
Selects the cipher specification for encrypting the session. cipher_spec is a comma-separated list of ciphers listed in order
of preference. See the Ciphers keyword in ssh_config(5) for more information.
Unfortunately this man information doesn’t tell you what the default ciphers are, but I looked at a different flavor of Linux and the defaults included the unwanted ChaCha20-Poly1305 cipher per some OpenSSH information on Ubuntu. It appears that AWS may be changing defaults for various crypto algorithms in Amazon Linux 2023 so your best option is to verify using the above commands what is actually running and available.
The above documentation says the configuration file for ssh is here:
/etc/ssh/ssh_config
But remember that we are interested in sshd.
If I run the man command for sshd it tells me that the configuration file is at /etc/ssh/sshd_config by default and can be overridden with -f.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*yohnJUCz4TBoz3bwgzL2-A.png)
It’s a good idea to verify what is running on your system. If someone ran SSH with -f you might be updating the wrong configuration file if the default is not in use.
I can see in the command above that ran my SSH server that the sshd service is running with the -D option but not the -f option so the default configuration file should be in use.
Now if you look at the configuration file for the ssh (no d) service you’ll see this line for Ciphers is commented out. You’d think that might be in the sshd service as well but it’s not.
# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
Also notice that the whole Host section is commented out.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*eRxjyU3ZEscUqXRcYev9Mw.png)
If you scroll down you will notice the following comment and configuration:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*CixaNaUVWgP67QthKUmCOg.png)
Well, I initially made a mistake and went down a rabbit hole looking at those files. But keep in mind that the configuration file may include other configuration files. So if the same setting is in both configuration files, which one takes precedence? That will be important to understand. It depends which file is read first and whether the software that reads it uses the first or last value it reads.
Let’s skip over to the sshd configuration file since that’s the one we care about.
sudo cat /etc/ssh/sshd_config
You’ll find a similar included at the top of that file.
# To modify the system-wide sshd configuration, create a *.conf file under
# /etc/ssh/sshd_config.d/ which will be automatically included below
Include /etc/ssh/sshd_config.d/*.conf
So what I did was jump through a series of hoops editing different files until I figured out which one I actually needed to modify to get the results I wanted.
Create a backup image of your EC2 instance
Before you make these changes you might want to make an image of your EC2 instance in case something goes wrong.
Check the box next to your EC2 instance in the list and choose these options:
Actions > Image and templates > Create image
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*XbrQvQIgg9u7OO7oPY5Kpw.png)
If something goes wrong you can deploy an image using your backup Amazon Machine Image (AMI):
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*IKZPO0zu1WHe-pSfPy1pvA.png)
Mind the cost when you start creating backups here due to the images and EBS volumes. But this is a good way to make sure you can get back to your image in the prior state should something go wrong.
Wait for the pending image creation to complete before proceeding:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*cNdJAa1ysZQZt1X635WsHg.png)
Note that when you refresh this page it reorders the AMIs for some odd reason so your pending item may not be at the top of the list anymore.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*ZznB_oiFYB6xWJe5THOizg.png)
OK once that is complete, reconnect to your linux host and return to the configuration files above and make some modifications.
Which encryption algorithms should we allow?
Recall that our server allows clients to connect with these encryption algorithms for encryption of data in transit:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Vo2rqizWz1bL4Zkrm__idA.png)
Well, I know I want to get rid of the cha-cha-cha algorithm. 💃
But what’s the difference between CTR and GCM and which one is recommended?
As noted before I tend to rely on NIST for encryption recommendations.
Well, the documentation from NIST is not exactly crystal clear.
It appears this document covers block cipher modes.
The latest version of this document was written in 2016: