A denial-of-service vulnerability in the gmp plugin was discovered in strongSwan. All versions patched with the fix for CVE-2018-16151/2 are affected.
Google's OSS-Fuzz revealed a bug in the gmp plugin that may lead to a denial-of-service attack. It's caused by the patch that fixes CVE-2018-16151/2, so any strongSwan version with that patch applied is affected.
The verification of a signature with a very short RSA public key causes an integer underflow in a length check that then results in a heap buffer overflow, which most likely crashes the IKE daemon.
CVE-2018-17540 has been assigned for this vulnerability.
As mentioned, the vulnerability was introduced with the patch that fixes CVE-2018-16151/2 (Bleichenbacher-style low-exponent signature forgery due to lenient parsing of RSA signatures). Instead of parsing PKCS#1 v1.5 RSA signatures to verify them, the plugin now calls code that was previously used only with private keys to generate signatures. The shared function produces the encoding of the plaintext signature
0x00 || 0x01 || PS || 0x00 || T
which is then simply compared to the decrypted signature value. Because the padding PS has to be at least 8 bytes and there are 3 marker bytes, the function contained the following check:
if (data.len > keylen - 11)
{
return FALSE;
}
Where data.len is the length of the signature value T (the ASN.1 encoded digestInfo containing the hash) and keylen is the length of the key's modulus in bytes. Because keylen is unsigned (size_t) this results in an integer underflow if keylen is lower than 11. This in turn causes a heap buffer overflow when the encoding is written to a buffer of only keylen bytes.
This code snippet wasn't a problem so far because it was only called for private keys. Besides that these are only loaded locally and with appropriate privileges, the gmp plugin also enforced a minimum length for the moduli of private keys (currently 64 bytes). However, that's not the case for public keys.
So when the signature verification, that operates on public keys, started using that code, the key length could suddenly take values that would trigger the integer underflow (the initial trigger on OSS-Fuzz was a key with a 4-bit modulus).
This in itself would still not yet be a serious issue in practice, because signature verification is generally only done with trusted public keys. So an attacker would have to get a certificate with a specifically crafted public key issued from a trusted CA, or get a certificate with such a key installed directly on the attacked host.
Unfortunately, the x509 plugin, after parsing a certificate, uses the contained public key to check if the certificate is self-signed. This means that an attacker can just send a crafted certificate to the IKE daemon to exploit the vulnerability.
The openssl plugin behaves the same way when parsing certificates, but depending on the plugin order the gmp plugin, even if loaded, won't be used for the signature verification. And the openssl plugin might even reject such certificates while parsing, at least that was the case for the OSS-Fuzz triggers. So unless the x509 plugin is loaded too, such installations might be less vulnerable (when using the openssl plugin, the gmp plugin is redundant anyway).
Since this is a heap buffer overflow the possibility of remote code execution can't be ruled out completely. However, due to the form of the data that's written to the buffer it seems difficult to actually exploit it in such a way.
As mentioned in the introduction, credit to OSS-Fuzz for finding this vulnerability.
Since only the gmp plugin is affected, installations that don't have this plugin enabled and loaded are not vulnerable.
Also not vulnerable are installations that have not been patched with the fix for CVE-2018-16151/2.
The just released strongSwan 5.7.1 fixes this vulnerability. For older releases that were patched with the fix for CVE-2018-16151/2 we provide a patch that fixes the vulnerability and should apply with appropriate hunk offsets.