A denial-of-service vulnerability in the gmp plugin was discovered in strongSwan. All versions since 4.4.0 are affected.
We recently started fuzzing some of our plugins using Google's OSS-Fuzz infrastructure. Among the bugs that were discovered two may lead to denial-of-service attacks. The one described in this article affects the gmp plugin, the other the ASN.1 parser in combination with the x509 plugin (described in a separate article).
RSA public keys passed to the gmp plugin aren't validated sufficiently before attempting signature verification, so that invalid input might lead to a floating point exception. Affected are all strongSwan versions since 4.4.0, up to and including 5.5.2.
CVE-2017-9022 has been assigned for this vulnerability.
With strongSwan 4.4.0 the gmp plugin started to use mpz_powm_sec(), if available, for side-channel-free exponentiation. Compared to mpz_powm() this function has two additional requirements regarding the passed exponent and modulus: The exponent must be larger than zero and the modulus must be odd. If these requirements are not met the calculations performed by libgmp will result in a floating point exception that crashes the whole process. Until now the plugin simply replaced mpz_powm() with mpz_powm_sec() without any additional input checks. So a certificate with an appropriately prepared public key sent by a peer could be used for a denial-of-service attack.
Remote code execution is not possible due to this issue.
Credit to OSS-Fuzz for finding this vulnerability, and to Sven Defatsch for setting up the integration and creating the fuzz target.
Installations that don't have the gmp plugin enabled and loaded are not vulnerable.
The just released strongSwan 5.5.3 fixes this vulnerability. For older releases we provide a patch that fixes the vulnerability and should apply with appropriate hunk offsets.