A denial-of-service vulnerability in the gmp plugin was discovered in strongSwan. All versions are affected.
Shortly after the last release the fuzzer running on Google's OSS-Fuzz infrastructure revealed another issue in the gmp plugin that may lead to a denial-of-service attack.
Specific RSA signatures passed to the gmp plugin for verification can cause a null-pointer dereference. Potential triggers are signatures in certificates, but also signatures used during IKE authentication. Affected are all strongSwan versions, up to and including 5.5.3.
CVE-2017-11185 has been assigned for this vulnerability.
To verify an RSA signature it is decrypted using the public key with the operation m^e mod n, where m is the signature value, and e and n are the exponent and modulus of the public key, respectively. The value m is an integer between 0 and n-1, however, the gmp plugin did not verify and enforce this. So if m equals n the calculation results in 0, in which case mpz_export() returns NULL. This result wasn't handled properly causing a null-pointer dereference, resulting in a denial-of-service vulnerability.
An attacker can exploit this by either setting the signature value during IKE to the modulus of any public key the responder trusts, or by sending a crafted certificate, in which the signature value again matches the modulus of a trusted public key. strongSwan will enumerate its trusted public keys and try to verify the signature. Once the key with the matching modulus is used the bug is triggered and the daemon crashes.
Remote code execution is not possible due to this issue.
Credit to OSS-Fuzz for finding this vulnerability.
Installations that don't have the gmp plugin enabled and loaded are not vulnerable.
The just released strongSwan 5.6.0 fixes this vulnerability. For older releases we provide a patch that fixes the vulnerability and should apply with appropriate hunk offsets.