Benjamin discovered the vulnerabilities using the OSS-Fuzz fuzzing service.
The first “moderate severity” issue, tracked as CVE-2017-3737, is related to an “error state” mechanism implemented since OpenSSL 1.0.2b.
“OpenSSL 1.0.2 (starting from version 1.0.2b) introduced an “error state” mechanism. The intent was that if a fatal error occurred during a handshake then OpenSSL would move into the error state and would immediately fail if you attempted to continue the handshake.” reads the security advisory.
“This works as designed for the explicit handshake functions (SSL_do_handshake(), SSL_accept() and SSL_connect()), however due to a bug it does not work correctly if SSL_read() or SSL_write() is called directly.”
“If SSL_read()/SSL_write() is subsequently called by the application for the same SSL object then it will succeed and the data is passed without being decrypted/encrypted directly from the SSL/TLS record layer,” OpenSSL said in its advisory.
The flaw has been rated “moderate severity” because the targeted application would need to be affected by a bug that causes a call to SSL_read() or SSL_write() after getting a fatal error.
The second flaw tracked as CVE-2017-3738 is an overflow vulnerability that could be exploited by an attacker to access TLS-protected communications. The flaw was rated as “low severity” because it is very difficult to trigger in a real attack scenario.
“There is an overflow bug in the AVX2 Montgomery multiplication procedure used in exponentiation with 1024-bit moduli. No EC algorithms are affected. Analysis suggests that attacks against RSA and DSA as a result of this defect would be very difficult to perform and are not believed likely.” continues the advisory. “Attacks against DH1024 are considered just feasible, because most of the work necessary to deduce information about a private key may be performed offline. The amount of resources required for such an attack would be significant.”
The OSS-Fuzz tool also allowed Google researchers to find two low and medium severity, tracked CVE-2017-3736 and CVE-2017-3732, vulnerabilities in early November.
This is the fourth OpenSSL security update in 2017, and it will likely be the last one.