openssl/ssl/statem
Matt Caswell 48c054fec3 Excessive allocation of memory in dtls1_preprocess_fragment()
This issue is very similar to CVE-2016-6307 described in the previous
commit. The underlying defect is different but the security analysis and
impacts are the same except that it impacts DTLS.

A DTLS message includes 3 bytes for its length in the header for the
message.
This would allow for messages up to 16Mb in length. Messages of this length
are excessive and OpenSSL includes a check to ensure that a peer is sending
reasonably sized messages in order to avoid too much memory being consumed
to service a connection. A flaw in the logic of version 1.1.0 means that
memory for the message is allocated too early, prior to the excessive
message length check. Due to way memory is allocated in OpenSSL this could
mean an attacker could force up to 21Mb to be allocated to service a
connection. This could lead to a Denial of Service through memory
exhaustion. However, the excessive message length check still takes place,
and this would cause the connection to immediately fail. Assuming that the
application calls SSL_free() on the failed conneciton in a timely manner
then the 21Mb of allocated memory will then be immediately freed again.
Therefore the excessive memory allocation will be transitory in nature.
This then means that there is only a security impact if:

1) The application does not call SSL_free() in a timely manner in the
event that the connection fails
or
2) The application is working in a constrained environment where there
is very little free memory
or
3) The attacker initiates multiple connection attempts such that there
are multiple connections in a state where memory has been allocated for
the connection; SSL_free() has not yet been called; and there is
insufficient memory to service the multiple requests.

Except in the instance of (1) above any Denial Of Service is likely to
be transitory because as soon as the connection fails the memory is
subsequently freed again in the SSL_free() call. However there is an
increased risk during this period of application crashes due to the lack
of memory - which would then mean a more serious Denial of Service.

This issue does not affect TLS users.

Issue was reported by Shi Lei (Gear Team, Qihoo 360 Inc.).

CVE-2016-6308

Reviewed-by: Richard Levitte <levitte@openssl.org>
2016-09-21 20:37:53 +01:00
..
README Add a state machine README 2015-10-30 08:38:18 +00:00
statem.c Excessive allocation of memory in tls_get_message_header() 2016-09-21 20:37:53 +01:00
statem.h Indent ssl/ 2016-08-18 14:02:29 +02:00
statem_clnt.c Quiet compiler warning about uninitialised variable 2016-09-20 21:59:48 +02:00
statem_dtls.c Excessive allocation of memory in dtls1_preprocess_fragment() 2016-09-21 20:37:53 +01:00
statem_lib.c Excessive allocation of memory in tls_get_message_header() 2016-09-21 20:37:53 +01:00
statem_locl.h Indent ssl/ 2016-08-18 14:02:29 +02:00
statem_srvr.c Convert ClientVerify Construction to WPACKET 2016-09-20 10:16:56 +01:00

State Machine Design
====================

This file provides some guidance on the thinking behind the design of the
state machine code to aid future maintenance.

The state machine code replaces an older state machine present in OpenSSL
versions 1.0.2 and below. The new state machine has the following objectives:
    - Remove duplication of state code between client and server
    - Remove duplication of state code between TLS and DTLS
    - Simplify transitions and bring the logic together in a single location
      so that it is easier to validate
    - Remove duplication of code between each of the message handling functions
    - Receive a message first and then work out whether that is a valid
      transition - not the other way around (the other way causes lots of issues
      where we are expecting one type of message next but actually get something
      else)
    - Separate message flow state from handshake state (in order to better
      understand each)
      - message flow state = when to flush buffers; handling restarts in the
        event of NBIO events; handling the common flow of steps for reading a
        message and the common flow of steps for writing a message etc
      - handshake state = what handshake message are we working on now
    - Control complexity: only the state machine can change state: keep all
      the state changes local to the state machine component

The message flow state machine is divided into a reading sub-state machine and a
writing sub-state machine. See the source comments in statem.c for a more
detailed description of the various states and transitions possible.

Conceptually the state machine component is designed as follows:

                        libssl
                           |
---------------------------|-----statem.h--------------------------------------
                           |
                    _______V____________________
                   |                            |
                   |    statem.c                |
                   |                            |
                   |    Core state machine code |
                   |____________________________|
        statem_locl.h     ^          ^
                 _________|          |_______
                |                            |
   _____________|____________   _____________|____________
  |                          | |                          |
  | statem_clnt.c            | | statem_srvr.c            |
  |                          | |                          |
  | TLS/DTLS client specific | | TLS/DTLS server specific |
  | state machine code       | | state machine code       |
  |__________________________| |__________________________|
               |        |_______________|__       |
               |        ________________|  |      |
               |       |                   |      |
   ____________V_______V________   ________V______V_______________
  |                             | |                               |
  | statem_both.c               | | statem_dtls.c                 |
  |                             | |                               |
  | Non core functions common   | | Non core functions common to  |
  | to both servers and clients | | both DTLS servers and clients |
  |_____________________________| |_______________________________|