In TLS mode of operation the padding value "pad" is obtained along with the
maximum possible padding value "maxpad". If pad > maxpad then the data is
invalid. However we must continue anyway because this is constant time code.
We calculate the payload length like this:
inp_len = len - (SHA_DIGEST_LENGTH + pad + 1);
However if pad is invalid then inp_len ends up -ve (actually large +ve
because it is a size_t).
Later we do this:
/* verify HMAC */
out += inp_len;
len -= inp_len;
This ends up with "out" pointing before the buffer which is undefined
behaviour. Next we calculate "p" like this:
unsigned char *p =
out + len - 1 - maxpad - SHA256_DIGEST_LENGTH;
Because of the "out + len" term the -ve inp_len value is cancelled out
so "p" points to valid memory (although technically the pointer arithmetic
is undefined behaviour again).
We only ever then dereference "p" and never "out" directly so there is
never an invalid read based on the bad pointer - so there is no security
issue.
This commit fixes the undefined behaviour by ensuring we use maxpad in
place of pad, if the supplied pad is invalid.
With thanks to Brian Carpenter for reporting this issue.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3832)
335d0a4646