DH_check is used to test the validity of Diffie-Hellman parameter sets (p, q, g). Among the tests performed are primality tests on p and q, for this BN_is_prime_ex is called with the rounds of Miller-Rabin set as default. This will therefore use the average case error estimates derived from the function BN_prime_checks_for_size based on the bit size of the number tested.
However, these bounds are only accurate on testing random input. Within this testing scenario, where we are checking the validity of a DH parameter set, we can not assert that these parameters are randomly generated. Thus we must treat them as if they are adversarial in nature and increase the rounds of Miller-Rabin performed.
Generally, each round of Miller-Rabin can declare a composite number prime with probability at most (1/4), thus 64 rounds is sufficient in thwarting known generation techniques (even in safe prime settings - see https://eprint.iacr.org/2019/032 for full analysis). The choice of 64 rounds is also consistent with SRP_NUMBER_ITERATIONS_FOR_PRIME 64 as used in srp_Verify_N_and_g in openssl/apps/s_client.c.
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8593)
2500c093aa