When the parameters are incorrect, all assumptions of (EC)DSA fly out
the window, including whether the retry loop actually terminates.
While ECDSA is broadly used with fixed, named groups, DSA was
catastrophically mis-specified with arbitrary parameters being the
default and only mode. Cap the number of retries in DSA_do_sign so
invalid DSA groups cannot infinite loop, e.g. if the "generator" is
really nilpotent.
This also caps the iteration count for ECDSA. We do, sadly, support
arbitrary curves via EC_GROUP_new_curve_GFp, to help Conscrypt remain
compatible with a badly-designed Java API. After
https://boringssl-review.googlesource.com/c/boringssl/+/51925, we
documented that untrusted parameters are not supported and may produce
garbage outputs, but we did not document that infinite loops are
possible. I don't have an example where an invalid curve breaks ECDSA,
but as it breaks all preconditions, I cannot be confident it doesn't
exist, so just cap the iterations.
Thanks to Hanno Böck who originally reported an infinite loop on
invalid DSA groups. While that variation did not affect BoringSSL, it
inspired us to find other invalid groups which did.
Thanks also to Guido Vranken who found, in
https://github.com/openssl/openssl/issues/20268, an infinite loop when
the private key is zero. That was fixed in the preceding CL, as it
impacts valid groups too, but the infinite loop is ultimately in the
same place, so this change also would have mitigated the loop.
Update-Note: If signing starts failing with ECDSA_R_INVALID_ITERATIONS,
something went horribly wrong because it should not be possible with
real curves. (Needing even one retry has probability 2^-256 or so.)
Change-Id: If8fb0157055d3d8cb180fe4f27ea7eb349ec2738
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/57228
Commit-Queue: David Benjamin <davidben@google.com>
Reviewed-by: Adam Langley <agl@google.com>
474ddf8ba9