Check for overflow in massive mallocs.

Hopefully it never happens, but a malloc of nearly the whole address
space should fail cleanly.

Change-Id: I82499e3236a1a485f5518b1c048899b1df3e8488
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/39864
Reviewed-by: David Benjamin <davidben@google.com>

crypto/mem.c

@@ -105,6 +105,10 @@ void sdallocx(void *ptr, size_t size, int flags) {
 }
 
 void *OPENSSL_malloc(size_t size) {
+  if (size + OPENSSL_MALLOC_PREFIX < size) {
+    return NULL;
+  }
+
   void *ptr = malloc(size + OPENSSL_MALLOC_PREFIX);
   if (ptr == NULL) {
     return NULL;