[klibc] [klibc:master] malloc: Fail if block size is out of range for sbrk

[klibc-bot for Ben Hutchings]

Commit-ID:  abe5c3477ffa5e91029ef040aede622145dcc777
Gitweb:     http://git.kernel.org/?p=libs/klibc/klibc.git;a=commit;h=abe5c3477ffa5e91029ef040aede622145dcc777
Author:     Ben Hutchings <ben at decadent.org.uk>
AuthorDate: Wed, 28 Apr 2021 04:39:25 +0200
Committer:  Ben Hutchings <ben at decadent.org.uk>
CommitDate: Thu, 29 Apr 2021 16:02:58 +0200

[klibc] malloc: Fail if block size is out of range for sbrk

sbrk() takes a parameter of type intptr_t.  We allow allocating up to
PTRDIFF_MAX (equal to INPTPTR_MAX), and then add a header to that, so
the result fsize can be > INTPTR_MAX.  The conversion of fsize to
intptr_t would then result in undefined behaviour (but probably
*lowering* the top of heap).  Fail cleanly before that happens.

This is currently a theoretical problem since we actually use mmap()
instead of sbrk() on all architectures.

Signed-off-by: Ben Hutchings <ben at decadent.org.uk>

---
 usr/klibc/malloc.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/usr/klibc/malloc.c b/usr/klibc/malloc.c
index abda84c2..09a596f1 100644
--- a/usr/klibc/malloc.c
+++ b/usr/klibc/malloc.c
@@ -171,6 +171,10 @@ void *malloc(size_t size)
 	fsize = (size + MALLOC_CHUNK_MASK) & ~MALLOC_CHUNK_MASK;
 
 #if _KLIBC_MALLOC_USES_SBRK
+	if (fsize > INTPTR_MAX) {
+		errno = ENOMEM;
+		return NULL;
+	}
 	fp = (struct free_arena_header *)sbrk(fsize);
 #else
 	fp = (struct free_arena_header *)