Entering into Out of Memory Condition

In this blog post I'm describing an approach to force the execution flow to enter into out of memory (OOM) error conditions when the amount of memory to allocate is not controlled by the attacker as in the example below.

#define MAX_SIZE 0x08000000 /* 128M */

ptr = malloc(MAX_SIZE)
if (ptr == NULL)
{
  return OUT_OF_MEMORY;
}

When testing software, it's reasonable to think that error conditions like above are unlikely to be reached as often as other error conditions. Some of them may never get reached because it's not obvious to make the allocation with fixed-size to fail.

OOM error condition could be a potential to enter into vulnerable code path. One example I've heard that the memory allocation had failed and the error condition led to call the cleanup functionality which caused to free of uninitialized pointer.

When testing, one possible way to make memory allocation to fail is to hook the memory allocation function and change the functionality to fail, that is to return with NULL pointer. However, I wanted to avoid this approach of testing because even if I find bug I'd still need a lot of work to reproduce it in real word scenario (if possible at all), when the hook is not applied.

Therefore, what I thought is that the other way to force fixed-size memory allocation to fail is to consume large amount of memory of the virtual address space, and as a consequence the program will be operating under low-memory conditions. This means if memory allocation occurs with reasonable small size it could fail because there is not much memory available. My theory is that any vulnerability found that we reach via OOM error conditions can be reproduced in a real word scenario because we always expect to find something that triggers to consume large amount memory in order to make other allocation to fail. This is pretty much possible in browsers where basically you can easily fill arbitrary amount of the heap memory.

Experimentally, I involved this approach in my fuzzing methodology. I attach the target process to Windbg and execute .dvalloc command to allocate arbitrary amount of memory in the virtual address space. The amount of memory available in the address space can be queried with !address -summary command, or can be seen in Process Explorer, and so you can allocate more for your test if needed.

When testing Firefox with this methodology I notice increased amount of int 3 crashes in mozalloc_abort() that is due to out of memory condition - those crashes are not exploitable though.