Problem description:
Quite a few programs fail to handle memory allocation errors correctly. Most programmers check the result code of alloc calls, but due to the problems testing such failures, they do not evaluate if the program reacts correct in case of failure. This might be due to the problems to unit-test memory allocation failures. Therefore in many applications and libraries, errors can be triggered by running programs under memory restrictions, e.g. using

bash -c "ulimit -v 50000; grep -E -e '.++++++++++++++++++++++' axb"

or

./ExecHelper 2084870 /usr/bin/sudoedit

Typical error pattern is double-free or free of uninitialized structure members or pointers, e.g.

function x(*struct) {
  struct->a=alloc()
  struct->b=alloc()
  if(!struct.a || !struct.b ) {
    free(a, b)
    // !!! No a=NULL, b=NULL
    return(-1)
  }
}
...
@exit: free(a,b);

or

function x(*struct) {
  struct.a=alloc()
  if(!a) return -1;
  ..
  struct.b=alloc()
  if(!b) return -1;
}
...
@exit: free(a,b);

The idea behind using memory limits is to

• have the ability to control the exact alloc() invocation that should fail by using an appropriate limit and program input of suitable length.
• Crash programs, that would never run into out of memory conditions on a normal system, e.g. passwd
• Avoid causing memory starvation issues on the target instance, which could be easily detected, so e.g. a machine might just 128MB ram would just lock up when using a very memory-consuming regular expression.

References:

• ExecHelper.c: Helper to execute software with memory constraints, usage ./ExecHelper [limit in bytes] [program] [args]
• SuidBinariesAndStackBaseRandomization: Get the stack layout of a suid-process if needed to guess addresses
• Ubuntu bugreport on libc regex: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/343894