Memory Allocation Failures

Introduction

Memory allocation operations are not guaranteed to succeed — if there is no memory available, or if the program has exceeded the amount of memory it is allowed to reference, these operations typically will return a null pointer. Many programs do not check for allocation failures. Such a program would attempt to use the null pointer returned by e.g. malloc as if it pointed to allocated memory. When a program instruction references a memory location that it is not allowed to access , the faulting application receives an exception. If the application cannot handle this exception, it will be terminated. In some circumstances, null pointer dereferences pose a security risk because the instruction flow of the application can be modified. It is desirable that failed memory allocations are handled by the application. A failed memory allocation can be simulated by hooking the memory allocation routines so they return 0 instead of a valid address. This document describes how to monitor the behaviour of an application in case of a memory allocation failure.

A Browser Crash

An example of a webpage that causes memory allocations to fail and subsequently causes an access violation firefox, can be found here or here.
When a browser is run in a debugger, the pydbg script, the faulting eip can be found:

$ python av_handler.py
...

EIP: 6e87c1c1 mov [eax+0x1c],edi
EAX: 00000000 (         0) -> N/A
EBX: 0032db08 (   3332872) -> ...1....p!...w.nP..1.0..P..1.0....................
v3....... .0......Rl|n...........1......2.J..n....P..1%........................0
ECX: 00000000 (         0) -> N/A
EDX: 00000000 (         0) -> N/A
EDI: 6e7c6dd8 (1853648344) -> N/A

...

Eip module path :  \Windows\SysWOW64\Macromed\Flash\NPSWF32.dll

What happened here is that the code from NPSWF32.dll referenced memory that has not been mapped in the process address space: [eax+0x1c]. Eax was expected to contain the address of a block of heap allocated memory, previously allocated by HeapAlloc. But HeapAlloc failed and returned 0 instead of a valid heap address. The program neglected to check whether the value returned by HeapAlloc was nonzero and used it as a memory location.
Now, if the same webpage is loaded again, the debugger script will report another instruction where an access violation occured. Actually, NPSWF32.dll generates access violations at dozens of different locations.

List Faulting Instructions

After monitoring a couple of hundred runs of the av_handler.py script, we get a list locations in NPSWF32.dll where access violations occured. For further analysis I loaded NPSWF32.dll in IDA pro and colorod all faulting instructions in ida with idapython:


import idc, idaapi

offsets = [ 0x00076b20,0x0008ccc2,0x001783bb,0x001b76c7,0x001b76cc,0x001b78c6,0x001b7cb1,
            0x001b7d00,0x001b85c5,0x001b95b5,0x001b972c,0x001bc699,0x001c9d4b,0x001ca281,
            0x001cfe99,0x001d2c4d,0x001d3107,0x001d44df,0x001d4fab,0x001d50cd,0x001d5117,
            0x001dddfb,0x001e4a28,0x001e53f8,0x001e5403,0x001e5c79,0x001e6ca6,0x001e7456,
            0x001e7bd1,0x001faaa8,0x001fbd8b,0x001fc025,0x00203333,0x00203a20,0x00203e82,
            0x0020484e,0x00205339,0x002059ee,0x002b93da,0x002b9448,0x002b946c,0x002b999a,
            0x002b9a6c,0x002b9a74,0x002b9a7c,0x002b9a84,0x002b9a8c,0x002b9a94,0x002bbfff,
            0x002bc1b7 ]

for off in offsets:
    rebased = idaapi.get_imagebase() + off
    print "%x"%rebased, idc.GetFunctionName(rebased), idc.GetDisasm(rebased)
    idc.SetColor(rebased,CIC_ITEM,0x2288ff)

output :

10076b20 sub_10076B10 mov     dword ptr [esi+40h], 1
1008ccc2 sub_1008CCBE mov     eax, [ebp+arg_0]
101783bb sub_1017838E mov     eax, [esi+4]
101b76c7 sub_101B76C0 mov     ecx, [eax]
101b76cc sub_101B76C0 mov     edx, [eax+4]
101b78c6 sub_101B78C0 cmp     dword ptr [edi+8], 4
101b7cb1 sub_101B7B90 mov     [edx+ecx*4], esi
101b7d00 sub_101B7CE0 mov     [ebp+eax*4+0], esi
101b85c5 sub_101B85C0 mov     eax, [ebp+0]
101b95b5 sub_101B95B0 mov     eax, [esi+8]
101b972c sub_101B9630 or      dword ptr [esi+8], 80000000h
101bc699 sub_101BC630 mov     [edx+eax*2+8], cx
101c9d4b sub_101C9D20 mov     ecx, [ebp+eax*4+0]
101ca281 sub_101CA270 mov     esi, [edx]
101cfe99 sub_101CFE80 mov     [eax+edx*4], ecx
101d2c4d sub_101D2C20 mov     edx, [ecx]
101d3107 sub_101D30F0 mov     ecx, [eax]
101d44df sub_101D44C0 mov     eax, [edi+18h]
101d4fab sub_101D4F20 mov     eax, [esi]
101d50cd sub_101D4F20 mov     ecx, [ecx]
101d5117 sub_101D4F20 mov     edi, [ecx]
101dddfb sub_101DDC50 mov     [edx], di
101e4a28 sub_101E49D0 mov     [ebx+3Ch], edi
101e53f8 sub_101E52F0 or      dword ptr [esi+18h], 8000000h
101e5403 sub_101E52F0 mov     [esi+20h], eax
101e5c79 sub_101E5530 mov     [ecx+edx], eax
101e6ca6 sub_101E6CA0 cmp     byte ptr [edi], 0
101e7456 sub_101E7450 cmp     dword ptr [eax], 0
101e7bd1 sub_101E7B90 mov     [esi+50h], ebp
101faaa8 sub_101FA800 mov     ecx, [edi+0Ch]
101fbd8b sub_101FA800 mov     [edx+eax+8], ecx
101fc025 sub_101FA800 mov     [eax-4], edx
10203333 sub_10203313 mov     ecx, [eax+4]
10203a20 sub_102039FF and     [eax], edx
10203e82 sub_10203E7A mov     [ecx], eax
1020484e sub_10204821 mov     [eax+ecx*8], edx
10205339 sub_102052F8 mov     ecx, [esi]
102059ee sub_102059D4 mov     [eax+10h], cx
102b93da unknown_libname_25 rep movsd
102b9448 unknown_libname_25 mov     [edi], al
102b946c unknown_libname_25 mov     [edi], al
102b999a unknown_libname_65 rep movsd
102b9a6c unknown_libname_65 mov     [edi+ecx*4-1Ch], eax
102b9a74 unknown_libname_65 mov     [edi+ecx*4-18h], eax
102b9a7c unknown_libname_65 mov     [edi+ecx*4-14h], eax
102b9a84 unknown_libname_65 mov     [edi+ecx*4-10h], eax
102b9a8c unknown_libname_65 mov     [edi+ecx*4-0Ch], eax
102b9a94 unknown_libname_65 mov     [edi+ecx*4-8], eax
102bbfff _fastzero_I movdqa  oword ptr [edi], xmm0
102bc1b7 _fastcopy_I movdqa  oword ptr [edi], xmm0

particularly vuln function : unknown_libname_65. Let's examine the disassembly:

Ida screen of the action:

Looking at the logs from the access violations at these addresses, it can be seen that edi was not initialized ( edi = 0 ).
Tracing the code in unknown_libname_65 It can be seen that edi has been passed as an argument to unknown_libname_65 :
.text:6EA6994B mov edi, [ebp+arg_0] ;
Looking at the code cross references to unknown_libname_65, we can see how arg_0 was initialized: by a call to HeapAlloc. As edi turned out to be 0, arg_0 must have been 0, so HeapAlloc must have failed.

Programming The Pattern

With the help of a debugger we can simulate failed memory allocations by modifying the return code of ntdll.RtlAllocateHeap to 0. Below I describe an example use of a script (or py2exe executable) that does this.
Atest Command line:


$ ./atest
Usage:
d:\build\atest.exe [ OPTIONS ] program arguments
OPTIONS :

        -a,  --address==address
                 address

        -h,  --help
                 print help

        -m,  --monitor
                 monitor what dlls are being loaded

        -o,  --outdir==dirname
                 output directory

        -t,  --target==dllname
                 test target dllname

        -v,  --verbose
                 verbode output

Atest listing calls from NPSWF32.dll that result in a call to ntdll.RtlAllocateHeap:


$ ./atest.exe firefox test.swf -t NPSWF32.dll | tee alloc.log
Loading  firefox test.swf
Using dll  NPSWF32.dll
base 0x6e7b0000
RtlAllocateHeap         at 0x779a2f1f
RtlReAllocateHeap       at 0x779b59e7
  Hit     Called from offset (hex, dec)   Instruction
- 001359 0x002bdb25 2874149             call [0x6ea77378]
- 001359 0x002baea5 2862757             call 0x6ea6db10
- 001359 0x002bb01e 2863134             call 0x6ea6ae95
- 001359 0x002bb0d9 2863321             call 0x6ea6afca
- 001363 0x002c0d8c 2887052             call [0x6ea773b0]
- 001363 0x002bcec5 2870981             call 0x6ea70d76
- 001391 0x002bf4b0 2880688             call 0x6ea6f084
- 001391 0x002bca92 2869906             call 0x6ea6f49f
- 001392 0x002bf142 2879810             call [0x6ea7736c]
- 001393 0x002baed6 2862806             call 0x6ea6fbb1
- 001395 0x002bf466 2880614             call 0x6ea6a320
- 001395 0x002bfc4e 2882638             call 0x6ea6f45a
- 001396 0x002ba397 2859927             call edi
- 001397 0x002bf604 2881028             call 0x6ea6f49f
- 001397 0x002baee0 2862816             call 0x6ea6f5dc
- 001399 0x002be6d5 2877141             call 0x6ea6f45a
- 001399 0x002be83f 2877503             call 0x6ea6e69a
- 001399 0x002baef0 2862832             call 0x6ea6faf6
- 001401 0x002bfb22 2882338             call [0x6ea772b8]
- 001405 0x002bfb74 2882420             call 0x6ea6f45a
- 001407 0x002bf8bc 2881724             call 0x6ea6f49f
- 001409 0x002bf8ea 2881770             call 0x6ea6f49f
- 001539 0x002bbfbb 2867131             call [0x6ea77190]
- 001539 0x002bdcc3 2874563             call [0x6eaf1598]
- 001539 0x002baf04 2862852             call 0x6ea6dca3
- 001541 0x002bae18 2862616             call 0x6ea6f49f
- 001541 0x002bdcd9 2874585             call 0x6ea6dc7f
- 001543 0x002c203f 2891839             call 0x6ea6f49f
- 001545 0x002c67c1 2910145             call [0x6ea7733c]
- 001545 0x002c685b 2910299             call 0x6ea767b0
- 001545 0x002bdcfa 2874618             call 0x6ea6dc62
- 001547 0x002c68ba 2910394             call [0x6ea7733c]
- 001572 0x00205b0c 2120460             call [0x6ea7736c]
- 001573 0x002037e5 2111461             call 0x6e9b2fb5
- 001573 0x002039a0 2111904             call 0x6e9b381f
- 001661 0x0014b858 1357912             call [0x6ea776c0]
- 001829 0x0014b7e0 1357792             call [ecx]
- 001843 0x0014b87e 1357950             call [0x6ea7733c]
- 001845 0x00152139 1384761             call [0x6ea7733c]

Now we know at what offsets NPSWF32 calls functions that rely on ntdll.RtlAllocateHeap. We can specify an offset to atest, atest will force RtlAllocateHeap to return 0 when it is called from that offset.


$ ./atest firefox test.swf -t NPSWF32.dll -a 921165 
Loading  firefox test.swf
Using dll  NPSWF32.dll
Setting breakpoint in NPSWF32.dll (base : 0x70280000 ) at offset 0x000e0e4d
\Windows\SysWOW64\Macromed\Flash\NPSWF32.dll 70360e4d
70360e40 cmp eax,ebx
70360e42 jz 0x70360e54
70360e44 mov ecx,[0x706af2d0]
70360e4a push ecx
70360e4b mov ecx,eax
70360e4d call 0x7048485f
70360e52 jmp 0x70360e56
70360e54 xor eax,eax
70360e56 push eax
70360e57 mov ecx,ebp
70360e59 call 0x704843d2
> 0x70360e4d call 0x7048485f
At 0x77d750a6L Change ret : 0x37c9640 >> 0
0x77d6fa56 from ntdll.dll, base at 0x77d30000
0x77d5f2b8 from ntdll.dll, base at 0x77d30000
0x70485b12 from NPSWF32.dll, base at 0x70280000
0x70360e52 from NPSWF32.dll, base at 0x70280000
Access violation
(1883792350L, 'call 0x70485e82')
(1883792355L, 'push esi')
(1883792356L, 'mov ecx,edi')
(1883792358L, 'call 0x70485ed5')
(1883792363L, 'mov esi,ebx')
(1883792365L, 'jmp 0x704863b8')
(1883792367L, 'pop edi')
(1883792368L, 'pop esi')
(1883792369L, 'pop ebx')
(1883792370L, 'ret')
(1883792371L, 'push ebp')
(1883792372L, 'mov ebp,esp')
(1883792374L, 'push ecx')
(1883792375L, 'push ebx')
(1883792376L, 'push esi')
(1883792377L, 'mov ebx,ecx')
(1883792379L, 'mov esi,[ebx+0x8]')
(1883792382L, 'test esi,esi')
(1883792384L, 'jnz 0x7048641c')
(1883792386L, 'cmp [ebx+0xc],esi')
(1883792389L, 'jnz 0x70486412')
(1883792391L, 'mov ecx,ebx')
(1883792393L, 'call 0x70485da3')
(1883792398L, 'test eax,eax')
(1883792400L, 'jz 0x7048643c')
(1883792402L, 'mov esi,[ebx+0x8]')
(1883792405L, 'test esi,esi')
(1883792407L, 'jnz 0x7048641c')
(1883792409L, 'mov esi,[ebx+0xc]')
(1883792412L, 'test byte [esi+0x2a],0x1')
(1883792416L, 'jz 0x7048644b')
(1883792418L, 'mov eax,[ebx+0x38]')
(1883792421L, 'cmp byte [eax+0x33c],0x0')
CONTEXT DUMP
  EIP: 704863fb mov esi,[ebx+0x8]
  EAX: 00000000 (         0) -> N/A
  EBX: 00000000 (         0) -> N/A
  ECX: 00000000 (         0) -> N/A
  EDX: 00000003 (         3) -> N/A
  EDI: 04f2b064 (  83013732) -> .S..............................................
................................................................................
................................................................................
................................................ (heap)
  ESI: 00000008 (         8) -> N/A
  EBP: 0013ec64 (   1305700) -> .....:Hp........`...-f4p................`.......
..4p....`.....5p......... ...........J.p. ...S................|.................
[.6p. ..v.6p.S........... ..p...@0..........................@0..!.1p......1p. ..
@0..@0............6p....t......p....t.... ...... (stack)
  ESP: 0013ec58 (   1305688) -> ......|..........:Hp........`...-f4p............
....`.........4p....`.....5p......... ...........J.p. ...S................|.....
............[.6p. ..v.6p.S........... ..p...@0..........................@0..!.1p
......1p. ..@0..@0............6p....t......p.... (stack)
  +00: 00000008 (         8) -> N/A
  +04: 037c92a8 (  58495656) -> ....................[...........................
................................................................................
................................................................................
.............................t.. ........... 0.. (heap)
  +08: 00000000 (         0) -> N/A
  +0c: 04f1c0e8 (  82952424) -> TT (heap)
  +10: 70483a8d (1883781773) -> N/A
  +14: 00000008 (         8) -> N/A

mod path :  \Windows\SysWOW64\Macromed\Flash\NPSWF32.dll
Eip module base : 0x70280000
Eip offset : 0x002063fb
call stack :
0x70483a8d from NPSWF32.dll, base at 0x70280000

Closure

In the previous paragraphs we obtained 2 result sets:

heay loaded flash page causing memory corruption issues (50 locations)
access violations obtained by setting the return value from RtlAllocateHeap to 0 (15 out of 160 caused an av)

Intersection:


Eip offset : 0x001c9d4b ; function : sub_101C9D20 ; mov ecx, [ebp+eax*4+0]
Eip offset : 0x001d2c4d ; function : sub_101C9D20 ; mov edx, [ecx]
Eip offset : 0x002b999a ; function : unknown_libname_65 ;  rep movsd
Eip offset : 0x002bbfff ; function : _fastzero_I ; movdqa  oword ptr [edi], xmm0

Conclusion

In the previous paragraphs we looked at a way to simulate memory allocation failures in applications. We have found that access violations that occured in a real crashing application (firefox with flash) could be identified by hooking the windows memory allocation routines.
The testing was done for flash 10. Similar testing with internet explorer and Flash10a.ocx, showed that, for the files tested, slightly less than 10% of all failed memory allocations caused access violations. For Flash9f.ocx, about 20% of failed memory allocations caused access violations. This means that adobe has somehow cleaned up the source code between these two releases.

References, Related