Several weeks ago we received a bug report from a customer that said their game was crashing when using IL2CPP scripting backend. QA verified the bug and assigned it to me for fixing. The project was quite big (although far from the largest ones); it took 40 minutes to build on my machine. The instructions on the bug report said: “Play the game for 5-10 minutes until it crashes”. Sure enough, after following instructions, I observed a crash. I fired up WinDbg ready to nail it down. Unfortunately, the stack trace was bogus: 0:049> k # Child-SP RetAddr Call Site 00 00000022`e25feb10 00000000`00000010 0x00007ffa`00000102 0:050> u 0x00007ffa`00000102 L10 00007ffa`00000102 ?? ??? ^ Memory access error in 'u 0x00007ffa`00000102 l10' Clearly, it tried executing an invalid memory address. Although the stacktrace had been corrupted, I was hoping that only a part of the whole stack got corrupted and that I should be able to reconstruct it if I look at memory contents past the stack pointer register. Surely enough, that gave me an idea where to look next: 0:049> dps @rsp L200 ............... 00000022`e25febd8 00007ffa`b1fdc65c ucrtbased!heap_alloc_dbg+0x1c [d:\th\minkernel\crts\ucrt\src\appcrt\heap\debug_heap.cpp @ 447] 00000022`e25febe0 00000000`00000004 00000022`e25febe8 00000022`00000001 00000022`e25febf0 00000022`00000000 00000022`e25febf8 00000000`00000000 00000022`e25fec00 00000022`e25fec30 00000022`e25fec08 00007ffa`99b3d3ab UnityPlayer!std::_Vector_alloc<std::_Vec_base_types<il2cpp::os::PollRequest,std::allocator<il2cpp::os::PollRequest> > >::_Get_data+0x2b [ c:\program files (x86)\microsoft visual studio 14.0\vc\include\vector @ 642] 00000022`e25fec10 00000022`e25ff458 00000022`e25fec18 cccccccc`cccccccc 00000022`e25fec20 cccccccc`cccccccc 00000022`e25fec28 00007ffa`b1fdf54c ucrtbased!_calloc_dbg+0x6c [d:\th\minkernel\crts\ucrt\src\appcrt\heap\debug_heap.cpp @ 511] 00000022`e25fec30 00000000`00000010 00000022`e25fec38 00007ffa`00000001 ............... 00000022`e25fec58 00000000`00000010 00000022`e25fec60 00000022`e25feca0 00000022`e25fec68 00007ffa`b1fdb69e ucrtbased!calloc+0x2e [d:\th\minkernel\crts\ucrt\src\appcrt\heap\calloc.cpp @ 25] 00000022`e25fec70 00000000`00000001 00000022`e25fec78 00000000`00000010 00000022`e25fec80 cccccccc`00000001 00000022`e25fec88 00000000`00000000 00000022`e25fec90 00000022`00000000 00000022`e25fec98 cccccccc`cccccccc 00000022`e25feca0 00000022`e25ff3f0 00000022`e25feca8 00007ffa`99b3b646 UnityPlayer!il2cpp::os::SocketImpl::Poll+0x66 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\socketimpl.cpp @ 1429] 00000022`e25fecb0 00000000`00000001 00000022`e25fecb8 00000000`00000010 ............... 00000022`e25ff3f0 00000022`e25ff420 00000022`e25ff3f8 00007ffa`99c1caf4 UnityPlayer!il2cpp::os::Socket::Poll+0x44 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\socket.cpp @ 324] 00000022`e25ff400 00000022`e25ff458 00000022`e25ff408 cccccccc`ffffffff 00000022`e25ff410 00000022`e25ff5b4 00000022`e25ff418 00000022`e25ff594 00000022`e25ff420 00000022`e25ff7e0 00000022`e25ff428 00007ffa`99b585f8 UnityPlayer!il2cpp::vm::SocketPollingThread::RunLoop+0x268 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 452] 00000022`e25ff430 00000022`e25ff458 00000022`e25ff438 00000000`ffffffff ............... 00000022`e25ff7d8 00000022`e25ff6b8 00000022`e25ff7e0 00000022`e25ff870 00000022`e25ff7e8 00007ffa`99b58d2c UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint+0xec [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 524] 00000022`e25ff7f0 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread 00000022`e25ff7f8 00007ffa`99b57700 UnityPlayer!il2cpp::vm::FreeThreadHandle [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 488] 00000022`e25ff800 00000000`0000106c 00000022`e25ff808 cccccccc`cccccccc 00000022`e25ff810 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread 00000022`e25ff818 000001c4`1705f5c0 00000022`e25ff820 cccccccc`0000106c ............... 00000022`e25ff860 00005eaa`e9a6af86 00000022`e25ff868 cccccccc`cccccccc 00000022`e25ff870 00000022`e25ff8d0 00000022`e25ff878 00007ffa`99c63b52 UnityPlayer!il2cpp::os::Thread::RunWrapper+0xd2 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\thread.cpp @ 106] 00000022`e25ff880 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread 00000022`e25ff888 00000000`00000018 00000022`e25ff890 cccccccc`cccccccc ............... 00000022`e25ff8a8 000001c4`15508c90 00000022`e25ff8b0 cccccccc`00000002 00000022`e25ff8b8 00007ffa`99b58c40 UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\vm\threadpool.cpp @ 494] 00000022`e25ff8c0 00007ffa`9da83610 UnityPlayer!il2cpp::vm::g_SocketPollingThread 00000022`e25ff8c8 000001c4`155a5890 00000022`e25ff8d0 00000022`e25ff920 00000022`e25ff8d8 00007ffa`99c19a14 UnityPlayer!il2cpp::os::ThreadStartWrapper+0x54 [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\threadimpl.cpp @ 31] 00000022`e25ff8e0 000001c4`155a5890 ............... 00000022`e25ff900 cccccccc`cccccccc 00000022`e25ff908 00007ffa`99c63a80 UnityPlayer!il2cpp::os::Thread::RunWrapper [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\thread.cpp @ 80] 00000022`e25ff910 000001c4`155a5890 ............... 00000022`e25ff940 000001c4`1e0801b0 00000022`e25ff948 00007ffa`e6858102 KERNEL32!BaseThreadInitThunk+0x22 00000022`e25ff950 000001c4`1e0801b0 00000022`e25ff958 00000000`00000000 00000022`e25ff960 00000000`00000000 00000022`e25ff968 00000000`00000000 00000022`e25ff970 00007ffa`99c199c0 UnityPlayer!il2cpp::os::ThreadStartWrapper [ c:\users\tautvydas\builds\bin2\il2cppoutputproject\il2cpp\libil2cpp\os\win32\threadimpl.cpp @ 26] 00000022`e25ff978 00007ffa`e926c5b4 ntdll!RtlUserThreadStart+0x34 00000022`e25ff980 00007ffa`e68580e0 KERNEL32!BaseThreadInitThunk Here’s a rough reconstructed stacktrace: 00000022`e25febd8 00007ffa`b1fdc65c ucrtbased!heap_alloc_dbg+0x1c [...\appcrt\heap\debug_heap.cpp @ 447] 00000022`e25fec28 00007ffa`b1fdf54c ucrtbased!_calloc_dbg+0x6c [...\appcrt\heap\debug_heap.cpp @ 511] 00000022`e25fec68 00007ffa`b1fdb69e ucrtbased!calloc+0x2e [...\appcrt\heap\calloc.cpp @ 25] 00000022`e25feca8 00007ffa`99b3b646 UnityPlayer!il2cpp::os::SocketImpl::Poll+0x66 [...\libil2cpp\os\win32\socketimpl.cpp @ 1429] 00000022`e25ff3f8 00007ffa`99c1caf4 UnityPlayer!il2cpp::os::Socket::Poll+0x44 [...\libil2cpp\os\socket.cpp @ 324] 00000022`e25ff428 00007ffa`99b585f8 UnityPlayer!il2cpp::vm::SocketPollingThread::RunLoop+0x268 [...\libil2cpp\vm\threadpool.cpp @ 452] 00000022`e25ff7e8 00007ffa`99b58d2c UnityPlayer!il2cpp::vm::SocketPollingThreadEntryPoint+0xec [...\libil2cpp\vm\threadpool.cpp @ 524] 00000022`e25ff878 00007ffa`99c63b52 UnityPlayer!il2cpp::os::Thread::RunWrapper+0xd2 [...\libil2cpp\os\thread.cpp @ 106] 00000022`e25ff8d8 00007ffa`99c19a14 UnityPlayer!il2cpp::os::ThreadStartWrapper+0x54 [...\libil2cpp\os\win32\threadimpl.cpp @ 31] 00000022`e25ff948 00007ffa`e6858102 KERNEL32!BaseThreadInitThunk+0x22 00000022`e25ff978 00007ffa`e926c5b4 ntdll!RtlUserThreadStart+0x34 Alright, so now I knew which thread was crashing: it was the IL2CPP runtime socket polling thread. Its responsibility is tell other threads when their sockets are ready to send or receive data. It goes like this: there’s a FIFO queue that socket poll requests get put in by other threads, the socket polling thread then dequeues these requests one by one, calls select() function and when select() returns a result, it queues a callback that was in the original request to the thread pool. So somebody is corrupting the stack badly. In order to narrow the search, I decided to put “stack sentinels” on most stack frames in that thread. Here’s how my stack sentinel was defined: