Dan McKinley
Math, Programming, and Minority Reports

Beware the OutOfMemoryException Red Herring
March 8th, 2006

Low memory conditions have a way of executing lightly-or-never-tested code paths. Bugs in these paths can sometimes cast a penumbra over the real reason your application is crashing.

Worse, the resulting exceptions or crashes are likely to be more or less nondeterministic. Worse still, the most obvious steps with SOS rarely work once the CLR or the IIS worker process has started to OOM and has tossed you an access violation.

As a concrete example—this is a !clrstack from an application that appears to be crashing with an IOException.

0:026> !clrstack
Thread 26
ESP         EIP
0x10e3ea90  0x77e55dea [FRAME: GCFrame]

This doesn’t tell us much, but if we look at the native stack:

0:026> k
ChildEBP RetAddr
10e3e9e8 79216aed kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
10e3ea40 79216a70 mscorsvr!GetAssemblyMDImport+0x2e1d4
10e3ea68 79216a24 mscorsvr!GetAssemblyMDImport+0x2e157
10e3ea78 79265ec2 mscorsvr!GetAssemblyMDImport+0x2e10b
10e3eab4 7924e9d3 mscorsvr!CoEEShutDownCOM+0x12dc1
10e3eac8 7c82eeb2 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x7e73
10e3eaec 7c82ee84 ntdll!ExecuteHandler2+0x26
10e3eb94 7c82ecc6 ntdll!ExecuteHandler+0x24
10e3eb94 77e55dea ntdll!KiUserExceptionDispatcher+0xe
10e3eee0 79216aed kernel32!RaiseException+0x53
10e3ef38 7924c8d0 mscorsvr!GetAssemblyMDImport+0x2e1d4
10e3efe8 791b3208 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x5d70
10e3eff4 791b3ad7 mscorsvr!Ordinal76+0x3208
10e3f108 791d3ef0 mscorsvr!Ordinal76+0x3ad7
10e3f1c4 791d3fa4 mscorsvr!GetCompileInfo+0x13e3
10e3f1ec 7931392f mscorsvr!GetCompileInfo+0x1497
10e3f21c 791cc3c8 mscorsvr!ReleaseFusionInterfaces+0x5f3d7
10e3f264 793139e1 mscorsvr!Ordinal76+0x1c3c8
10e3f2d8 7924a3f2 mscorsvr!ReleaseFusionInterfaces+0x5f489
10e3f2f0 791d4096 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x3892

We can see from the frames in the middle (the lower kernel32!RaiseException through the subsequent ntdll calls) that the current exception is being raised from inside another exception handler (a catch block, in managed terms). The current exception doesn’t appear to reflect this:

0:026> !cen
System.IO.IOException (0xbab602fc)
 
0:026> !do!do 0xbab602fc
Name: System.IO.IOException
-----------------
Exception bab602fc in MT 79bd1234: System.IO.IOException

But with some wizardry, we can coerce the relevant information. We can hypothesize that the original exception would be passed as a parameter somewhere before the original call to kernel32!RaiseException. Isolating just that frame on the stack, and the frame before it:

0:026> kb
…
10e3eee0 79216aed e0434f4d 00000001 00000000 kernel32!RaiseException+0x53
…
10e3ef38 7924c8d0 bab5ce60 00000000 bab5ce60 mscorsvr!GetAssemblyMDImport+0x2e1d4

The address 0xbab5ce60 (second line) looks like a good candidate, being close to the address of the current exception while not being equal to it.

0:026> !do bab5ce60
Name: System.IO.IOException
-----------------
Exception bab5ce60 in MT 79bd1234: System.IO.IOException
_message: Unable to create a transport connection.
_innerException:
    Exception 06c6003c in MT 79b96cec: System.OutOfMemoryException
    _remoteStackTraceString:

Bingo. (Mind you, we haven’t taken any steps yet to actually fix the problem.) This was relatively easy, since all of the relevant information could be reconstructed from the offending stack. In other situations, you may need to !DumpAllExceptions to see what has been happening recently in the application.

Note: a very good hint that you are looking at an OOM situation is that your crash dump is exactly two gigabytes in size.

The Russian Doll Approach to Web Services
March 6th, 2006

Here’s an anecdote for the WTF inbox. I assure you this is very real, but I cannot divulge any of the specifics.

Some time ago a friend of mine was talking to a web services vendor, who was explaining his versioning scheme. The vendor’s approach was to make all of the web service functions accept a single parameter describing the function being called, and the version of the function requested. Prototypically:

public object Foo(FunctionCallInfo)

And what is FunctionCallInfo, you ask? Why, it is a strict XML document adhering to a schema that he would provide.

My observation, which my friend also arrived at independently, was that this person was basically creating an implementation of web services inside of web services. So nat’ralists observe, a flea / Hath smaller fleas that on him prey.

XML: the cause of, and solution to, all of your development problems.

C++ and VS2005 Deployment PSA
March 2nd, 2006

This is a straightforward fix, but I couldn’t find anything about it anywhere else. I have a handful of old ATL projects that have been upgraded over the years from Visual C++ 6.0, through VS2003 and are now being built using Visual C++ 8. I was trying to add them to a deployment project today, and kept running into this error:

Two or more objects have the same target location
('[targetdir]\regsvr32.trg').

The issue is that the way libraries are registered has changed over the various incarnations of Visual Studio. My old projects were still using a post-build step to perform registration:

This apparently can create the output file that the deployment package is complaining about. The fix is to delete the post build steps and enable the modern registration option on the Linker’s General tab.