Archive for the ‘Microsoft Technology’ Category

Debugger Exercise: Displaying a Function’s Return Value

I found myself needing to automatically manipulate the return value of a managed function in release code today. I thought this would make an interesting little writeup.

For the purposes of the demonstration, let's use this example program:

class Program
{
    static void Main()
    {
        while (!Console.KeyAvailable)
        {
            Console.WriteLine(FooBar(false));
            Console.WriteLine(FooBar2());
        }
    }

    static string FooBar(bool b)
    {
        if (b)
        {
            return "b";
        }
        return "a";
    }

    static string FooBar2()
    {
        return FooBar(true);
    }
}

The function we're interested in will be FooBar. I've made things interesting by calling the function from two places–let's assume we're not sure where the function is called from. We'll make things more interesting still by accepting the limitation that we won't cheat and set a breakpoint on the ret instruction in FooBar. The function I needed to instrument today was a little monolithic, so let's take it as a given that we need to be able to handle a function that might be long and might have more than one exit point.

Compile this program and start it in WinDbg. Start by loading SOS:

0:003> .loadby sos mscorwks

The first thing we'll do is set a breakpoint on FooBar. (I'm going to do this manually so that these steps will work with the CLR v1.1. The new version of SOS makes this slightly easier. I'm also going to cover this quickly, so if you want a clearer explanation of what's going on in this next step see Eran Sandler's writeup here.)

To set the breakpoint, let the program run for a second so that the method is compiled by the JIT. Then we need to find and dump out the MethodTable for the class:

0:003> !name2ee retaddr!retaddr.Program
Module: 00912d5c (retaddr.exe)
Token: 0x02000002
MethodTable: 00913140
EEClass: 009112ec
Name: retaddr.Program

0:003> !dumpmt -md 00913140
EEClass: 009112ec
Module: 00912d5c
Name: retaddr.Program
mdToken: 02000002
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 8
--------------------------------------
MethodDesc Table
   Entry MethodDesc      JIT Name
...
00d00070   00913120      JIT retaddr.Program.Main()
00d000d8   00913128      JIT retaddr.Program.FooBar(Boolean)
00d00130   00913130      JIT retaddr.Program.FooBar2()
...

Now that we have the MethodTable, we need to dump out the MethodDesc for FooBar to get the virtual address of the jitted code.

0:003> !dumpmd 00913128
Method Name: retaddr.Program.FooBar(Boolean)
Class: 009112ec
MethodTable: 00913140
mdToken: 06000002
Module: 00912d5c
IsJitted: yes
m_CodeOrIL: 00d000d8

The compiled method begins at m_CodeOrIL (this is called "Method VA" in SOS 1.1). We can set a breakpoint there with "bp 00d000d8."

Here comes the trick. We know that the return address to the method should be on the top of the stack when the method is called. In other words, the ESP register points at the return address. We can set a dynamic breakpoint on that return address using a command like this:

bp 00d000d8 "bp poi(@esp); g"

If you try this, you'll notice that we break right after FooBar has returned to its caller. As with any function obeying one of the standard calling conventions, the return value is stored in the EAX register. We can give the dynamic breakpoint a command of its own that manipulates this value like this:

bp 00d000d8 "bp poi(@esp) \"!do -nofields @eax; g\"; g"

If we run the sample program now, we'll see this output repeating itself:

breakpoint 1 redefined
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 20(0x14) bytes
String: a

breakpoint 2 redefined
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 20(0x14) bytes
String: b

Voila.

Beware the OutOfMemoryException Red Herring

Low memory conditions have a way of executing lightly-or-never-tested code paths. Bugs in these paths can sometimes cast a penumbra over the real reason your application is crashing.

Worse, the resulting exceptions or crashes are likely to be more or less nondeterministic. Worse still, the most obvious steps with SOS rarely work once the CLR or the IIS worker process has started to OOM and has tossed you an access violation.

As a concrete example–this is a !clrstack from an application that appears to be crashing with an IOException.

0:026> !clrstack
Thread 26
ESP         EIP
0x10e3ea90  0x77e55dea [FRAME: GCFrame]

This doesn't tell us much, but if we look at the native stack:

0:026> k
ChildEBP RetAddr
10e3e9e8 79216aed kernel32!RaiseException+0x53
WARNING: Stack unwind information not available. Following frames may be wrong.
10e3ea40 79216a70 mscorsvr!GetAssemblyMDImport+0x2e1d4
10e3ea68 79216a24 mscorsvr!GetAssemblyMDImport+0x2e157
10e3ea78 79265ec2 mscorsvr!GetAssemblyMDImport+0x2e10b
10e3eab4 7924e9d3 mscorsvr!CoEEShutDownCOM+0x12dc1
10e3eac8 7c82eeb2 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x7e73
10e3eaec 7c82ee84 ntdll!ExecuteHandler2+0x26
10e3eb94 7c82ecc6 ntdll!ExecuteHandler+0x24
10e3eb94 77e55dea ntdll!KiUserExceptionDispatcher+0xe
10e3eee0 79216aed kernel32!RaiseException+0x53
10e3ef38 7924c8d0 mscorsvr!GetAssemblyMDImport+0x2e1d4
10e3efe8 791b3208 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x5d70
10e3eff4 791b3ad7 mscorsvr!Ordinal76+0x3208
10e3f108 791d3ef0 mscorsvr!Ordinal76+0x3ad7
10e3f1c4 791d3fa4 mscorsvr!GetCompileInfo+0x13e3
10e3f1ec 7931392f mscorsvr!GetCompileInfo+0x1497
10e3f21c 791cc3c8 mscorsvr!ReleaseFusionInterfaces+0x5f3d7
10e3f264 793139e1 mscorsvr!Ordinal76+0x1c3c8
10e3f2d8 7924a3f2 mscorsvr!ReleaseFusionInterfaces+0x5f489
10e3f2f0 791d4096 mscorsvr!GetMetaDataPublicInterfaceFromInternal+0x3892

We can see from the frames in the middle (the lower kernel32!RaiseException through the subsequent ntdll calls) that the current exception is being raised from inside another exception handler (a catch block, in managed terms). The current exception doesn't appear to reflect this:

0:026> !cen
System.IO.IOException (0xbab602fc)

0:026> !do 0xbab602fc
Name: System.IO.IOException
-----------------
Exception bab602fc in MT 79bd1234: System.IO.IOException

But with some wizardry, we can coerce the relevant information. We can hypothesize that the original exception would be passed as a parameter somewhere before the original call to kernel32!RaiseException. Isolating just that frame on the stack, and the frame before it:

0:026> kb
…
10e3eee0 79216aed e0434f4d 00000001 00000000 kernel32!RaiseException+0x53
…
10e3ef38 7924c8d0 bab5ce60 00000000 bab5ce60 mscorsvr!GetAssemblyMDImport+0x2e1d4

The address 0xbab5ce60 (second line) looks like a good candidate, being close to the address of the current exception while not being equal to it.

0:026> !do bab5ce60
Name: System.IO.IOException
-----------------
Exception bab5ce60 in MT 79bd1234: System.IO.IOException
_message: Unable to create a transport connection.
_innerException:
   Exception 06c6003c in MT 79b96cec: System.OutOfMemoryException
   _remoteStackTraceString:

Bingo. (Mind you, we haven't taken any steps yet to actually fix the problem.) This was relatively easy, since all of the relevant information could be reconstructed from the offending stack. In other situations, you may need to !DumpAllExceptions to see what has been happening recently in the application.

Note: a very good hint that you are looking at an OOM situation is that your crash dump is exactly two gigabytes in size.

The Russian Doll Approach to Web Services

Here's an anecdote for the WTF inbox. I assure you this is very real, but I cannot divulge any of the specifics.

Some time ago a friend of mine was talking to a web services vendor, who was explaining his versioning scheme. The vendor's approach was to make all of the web service functions accept a single parameter describing the function being called, and the version of the function requested. Prototypically:

public object Foo(FunctionCallInfo)

And what is FunctionCallInfo, you ask? Why, it is a strict XML document adhering to a schema that he would provide.

My observation, which my friend also arrived at independently, was that this person was basically creating an implementation of web services inside of web services. So nat'ralists observe, a flea / Hath smaller fleas that on him prey.

XML: the cause of, and solution to, all of your development problems.

What would do in this situation?

C++ and VS2005 Deployment PSA

This is a straightforward fix, but I couldn't find anything about it anywhere else. I have a handful of old ATL projects that have been upgraded over the years from Visual C++ 6.0, through VS2003 and are now being built using Visual C++ 8. I was trying to add them to a deployment project today, and kept running into this error:

Two or more objects have the same target location
('[targetdir]\regsvr32.trg').

regsvr32 error building a migrated C++ project

The issue is that the way libraries are registered has changed over the various incarnations of Visual Studio. My old projects were still using a post-build step to perform registration:

VC++ project using a post-build step to register itself

This apparently can create the output file that the deployment package is complaining about. The fix is to delete the post build steps and enable the modern registration option on the Linker's General tab.

VC++ project being registered via the new method.

A Real Head-Scratcher Courtesy The CLR And Office Teams

"Why the hell is this application insisting on loading an old version of the CLR?" I'm guessing that's what you're asking yourself if you've gotten here through your search engine of choice. Well, relax. I'm about to explain.

It is reasonably well-known that there are .config file settings that can force an application to load a specific CLR version. It is also possible to write unmanaged code using the hosting API's like CorBindToRuntimeEx to bind to a specific version. I'm not talking about either of these scenarios.

There is a set of undocumented (as far as I know) registry keys that can override either of these vectors. They are located here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\policy\AppPatch\v2.0.50727.00000

If you take a look at this key on your own machine, you'll probably see a handful of applications listed.

The root of the Microsoft Office .NET Framework version bug

Applications can specify a target version of the framework that will be loaded into the process, even if there is no .config file present, and even if you write some C++ to call CorBindToCurrentRuntime. Mscoree will simply report that the current runtime is 1.1.4322, even if the user has 2.0 installed on their machine. This is, I suppose, superfically similar to the strategy used in the Image File Execution Options built into NT.

I can see where this would be handy–presumably Microsoft has a handful of .NET 1.1 apps in the wild that showed bugs against 2.0, and it's not exactly practical or foolproof to try to drop a .config file on those apps when installing the framework (naturally, somebody can just install the application after the framework, then be puzzled as to why it isn't working).

Good solution, right? It's a shame they fouled it up.

Notice that Word and Excel are both listed as special applications on my machine. I have these values on my laptop:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\policy\ AppPatch\v2.0.50727.00000\excel.exe\{2CCAA9FE-6884-4AF2-99DD-5217B94115DF}]
"Minimum File Version Number"="11.0.0.0"
"Maximum File Version Number"="11.0.9999.9999"
"Target Version"="v1.1.4322"

It appears as though the someone is trying to force a relatively older build of Excel/Word 2003 to load v1.1. Fair enough, the Excel build on my machine is up to 11.6 at this point. Unfortunately, those developers failed miserably in the attempt. Every single other application listed under AppPatch in my registry looks like this:

"Minimum File Version"="11.0.0.0"
"Maximum File Version"="11.0.9999.9999"
"Target Version"="v1.1.4322"

When the version filter isn't specified (or actually in this case, miswritten as "Minimum/Maximum File Version Number" instead of "Minimum/Maximum File Version"), all versions of the executable will load the target version. So these keys effectively break 2.0 addins for all versions of Word and Excel.

There is an update available that fixes this issue, here. The KB article that accompanies it doesn't really explain what the issue is, which is why I've summarized it here. Office update is not exactly an automatic affair like its Windows counterpart, so this bug makes it essentially impossible to write a fault-free Office addin that targets 2.0 and does not hack up the user's registry upon installation. I can't say I advocate doing that, but I would understand if you did. As for myself, I am most likely going to target 1.1 much longer than I had hoped would be necessary.

Generic and Threadsafe Singleton Implementation

I googled around, and couldn't find a generic singleton implementation that was 1) correct and 2) met all of my needs. This is a clever approach, but unfortunately it is limited to objects that are created by calls to constructors.

So I went ahead and wrote the singleton generic that I fully expect to be included in the next BCL. (Not really. It's completely straightforward. However, the Steelers won the super bowl yesterday, so I'm not really in a modest mood.)

I started out defining a class factory type, which is responsible for creating the an instance in a purposefully vague way. I also wrote the default version which just calls new().

ClassFactory object model

/// <summary>
/// Interface for objects that create instances of
/// another type.
/// </summary>
public interface IClassFactory<T> where T : class
{
        T CreateInstance();
}

/// <summary>
/// A default <see cref="IClassFactory"/> implementation,
/// which uses a parameterless constructor to create the
/// instance.
/// </summary>
public class DefaultClassFactory<T> : IClassFactory<T>
        where T : class, new()
{
        public T CreateInstance()
        {
                return new T();
        }
}

From there, I went for the slam dunk in writing both a singleton class with a class factory and a default version that doesn't require one.

Singleton object model

/// <summary>
/// A base (or helper) singleton class. Defines the
/// singleton instance.
/// </summary>
/// <typeparam name="T">
/// The type of the singleton object.
/// </typeparam>
/// <typeparam name="class_factory">
/// The type of the class factory to use to create an
/// instance of type <typeparamref name="T"/>.
/// </typeparam>
public class Singleton<T, class_factory>
    where T : class
    where class_factory : IClassFactory<T>, new()
{
    private static object _sync = new object();
    private static T _default;

    /// <summary>
    /// Gets the singleton instance.
    /// </summary>
    public static T Default
    {
        get
        {
            EnsureDefault();
            return _default;
        }
    }

    /// <summary>
    /// Ensures that the singleton has been created.
    /// </summary>
    private static void EnsureDefault()
    {
        if (_default == null)
        {
            lock (_sync)
            {
                if (_default == null)
                {
                     CreateDefault();
                }
             }
        }
    }

    /// <summary>
    /// Uses the class factory to create the instance.
    /// </summary>
    private static void CreateDefault()
    {
        class_factory cf = new class_factory();
        T value = cf.CreateInstance();

        // This ensures that writes in the creation of
        // the default instance won't be shuffled beyond
        // the write to _default. Only matters on multiproc
        // machines where the hardware allows this. Does
        // nothing on an x86.
        Thread.MemoryBarrier();
        _default = value;
    }
}

/// <summary>
/// A basic singleton type that can be used with objects
/// that are created with a parameterless constructor.
/// </summary>
public class Singleton<T> :
    Singleton<T, DefaultClassFactory<T>>
    where T : class, new()
{
}

(For more on why I added that call to Thread.MemoryBarrier(), see this writeup.)

Here's a really simple example that does not take advantage of the customizability:

public class Foo : Singleton<Foo>
{
}

If you can't use Singleton as a base class, you have to write a little more code.

public class Foo : Bar
{
    public static Foo Default
    {
        get { return Singleton<Foo>.Default; }
    }
}

Debugger Feature Request

Mike Stall has had a recent string of posts about debugger features, which got me thinking about what I waste time doing in debuggers these days.

Omniscient debuggers would be great, but what I would really like is a memory window that I can annotate. I don't care if this makes it into Visual Studio or WinDbg, but it would save me time that I currently waste scribbling on notepads.

I should be able to stick some comments in the memory window, and they should stay at that address for the extent of the debugger run. This is what I mean:

Visual Studio memory window with annotation

In this case, since my memory window is @ESP, I should be able to see my annotations being pushed down on the stack as I step.

I realize maybe one percent of developers could use this, but dammit, we're the coolest one percent.

Psychic Debugging: Random Crashes

Raymond Chen has had an entry or two about "Psychic Debugging." This skill is a lot like everyday psychic abilities by virtue of being really just intuition and guesswork, but dissimilar in that it produces results that benefit society.

Today one of the developers was working on a weird problem he found in his development environment. He could test his functionality once (it's an ASPX page), but the second time he tried to use it there was usually a "random" exception on the page that didn't make any sense. Sometimes, instead of an exception, his worker process just crashed. He also told me that he was trying to use a new API that I knew mapped to a third-party COM object.

This is where the mystical powers came in handy. I tried running his page once, then running a different page–same weird exceptions and crashes. It was here that I started to suspect that the heap was being corrupted by the COM library.

(Let me quickly explain what heap corruption is: suppose I have two objects, X and Y, next to one another in memory. Now suppose I make a mistake writing my program and accidently write beyond the end of object X. I've now screwed up object Y, but we might not know about it until we try to use object Y. We might not try to use object Y for quite a while, so this makes the real cause of the bug awfully difficult to find. To make matters worse, the next time we run the program our bug writing to object X might scribble all over a completely different object that will crash the program somewhere else.)

On a hunch I attached a debugger after the first load of the page. When the exception occurred, I ran the !verifyheap command in the SOS extension. This reported that there was a corrupt object on the heap. I ran the same test a few times to confirm the theory.

If you're lucky, you might see a debug break like this in one of the Windows Debuggers as soon as the heap corruption occurs:

HEAP[heapcorrupt.exe]: Heap block at 001AD700 modified at 001AD780
  past requested size of 78
...
ntdll!DbgBreakPoint:

This is just a debug break, so it won't tear down the process immediately. Depending on the application you're working on the real problems could start much later, long after the offending code has finished executing.

All I can say is that it's a goddamned miracle that the developer was doing a good job testing his functionality and noticed this. The mayhem that would have ensued if this made it to QA would have made it far more difficult to track down the source of the problem. The happy ending here is that the vendor has a fully-managed version of their component that we can switch to–hopefully that will be more difficult for them to screw up.

Quick Debugger Tips

In my javascript debugging post I showed a stack trace and mentioned only that it seemed to be running javascript code. Here it is:

0:009> ~0k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f97c 6009db8b js3250!JSLL_MinInt+0x45e2
0012f9c8 6009cdd9 js3250!js_GetSrcNoteOffset+0x5358
0012f9f0 600c7c01 js3250!js_GetSrcNoteOffset+0x45a6
0012fa08 600856b5 js3250!js_GetScriptLineExtent+0x39e6
0012fa28 600b036f js3250!JS_NewStringCopyZ+0x44
0012fa40 600b3e93 js3250!js_FindProperty+0x26c5

This is a good stack to use to point out a few things about debugging without symbols.

If you're used to stepping through debug code in Visual Studio, your first instict may be to trust the stack printed out here. This would be wrong, for a few reasons.

First, there's the line telling us: "Stack unwind information not available. Following frames may be wrong." Stop me if I've covered this before, but it probably means that we're looking at optimized code. Optimized stack frames often have the pointers to each previous stack frame left out (you'll see this called "frame pointer omission," or FPO) which makes walking the stack much more difficult for the debugger.

Second, notice the offsets following the function names - most are very high. For example,

js3250!js_GetSrcNoteOffset+0x5358

The +0×5358 indicates that the return address is to a point more than 20KB after the export function given. 20KB is a lot of machine code, so this almost definitely an internal function call and not a return address into the body of js_GetSrcNoteOffset.

I say "almost definitely" instead of "definitely" with good reason. I have personally seen some VB6 functions several thousands of lines long that would certainly generate more than 20KB of machine code. But I guess if you're dealing with that then you have enough of your own problems without worrying about symbol subtleties in the Windows debuggers.

So the debugger here is showing us the closest symbol to the return address available, which turns out to not be very close at all.

What can we say about this thread's stack, given just this information? Only that it is very likely to be running javascript code. Not much else.

Conventional Javascript Debugging is for Wimps

Recently, Firefox has been maxing out my CPU "when I have more than three tabs open." As is typical with bug reports from users, this one is very poorly worded and essentially useless. It turns out that that was not a good explanation of what was going on. I messed around a little more and figured out that the problem had these characteristics:

  • The CPU was maxed out on any ESPN.com page.
  • Thread #0 was the offending thread.
  • Firefox's working set increased steadily while looking at ESPN.
  • Firefox eventually crashes with an A/V if the browser is left on ESPN.

Firefox 1.5 CPU Spike

I'm not sure if this is some strange interaction of my specific combination of extensions and settings, or if this happens for all Firefox users. This seems like a possible security vulnerability, but I can't say one way or the other.

I suppose I could have downloaded and compiled the source code for Firefox to figure things out, but that was way more effort than I felt like giving. I did managed to debug and fix this issue for myself–without source or symbols–which I think makes for an interesting writeup.

Since the problem was on every page on the website, and continued after the page finished loading, I assumed it had to be a javascript or Flash problem (both of which are abused by ESPN to an irritating degree). I don't even have the Flash plugin for Firefox installed, so that narrowed the scope of my investigation.

The first thing I did was try to debug the Javascript normally; this was hampered by several factors:

  • The Javascript debugger for Firefox, Venkman, hasn't officially come out for version 1.5 yet. I've complained about broken extensions before, but let me just reiterate how stupid and unprofessional I think broken extensions are.
  • Although this kind person has done his own workaround for 1.5, it was either brought to its knees by the ESPN site or was broken by something else.

To summarize the story to this point, I had a (likely) javascript problem on a page I visit very frequently, coupled with a browser bug that maxed the CPU and made the issue difficult to diagnose through normal means.

I attached to Firefox.exe in Windbg and started randomly breaking in and checking on thread zero while the spike was in progress. Sometimes, frames in the js3250 module were on the stack:

0:009> ~0k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f97c 6009db8b js3250!JSLL_MinInt+0x45e2
0012f9c8 6009cdd9 js3250!js_GetSrcNoteOffset+0x5358
0012f9f0 600c7c01 js3250!js_GetSrcNoteOffset+0x45a6
0012fa08 600856b5 js3250!js_GetScriptLineExtent+0x39e6
0012fa28 600b036f js3250!JS_NewStringCopyZ+0x44
0012fa40 600b3e93 js3250!js_FindProperty+0x26c5

The information for js3250 confirms (if the module name and the names of the export functions weren't enough for you) that it is the Mozilla Javascript implementation.

0:009> lmv m js3250
start    end        module name
60080000 600e9000   js3250     (export symbols)
   C:\Program Files\Mozilla Firefox\js3250.dll
    Loaded symbol image file:
        C:\Program Files\Mozilla Firefox\js3250.dll
    Image path: C:\Program Files\Mozilla Firefox\js3250.dll
    Image name: js3250.dll
    Timestamp:        Fri Nov 11 20:05:34 2005 (43753FDE)
    CheckSum:         00073A1C
    ImageSize:        00069000
    File version:     4.0.0.0
    Product version:  4.0.0.0
    File flags:       0 (Mask 3F)
    File OS:          10004 DOS Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04e4
    CompanyName:      Netscape Communications Corporation
    ProductName:      NETSCAPE
    InternalName:     JS3240
    OriginalFilename: js3240.dll
    ProductVersion:   4.0
    FileVersion:      4.0
    FileDescription:  Netscape 32-bit JavaScript Module
    LegalCopyright:   Copyright Netscape Communications. 1994-96
    LegalTrademarks:  Netscape, Mozilla

All of this is reasonably good evidence that javascript is the right place to start. I took a look at the exports for js3250 ("x js3250!*" in Windbg) — it appeared as though this module was implemented as a few dozen C-style exports.

I thought that a logical thing to look for was an "execute a javascript function" function of some kind, and there were a few exports named some variation of "Call function."

0:009> x js3250!*Call*Function*
6008541f js3250!JS_CallFunction (<no parameter info>)
60085464 js3250!JS_CallFunctionName (<no parameter info>)
600854ff js3250!JS_CallFunctionValue (<no parameter info>)

I set a breakpoint on all of these.

0:009> bm js3250!*Call*Function*
  1: 6008541f @!"js3250!JS_CallFunction"
  2: 60085464 @!"js3250!JS_CallFunctionName"
  3: 600854ff @!"js3250!JS_CallFunctionValue"

After resuming the program, I started hitting these breakpoints constantly. I did a sanity check and made sure that this didn't occur on a cleaner site (google), and this was the case. I didn't have any immediate success figuring out what script functions were being called. You can dig up the source for these API's if you like, but I'm guessing that the script is already processed into different data structures by the time the javascript engine gets here.

I looked at some of the stacks when these functions were called, and I noticed this pretty far down:

0012fc7c 00534e18 js3250!JS_EvaluateUCScriptForPrincipals+0x70

I hoped this would lead me to some bits of javascript pointed to from nearby locations on the stack, so I set a breakpoint there. When it was hit, I started dumping out strings (using "dda @esp" and and "ddu @esp") and found this:

http://espn-att.starwave.com/motion/fsp/fsp.js

As an only slightly educated guess, I used the Adblock extension to block this script.

After reloading the page, the problem was gone! There don't seem to be any site-breaking problems associated with turning this script off. This is all or part of the "ESPN Motion" business that tries to display sound and video on the site. Hey, ESPN: this is a terrible idea in the first place. Your website shouldn't start yelling at me when I visit it. It's no excuse for Firefox to A/V, but I thought everyone with a three-digit IQ stopped doing this in 1996.