Posts Tagged ‘WinDbg’

Using WinDbg to Log Exceptions

Since I discovered that I am currently—and quite inexplicably—the #1 Google Blog Search result for the term “WinDbg,” I decided that I would attempt to actually produce some WinDbg content.

If you are not familiar with WinDbg, check it out with the rest of the Debugging Tools for Windows here.

To get things started, I thought I would post some examples that I’ve found useful lately.

Introduction

I use this technique quite a bit when I’m doing QA work on an application. I can think of a bunch of other scenarios where it might be useful, but one chore in particular that is well suited for this is finding ASP.NET page compilation exceptions.

There isn’t really a good way to clean up page compilation exceptions in a massive ASP.NET application. Why would you bother? Well, there are a few reasons.

  1. They encourage you to disable break-on-exceptions in a debugger like VS.NET, since you’ll be inundated with a lot of other people’s exceptions when trying to attach to the worker process. If you do that, you might miss swallowed exceptions or exceptions on background threads that you would have otherwise noticed and fixed.
  2. They might indicate a problem in the HTML being sent back to the user.
  3. They cause your application to take longer to start up.
  4. Some of us are just sticklers for the details, ok?

So, one good way to trim them down to a bearable amount is to log all of the exceptions that happen when the app starts.

Getting Started

The first thing you need to do is open up a log:

0:003> .logopen c:\temp\exceptions.txt
Opened log file 'c:\temp\exceptions.txt'

And if this is managed code you’ll want to load the SOS extension:

.load clr10\sos

We’ll use a breakpoint with a custom command to print out the stack when the exception occurs.

0:003> bp kernel32!RaiseException "!clrstack; g"

The !clrstack command will print out the stack, and g (go) resumes execution. If the application is unmanaged, you will want to replace !clrstack with one of the k (display stack) commands. Both managed and unmanaged code will use the same underlying call to the kernel32!RaiseException function.

Once you have this set up, you’ll see a trickle or a flood of stacks, depending on the relative health of your app. They’ll probably be more interesting than this:

(848.d18): CLR exception - code e0434f4d (first chance)
Thread 0
ESP         EIP
0x0012f648  0x7c81eae1 [FRAME: HelperMethodFrame]
0x0012f674  0x02e000a0 [DEFAULT] Void CsConsoleApp.Program.Main(SZArray String)
  at [+0x48] [+0xf] c:\src\scratch\csconsoleapp\class2.cs:26
0x0012f8b0  0x791d94bc [FRAME: GCFrame]
0x0012f9b0  0x791d94bc [FRAME: GCFrame]
0x0012fa94  0x791d94bc [FRAME: GCFrame]

But you get the idea.

Adding More Information

Now, notice that in the case of a managed exception, WinDbg didn’t give us much information, other than the fact that it occurred:

(848.d18): CLR exception - code e0434f4d (first chance)

It would be nice to get the exception text, right? You can do that by changing the breakpoint to this:

0:000> bp mscorwks!JIT_Throw "du poi(@ecx+10)+c; !clrstack; g"

Homework: explain this command and how I came up with it. I will post the answer soon.

Now we’ll get something more useful:

(848.d18): CLR exception - code e0434f4d (first chance)
00aaab88  "You stink!"

Other Mods

One thing you might want to do in the case of rarer exceptions is create a dump of the process. This is especially useful if your issue only happens in production. The simplest way to do that is very similar to what we just did:

0:003> bp kernel32!RaiseException ".dump /ma /o c:\\temp\\myapp.dmp; g"
breakpoint 0 redefined

The /ma switch produces a full dump. I set this breakpoint to overwrite a single dump file.

More homework: modify the breakpoint to create a series of dumps. The answer will be posted in the near future.

A Word for OllyDbg

OllyDbg is a very nice tool for debugging other people’s code. While I definitely still prefer WinDbg in most situations, OllyDbg is great for stepping through assembly.

OllyDbg

I had a good reason to use it last week. I have an old VB6 application that needs to interop with new .NET applications. The VB6 code is using an ancient COM library for encryption; this library is fairly opaque in regards to what it is actually doing.

Without giving away too many details, there’s no way I can move the old code to over to anything else. I wish the original author had just imported the advapi32 Crypt* functions, but it’s too late for that now.

So, needing to decrypt data coming from the VB6 side, I was left with a few choices. I could just use COM Interop and reference the old library in my new code. But I wasn’t really happy with that, mostly because of the complexity that it adds to deployment.

I was familiar enough with the Crypto API to know that the COM library couldn’t be doing anything too complicated. This is where OllyDbg comes in.

I stepped through the library call in assembly, stopping when it made Crypt* calls. I got the parameters from the stack, and wrote them all down.

From there, I wrote a quick C++ console app to test out recreating the calls and decrypting some sample data. As it turned out, the specific algorithms that the COM library was using weren’t exposed in System.Security.Cryptography, so the C# version I ended up with had to use P/Invoke with advapi32.

But anyway, I got rid of an annoying dependency. A very satisfying hack. You can read about an even better one here—Lee Holmes uses OllyDbg to crack a program to run as a non-admin.

Annoy Your Friends and Colleagues with Global Flags

Step 1

Open gflags.exe. It comes with the Debugging Tools for Windows.

Step 2

On the Image File tab, enter the name of a popular executable. Suggestions:

  • devenv.exe [developers]
  • outlook.exe [human resources]
  • excel.exe [bankers and finance dweebs]
  • iexplore.exe / firefox.exe [catch-all]
  • Step 3

    Press tab to refresh the form. Check “Debugger,” and enter sol.exe in the textbox. Press ok.

    Step 4

    Bask in the hilarity of your sophisticated prank.

    gflags.exe

Things I need

It would be really great if any of these things existed. Some of them might, but I’ve been unsuccessful in finding any of them.

  1. A utility that finds/cleans files that are in Visual Sourcesafe, but are not found in Visual Studio projects and/or solutions. When a file is ‘deleted’ in VS2003, it’s not removed from VSS. Repeated on a vast scale, this gets annoying and throws a wrench into some other things that I’m trying to do.
  2. A NAnt task that lets me use csc/vbc for all files in a Visual Studio project. I need the command line arguments (ie, /debug:pdbonly) for both of those, and the convenience of the solution task. (Of all of these, it seems like this is the most likely to exist and I just haven’t found it.)
  3. DebugEngine extensions intended for ASP.NET apps. They should be able to show me the pages that are running, the HttpContext for each request, the items that are in session, and so forth.
  4. Someone who can take difficult, complex programming tasks from me. Ideally they would be smart enough that I would trust them implicitly, rather than agonize about the projects on a daily basis.
  5. A Visual Studio project type for managing a number of XML files. By that I mean, a bunch of NAnt scripts. Actually, any tool that has a solution-like treeview and a document outline utility will work.
  6. PowerCollections, but for .NET 1.1. (Yeah, yeah, it might not be too hard to port them myself).
  7. An “Add New Item…” item in Visual Studio that has NO default name set (like “Class1.cs”), NO autogenerated comments, and has internal access by default. If I call the item IAnything, it should figure out that the item is an interface and not a class.

I’ve had some success lately eliciting comments from knowledgeable googlers, and I have high hopes for this post.

Getting some grease under my fingernails

I decided I needed some debugger extensions specifically intended for ASP.NET hang dumps. Psscor has a few nice ones but I figured I would learn a lot in the attempt, anyway.

Here’s how the project is breaking down.

Days 1-2: Messed around trying to learn how to use the build utility that comes with the Windows 2003 DDK. This would have been faster if I had looked at a makefile more than once in the previous five years.

Day 3: Toyed with the idea of using LoadLibrary on psscor and filtering the output, but instead decided that I was insane.

Days 4-5: Got pretty familiar with writing an extension on my own, and what the implementation details would be. It was becoming pretty clear that I would need to get some info on the structures if I was to have any hope of getting finished.

Days 5-7: Downloaded the SSCLI and realized that it’s got a (pretty old) version of SOS.dll in it. I have a few moments of euphoria here and radically switch approaches: I will now start with this and extend it to suit my needs.

I pretty quickly found out that this DLL is assuming the SSCLI execution engine and won’t work with the Microsoft production ones (mscorsvr in my case, or mscoree). No problem, I assume that won’t take too long to fix.

The problem is that this thing is filled with code like this:

// Note this is possible to be spoofed, but pretty unlikely
// call XXXXXXXX
if (spot[-5] == 0xE8) {
    move (*whereCalled, retAddr-4);
    *whereCalled += retAddr;
    //*whereCalled = *((int*) (retAddr-4)) + retAddr;
    if (*whereCalled < 0x80000000 && *whereCalled > 0x1000)
        return;
    else
        *whereCalled = 0;
}

(your guess is as good as mine).

I’m not sure if they had their best programmers working on this thing. I don’t know. Perhaps they just never expected someone like me to look at it in depth.

So that’s where I’m stuck for the moment. I’m going through it, commenting it and trying to understand it. I will eventually refactor it quite a bit. At the moment there are a lot of very long, very hard to follow functions in it. But, it’s good to look at some unmanaged code for a change.

Managed Debugging with WinDbg, Part 1 of N

There seems to be relatively little information about WinDbg available, so I will try to post some things as I figure them out myself.

We had a severe-to-extremely-severe production problem last week, and my recent activity with WinDbg was another example of “learning with a gun to your forehead.”

Getting the Right Extensions

If you are doing any kind of managed debugging, you will want the current set of extensions which can be found here.

The simplest thing to do is just drop the .dlls in the install directory for WinDbg. The two dll’s in this set that will be most important to you are psscor.dll and sieextpub.dll. Psscor has a lot of tools for dumping the contents of managed objects, and the second has some powerful functions for showing application and thread state.

There is a third extension, sos.dll, whose functionality is mostly overlapping with psscor.

Extensions Basics

Load an extension into WinDbg like so:

0:000> .load psscor

At any point, you can see the extensions you have loaded with this command:

0:000> .chain

All of the extensions I’ve mentioned come with help commands. The help for the topmost extension in the chain can be called like this:

0:000> !help

But you can always refer back to extensions further down in this way:

0:000> !sieextpub.help

An Example – Debugging an ASP.NET Hang

This was the scenario we found ourselves in last week. The first thing you will need to do is get a hang dump of the worker process using AdPlus, which I don’t have time to cover here. However, it is relatively straightforward.

Once we’ve got the dump, the most obvious thing to try is to see what the process is doing. To do that, load psscor and use this command:

0:000> ~*e!clrstack

This will dump out the managed stack of all of the threads in the process (the ~*e means that we want to iterate through all of the threads and perform the specified action for each).

That will give us a general idea of what is going on. The stack of one or more particular threads is bound to be interesting, and we can narrow it down to the stack trace of a single thread using:

0:000> ~113e!clrstack

Here I’ve replaced * (all threads) with a single thread, 113. Assuming this is a managed thread, you should see some output like this.

Thread 113
ESP         EIP     

…

0x0dc5f6cc  0x0fa3bb17 [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequestMain()
0x0dc5f710  0x0fa3aedf [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequest()
0x0dc5f74c  0x0fa3a94b [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)

…

For this example, I’ll just show how to figure out which page in the application is executing. Since we’re running inside a page class for a lot of the response, the this pointer should be all we need.

Grab two stack pointers (the ESP register) and use the psscor.DumpStackObjects command, or dso for short:

0:000> !dso 0x0dc5f6cc  0x0dc5f74c 

Thread 0
ESP/REG    Object     Name
0x0dc5f6cc 0x31f0d208 System.Collections.Specialized.HybridDictionary
0x0dc5f6d0 0x0371c55c _ASP.incomeStatement_aspx
0x0dc5f6d8 0x0371c55c _ASP.incomeStatement_aspx
0x0dc5f714 0x31f0d208 System.Collections.Specialized.HybridDictionary
0x0dc5f720 0x070ca198 System.Globalization.CultureInfo
0x0dc5f724 0x070fb010 System.Threading.Thread
0x0dc5f728 0x0371c55c _ASP.incomeStatement_aspx
0x0dc5f73c 0x31f0d208 System.Collections.Specialized.HybridDictionary

Bingo – in this case, _ASP.incomeStatement_aspx is an instance of the page class.

Note that the first argument to dso is the upper stack pointer, and the second is the lower stack pointer. I’ll post some more stuff when I have time.