Dan McKinley

Occasionally, I write things for Etsy's news blog thingamajig, the Storque. If this site isn't enough excitement for you, you can see what I am doing over there by following this link. End communication.

So here's my method for determining whether or not a particular platform is worthwhile.

I ask myself, "how would David Letterman summarize this?" I know this doesn't seem very scientific, but it's a mask I have years of experience wearing. Trust me on this one.

When the curtain is raised to reveal a total square dancing around with a sign that says, "it's simply XML!" I ask myself, "is this anything?" Then I reflect for a second or two and decide, dismissively, that no. This is nothing.

Lakin Wecker has an alternative approach that involves thinking carefully and considering the lessons of history. I will re-evalutate my approach if our results ever differ significantly.

In front of City Hall, a man plays the steel drums in front of the ZipCar tent. Increased access to the birthright of every citizen (the automobile) was advocated. Free watermelon was served.


Nearby a person reminds us (via bumper sticker) to drive safely, while parked in a painted bike lane.


The placard in his window lets us know that this person is a hero, rather than an ordinary disgrace.


Back at Etsy Labs, time for the deliveries (that's the sidewalk).


Also near the Labs, signs that Brooklyn's repressed SUV-owning minority are joining forces and taking back the sidewalks.


Meanwhile, on Brooklyn Bridge Boulevard, a woman is brutally killed.

So I wrote this replacement version of Condition using the native posix support. Event and Semaphore are both written in terms of Condition, so you can use this as a fast route to getting native versions of those synchronization primitives. (Note, though, that there is a native posix semaphore, so implementing it in terms of a condition variable is not really necessary.)

I have trained myself to ask, "why hasn't anybody done this before?" when writing this sort of thing. And as always there's a really good reason: for a percentage of applications that is probably very close to one hundred percent, the difference in performance between this and the pure python version (which is implemented using polling) is not going to amount to a hill of beans. That was indeed the case for the application where I was trying to put this to use, and I reverted to the much simpler python Condition.

But there was no way to know for sure that that was the case without trying this, so if you find yourself in a similar situation, here is the code.


#include "Python.h"
#include "pthread.h"
#include "structmember.h"

typedef struct {
  PyObject_HEAD
  int set;
  pthread_cond_t cond;
  pthread_mutex_t lock;
} ConditionObject;


static int cond_init( ConditionObject* self, PyObject* args,
              PyObject* kwargs );
static void cond_free( ConditionObject* self );
static PyObject* cond_acquire( ConditionObject* self );
static PyObject* cond_release( ConditionObject* self );
static PyObject* cond_wait( ConditionObject* self, PyObject* args );
static PyObject* cond_notify( ConditionObject* self );
static PyObject* cond_notifyAll( ConditionObject* self );

static PyMemberDef cond_members[] = {
  {NULL}
};

static PyMethodDef cond_methods[] = {
  { "acquire", (PyCFunction)cond_acquire, METH_NOARGS, "" },
  { "release", (PyCFunction)cond_release, METH_NOARGS, "" },
  { "wait", (PyCFunction)cond_wait, METH_VARARGS, "" },
  { "notify", (PyCFunction)cond_notify, METH_NOARGS, "" },
  { "notifyAll", (PyCFunction)cond_notifyAll, METH_NOARGS, "" },
  { NULL }
};

static PyTypeObject ConditionType  = {
  PyObject_HEAD_INIT(NULL)
  0,                         /*ob_size*/
  "_pthread_cond.Condition", /*tp_name*/
  sizeof(ConditionObject),   /*tp_basicsize*/
  0,                         /*tp_itemsize*/
  (destructor)cond_free,     /*tp_dealloc*/
  0,                         /*tp_print*/
  0,                         /*tp_getattr*/
  0,                         /*tp_setattr*/
  0,                         /*tp_compare*/
  0,                         /*tp_repr*/
  0,                         /*tp_as_number*/
  0,                         /*tp_as_sequence*/
  0,                         /*tp_as_mapping*/
  0,                         /*tp_hash */
  0,                         /*tp_call*/
  0,                         /*tp_str*/
  0,                         /*tp_getattro*/
  0,                         /*tp_setattro*/
  0,                         /*tp_as_buffer*/
  Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,        /*tp_flags*/
  "",                        /* tp_doc */
  0,                       /* tp_traverse */
  0,                       /* tp_clear */
  0,                       /* tp_richcompare */
  0,                       /* tp_weaklistoffset */
  0,                       /* tp_iter */
  0,                       /* tp_iternext */
  cond_methods,              /* tp_methods */
  cond_members,              /* tp_members */
  0,                         /* tp_getset */
  0,                         /* tp_base */
  0,                         /* tp_dict */
  0,                         /* tp_descr_get */
  0,                         /* tp_descr_set */
  0,                         /* tp_dictoffset */
  (initproc)cond_init,       /* tp_init */
  0,                         /* tp_alloc */
  0,                         /* tp_new */
};

static int cond_init( ConditionObject* self, PyObject* args,
              PyObject* kwargs ) {
  self->set = 0;
  int err = pthread_mutex_init( &self->lock, NULL );
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return -1;
  }
  err = pthread_cond_init( &self->cond, NULL );
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return -1;
  }
  return 0;
}

static void cond_free( ConditionObject* self ) {
  pthread_mutex_destroy( &self->lock );
  pthread_cond_destroy( &self->cond );
}

static PyObject* cond_acquire( ConditionObject* self ) {
  int err = 0;
  Py_BEGIN_ALLOW_THREADS;
  err = pthread_mutex_lock( &self->lock );
  Py_END_ALLOW_THREADS;
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return NULL;
  }
  Py_RETURN_NONE;
}

static PyObject* cond_release( ConditionObject* self ) {
  int err = pthread_mutex_unlock( &self->lock );
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return NULL;
  }
  Py_RETURN_NONE;
}

static PyObject* cond_wait( ConditionObject* self, PyObject* args ) {
  // for now timed waits are not supported (it's ignored)
  int err = 0;
  while( !self->set && err != EINVAL ) {
    Py_BEGIN_ALLOW_THREADS;
    err = pthread_cond_wait( &self->cond, &self->lock );
    Py_END_ALLOW_THREADS;
    if( PyErr_CheckSignals() ) {
      return NULL;
    }
  }
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return NULL;
  }
  Py_RETURN_NONE;
}

static PyObject* cond_notify( ConditionObject* self ) {
  self->set = 1;
  int err = 0;
  Py_BEGIN_ALLOW_THREADS;
  err = pthread_cond_signal( &self->cond );
  Py_END_ALLOW_THREADS;
  self->set = 0;
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return NULL;
  }
  Py_RETURN_NONE;
}

static PyObject* cond_notifyAll( ConditionObject* self ) {
  self->set = 1;
  int err = 0;
  Py_BEGIN_ALLOW_THREADS;
  err = pthread_cond_broadcast( &self->cond );
  Py_END_ALLOW_THREADS;
  self->set = 0;
  if( err != 0 ) {
    PyErr_SetFromErrno( PyExc_OSError );
    return NULL;
  }
  Py_RETURN_NONE;
}

static PyMethodDef module_methods[] = {
  {NULL}
};

PyMODINIT_FUNC
init_pthread_cond(void)
{
  ConditionType.tp_new = PyType_GenericNew;
  if( PyType_Ready( &ConditionType ) < 0 ) {
    return;
  }
  PyObject* mod = Py_InitModule( "_pthread_cond", module_methods );
  if( mod == NULL ) {
    return;
  }
  Py_INCREF( &ConditionType );
  PyModule_AddObject( mod, "Condition", (PyObject*)&ConditionType );
}

If I believed in that sort of thing, I would guess that it said "leave Alphabet City immediately."

Some people sing carols every XXX-mas, I get bored and write the Mandelbrot Set program in whatever my favorite language happens to be that year. I thought the brevity of the output this year (Haskell) was kinda neat.

import Graphics.UI.GLUT
import Control.Monad
import Data.Int
import Data.Complex

iterations = 400

x // y = fromIntegral x / fromIntegral y

-- Divides [a] into [[a], [a], ...] with each sublist of length n,
-- except the last sublist which has length <= n.
chunkify n [] = []
chunkify n xs = let (xs', rest) = splitAt n xs
                in xs' : chunkify n rest

-- Converts a coordinate in screen space to a vertex.
pix2vert (Size w h) (x, y) = Vertex2 ((3 // w * fromIntegral x) - 2.0)
                             ((2 // h * fromIntegral y) - 1.0)

-- List of all of the vertices that represent screen pixels.
vertices :: IO [Vertex2 GLfloat]
vertices = get windowSize >>= \(Size w h) ->
           return $ [pix2vert (Size w h) (x, y) | x <- [0..w-1], y <- [0..h-1]]

-- Gets the color for a number of iterations.
color3 r g b = Color3 r g b
getcolor :: Int -> Color3 Float
getcolor iter | iter == iterations = color3 0 0 0
              | otherwise          = color3 (amt*0.5) amt (amt*0.5)
              where amt = iter // iterations

-- Returns the number of iterations <= the maximum iterations of the
-- Mandelbrot set at the given vertex.
mandel (Vertex2 r i) = length . takeWhile (\z -> magnitude z <= 2) .
                       take iterations $ iterate (\z -> z^2 + (r :+ i)) 0

-- plots one point.
drawVert v = do color . getcolor $ mandel v
                vertex v

-- draws all the vertices in slices (to update the display while drawing).
display' chunks = do mapM_ (\vs -> do renderPrimitive Points $ mapM_ drawVert vs
                                      flush) chunks
                     displayCallback $= display

-- draws the whole fractal
display = do clear [ ColorBuffer ]
             displayCallback $= (vertices >>= display' . chunkify 256)
             get currentWindow >>= postRedisplay

main = do
   getArgsAndInitialize
   initialDisplayMode $= [ SingleBuffered, RGBMode ]
   initialWindowSize $= Size 1200 1024
   initialWindowPosition $= Position 100 100
   createWindow "Mandelbrot"
   clearColor $= Color4 0 0 0 0
   matrixMode $= Projection
   loadIdentity
   ortho (-2) 1 (-1) 1 (-1) 1
   displayCallback $= display
   mainLoop

Screenshot:

Although I haven't found this to be necessary nearly as often as Windbg was for me on Windows, it's still somewhat handy to be able to look at a Python daemon or multithreaded program in GDB. I've had to set this up a few times now, and I've forgotten the steps each time, so here they are in one place.

Step 1 - Get a debug build of Python.

You need a debug version of the interpreter. This is what I do from source on *nix:

./configure --prefix=/usr/local/dbg
make OPT=-g
sudo make install

sudo ln -s /usr/local/bin/dbgpy /usr/local/dbg/bin/python

Step 2 - Fix the botched .gdbinit that comes with the Python source.

The Python source comes with a .gdbinit file (Misc/gdbinit), but it's broken in 2.5. Apply these changes, which will fix pystack to 1) work and 2) work on threads other than the main thread.

@@ -119,8 +122,8 @@
 
 # print the entire Python call stack
 define pystack
-    while $pc < Py_Main || $pc > Py_GetArgcArgv
-        if $pc > PyEval_EvalFrame && $pc < PyEval_EvalCodeEx
+    while ($pc < Py_Main || $pc > Py_GetArgcArgv) && ($pc < t_bootstrap || $pc > thread_PyThread_start_new_thread)
+        if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx
         pyframe
         end
         up-silently 1
@@ -130,8 +133,8 @@
 
 # print the entire Python call stack - verbose mode
 define pystackv
-    while $pc < Py_Main || $pc > Py_GetArgcArgv
-        if $pc > PyEval_EvalFrame && $pc < PyEval_EvalCodeEx
+    while ($pc < Py_Main || $pc > Py_GetArgcArgv) && ($pc < t_bootstrap || $pc > thread_PyThread_start_new_thread)
+        if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx
         pyframev
         end
         up-silently 1

Note that the locals still won't work. I'll post a diff that fixes those when I get around to doing that. In the meantime it's not too hard to poke through the python structures to get them.

On OS X you can just copy the result of this to ~/.gdbinit. Some linux distros might be cranky about where you put this and where you launch gdb.


Step 3 - Launch your program with the debug interpreter and attach.

From the command line you can attach with "gdb /usr/local/bin/dbgpy [pid]." I like gdba mode in emacs (see below - M-x gdba, then the same kind of command line).

 

That site I have been working on in semi-secret for the last six or eight months is now live. Here it is:

http://feedeachother.com

Explanations, walkthroughs, etc. can all be found there.

Most of the credit for this thing really goes to Udi Falkson, one of my old Cornell friends who actually had the balls to quit a great job at Yahoo! to work on this full time. The lion's share of the code is his. I've seen it, and trust me, his dried blood and guts are all over this thing. At least I am pretty sure that's what it is.

Please check it out if you have not already, and add me as a contact. I am very proud of this site--working on something that you personally love and want to use is really satisfying.

People that I would like to thank, some of whom are pretty unlikely to ever notice this unless they google themselves:

• Darrel Herbst, an early adopter and the source of a lot of typically brilliant feedback.
  
  
• Katya Bassil, for being the best tester ever. You can thank her for Cute Overload working flawlessly.
  
  
• Evan Reiser, for tons of good feedback, visual nitpickery, and for scouring the interweb for competitors.
  
  
• Ron Eigen, for general enthusiasm and inviting a million people.
  
  
• Rich Hedge, for awesome legal advice.
  

Hey, by the way, the last post is likely to be my last Windows-oriented post for the forseeable future. I have recently quit the financial sector and will be starting a fantastically cool new job here in a few weeks. I no longer have a Windows machine at home at all, so unless I take up Mono, expect a lot of posts that mention emacs from now on. I think this will be for the best, since I've been kind of bored and nonplussed with .NET stuff lately, F# notwithstanding.

Giving up on Microsoft development entirely was a little bittersweet until someone showed me XMLScript a week or two ago--clearly, a lot of people in that world are out of their god damned minds. Best of luck writing xhtml-serving web applications defined in xml markup with client script written declaratively in more xml, also exposing xml web services to rich client applications defined largely in XAML, updated automatically as defined in xml configuration files, to those of you lucky enough to get to do this.

Here's a problem:

1. You have an application that's hanging permanently or temporarily.
2. The hang does not occur in any synchronization code that you have written yourself.
3. You are writing a windows forms application, or any kind of application using COM interop.

Let me present an educated guess as to what is happening. Note to people getting here by googling: there are actually many other reasons this might happen in addition to the one I am going to tell you about. I merely suggest that you spend more time thinking before being led too far down a rabbit hole by what I am about to write. Experience has taught me that people tend to lock on to one internet analysis (say, an unrelated KB article mentioning the same exception text) and waste a lot of time on it. None of the other sites seem to be nice enough to explicitly warn you about your own tendencies--I am, you should thank me.

With those disclaimers, here is my psychic debugging response:

1. You are inadvertently calling a method on an STA object created on a different thread.
2. That other thread is busy doing something else.

For people that managed (har har) to avoid COM development, or stayed completely in the insulated VB6 world, I will now explain briefly everything you really need to know about threading models in this, our fantastic futuristic age.

Methods on single threaded apartment (STA) objects are synchronized. The object receives all calls to its methods on the thread that owns it. This is achieved by creating a hidden window that receives messages for any calls made on those objects. Multi-threaded apartment (MTA) objects behave more or less like any .NET object you create: calls are not automatically synchronized, and you are responsible for doing any synchronization necessary yourself.

So in order for calls to STA objects to work correctly, the thread that owns them has to be free to run its message loop a sufficient percentage of the time. Running a computationally-intensive process on an STA thread is a great way to hang the other threads in your application, if you are careless or ignorant of where the STA calls are.

If you attach with Windbg and dump out the native stacks (~*k), you will probably see threads in calls to WaitOne that got there through functions named things like "SendReceive" above some interop marshalling code if this is indeed your problem. If you are lucky you might see a function called "GetToSTA," however, the stacks unfortunately do not always make things that easy.

This is one of those things that I have seen a bunch of times, but seems to be on the radar of an increasingly small number of people. It often takes me longer than it should to figure it out, too. If you are doing any kind of practical Windows development, you still need to understand the basics of STA/MTA. That's because any development that is "practical" will almost necessarily involve interoperating with old code.

Here's a related post that I wrote, about how STA objects can result in blocked finalizer threads.

Hey, nifty, fake rocks (click for larger versions).





I assume it's either related to this, or it's "Movie Where Manhattan Explodes #957." (They are getting less and less creative with the titles these days.)

Let me be the first to say that this is probably a really bad idea, unless you are very desperate.

On the other hand, this was really fun to write.

Today I wrote some code that patches the first few bytes of user32!MessageBoxExW, in order to keep a pesky third-party library from showing a prompt when nobody was around to click on it. This overwrites the start of that function with a relative jump to one of my own, which then forwards to a managed function that logs what the message box was trying to tell us.

Actually, this was for a friend's program. I would probably shy away from doing this if I weren't only indirectly responsible.

void InstallHook()
{
    HMODULE hUser32 = LoadLibrary(L"user32.dll");
    PROC pMsgBox = GetProcAddress(hUser32, "MessageBoxExW");
    FailIf(pMsgBox == NULL);

    // Need to change the memory protection for the
    // (read/execute) page in user32.dll before we
    //write to it.
    MEMORY_BASIC_INFORMATION mbi = {0};
    VirtualQuery(pMsgBox, &mbi,
        sizeof(MEMORY_BASIC_INFORMATION));
    FailIf(!VirtualProtect(mbi.BaseAddress, mbi.RegionSize,
        PAGE_READWRITE, &mbi.Protect));

    // Copy in the jump instruction... you could of course
    // opt to not write this in assembly, but if we've gone this
    // far, what's the difference?
    __asm
    {
         // edi <- user32!MessageBoxExW
        mov edi, pMsgBox;              
       
        // write the opcode for JMP rel32
        mov byte ptr [edi], 0xe9;       

        // eax <- _MessageBoxEx, then
        // eax <- _MessageBoxEx - user32!MessageBoxExW
        mov eax, offset _MessageBoxEx;   
        sub eax, edi;                   
       
        // One byte for jmp, four for the address.
        sub eax, 5;                       
        inc edi;                       
        stosd;
    };

    // Restore the old protection
    DWORD oldProtect = 0;
    FailIf(!VirtualProtect(mbi.BaseAddress,
        mbi.RegionSize, mbi.Protect, &oldProtect));
}

With that in place it's not hard to erect some managed scaffolding around it, and nobody needs to know your secret. Until they try to run it on an x64, I guess:

static void Main(string[] args)
{
    HookMonkey.Install(delegate(MessageBoxInfo mbi)
    {
        Console.WriteLine(mbi.Text);
        return DialogResult.OK;
    });
    MessageBox.Show("foo bar");
}

You can download some sample code here.

Don Box says:

As a WPF user, I spend at least as much time reading and writing XAML as I do reading/writing C# code that does WPF-isms. I do spend a lot of time in C#, but little of it is WPF specific, which arguably is one of the strengths of WPF’s data/content facilities.

In my opinion, the sooner folks get thrown into the XAML pool the sooner they learn to swim.

(Emphasis mine.)

I thought the whole point of selling our souls to the XML devils was that development tools would deal with that slop for us.

Forget swimming. I am having trouble seeing any non-masochistic reason for me to learn this at all. I am just not getting through the Microsoft marketing on this one. I say this as somebody with deep-to-extremely-deep knowledge of most of the Microsoft tools of the last 10 years--I'm not Don Box, but I'm also no slouch.

Give me one good reason why I should spend my spare time learning an API that looks like this. (And it would be my spare time. I am still living here in the real world, where it still seems to be occasionally more practical to write Windows platform code in C++, for god's sake.)

Wikipedia reassures me--in a rather hilarious, pleonastic way--that this is "simply XML."

[A] key aspect of the technology is the reduced complexity needed for tools to process XAML, because it is simply XML. As a result, a variety of products are emerging, particularly in the WPF space, which create XAML-based applications. As XAML is simply based on XML, developers and designers are able to share and edit content freely amongst themselves without requiring compilation.

(Emphasis mine.)

Show me one XML-based API that does not, in one way or another, prove this to be an oxymoron. (See also: "Java programmer writes application which reads 243 XML files on startup then wonders why it takes 30 seconds to start." Thanks, Slava.)

Sometimes trite observations become trite because they are obviously correct. Who created the creator? Why re-invent the s-expressions that have been around since the 50's, but poorly? Et cetera. In these cases the fact that the criticism is well-worn does nothing to detract from it.

Maybe I am just being a grizzled old fogey (at the incredibly advanced age of 27.)  It wouldn't be the first time that I have been described as a crotchety septugenarian in the body of a young adult. If I am way off the mark here, please enlighten me.

Update, May 20th: Jon Harrop avoids my bitching and gives solutions. See XAML or F#.

I suppose I finally had enough of the carpal tunnel caused by typing '\' to delimit directories (yes, powershell helps a little, but not enough).

Or, I decided I knew enough about Windows for now and needed to branch out.

Or maybe I was just in a rut.

Well, I'm happy about whatever it was that caused me to make my latest impulse purchase.