Performance impact of running under MDbgEngine

As promised, this post will present the results of my investigation into the performance impact of running under the simplest possible debugger written using MDbgEngine.

Timing method

For timing of all the samples, I used this method inside the debugee:

private static void Time(Action action, int count)
{
    var sum = 0L;

    for (var i = 0; i < count; i++)
    {
        var s = new Stopwatch();
        s.Start();
        action();
        s.Stop();
        sum += s.ElapsedMilliseconds;
        Thread.Sleep(1000);
    }
    Console.WriteLine(sum/count);
}

To minimize the impact of random system events, I run all of the samples 10 times, with a 1 second pause between each invocation, and then took an arithmetic average of the execution times.

CPU-bound code

private static void CPU(int count)
{
    Array.Sort(Enumerable.Range(1, count).Reverse().ToArray());
}
count = 1 000 000 count = 10 000 000
No debugger 106 ms 1048 ms
Visual Studio 105 ms 1085 ms
MDbgEngine 112 ms 1067 ms

Conclusion: Running under a debugger has no impact on CPU-bound code. All measurements are within the margin of error.

IO-bound code

Base on the above findings, and a personal hunch, I decided to skip this one.

Exception-bound code

This test should prove most interesting, as it’s the only that has code really interfacing with the debugger.

The sample code throws exceptions in a loop:

private static void Exception(int count)
{
    for (int i = 0; i < count; i++)
    {
        try
        {
            throw new Exception();
        }
        catch
        {
        }
    }
}

And the debugger inspects them and does nothing:

process.PostDebugEvent +=
    (sender, e) =>
        {
            if (e.CallbackType == ManagedCallbackType.OnBreakpoint)
                process.Go();

            if (e.CallbackType == ManagedCallbackType.OnException2)
            {
                var e2 = (CorException2EventArgs)e.CallbackArgs;
            }

            if (e.CallbackType == ManagedCallbackType.OnProcessExit)
                stop.Set();
        };
count = 1 000
No debugger 41 ms 1.0
Visual Studio 6380 ms 155.6
MDbgEngine 1322 ms 32.2

Conclusion: Going back and forth between the debugger and its debuggee, causes a performance drop of 2 orders of magnitude. I guess Visual Studio is more because of the Output window, which logs all exceptions.

The question is: how many exceptions does code generate on average? Instead of guesstimating, I decided to guessmeasure. All of the code indexed by SymbolSource, including only the most recent version of each project, consists of more than 1.5 million lines of code (that’s where I stopped counting). Of the lines used for this measurement, there were 12 thousand lines that contained the word “throw”. Therefore, I have guessmeasured, that all code is 0.8% exception on average.

Final conclusion: running under MDbgEngine causes a 24.8% drop in performance.

Would you be willing to take that, for the promise of gaining access to detailed call stacks and being able to generate minidumps for really hard to catch bugs?

Personally, I’ll shout a loud yes for all development and integration, and for occasional production too.

Advertisements
Leave a comment

2 Comments

  1. Mike Volodarsky

     /  March 15, 2012

    Hey Marcin,

    We are doing a lot with debugging, and would love to play with PADRE if you are interested in chatting about it over email.

    Best,
    Mike

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: