Moving to the new SymbolSource engine

A few months ago I promised we would be launching a new version of SymbolSource. Today I’m very happy to make the first part of the new service public, the one you’ve been waiting for most impatiently.

A new package repository has been deployed to production at and contains all packages from (around 50 thousand of them!). In other words this is the new complementary service to, running on a codebase rewritten from scratch – something we needed to do to solve all of the performance issues you have been facing with SymbolSource in the past.

What about the SymbolSource repository for MyGet, or the other company instances, you might ask? Those will be migrated to the new service too, and I will report as soon as we will have done that.

For now let’s see how the new service operates. You can try all of these commands yourself, and update all of your scripts and debuggers accordingly.

Publishing a symbol package

To push a symbol package run this command:

> nuget push SymbolSource.DemoLibrary- a0f4e24d-851f-4327-a9ec-181d335e7e95 -source
Pushing SymbolSource.DemoLibrary to ''...
Your package was pushed.

The command will succeed immediately without any timeouts, it just sends data to Azure storage. Packages are queued and processed asynchronously. That means that you will need to wait a bit before the symbols and sources become available, but we’ll be monitoring the queue length to make sure it never takes more than a few minutes.

Waiting and refreshing isn’t much fun, so we included a simple notification service, which you can use directly from the command prompt. Just append your Twitter username to the API key after a slash, like  this:

> nuget push SymbolSource.DemoLibrary- a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source
Pushing SymbolSource.DemoLibrary to ''...
Your package was pushed.

You’ll get tweeted by our friendly SymbolSource Bot (@smbsrc) when the package starts processing and when it finishes. The tweet will also tell you if anything went wrong, in which case @SymbolSource will also be mentioned to alert us about the issue.

One thing you’ll notice immediately is that we’ve added a suffix to the version number. Each upload to SymbolSource is identified separately. We then do a form of reference counting on all of the symbols and sources, so there is no cost to that. But in case you upload different builds with the same version number, all of them will be available to clients. Unless of course you decide to delete a package. Which is now a fully supported self-service operation!

Deleting a symbol package

If you know the SymbolSource version string of your package, deleting is as simple as running this command:

> nuget delete SymbolSource.DemoLibrary a0f4e24d-851f-4327-a9ec-181d335e7e95/@TripleEmcoder -source
Deleting SymbolSource.DemoLibrary from the ''.
SymbolSource.DemoLibrary was deleted successfully.

Once again NuGet will lie a bit, as this is an asynchronous operation. But you will get the same friendly tweets if you include your username with the API key, as I did above.

But what if you didn’t push with Twitter notifications enabled, and are now at a loss as to what the version string is? No worries, you can query SymbolSource as any other package source and find out what the version is.

Before we talk about that, however, if you really want to remove your content from SymbolSource, run this too:

> nuget delete SymbolSource.DemoLibrary a0f4e24d-851f-4327-a9ec-181d335e7e95 -source,all,original
Deleting SymbolSource.DemoLibrary from the ''.
SymbolSource.DemoLibrary was deleted successfully.

SymbolSource saves a copy of all uploaded packages, in case there is a change in the indexing algorithm and we need to reprocess. They are hosted in subfeed called original – see below for an explanation of what that is.

Listing symbol packages

By now you probably have already guessed the command to list packages from SymbolSource:

> nuget list SymbolSource.DemoLibrary -source
No packages found.

Well, that result might seem unexpected. But remember that we added an additional version string and therefore promoted all packages to semantic versioning. NuGet treats those as prerelease packages. A quick fix:

> nuget list SymbolSource.DemoLibrary -prerelease -source

There it is! Just remember about that NuGet will only show the most recent package, but you can show all of the uploads:

> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source

All of this works in NuGet Package Explorer too! As an exercise try downloading one of those packages. What you will get is a status package, useful in determining what symbols and source actually have been indexed by SymbolSource. More on this next time.

And in case you’re wondering… That version string is just a timestamp, encoded into an alphabet-based number system to save length. Unfortunately NuGet imposes a 20-character limit on all semantic versions.

Advanced listing by status and ownership

In my last post I also mentioned subfeeds, which let you query by status and ownership. To recap here are all of the statuses – they’ve slightly changed since our original spec:

  • new – not yet opened, random package name and unknown version,
  • original – a copy of the uploaded package, without any processing, useful if we improve the indexer and need to resubmit packages without any users’ actions,
  • indexingqueued – indexing will soon start asynchronously,
  • indexing – processing is in progress as you are looking at the list,
  • succeeded – the package has been indexed without any issues,
  • deletingqueued – deleting with soon start asynchronously,
  • deleting – deleting is in progress as you are looking at the list,
  • deleted – the package has been completely removed from the index,
  • partial – something went wrong, either during indexing or deleting, and some symbols and sources might be available, but some may not,
  • damagednew – package could not be read with a standard NuGet library,
  • damagedindexing, damageddeleting – these will probably indicate an error in SymbolSource.

For example, if you wanted to confirm that a package has been deleted, you would need to run this command:

> nuget list SymbolSource.DemoLibrary -prerelease -allversions -source,all,deleted

As you can see, the list contains a package we did delete a few paragraph earlier – version There are however also some old versions, which I deleted before writing this post. I can now delete them again, sort of, to simply remove the status package. Note that at this point symbols and sources have already been deleted, this is just similar to cleaning log files:

> nuget delete SymbolSource.DemoLibrary a0f4e24d-851f-4327-a9ec-181d335e7e95 -source
Deleting SymbolSource.DemoLibrary from the ''.
Failed to process request. 'Not Found'.
The remote server returned an error: (404) Not Found..

A 404 error is a bit odd, isn’t it? Well not exactly. Remember, we have already deleted this package. It doesn’t show in the feed. To remove the status package we need to address the subfeed specifically.

> nuget delete SymbolSource.DemoLibrary a0f4e24d-851f-4327-a9ec-181d335e7e95 -source,all,deleted
Deleting SymbolSource.DemoLibrary from the ',all,deleted'.
SymbolSource.DemoLibrary was deleted successfully.

If you don’t specify a subfeed at all, SymbolSource defaults to succeeded state and all packages (as opposed to own packages). The first comma separates the feed name, which in case of is empty. Things will be different for, but more on that in a future post.

So far we have listed all packages, regardless of who uploaded them. Of course pushing and deleting is permitted according to package ownership. But you can also limit the packages listed by a query, using the following command:

nuget list -prerelease -allversions -source,own,succeded
Please provide credentials for:,own,succeded
UserName: @TripleEmcoder
Password: ************************************

SymbolSource will ask for credentials this time, as it needs to known who the user is, to list only the right packages. You can pass anything as the username, but if you give us something meaningfull like a Twitter handle, we will be able to get back to you in case of problems. The password is your NuGet API key.

Debugging in Visual Studio

And last, but not least, the new URL for Visual Studio is…

Just replace the old URL in your Visual Studio options and you’re done. Happy (and fast) debugging 🙂

Wrapping it up

In this post I described the core features of our new SymbolSource engine. I hope you will find the new service much faster and much more stable than the previous one. We will be gradually upgrading other parts of the service over the next weeks and months. If you have any questions or feedback please leave them in the comment section.

The Orchard packaging experiment

I’ve written about plugins based on NuGet before. One piece of software that manages its pluggable parts with NuGet is Orchard – a CMS based on .NET and MVC, which from the experience of my company, Caliper, also serves as a nice framework for building web applications without any CMS functionality.

There’s a couple of things that Orchard solves for you, that make it great as a general framework:

  • database connectivity (with NHibernate), including out-of-the-box support for local, automatic SqlCe databases,
  • data migrations, for upgrading table schemas with new releases of your web applications or modules,
  • theme and module management, with the use of NuGet packages and feeds,
  • dynamic compilation, which lets you do in place changes to your web site or application, even through FTP, and see changes instantenously.

Orchard is distributed as a ZIP file that you can deploy as an application in IIS or open as a web application in Visual Studio. Module and theme packages are only managed through Orchard’s web UI, which gets them from a special Orchard Gallery Feed URL: If you open a module from that feed you’ll see that it contains all of its sources (as in *.cs files), for Orchard’s dynamic compilation to take care of. Open-source at its best. Or is it?

As much as we love having access to sources at Caliper, we love having them on-demand in our debugging sessions even more – instead of moving thousands of them around on disk for no reason. And we all have SymbolSource to handle that for free 🙂

So we set out to try and make Orchard and its modules more vanilla-nuggety and symbol/source server friendly.

Step 1: Packaging Orchard

First, we decided to try putting Orchard itself into packages. The end-result experience should speak for itself:

  1. Open Visual Studio and create an ASP.NET Empty Web Application (yes, no MVC, no Web Forms, no nothing).
  2. Add as a new package source to Visual Studio (yes, SymbolSource can host and serve NuGet packages as a package repository)
  3. Install the Orchard package from this feed in the empty project.
  4. Run the project with F5. You now have the Orchard setup page ready for action.

Things to note, as this should be considered work-in-progress still:

  • All standard modules and themes are downloaded as dependencies. We haven’t tried yet to determine what subset is needed for setup, and which of them could be installed an on as-needed basis later with NuGet in Visual Studio or Orchard through the web UI.
  • External dependencies are taken from the official NuGet feed only if they were taken by the Orchard team as-is, without modification or recompilation. All the rest is in Orchard.External.
  • SqlCe native binaries are not included yet. But you should be good to go on your development machine, as you probably have in installed globally in your system.
  • We haven’t yet figured out how to do versioning, so all packages have strict dependencies (==, i.e. [x.y.z] in NuGet).
  • You can easily create your own binary module packages that take other Orchard modules as NuGet dependencies and get compiled with the right references.

Now for the best part:

  • No *.cs files anywhere! Less files is more speed, less waiting, less transfers, less comparing on updates, less temptation to modify modules in place.
  • Debuggable – with SymbolSource! All symbols (PDB files) and sources are indexed and accessible with Visual Studio using the regular, public SymbolSource URL. This applies only to Orchard.Framework, Orchard.Core and Orchard.Web projects at the moment, but all Orchard modules will follow soon.
  • No dynamic compilation! No more screwed up assembly names and WCF services unable to load. Yes, you can disable dynamic compilation in configuration, but then you’re still required to compile modules and cleanup sources yourself.
  • All views are included in the main project as content files, so it’s easy to find, edit or copy them to your theme.

Step 2: Repackaging modules

Since a lot of the standard Orchard distribution is built using modules, we had to figure out how to transform them form source form into binary form. We have a small application that interfaces with NuGet and MSBuild to get the modules, guess their interdependencies from project references, then compile and repackage them. All of the standard modules have been uploaded in repackaged, binary form to the SymbolSource Orchard repository.

So far we haven’t touched any contributed modules from the Orchard Gallery, but our plan is to either create a SymbolSource gateway that would do that compilation on the fly (module authors would decide to push to it additionally), or monitor the gallery feed and repackage new modules into our Orchard repository automatically.

What do you think?

All the tooling that we created for this is available on GitHub: To run it, just clone, open a Visual Studio prompt and do msbuild Orchard.proj. The output will be in pkg (main Orchard packages) and pkgbin (module packages).

A world of debuggable open-source software – Part 3: Applications

In my last post, A world of debuggable open-source software – Part 2: Plugins, I used NuGet Package Explorer as an example of an application that uses NuGet-based plugins, and how that makes it very easy to publish their symbols and sources to SymbolSource, for a great debugging experience in case anything goes wrong at runtime. If you’re interested in creating your own plugin system based on NuGet, have a look at this post by Aaron Powell: Creating a NuGet-based plugin engine.

I also mentioned in that post that you can push any PDB files and sources to SymbolSource, including those of entire applications. Because NuGet is still the easiest way of pushing symbol packages, what we would need is a NuGet-based method of distributing programs. Fortunately such tool exists already and is called Chocolatey. If you haven’t heard of it before, here’s how it’s described by its authors:

Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with windows in mind.

In this post I will show you how Chocolatey and SymbolSource can be used to publish open-source applications with full debugging support, allowing users with programming skills to analyze problems through code and provide much more useful bug reports. Or even patches and pull requests!

A world of debuggable open-source software – Part 2: Plugins

Apart from commonly being free-as-in-beer, open-source is also great because it enables us geeks to have a deep look into the inner-workings of software that we use. How many times have you wondered why a program fails, only to find that it has no logging capabilities, or produces useless output? Open-source often lets you avoid banging your head on the wall too hard, because you can always have a look at the code and try to figure out what the problem is.

But what if the reason still remains a mystery after reading the code? Fortunately then you have the most powerful tool in your development toolbox still left – the debugger. Let’s recap what it usually takes to start debugging third-party code?

Creating a custom PostSharp aspect (with IL transformation)

Last time I wrote about my efforts to create minidumps with PostSharp aspects. I showed some ways of achieving that, but the main conclusion was that no out-of-the-box aspect in PostSharp provides a transformation that would allow inserting minidump generation code in the optimal place: just before throw instructions. Fortunately, it isn’t too hard to create such an aspect, even though the required PostSharp SDK is unsupported and undocumented.

Let’s start by analyzing how PostSharp aspects are architected. Then we’ll move to implementing the actual IL transformation required to temporarily store the exception about be thrown, pass it to the aspect code, and then actually throw it expected.


Writing minidumps with PostSharp aspects

In the comments for one of my previous posts, Writing an automatic debugger in 15 minutes, a reader suggested creating minidumps with PostSharp aspects. Although I proved (by the very scientific method of guesstimation – see Performance impact of running under MDbgEngine) that running under a debugger introduces only a 25% performance penalty, it is reasonable to try to avoid it altogether. Today’s post will be an investigation into using PostSharp as a means of injecting minidump creation code into exception handlers.


Performance impact of running under MDbgEngine

As promised, this post will present the results of my investigation into the performance impact of running under the simplest possible debugger written using MDbgEngine.

Accessing stack traces with MDbgEngine and PADRE

In my previous post, I showed how MDbgEngine (available on NuGet) can be used to stop a process when an exception is thrown, and how a minidump can then be created to enable post-mortem debugging.

Using MDbgEngine, you can do much, much more – and all with a very high-level API. (more…)

Writing an automatic debugger in 15 minutes (yes, a debugger!)

Seriously, it will take you longer to read this long introduction, than to code a working debugger in C#.

You may also want to check out:

Bugs in production

Remember all those reports coming back form clients saying “hey, your program crashed” or “hey, your is site showing these ugly yellow pages at random moments”? So do I. Unfortunately, there isn’t much that can be done to diagnose problems in running software, basically we’re at the mercy of our clients’ reports (which vary in quality, most of the time tending towards useless) or more or less verbose logging. Having those, we can reproduce issues during a debugging session to see what code causes them. If that fails,  we can go to the extremes of attaching a debugger in a production environment. Not feeling enough pressure today? Attach to a live site and try to setup breakpoints the exact way needed to only catch the error, and not stop a milion people from doing their daily work. At first try. Logging, on the other hand is very safe, it but will only tell you as much as you predicted that would be needed.

Now what if we could have something in the middle? More than logging, but less than an interactive debugging session?