The Orchard packaging experiment

I’ve written about plugins based on NuGet before. One piece of software that manages its pluggable parts with NuGet is Orchard - a CMS based on .NET and MVC, which from the experience of my company, Caliper, also serves as a nice framework for building web applications without any CMS functionality.

There’s a couple of things that Orchard solves for you, that make it great as a general framework:

  • database connectivity (with NHibernate), including out-of-the-box support for local, automatic SqlCe databases,
  • data migrations, for upgrading table schemas with new releases of your web applications or modules,
  • theme and module management, with the use of NuGet packages and feeds,
  • dynamic compilation, which lets you do in place changes to your web site or application, even through FTP, and see changes instantenously.

Orchard is distributed as a ZIP file that you can deploy as an application in IIS or open as a web application in Visual Studio. Module and theme packages are only managed through Orchard’s web UI, which gets them from a special Orchard Gallery Feed URL: http://packages.orchardproject.net/FeedService.svc. If you open a module from that feed you’ll see that it contains all of its sources (as in *.cs files), for Orchard’s dynamic compilation to take care of. Open-source at its best. Or is it?

As much as we love having access to sources at Caliper, we love having them on-demand in our debugging sessions even more – instead of moving thousands of them around on disk for no reason. And we all have SymbolSource to handle that for free :)

So we set out to try and make Orchard and its modules more vanilla-nuggety and symbol/source server friendly.

Step 1: Packaging Orchard

First, we decided to try putting Orchard itself into packages. The end-result experience should speak for itself:

  1. Open Visual Studio and create an ASP.NET Empty Web Application (yes, no MVC, no Web Forms, no nothing).
  2. Add http://nuget.gw.symbolsource.org/Public/Orchard/FeedService.mvc as a new package source to Visual Studio (yes, SymbolSource can host and serve NuGet packages as a package repository)
  3. Install the Orchard package from this feed in the empty project.
  4. Run the project with F5. You now have the Orchard setup page ready for action.

Things to note, as this should be considered work-in-progress still:

  • All standard modules and themes are downloaded as dependencies. We haven’t tried yet to determine what subset is needed for setup, and which of them could be installed an on as-needed basis later with NuGet in Visual Studio or Orchard through the web UI.
  • External dependencies are taken from the official NuGet feed only if they were taken by the Orchard team as-is, without modification or recompilation. All the rest is in Orchard.External.
  • SqlCe native binaries are not included yet. But you should be good to go on your development machine, as you probably have in installed globally in your system.
  • We haven’t yet figured out how to do versioning, so all packages have strict dependencies (==, i.e. [x.y.z] in NuGet).
  • You can easily create your own binary module packages that take other Orchard modules as NuGet dependencies and get compiled with the right references.

Now for the best part:

  • No *.cs files anywhere! Less files is more speed, less waiting, less transfers, less comparing on updates, less temptation to modify modules in place.
  • Debuggable – with SymbolSource! All symbols (PDB files) and sources are indexed and accessible with Visual Studio using the regular, public SymbolSource URL. This applies only to Orchard.Framework, Orchard.Core and Orchard.Web projects at the moment, but all Orchard modules will follow soon.
  • No dynamic compilation! No more screwed up assembly names and WCF services unable to load. Yes, you can disable dynamic compilation in configuration, but then you’re still required to compile modules and cleanup sources yourself.
  • All views are included in the main project as content files, so it’s easy to find, edit or copy them to your theme.

Step 2: Repackaging modules

Since a lot of the standard Orchard distribution is built using modules, we had to figure out how to transform them form source form into binary form. We have a small application that interfaces with NuGet and MSBuild to get the modules, guess their interdependencies from project references, then compile and repackage them. All of the standard modules have been uploaded in repackaged, binary form to the SymbolSource Orchard repository.

So far we haven’t touched any contributed modules from the Orchard Gallery, but our plan is to either create a SymbolSource gateway that would do that compilation on the fly (module authors would decide to push to it additionally), or monitor the gallery feed and repackage new modules into our Orchard repository automatically.

What do you think?

All the tooling that we created for this is available on GitHub: http://github.com/SymbolSource/Orchard. To run it, just clone, open a Visual Studio prompt and do msbuild Orchard.proj. The output will be in pkg (main Orchard packages) and pkgbin (module packages).

A world of debuggable open-source software – Part 3: Applications

In my last post, A world of debuggable open-source software – Part 2: Plugins, I used NuGet Package Explorer as an example of an application that uses NuGet-based plugins, and how that makes it very easy to publish their symbols and sources to SymbolSource, for a great debugging experience in case anything goes wrong at runtime. If you’re interested in creating your own plugin system based on NuGet, have a look at this post by Aaron Powell: Creating a NuGet-based plugin engine.

I also mentioned in that post that you can push any PDB files and sources to SymbolSource, including those of entire applications. Because NuGet is still the easiest way of pushing symbol packages, what we would need is a NuGet-based method of distributing programs. Fortunately such tool exists already and is called Chocolatey. If you haven’t heard of it before, here’s how it’s described by its authors:

Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with windows in mind.

In this post I will show you how Chocolatey and SymbolSource can be used to publish open-source applications with full debugging support, allowing users with programming skills to analyze problems through code and provide much more useful bug reports. Or even patches and pull requests!
Read the full post »

A world of debuggable open-source software – Part 2: Plugins

Apart from commonly being free-as-in-beer, open-source is also great because it enables us geeks to have a deep look into the inner-workings of software that we use. How many times have you wondered why a program fails, only to find that it has no logging capabilities, or produces useless output? Open-source often lets you avoid banging your head on the wall too hard, because you can always have a look at the code and try to figure out what the problem is.

But what if the reason still remains a mystery after reading the code? Fortunately then you have the most powerful tool in your development toolbox still left – the debugger. Let’s recap what it usually takes to start debugging third-party code?
Read the full post »

Creating a custom PostSharp aspect (with IL transformation)

Last time I wrote about my efforts to create minidumps with PostSharp aspects. I showed some ways of achieving that, but the main conclusion was that no out-of-the-box aspect in PostSharp provides a transformation that would allow inserting minidump generation code in the optimal place: just before throw instructions. Fortunately, it isn’t too hard to create such an aspect, even though the required PostSharp SDK is unsupported and undocumented.

Let’s start by analyzing how PostSharp aspects are architected. Then we’ll move to implementing the actual IL transformation required to temporarily store the exception about be thrown, pass it to the aspect code, and then actually throw it expected.

Read the full post »

Writing minidumps with PostSharp aspects

In the comments for one of my previous posts, Writing an automatic debugger in 15 minutes, a reader suggested creating minidumps with PostSharp aspects. Although I proved (by the very scientific method of guesstimation – see Performance impact of running under MDbgEngine) that running under a debugger introduces only a 25% performance penalty, it is reasonable to try to avoid it altogether. Today’s post will be an investigation into using PostSharp as a means of injecting minidump creation code into exception handlers.

Read the full post »

Performance impact of running under MDbgEngine

As promised, this post will present the results of my investigation into the performance impact of running under the simplest possible debugger written using MDbgEngine.
Read the full post »

Accessing stack traces with MDbgEngine and PADRE

In my previous post, I showed how MDbgEngine (available on NuGet) can be used to stop a process when an exception is thrown, and how a minidump can then be created to enable post-mortem debugging.

Using MDbgEngine, you can do much, much more – and all with a very high-level API. Read the full post »

Writing an automatic debugger in 15 minutes (yes, a debugger!)

Seriously, it will take you longer to read this long introduction, than to code a working debugger in C#.

You may also want to check out:

Bugs in production

Remember all those reports coming back form clients saying “hey, your program crashed” or “hey, your is site showing these ugly yellow pages at random moments”? So do I. Unfortunately, there isn’t much that can be done to diagnose problems in running software, basically we’re at the mercy of our clients’ reports (which vary in quality, most of the time tending towards useless) or more or less verbose logging. Having those, we can reproduce issues during a debugging session to see what code causes them. If that fails,  we can go to the extremes of attaching a debugger in a production environment. Not feeling enough pressure today? Attach to a live site and try to setup breakpoints the exact way needed to only catch the error, and not stop a milion people from doing their daily work. At first try. Logging, on the other hand is very safe, it but will only tell you as much as you predicted that would be needed.

Now what if we could have something in the middle? More than logging, but less than an interactive debugging session?
Read the full post »

Embedding an ASP .NET application in a unit test (part 2)

In the previous part of this series I talked about how we set up a unit test to run a web application under an embedded instance of the Cassini web server. This takes care of the central component of the diagram below that illustrates our architecture.

So what about mocking the entire backend?
Read the full post »

Embedding an ASP .NET application in a unit test (part 1)

We came across an interesting problem in SymbolSource recently. In the current architecture, the support for NuGet and OpenWrap is provided though so called gateways, that act as protocol adapters connecting to our SOAP API. In theory this provides a great testing opportunity, because it is easy to mock a set of relatively simple, procedural APIs.

A lot has been written about testing MVC applications: that this paradigm of programming is unit test friendly, because you can test the controller in isolation, or you could even try testing views. But what about configuration? Routing? Handlers? That can only be tested in an environment with a fully running web application. Because of this, I should probably have put integration testing in the post’s title, but since we wanted to focus on fully automating our gateway tests and running them with xUnit in CI, let’s keep the original term.
Read the full post »

Follow

Get every new post delivered to your Inbox.

Join 312 other followers