WebHooks vs Message Bus

WebHooks are a relatively new concept in ASP.NET but they have been around for a while with other platforms. GitHub, PayPal and even Dropbox support them already. If you are unfamiliar with the concept of a webHook then go read the introduction first. Recently I needed to solve a problem for which WebHooks seemed like a possible solution. This is my journey through that decision process.

Publisher-Subscriber Model

WebHooks are designed to solve the classic publisher/subscriber problem. The concept is pretty straightforward: a publisher raises some sort of event and zero or more subscribers get notified that the event occurred. For example you might have a public facing site that allows a user to submit an order. When the order is submitted an event is raised that notifies the warehouse to process the order. At the same time a separate system might send an email to the customer notifying them that their order is being processed. Later when the order is shipped another event is raised that allows the email system to send a follow up email.

In a pure pub/sub system the publisher doesn’t actually contact the subscribers directly but rather notifies a third-party system that is responsible for tracking the subscribers. This allows the publisher and subscribers to be decoupled allowing each to evolve separately.

A typical pub/sub model has three entities: publisher, subscriber, registrar.  The registrar is responsible for managing the events that can be raised by publishers and the events that subscribers are interested in. The registrar generally stores this data to a database or other persisted storage. A good registrar is fault tolerant, highly reliable and may support advanced features like prioritization, retry and error processing.

The publisher is responsible for 2 things: registering an event and raising an event. It is important for a publisher to register an event because that is what a subscriber will be listening for. Without a registration system a subscriber has no way of indicating what event(s) they are interested in. In general a publisher will register events at startup or during installation. Once events are registered the publisher can raise an event at any point. Each event generally has some data provided by the publisher to help identify why the event was raised. The event and data combined are generally referred to as a message.

The subscriber also has 2 responsibilities: subscribe to an event and handle the event. Generally when a subscriber comes online (or is installed) it will register for the events it is interested in. It is critical that the publisher has already registered the event otherwise the subscriber has no way of doing this. After that the subscriber simply waits to be notified by the registrar.

Message Bus

A typical pub/sub model will likely use a message bus as the registrar. While message buses can support more than the pub/sub model we will limit our discussion to this use case. A message bus typically stores a copy of a message for each subscriber. This allows subscribers to handle messages independently of each other. This also means a bus can notify subscribers in any order and possibly in parallel leading to a more robust, fast and reliable messaging system.

Most buses support error handling. If a subscriber fails to properly handle a message it may be given a couple of opportunities to retry before the message is marked as error. Since each subscriber gets its own copy of the message an error in one subscriber would not impact the other subscribers.

One feature that most buses support is the reactive notification of subscribers. Subscribers may not be available when the message is received by the bus so the message goes into a queue until the subscriber comes online. This is both a plus and a minus. The plus side is that the subscriber will eventually get notified and is not necessary for the initial request. The downside is that something has to wake up the subscriber so that it knows there are messages. In many implementations the subscriber is a process that runs frequently (perhaps a service or scheduled task). The downside to this is that you can end up with lots of little processes running to handle various messages unless they are somehow related. Additionally if something prevents the subscriber from running then something outside the bus infrastructure would need to report it because the bus will basically wait forever.

Because subscribers are notified reactively however, messages are generally received in the same order they are sent. There is no guarantees on this but most buses provide some functionality to ensure this. This makes it easier for a subscriber to process messages that are order dependent.

WebHooks

WebHooks are designed for solving the pub/sub model but it is important to understand their limitations and the current limits of the ASP.NET implementation. Events still need to be registered in some sort of registrar. Subscribers, known as receivers, contact the registrar to subscribe to the events. When an event is raised by the publisher, known as senders,  the registrar contacts each subscriber and passes them the data. So far this is basically like a message bus but the implementation details make it dramatically different.

One of the first differences is that WebHooks is proactive whereas buses are reactive. When an event is received WebHooks immediately contacts each subscriber to let them know about the event. If the subscriber is not available then an error occurs (more on this later). While the publishers and subscribers are still decoupled, the subscriber must be available at the time the event is raised. If the subscriber is not available you may potentially lose the event. However because WebHooks will “call” the subscriber, this approach fits more naturally with making REST calls rather than having to write programs that run continuously. Nevertheless maintenance windows for subscribers needs to be carefully managed.

In regards to errors there are also additional concerns around timeouts. Since WebHooks will immediately call a subscriber, it is important that the subscriber process the request quickly. If it doesn’t then WebHooks will potentially run out of resources waiting for subscribers to finish. As such subscribers need to either quickly process each event or store the event for later processing. As soon as you start down the “process it later” route you start to lose the benefits of a messaging system. It is up to WebHooks to decide how to handle errors. It could be as simple as logging the error, subscribe and event. Since timeouts could be a common problem some sort of retry logic should also be implemented.

Partially because of retries and partially because of the nature of making asynchronous calls to subscribers, there is no real ordering guarantees on events. If ordering is important then it must be built into the event data. This complicates subscribers for events that do have ordering requirements.

A final consideration for WebHooks is the processing of results. WebHooks only has 2 possible outcomes from calling a subscriber: success or failure. In the case of an error WebHooks will likely try to send the request again. But an error may simply mean that the event was received but could not be processed (bad data or something). So it is a failure but should not be sent again. It is up to the subscriber to raise errors back to the publisher and not fail the call. In a bus system the subscriber could have simply failed the call and the bus could move the message to an error queue.

ASP.NET WebHooks

Now let’s look at the current implementation of WebHooks in ASP.NET. A word on terminology. Publishers are referred to as senders in WebHooks and subscribers as receivers. We will use that terminology for this discussion.

All the implementation is available via NuGet.

• Microsoft.AspNet.WebHooks.Custom – Need for all WebHook functionality.
• Microsoft.AspNet.WebHooks.Custom.* – Need for whichever store you will be using.
• Microsoft.AspNet.WebHooks.Custom.Api – Need for the Web API functionality in receivers.
• Microsoft.AspNet.Webhooks.Receivers.* – Custom receivers for various existing implementations.

The first thing to note is that it is in preview as of 7 Nov 2015. If you are not comfortable with using pre-release software then WebHooks are not an option at this point. But, having used it in a proof of concept, I can say it is solid. But the APIs are changing so be prepared if you go this route. Also note that you’ll need to enable NuGet to retrieve pre-release software.

The core of ASP.NET WebHooks is the WebHookManager type. This is the registrar of the system. It is responsible for both the sender and receiver processing and therefore is needed by everyone. To use it you must specify the IWebHookStore type to use for the events. There are several different one’s available including memory, Azure and now SQL. Since publishers and subscribers are probably not running in the same process the memory store is not that useful. It is critical that the senders and receivers use the same store and configuration otherwise they will not be able to communicate.

When a sender comes online it registers the events it will send with WebHookManager. To raise an event it uses the NotifyAsync method on the manager. The existing implementation makes the assumption that you’re using this functionality from an MVC or Web API project and therefore it provides extension methods in controllers to simplify this logic. You can technically use WebHooks in any project type but you’ll have to write your own extensions.

When WebHookManagergets notified of an event it uses the registered store to enumerate the subscribers. For each receiver it makes the appropriate call to the receiver (assumes a REST call). An important thing to note is that this spawns tasks that notify each of the receivers. If the process terminates before all receivers have been notified then you will lose events. It’s also important to note that, since the notification happens from the manager, that the server raising the events must be able to contact each receiver. For servers behind a firewall it might be necessary to allow additional URLs through for this to work.

Each receiver has a fixed amount of time to respond. In the current implementation the manager will retry twice with a timeout of 1 minute and 4 minutes respectively. As of this writing, a new constructor overload has been added allowing you to customize the timeout intervals. Ultimately any errors will be ignored unless the sender checks the returned task.

In regard to ordering, the current implementation mostly ensures that events are received in order but because everything is async a receiver may receive multiple messages at the same time. Care should be taken to ensure ordering requirements are met.

Final Thoughts

As I mentioned in the beginning, I started to take a look at WebHooks as a potential replacement for a more complicated message bus in the cases where I needed the pub/sub model. But after looking at the current implementation I cannot really see it as replacing a message bus except in the cases where notifications are not critical or where subscribers need to do any lengthy work. There are simply too many cases where WebHooks would be unreliable. I could see it as being useful for raising toast messages or other alerts but not for mission critical events, at least without a lot of extra work. If I’m going to put in a lot of extra work just to get WebHooks working reliably then I might as well use a message bus. I’m still interested in the concept but I don’t think, at this point, it is a replacement for applications needing a solid pub/sub system.