WebHook Design with ServiceStack

ServiceStack
1

What would be a good direction to go to implement WebHooks in a ServiceStack stack (as a producer)?
For cloud hosted REST services (we are on Azure) are we better to integrate with what the cloud platform providers have instead of building our own stuff (preferred)?

Has anyone got any learning from going there?

Clearly, we are interested in the basic three things:

  1. how a SS service is going to publish an event to the webhook.
  2. how to manage the external subscriptions to the web hooks?
  3. how and when to fire the webhook to the subscribers?
1 Like
2

Not sure if it’s relevant but @layoric has developed a Discourse WebHook with ServiceStack to sync with this forum over at DiscourseWebHook. The incoming WebHook didn’t map cleanly to a ServiceStack DTO so it needed to implement IRequiresRequestStream so the body could be parsed manually.

I don’t know of any example using ServiceStack to produce web hooks, although I’d expect you’re just going to want to publish a DTO to a specified endpoint. The closest thing we’ve got for behavior like this in the ServiceStack framework itself is being able to specify a HTTP URL as the replyUri in a MQ Message in which case it will POST the Response DTO of the MQ Service to that HTTP URL.

I’d imagine you’d want to do something similar where you’d provide an API to register URLs against different WebHook events then when it’s time (i.e. event is triggered), go through each registered URL and POST a DTO to it which you can easily do with HTTP Utils.

I don’t have any experience with creating WebHooks but I thought it’d be fairly straight-forward, provide an API that lets users register for an event, then when an event happens (e.g. Customer is created) go through each url registered for that event and post the same Response DTO to each registered URL (incidentally a task like this would be an ideal role for an MQ).

But depending on your requirements you may want to develop a more sophisticated Web Hooks implementation in which case I’d look into Github’s WebHooks API which has a fairly vast and elaborate WebHooks API.

3

Thanks, Ill digest the references you gave here.

WebHooks just like github is what we are after.

I am kind of hoping there is somekind of well-known cloud architecture we can leverage. i.e. a combo of queues/functions/whatever, that makes it easy to publish events (via a queue), dispatch events to subscribers and manage their subscriptions without having to support all that in my own service.

Actually having said that, having an encapsulated WebHookFeature that you can just register in your own SS service that gives all the endpoints and configuration of events would be very neat indeed.
Would make a great SS project! (i.e. ServiceStack.WebHooks,or ServiceStack.WebHooks.Azure or ServiceStack.WebHooks.Aws)

thoughts?

4

Yeah I definitely think a lot of the basic WebHooks functionality could be encapsulated in a reusable Plugin. You’d need a data store to maintain the WebHook registrations which I’d do in OrmLite so it can integrate with any existing major RDBMS. Firing the web hooks would need to be done in a background thread or for bonus points you can check if they’re running an MQ Service and if they are publish the Request DTO that will fire the WebHook events, or if not, falling back to a background thread if not.

5

I might be up for helping to create such a thing.

Give me a couple hours, I’ll sketch out a notional ‘WebHookFeature’ and see what the variability/extensibility points may look like, with some suggestions of notional technology implementations that could be supported by it.

6

I think question 3 is up to what purpose the webhooks are serving.

If you want to fire a webhook on an action of another service, just firing HTTP request using the HTTP Utils on a background thread might be the way to go (simple but no redundancy/retry/backoff etc). Could wrap it into an attribute filter, eg:

public class WebHookFilterAttribute : ResponseFilterAttribute
{
	public override void Execute(IRequest req, IResponse res, object responseDto)
	{
		var dbConnectionFactory = req.TryResolve<IDbConnectionFactory>();
		var session = req.GetSession();
		if (session == null || session.IsAuthenticated == false)
		{
			return;
		}

		string verb = req.Verb;
		string typeName = req.Dto.GetType().Name;

		using (var db = dbConnectionFactory.OpenDbConnection())
		{
			var webHooks =
				db.Select<WebHookInstance>(x => x.UserName == session.UserName).ToList();

			if (webHooks.Count == 0)
			{
				return;
			}

			if (webHooks.Any(x => x.IncomingVerb == verb && x.IncomingRequestType == typeName) == false)
			{
				return;
			}
			foreach (var webHookInstance in webHooks)
			{
				var request = new WebHookData
				{
					Request = req.Dto,
					Response = res.Dto
				};
				WebHookInstance webHook = webHookInstance;
				new Task(() => NotifyExternalListener(webHook.Url, request)).Start();
			}
		}
	}

	private static void NotifyExternalListener(string url, WebHookData request)
	{
		JsonServiceClient client = new JsonServiceClient();
		client.Post<WebHookDataResponse>(url, request);
	}
}

You just need a could of DTOs and services to manage the CRUD of the webhooks into DB of your choice to let users manage them, eg:

[Authenticate]
public object Get(WebHook request)
{
	if (request.Id == null)
	{
		//Get all for user and return
		var webHook = Db.Single<WebHookInstance>(x => x.UserName == SessionAs<AuthUserSession>().UserName && x.Id == request.Id);
		return new WebHookResponse
		{
			WebHook = webHook
		};
	}

	var result = Db.Select<WebHookInstance>(x => x.UserName == SessionAs<AuthUserSession>().UserName);
	return new WebHookResponse
	{
		WebHooks = result.ToList()
	};
}

[Authenticate]
public object Post(CreateWebHook request)
{
	WebHookInstance webHookInstance = request.ConvertTo<WebHookInstance>();
	webHookInstance.UserName = SessionAs<AuthUserSession>().UserName;
	Db.Insert(webHookInstance);

	return new WebHookResponse
	{
		WebHook = webHookInstance
	};
}

DTOs

[Route("/webhooks")]
[Route("/webhooks/{Id}")]
public class WebHook : IReturn<WebHookResponse>
{
	public int? Id { get; set; }
}

[Route("/webhook",Verbs = "POST")]
public class CreateWebHook : IReturn<WebHookResponse>
{
	public string Url { get; set; }
	public string Name { get; set; }
	public string IncomingVerb { get; set; }
	public string IncomingRequestType { get; set; }
}

public class WebHookResponse
{
	public List<WebHookInstance> WebHooks { get; set; }
	public WebHookInstance WebHook { get; set; }
}

public class WebHookInstance
{
	[AutoIncrement]
	public int Id { get; set; }
	public string UserName { get; set; }

	public string Url { get; set; }
	public string Name { get; set; }
	public string IncomingVerb { get; set; }
	public string IncomingRequestType { get; set; }
}

public class WebHookData
{
	public object Request { get; set; }
	public object Response { get; set; }
}

This is a very simple approach for allowing users to hook into your other exposed services, but obviously doesn’t handle any retrying, backoff, timeouts etc that a cloud solution might (AWS SNS). So the other end of the extreme would be I guess wrapping AWS SNS creation of Topics + HTTP/HTTPS/Other Subscriptions which could be also wrapped into a nice IPlugin feature.

2 Likes
7

Just a first cut at the shape of a candidate WebhookFeature.

https://gist.github.com/jezzsantos/8ef7e66c8c1e75823eb1a66af8379345

The idea being that this would be a ServiceStack IPlugin with one-line registration

appHost.Plugins.Add(new WebhookFeature());

that includes a built-in webhook subscription web service, that consumers can register and manage callbacks to be POSTED events raised by your own service.

We would need to make it highly extensible so ServiceStack developers can control how the main components work in their architectures, and make it a highly testable and easy to stub the various components in testing environments.

How It Works

Your service (internally) uses a IWebhookPublisher.Publish<T>(string eventName, T event) to publish things that happen in your service (each with a unique eventName and unique POCO event (containing event data).

The WebhookFeature then publishes your events to a reliable storage mechanism of your choice (i.e. queue, DB, remote cloud service, etc). This is configurable in the WebhookFeature

Then depending on your architecture, you can have any number of workers relay those events to subscribers to your events.

Note: not quite resolved the relay end of things yet (ran out of time). For example, in an Azure cloud architecture you might use a WorkerRole to check the queue every second, and then relay to all subscribers. In AWS you might use Lambda to relay to all subscribers. In another architecture, you might do things on a background thread, triggered by the `IWebhookPublisher.Publish(). In any case we need a cross-process way that relays can find out who the subscribers are and easily dispatch to them.

Another thing to thing about is whether to include those things in a separate nuget (specific to a architecture) rather than bundling everything in to the main one. I can see people wanting flexibility in the following things: the kind of storage for subscriptions (MemoryCache/DB by default), the kind of reliable storage of events (MemoryCache/DB by default), and the relay component (in memory, background thread by default).

I guess what I am looking for at this point is if there is anyone with an appetite to co-design/develop this further??, and whether we should create a new repo for it, and get started? @mythz, @layoric?

8

@mythz @layoric were you guys keen on this?
(please see the end of the last post)

9

Come down to time, but happy to help when I can. @jezzsantos maybe create a separate repo for it on GitHub to get things started :+1:

10

will do, my github I presume, or SS github?

11

I think we can start with just yours for now I guess (@mythz ?), see where it goes.

12

It would need to be in your own repo, we’d have to support anything in ServiceStack’s repos and I don’t see creating WebHooks would be popular enough to justify the time to maintain/support it. It would be ideal if you’re be able to find someone else here who also needs to use this feature as it will help with designing something generic enough to meet each of your requirements. Failing that just design something that works for you, as additional requirements will start to manifest once you start making use of it.

13

Righto. Thanks guys.

Moving the project over here: https://github.com/jezzsantos/ServiceStack.Webhooks

14

Just circling back on this thread.

We now have a working webhook framework encapsulated in a WebhookFeature plugin over at: ServiceStack.Webhooks, which permits other packages to plugin to it for various architectural components of people’s own services.
We have demonstrated that for some Azure technologies in the ServiceStack.Webhooks.Azure package.

Now looking to see what others might need.
Also looking for contribs.

1 Like
15

Nice! We will (most probably) be using this for our external services - we do not want to expose our MessageQueue and Webhooks is a good candidate.

16

That’s great @Rob,
Looking forward to supporting you.

We should probably move this discussion over to the project, but can I ask?
Can I ask what platform you are on (to relay from the queue to the subscribers?).
If it one we dont yet have a relay for, then I’d be keen to either help create it or show you the way.

I’m creating this page right now to show the way. Building Your Own Plugin

17

Little late to be jumping back into this thread, but thought I’d add my 2 cents. @jezzsantos the solution looks pretty good; however, I wonder if you’ve considered some of the approaches suggested by @layoric … in that the triggering of the hooks is automatic based on the request type that can be applied in the request pipeline. The manual publishing/trigger of the hook in the service handling leaves me thinking that over time, it might be harder to maintain consistency of triggers, although it does lend itself to full control over what is included in it.

Thoughts now that you’re 1 year into it?

18

Hey @sirthomas,

Our use of it has been pretty basic, and have not done much to extend.

However, would be good to hear back from others if they feel the need for what you propose. Might be good if you had a concrete example of what you are asking for with it.

1 Like