ServerEvents Not Working

I have added the ServerEventsFeature to my service, to use Redis:

            Plugins.Add(new ServerEventsFeature
            {
                OnPublish = (res, msg) =>
                { //fired after ever message is published
                    res.Write("\n\n\n\n\n\n\n\n\n\n");
                    res.Flush();
                },
            });

            container.Register<IRedisClientsManager>(x => new PooledRedisClientManager("localhost:6379"));
            container.Register<IServerEvents>(x => new RedisServerEvents(x.Resolve<IRedisClientsManager>()));
            container.Resolve<IServerEvents>().Start();

and in one of my services, I am calling: IServerEvents.NotifyChannel("MyChannel", new MyDto());

Our services are hosted in a WebRole in the Azure Emulator (in IISExpress) when we are testing.

In an integration test, I am calling my service and trying to test if the server event is being sent to a client.

            [TestMethod, TestCategory("Integration")]
            public async Task WhenPutSomethingAndServerEventClient_ThenNotifiesClient()
            {
                using (var eventClient = new ServerEventsClient("https://localhost:4445/api", "MyChannel"))
                {
                    var connect = await eventClient.Connect();
                    var command = await eventClient.WaitForNextCommand();

                    // Calling service
                    var result = ServiceClient.Put(new MyApiCall());

                    var message = await eventClient.WaitForNextMessage();

                    Assert.Equal(result.Data, message.Data);
                }

So the call to var connect = await eventClient.Connect(); returns after a bit, but the call to var command = await eventClient.WaitForNextCommand(); never returns - ever!

What am I missing?

Do you have dynamic compression disabled?

If you don’t have ServerEventsFeature.NotifyChannelOfSubscriptions=false then the first command should be JOIN Channel event notification. I would remove that OnPublish callback because if you need that it’s because your host has buffering enabled which hopefully disabling dynamic compression above should prevent it buffering.

I’ve just tried this in Server Events Test using RedisServerEvents and it’s firing both callbacks:

[Test]
public async Task WhenPutSomethingAndServerEventClient_ThenNotifiesClient()
{
    using (var client = CreateServerEventsClient())
    {
        var connect = await client.Connect();
        var command = await client.WaitForNextCommand();

        "GOT HERE".Print();
    }
}

Yes, I had tried that suggested configuration in the web.config. No dice.

I put a breakpoint in ServerEventsFeature.OnPublish AND on ServerEventsFeature.OnConnect, during the test, and what I found was that the OnConnect fires just fine, then OnPublish fires with some ‘OnConnect’ command, then it happens again: OnConnect and OnPublish for another user, and this just goes on and on creating more and more random users (with negative user ID’s). Which I can see getting created in Redis as well (I think). This loop cycles every few seconds.

Negative UserIds are assigned to anonymous (i.e non-authenticated) users. Sounds like the long-lived connection is being continually being disconnected and the new users are auto reconnections but I don’t know why it’s not sustaining the ServerEvents HTTP connection, I’m assuming it’s something to do with IIS Express configuration.

Don’t think it’s related to RedisServerEvents backend impl, but do you get the same behavior with the default MemoryServerEvents? i.e. try commenting out:

//container.Register<IServerEvents>(x => 
//    new RedisServerEvents(x.Resolve<IRedisClientsManager>()));

But it should work fine with IIS Express with doDynamicCompression=false in the Web.config since most of the Server Events Examples are running fine on IIS Express during development then we deploy on IIS.

So I’d recommend checking out the Chat Example to see if it’s working for you, if it is compare with your Web.config to see what’s different.

Yep, that works! removing the ‘RedisServerEvents’ registration and falling back to ‘MemoryServerEvents’ seems to work fine.

This is with the:

    <urlCompression doStaticCompression="true" doDynamicCompression="false" />

in the web.config

hmmm, what to try now? we are using v.55.

We would want to use Redis. since we will have server events for many of our services, and we deploy Redis for response/client caching.

(Also, I am presuming that the same Redis instance can accommodate multiple services with SSE?)

Then I’ll need a repro I can run locally. You can change Chat to run using RedisServerEvents by adding a RedisHost AppSetting, I’ve added an example of this in the latest Chat commit:

<add key="RedisHost" value="127.0.0.1:6379" />

Which you can uncomment and change to point to your Redis instance to get Chat running using RedisServerEvents thanks to the RedisHost check in AppHost.

I’ve tested this again and it’s working fine with RedisServerEvents, let me know what errors/unexpected behavior you’re seeing and any changes you’ve made to Chat to get them.

RedisServerEvents enable clients to connect to multiple load-balanced app servers, so you can connect to a channel on any App Server, not sure if that’s what you mean by multiple services with SSE?

I meant, if I configure SSE for multiple AppHosts?
(i.e. different machines with different sets of SS services on them).
Can they use the same Redis instance happily?

Using RedisServerEvents is meant to enable sharing of the same logical “Server Events” configuration across multiple app servers. Different AppHosts can communicate together using a different channel, but they’re not isolated, i.e. a Server Event Subscription in one AppHost is visible to any other AppHost configured to use the same Redis Instance.

“Different sets of Services” is a bit of a misnomer since the existence of Services doesn’t impact SSE configuration. If you’re asking if different AppHosts will be isolated, they wont be, they’re logically in the same group by design. But you can communicate on a different channel and only clients subscribed to the same channel will receive messages published to that channel, but you can for instance still send a direct message to a user by userId, userName, sessionId or subscriptionId even if they’re not on the same channel.

If you want “complete isolation” you would need to connect to a different Redis Instance or Redis Database.

Do need to share channels between AppHosts. And each AppHost that creates events will certainly be doing that on different channels, or even the same channel (although that is YAGNI right now).

Don’ t believe I need isolation. So that’s good.

thanks.

Hey Mythz,

Could it be possible that the Redis registration for SSE is somehow being confused with the Redis registration we are using for ICacheClient in server/client caching?

We have this right now in AppHost.Configure() in this order.

            // Needed for ICacheClient caching
            container.Register<IRedisClientsManager>(x => new PooledRedisClientManager("localhost:6379"));
            container.Register(x => x.Resolve<IRedisClientsManager>().GetCacheClient());

            // Needed for SSE
            container.Register<IRedisClientsManager>(x => new RedisManagerPool("localhost:6379"));
            container.Register<IServerEvents>(x => new RedisServerEvents(x.Resolve<IRedisClientsManager>()));
            container.Resolve<IServerEvents>().Start();

Actually, I have no idea the difference in use between PooledRedisClientManager("localhost:6379") and RedisManagerPool("localhost:6379"), I am just copying what the wiki tells me to do (for caching and for SSE).

But perhaps these need unifying somehow?

You should only have 1 Redis Manager registered. I think your question originally had a different registration so my previous answer doesn’t make sense anymore. I’ll start again:

You should remove the second duplicate registration, i.e:

//container.Register<IRedisClientsManager>(
//    x => new RedisManagerPool("localhost:6379"));

Since it will just be overriding the first.

This means:

container.Register<IServerEvents>(x => 
    new RedisServerEvents(x.Resolve<IRedisClientsManager>()));

SSE will use the existing IRedisClientsManager registration. It doesn’t matter which RedisManager you end up using.

OK Mythz,

I have a clue.

At the start of every integration test (when we were just using Redis as a ICacheClient) we are doing this:

                var redisClientsManager = appHost.Container.Resolve<IRedisClientsManager>();
                if (redisClientsManager != null)
                {
                    var redisCache = redisClientsManager.GetClient();
                    if (redisCache != null)
                    {
                        Recorder.DebugWithTimer("Flushing Redis Cache", () =>
                        {
                            redisCache.FlushDb();
                        });
                    }
                }

Now that we are using Redis as the SSE server as well, I suspect we are mucking up SSE in some way by blowing away its storage each test, which may be why it maybe not joining our subscription (and looping creating new users each time).

I added this, but seemed not to fix it.

                var redisSse = appHost.Container.Resolve<IServerEvents>();
                if (redisSse != null)
                {
                    Recorder.DebugWithTimer("Restarting Redis SSE", () =>
                    {
                        redisSse.Stop();
                        redisSse.Reset();
                        redisSse.Start();
                    });
                }

Is there something else I should be doing?

Have a look at our existing RedisServerEventTests we’re not doing anything different just starting and disposing the AppHost as normal. If anything I’d only be flushing Redis at the start of the TestFixture.

Yep, I got Chat working locally, and working with Redis going too!
I even got it going with v.55 of SS.

Can’t see any difference between what we have and what Chat has…

Looks like I am on my own to to why Redis keeps creating anonymous users every couple seconds and never lets my client join the channel!!! boo!

Almost as if every time my client sends a heartbeat, the RedisServerEvents creates a new user account, but does not join it to the channel. MemoryServerEvents works just fine.

I have the <urlCompression doStaticCompression="true" doDynamicCompression="false" /> in my web.config at both the root <system.WebServer> AND under the location path “api”

  <location path="api">
    <system.web>
      <httpHandlers>
        <add path="*" type="ServiceStack.HttpHandlerFactory, ServiceStack" verb="*" />
      </httpHandlers>
    </system.web>
    <system.webServer>
      <modules runAllManagedModulesForAllRequests="true" />
      <validation validateIntegratedModeConfiguration="false" />
      <handlers>
        <add path="*" name="ServiceStack.Factory" type="ServiceStack.HttpHandlerFactory, ServiceStack" verb="*" preCondition="integratedMode" resourceType="Unspecified" allowPathInfo="true" />
      </handlers>
      <!-- ServiceStack: Disable automatic buffering, see: https://github.com/ServiceStack/ServiceStack/wiki/Server-Events#response-buffering-delaying-events -->
      <urlCompression doStaticCompression="true" doDynamicCompression="false" />
    </system.webServer>
  </location>

  <system.web>
    <compilation targetFramework="4.5" />
    <httpRuntime targetFramework="4.5" />
    <httpHandlers>
      <add path="api*" type="ServiceStack.HttpHandlerFactory, ServiceStack" verb="*" />
    </httpHandlers>
  </system.web>

  <!-- Required for IIS 7.0 -->
  <system.webServer>
    <!-- ServiceStack: Required -->
    <modules runAllManagedModulesForAllRequests="true" />
    <validation validateIntegratedModeConfiguration="false" />
    <!-- ServiceStack: Disable automatic buffering, see: https://github.com/ServiceStack/ServiceStack/wiki/Server-Events#response-buffering-delaying-events -->
    <urlCompression doStaticCompression="true" doDynamicCompression="false" />
    <handlers>
      <!-- ServiceStack: Only required for IIS 7.0 -->
      <!--<add name="ServiceStack.Factory" path="api" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*" preCondition="integratedMode" resourceType="Unspecified" allowPathInfo="true"/>-->
    </handlers>
  </system.webServer>

Without being able to see the issue I can’t speculate any further about what behavior you’re seeing, so I’d say it’s time to break out the packet sniffer and see what’s really going on, I.e. which HTTP requests are failing and why. The easiest thing might be to copy Chat’s default.cshtml into your project, then have a look at the Network requests in Chrome WebInspector for any failed requests, if that doesn’t help identify the issue then I’d look at Fiddler to capture the HTTP traffic, although the buffering in Fiddler could affect Chat from functioning by itself, but hopefully it is still able to capture failed requests.

OK, one breakthrough.

A little hard to spot because we have a base AppHost class for common stuff in our services, but we had the Plugins.Add(new ServerEventsFeature()); call happening after the registration of RedisServerEvents.

Now that we have corrected that, we don’t get the users being recreated all the time! which is good.

But we still hang waiting for this call await eventClient.WaitForNextCommand(); in our test. So I guess we are still not joining the channel.

It is progress, but not enough.