We have an app that has to have nearly 100% uptime. It’s currently hosted in Azure, and the only way we’ve been able to achieve this is by running two instances of the app and persistence, then replicating data between the secondary in cases where it has to fail over.
I’d like to be able to distribute the components of the application and not have any single point of failure, but I’d also like to minimize or eliminate the manual data replication I have to conduct now to merge those failed-over transactions into the central primary once it’s back online.
Would there be issues if I had multiple AppHost instances running against a single pooled Redis cache? Is there a better way to implement redundancy with ServiceStack?
You can only have a single AppHost instance per AppDomain/process so you can’t share the same Redis client connection pool across multiple AppHost instances (and App Domains). If your instead talking about a Redis server instance, you can have many load balanced App server instances accessing the same Redis instance (if that’s what you mean, then pooled Redis cache doesn’t make sense here).
Redis supports 2 availability configurations, Redis Sentinel and Redis cluster, ServiceStack.Redis only supports Redis Sentinel. You can explore more about a Redis Sentinel configuration at: https://github.com/ServiceStack/redis-config
So the natural architecture for ServiceStack would be multiple load balanced App Servers configured to use the same Redis Sentinel configuration.
I need some clarification…when you say “you can have many load balanced App server instances accessing the same Redis instance,” does that mean that I could have two separate web apps on different servers accessing a shared Redis data store (for my purposes, an Azure Premium Redis store)? Basically, does “App Server” equal a process running a ServiceStack AppHost?
In that scenario the web apps would each have their own ServiceStack AppHost. My Redis environment would be the point of failure, but if either web environment went down I’d still be good to go.
Please read the links to Redis Sentinel above, Redis always has a single master and can have multiple read only Redis slave replica’s. The Redis Sentinel process monitors the Redis instances and will fail over and promote one of the slaves to master if it has reached consensus that the master has failed, ServiceStack RedisSentinel monitors the Redis Sentinel process and will automatically fail over to the newly promoted master when that happens.
You can have multiple Redis master instances but they would be completely disconnected (I.e. maintain independent datasets) which is highly unusual as it would mean load balanced App Servers aren’t identical as they’d be configured to connect to different redis masters that would maintain completely different data sets.
I understand Redis’s high availability options, and I understand ServiceStack’s ability to interact with Sentinel to accomplish failover, but I’m still unclear on my original question – can I have multiple instances of my app (including the ServiceStack app host) running off that single Redis instance?
To make it more concrete, I have an app that follows the recommended structure (web/API, service interface, and service model). Can I publish that package to two separate app services in Azure and have them both run in parallel off the same Redis/SQL environment? And even if I can, should I, or is there a better way to accomplish the overall objective of fault tolerance?
Yes you can and should have multiple App Servers connect to the same Redis instance, you would typically do this as the alternative of having it connect to different masters is highly unusual as explained in my previous comment.
Azure explains in this video that their managed Redis cache will automatically fail over:
Basically have all apps connect to the same Redis endpoint and Azure will automatically fail over, ServiceStack.Redis has a built-in feature in which it attempts to retry failed commands up to RedisConfig.DefaultRetryTimeout.
The alternative for using ServiceStack.Redis in Azure is to ignore Azure managed cache and setup a Redis Sentinel configuration running inside Linux VMs following our recommended Redis Sentinel configuration: https://github.com/ServiceStack/redis-config