Connection pool service exceptions

ServiceStack
1

Hi,

I have a problem with one of our services.

We are seeing a number of errors in the exception logs, and over time (1-2 days) eventually causes the service to stop.

the errors appear to be one of three types:

  1. Forcibly closed connection
  2. Unable to bind
  3. Unable to get connection from pool

I have reviewed the code and regarding #3, all code appears to be safely within the using() pattern, the other two I am unsure about.

Service is currently running 5.5 and we are about to update it.

I intend to dig into code further, but any thoughts on potential causes of these three errors ? I am assuming that #3 is the result of either of #1,#2 or both.

020-05-15 00:24:35Z|INFO|An existing connection was forcibly closed by the remote host - An existing connection was forcibly closed by the remote host

2020-05-18 22:13:24Z|INFO|Unable to bind to request ‘GetSecurityUserByEmail’ - { at ServiceStack.Serialization.StringMapTypeDeserializer.PopulateFromMap(Object instance, IDictionary2 keyValuePairs, List1 ignoredWarningsOnPropertyNames) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack.Client\Serialization\StringMapTypeDeserializer.cs:line 106
at ServiceStack.Host.RestPath.CreateRequest(String pathInfo, Dictionary2 queryStringAndFormData, Object fromInstance) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\RestPath.cs:line 544 at ServiceStack.Host.RestHandler.CreateRequest(IRequest httpReq, IRestPath restPath, Dictionary2 requestParams, Object requestDto) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\RestHandler.cs:line 156
at ServiceStack.Host.RestHandler.CreateRequestAsync(IRequest httpReq, IRestPath restPath, Dictionary`2 requestParams) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\RestHandler.cs:line 145
at ServiceStack.Host.RestHandler.CreateRequestAsync(IRequest httpReq, IRestPath restPath) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\RestHandler.cs:line 136
at ServiceStack.Host.RestHandler.ProcessRequestAsync(IRequest req, IResponse httpRes, String operationName) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\RestHandler.cs:line 89}

2020-05-18 22:15:00Z|INFO|error connecting: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. - { at MySql.Data.MySqlClient.MySqlPool.GetConnection()
at MySql.Data.MySqlClient.MySqlConnection.Open()
at ServiceStack.OrmLite.OrmLiteConnection.Open() in C:\BuildAgent\work\27e4cc16641be8c0\src\ServiceStack.OrmLite\OrmLiteConnection.cs:line 79
at ServiceStack.OrmLite.OrmLiteConnectionFactory.OpenDbConnection() in C:\BuildAgent\work\27e4cc16641be8c0\src\ServiceStack.OrmLite\OrmLiteConnectionFactory.cs:line 95
at AccountManagement.ServiceInterface.AccountService.Any(GetClassByClassCode request) in C:\Users\Public\workspace\mico-2\Account\Account.ServiceInterface\School\SchoolClass.cs:line 41
at lambda_method(Closure , Object , Object )
at ServiceStack.Host.ServiceRunner`1.ExecuteAsync(IRequest req, Object instance, TRequest requestDto) in C:\BuildAgent\work\3481147c480f4a2f\src\ServiceStack\Host\ServiceRunner.cs:line 133}

2

OrmLite doesn’t manage the connection handling, which is all handled by the underlying ADO .NET provider for the RDBMS that’s used. What you’d need to make sure is that whenever you open a db connection that you also dispose it, so the connection is returned to the pool. If you’re using the base.Db in your Service for example, it’s being automatically disposed after the Service is executed.

This says the max connection pool size was exceeded and that the timeout expired before another open db connection was made available. Note that where the Exception occurred doesn’t indicate the source of the connection leak, only that it was unable to retrieve an unused connection.

This error is when the Request is unable to be serialized into the GetSecurityUserByEmail which could be because the request was corrupted, I don’t know if it’s related to your other Exception.

3

Thanks Demis for the insight. I will report back when I know more about what is the root cause of the issue.

4

I have been having this issue with PostgreSQL. For NPGSQL there is a setting:

Connection Idle Lifetime : The time (in seconds) to wait before closing idle connections in the pool if the count of all connections exceeds Minimum Pool Size

It’s default is a bit long I think (5 minutes) so I set it to 10 seconds and set the max pool size to 20 less than my actual limit and it seems to have fixed issue.

Could be the same thing happening in MySQL perhaps, sorry I don’t know the settings for that I don’t use it.

1 Like
5

Thank you very much, will give this a try and run some tests.