AutoQuery for User gives random Collection modified error

We have an AutoQuery setup for our User Class:

[Route("/users", "GET")]
    public partial class UserQuery : QueryDb<User, UserResponse>
    {
        public int? Id { get; set; }
        public int? AgencyId { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string DisplayName { get; set; }
        public string Email { get; set; }
        public string PhoneNumber { get; set; }
        public string BadgeNumber { get; set; }
        public bool? SMSNotificationsEnabled { get; set; } = false;
        public bool? EmailNotificationsEnabled { get; set; } = false;
        public int? AccountState { get; set; }
        public bool? IsActive { get; set; }
        public bool? Deleted { get; set; }
    }

    public class UserResponse
    {
        public string BadgeNumber { get; set; }
        public int AgencyId { get; set; }
        public bool SMSNotificationsEnabled { get; set; } = false;
        public bool EmailNotificationsEnabled { get; set; } = false;
        public int AccountState { get; set; }
        public string Email { get; set; }
        public string PhoneNumber { get; set; }
        public string UserName { get; set; }
        public bool IsActive { get; set; }
        public int Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }

//        public List<string> Permissions { get; set; } = new List<string>();
//        public List<string> Roles { get; set; } = new List<string>();
//        public List<string> Divisions { get; set; } = new List<string>();
    }

The User class is a custom class that implements IUserAuth

[Alias("AspNetUsers")]
    public class User : IUserAuth
    {
        public string BadgeNumber { get; set; }

        [References(typeof(Agency))]
        [Required]
        public int AgencyId { get; set; }

        [Reference]
        public Agency Agency { get; set; }

        [Required]
        public bool SMSNotificationsEnabled { get; set; } = false;

        [Required]
        public bool EmailNotificationsEnabled { get; set; } = false;

        [References(typeof(AccountState))]
        [Required]
        public int AccountState { get; set; }

        [Index]
        public string Email { get; set; }

        [Required]
        public bool EmailConfirmed { get; set; } = false;

        public string PasswordHash { get; set; }

        public string SecurityStamp { get; set; }

        public string PhoneNumber { get; set; }

        [Required]
        public bool PhoneNumberConfirmed { get; set; }

        [Required]
        public bool TwoFactorEnabled { get; set; }

        public DateTime? LockoutEndDateUtc { get; set; }

        [Required]
        public bool LockoutEnabled { get; set; }

        [Required]
        public int AccessFailedCount { get; set; }

        [Required]
        [Index]
        public string UserName { get; set; }

        [Required]
        public bool IsActive { get; set; }

        [Required]
        public bool Deleted { get; set; }

        [Reference]
        public List<UserPermission> UserPermissions { get; set; } = new List<UserPermission>();

        [Reference]
        public List<AspNetUserRole> UserRoles { get; set; } = new List<AspNetUserRole>();

        // Properties from IUserAuth
        public string StateUserName { get; set; }

        public string StatePassword { get; set; }

        [AutoIncrement]
        public int Id { get; set; }
        public string PrimaryEmail { get; set; }
        public string Salt { get; set; }
        public string DigestHa1Hash { get; set; }
        public List<string> Roles { get; set; }
        public List<string> Permissions { get; set; }
        public int? RefId { get; set; }
        public string RefIdStr { get; set; }
        public int InvalidLoginAttempts { get; set; }
        public DateTime? LastLoginAttempt { get; set; }
        public DateTime? LockedDate { get; set; }
        public DateTime CreatedDate { get; set; }
        public DateTime ModifiedDate { get; set; }
        public string DisplayName { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Company { get; set; }
        public DateTime? BirthDate { get; set; }
        public string BirthDateRaw { get; set; }
        public string Address { get; set; }
        public string Address2 { get; set; }
        public string City { get; set; }
        public string State { get; set; }
        public string Country { get; set; }
        public string Culture { get; set; }
        public string FullName { get; set; }
        public string Gender { get; set; }
        public string Language { get; set; }
        public string MailAddress { get; set; }
        public string Nickname { get; set; }
        public string PostalCode { get; set; }
        public string TimeZone { get; set; }
        public Dictionary<string, string> Meta { get; set; }
    }

But intermittently we get a “Collection was modified; enumeration operation may not execute” error. It’s odd though, it doesn’t happen every time, and it returns only one user (I have 10-15 in the system), using no query params.

Error

[12:09:57 INF] Request starting HTTP/1.1 GET http://localhost:5000/users application/json
[12:09:57 INF] Request finished in 174.3782ms 200 application/json; charset=utf-8
[12:10:01 INF] Request starting HTTP/1.1 GET http://localhost:5000/users application/json
[12:10:01 ERR] Error when Disposing Request Context
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException_InvalidOperation_EnumFailedVersion()
   at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext()
   at ServiceStack.ServiceStackHost.OnEndRequest(IRequest request)
[12:10:01 INF] Request finished in 161.2365ms 200 application/json; charset=utf-8

I know User is a little more special that my normal Models, but should it cause AutoQuery to fail?

Not sure why this would be happening

There’s something else weird that’s going on, the Exception suggests the RequestContext collection is being modified whilst the Request deps are being disposed of at the end of the request. Do you have any OnEndRequestCallbacks or are you using Request Scoped deps in async or a background thread or something?

No I don’t have any abnormal registrations, just typical service stack Interfaces and some custom interfaces.

Something is mutating the RequestContext when ServiceStack is trying to dispose of the request dependencies at the end of the request, I don’t really want to take a snapshot of the RequestContext Dictionary values before disposing it as it would mask the real issue - that something is mutating it when there shouldn’t be any other threads accessing that RequestContext at the same time.

Maybe enabling debug logging will help identify the issue, but it wont be with just this Service, something else in your App is mutating the RequestContext dependencies which we wont be able to see from here. You can look at doing some trial and error by commenting out some functionality to see if you can localize the issue. If you have Request Scoped dependencies you can change them to transient to see if it resolves the issue otherwise if you can put together a small stand-alone repro I can investigate further.