AutoGen AutoCrud Services

Announcements
1

Happy to announce a few exciting new features today designed to give even more value and productivity around your AutoQuery Services. They’re distinct but inter-related features so I’ll try do my best to describe them with minimal confusion.

All features are going to require the latest v5.8.1 of ServiceStack on MyGet.
If you already had v5.8.1 installed, you’ll need to clear your NuGet cache:

$ nuget locals all -clear

You’ll also need to update your x dotnet tool:

$ dotnet tool update -g x

Database-first Models

Long time users of ServiceStack will know it’s a staunch proponent of code-first development where your C# Types retains the master authority of your App’s logic, although there are a number of times where you have to work with existing databases which would require significant effort to create the initial code-first Data Models. Historically we’ve pointed people to use OrmLite’s T4 Template Support which provides a decent intitial stab, however it’s limited in its capability and offers a sub par development experience.

Code Generation of AutoQuery & Crud Services

With the preview of AutoCrud we could add a lot more value in this area as AutoCrud’s declarative nature allows us to easily generate AutoQuery & Crud Services by just emiting declarative Request DTOs.

You can then add the generated DTOs to your ServiceModel’s to quickly enable AutoQuery Services for your existing databases.

To enable this feature you you just need to initialize GenerateCrudServices in your AutoQueryFeature plugin, e.g:

Plugins.Add(new AutoQueryFeature {
    MaxLimit = 100,
    GenerateCrudServices = new GenerateCrudServices {}
});

If you don’t have an existing database, you can quickly test this out with a Northwind SQLite database available from
https://github.com/NetCoreApps/NorthwindAuto:

$ x download NetCoreApps/NorthwindAuto

As you’ll need to use 2 terminal windows, I’d recommend opening the project with VS Code which has great multi-terminal support:

$ code NorthwindAuto

The important parts of this project is the registering the OrmLite DB Connection, the above configuration and the local northwind.sqlite database, i.e:

container.AddSingleton<IDbConnectionFactory>(c =>
    new OrmLiteConnectionFactory(MapProjectPath("~/northwind.sqlite"), SqliteDialect.Provider));

Plugins.Add(new AutoQueryFeature {
    MaxLimit = 100,
    GenerateCrudServices = new GenerateCrudServices {}
});

Generating AutoQuery Types & Services

The development experience is essentially the same as Add ServiceStack Reference where you’ll need to run the .NET Core App in 1 terminal:

$ dotnet run

Then use the x dotnet tool to download all the AutoQuery & Crud Services for all tables in the configured DB connection:

$ x csharp https://localhost:5001 -path /crud/all/csharp

Updating Generated Services

If your RDBMS schema changes you’d just need to restart your .NET Core App, then you can update all existing dtos.cs with:

$ x csharp

i.e. the same experience as updating normal DTOs.

Generate Serializable Models for other Languages

Although it wasn’t a goal, since the implementation leverages ServiceStack’s existing infrastructure it’s able to easily generate Models for all ServiceStack Reference Supported Languages.

Where it becomes a useful tool to generate Serializable models for multiple languages, although in this case you’re likely just interested in the DTO types and not the AutoQuery operations which you can exclude with:

$ x typescript https://localhost:5001 -path /crud/all/typescript?ExcludeTypes=services

Using a Sharp App to generate DTO Data Models

If generating DTO Models for an existing database was you’re only goal, you don’t even need a .NET Core project as you could more easily do this with an Empty Sharp App with a single app.settings file that enables the above plugin.

Thanks to the mix gists support in all dotnet tools, you can download and configure an empty Sharp App with a single command.

Here’s the command I used to configure it to use techstacks.io PostgreSQL Database:

$ x mix autodto -replace DIALECT=postgresql -replace CONNECTION_STRING=$TECHSTACKS_DB

Where it just downloads the authdto gist app.settings and replaces the tokens.

It’s preferable to use an Environment Variable to configure your connection string, if it doesn’t exist you can set it for your terminal session:

$ set DB=My Connection String...
$ x mix autodto -replace DIALECT=postgresql -replace CONNECTION_STRING=$DB

Otherwise just download the app.settings and use VS Code to replace the DIALECT and CONNECTION_STRING placeholders manually:

$ x mix autodto

In any case you can run the dotnet tool in the directory with app.settings to run your Sharp App:

$ x

And just like above you can use the dotnet tool to generate Serializable DB Models in all supported languages:

$ x csharp https://localhost:5001 -path /crud/all/csharp?ExcludeTypes=services
$ x typescript https://localhost:5001 -path /crud/all/typescript?ExcludeTypes=services
$ x dart https://localhost:5001 -path /crud/all/dart?ExcludeTypes=services
$ x java https://localhost:5001 -path /crud/all/java?ExcludeTypes=services
$ x kotlin https://localhost:5001 -path /crud/all/kotlin?ExcludeTypes=services
$ x swift https://localhost:5001 -path /crud/all/swift?ExcludeTypes=services
$ x vbnet https://localhost:5001 -path /crud/all/vbnet?ExcludeTypes=services
$ x fsharp https://localhost:5001 -path /crud/all/fsharp?ExcludeTypes=services

Which can be updated as normal using their full language name or their wrist-friendly 2-letter abbreviation:

x cs
x ts
x da
x ja
x kt
x sw
x vb
x fs

Or put this in a .bat or .sh script to automate re-generation for all languages in a single command.

Customizing Code Generation

Being able to instantly generate AutoQuery Services for all your RDBMS tables is pretty cool, but it would be even cooler if you could easily customize the code-generation :slight_smile:

Together with the flexibility of the new declarative validation support you can compose a surprisingly large amount of your App’s logic using the versatility of C# to automate embedding your App’s conventions by annotating the declarative Request DTOs.

The existing code-generation already infers a lot from your RDBMS schema which you can further augment using the available GenerateCrudServices filters:

  • ServiceFilter - called with every Service Operation
  • TypeFilter - called with every DTO Type
  • IncludeService - a predicate to return whether the Service should be included
  • IncludeType - a predicate to return whether the Type should be included

For an example of this in action, here’s a typical scenario of how the Northwind AutoQuery Services could be customized:

  • Controlling which Tables not to generate Services for in ignoreTables
  • Which tables not to generate Write Crud Services for in readOnlyTables
  • Which tables to restrict access to in different roles in protectTableByRole
  • Example of additional validation to existing tables in tableRequiredFields
    • Adds the [ValidateNotEmpty] attribute to Services accessing the table and the [Required] OrmLite attribute for the Data Model DTO Type.
var ignoreTables = new[] { "IgnoredTable", }; // don't generate AutoCrud APIs for these tables
var readOnlyTables = new[] { "Region" };
var protectTableByRole = new Dictionary<string,string[]> {
    ["Admin"]    = new[] { nameof(CrudEvent), nameof(ValidationRule) },
    ["Accounts"] = new[] { "Order", "Supplier", "Shipper" },
    ["Employee"] = new[] { "Customer", "Order", "OrderDetail" },
    ["Manager"]  = new[] { "Product", "Category", "Employee", "EmployeeTerritory", "UserAuth", "UserAuthDetails" },
};
var tableRequiredFields = new Dictionary<string,string[]> {
    ["Shipper"] = new[]{ "CompanyName", "Phone" },
};

Plugins.Add(new AutoQueryFeature {
    MaxLimit = 100,
    GenerateCrudServices = new GenerateCrudServices
    {
        ServiceFilter = (op,req) => 
        {
            // Require all Write Access to Tables to be limited to Authenticated Users
            if (op.IsCrudWrite())
            {
                op.Request.AddAttributeIfNotExists(new ValidateRequestAttribute("IsAuthenticated"), 
                    x => x.Validator == "IsAuthenticated");
            }

            // Limit Access to specific Tables
            foreach (var tableRole in protectTableByRole)
            {
                foreach (var table in tableRole.Value)
                {
                    if (op.ReferencesAny(table))
                        op.Request.AddAttribute(new ValidateHasRoleAttribute(tableRole.Key));
                }
            }

            // Add [ValidateNotEmpty] attribute on Services operating Tables with Required Fields
            if (op.DataModel != null && tableRequiredFields.TryGetValue(op.DataModel.Name, out var requiredFields))
            {
                var props = op.Request.Properties.Where(x => requiredFields.Contains(x.Name));
                props.Each(x => x.AddAttribute(new ValidateNotEmptyAttribute()));
            }
        },
        TypeFilter = (type, req) => 
        {
            // Add OrmLite [Required] Attribute on Tables with Required Fields
            if (tableRequiredFields.TryGetValue(type.Name, out var requiredFields))
            {
                var props = type.Properties.Where(x => requiredFields.Contains(x.Name));
                props.Each(x => x.AddAttribute(new RequiredAttribute()));
            }
        },
        //Don't generate the Services or Types for Ignored Tables
        IncludeService = op => !ignoreTables.Any(table => op.ReferencesAny(table)) &&
            !(op.IsCrudWrite() && readOnlyTables.Any(table => op.ReferencesAny(table))),

        IncludeType = type => !ignoreTables.Contains(type.Name),
    }
});

To assist in code-generation a number of high-level APIs are available to help with identifying Services, e.g:

  • operation.IsCrud() - Is read-only AutoQuery or AutoCrud write Service
  • operation.IsCrudWrite() - Is AutoCrud write Service
  • operation.IsCrudRead() - Is AutoQuery read-only Service
  • operation.ReferencesAny() - The DTO Type is referenced anywhere in the Service (e.g. Request/Response DTOs, Inheritance, Generic Args, etc)
  • type.InheritsAny() - The DTO inherits any of the specified type names
  • type.ImplementsAny() - The DTO implements any of the specified interface type names

Typed Declarative Validate Attributes

Since the declarative Validation was announced there are now Typed Validation Attributes for all built-in Validators which you can use
instead of the string based [ValidateRequest] Type and [Validate] Property validator which they wrap, e.g:

[ValidateIsAuthenticated]            // or [ValidateRequest("IsAuthenticated")]
[ValidateIsAdmin]                    // or [ValidateRequest("IsAdmin")]
[ValidateHasRole(role)]              // or [ValidateRequest($"HasRole(`{role}`)")]
[ValidateHasPermission(permission)]  // or [ValidateRequest($"HasPermission(`{permission}`)")]

Property Validate Attributes

[ValidateNull]                       // or [Validate("Null")]
[ValidateNotEmpty]                   // or [Validate("NotEmpty")]
[ValidateLength(min,max)]            // or [Validate($"Length({min},{max})")]
//... same typed wrappers exist for all other built-in Fluent Validation Attributes

You can also do this for your own custom validators by sub classing the existing [Validate*] attributes, e.g:

public class ValidateIsAuthenticatedAttribute : ValidateRequestAttribute
{
    public ValidateIsAuthenticatedAttribute() : base("IsAuthenticated") { }
}

For more examples all built-in Validate attributes are defined in ValidateAttribute.cs.

Mixing generated AutoQuery Services & existing code-first Services

The expected use-case for these new features is that you’d create a new project that points to an existing database to bootstrap your project with code-first AutoQuery Services using the dotnet tool to download the generated types, i.e:

$ x csharp https://localhost:5001 -path /crud/all/csharp

At which point you’d “eject” from the generated AutoQuery Services (forgetting about this feature), copy the generated types into your ServiceModel project and continue on development as code-first Services just as if you’d created the Services manually.

But the GenerateCrudServices feature also supports a kind of “hybrid” mode where you can also just generate Services for any new AutoQuery Services for tables for which there are no existing services which you can download instead using the path:

$ x csharp https://localhost:5001 -path /crud/new/csharp

The existing /crud/all/csharp Service continues to return generated Services for all Tables but will stitch together and use existing types where they exist.

Auto Register generated AutoQuery Services

To recap we’ve now got an integrated scaffolding solution where we can quickly generate code-first AutoQuery Services and integrate them into our App to quickly build an AutoQuery Service layer around our existing database.

But can we raise the productivity level any higher? How about instead of manually importing the code-generated Services into our project we just tell ServiceStack to do it for us instead?

That’s what the magical AutoRegister flag does for us:

Plugins.Add(new AutoQueryFeature {
    GenerateCrudServices = new GenerateCrudServices {
        AutoRegister = true,
        //....
    }
});

It lets us tell ServiceStack to just auto register the new AutoQuery Services it already knows about and register them as if they were normal code-first Services that we had written ourselves.

As it’s important to understand how things work, it’s more accurately behind-the-scenes using the Metadata type structure it constructed in generating the Services & Types, i.e. the same structure of which it uses to project into its generated C#, TypeScript, (and other languages), that’s also what you manipulate in order to customize code-generation, is now also being used to generate .NET Types in memory using Reflection.Emit.

Barring any issues with the projection into IL, the end result is indistinguishable to a normal code-first ServiceStack Service created manually by a developer. Which is an important point as to why these solutions compose well with the rest of ServiceStack, just as an AutoQuery Service is a normal ServiceStack Service, these auto generated & auto registered ServiceStack Services are also regular Auto Query Services. The primary difference is that they only exist in a .NET Assembly in memory, not in code so they’re not “statically visible” to a C# compiler, IDE, tools, etc. But otherwise they’re regular typed ServiceStack Services and can take advantage of the ecosystem around Services including Add ServiceStack Reference & other Metadata Pages and Services, etc.

CreateCrudServices Instructions

Peeking deeper behind the AutoRegister flag will reveal that it’s just a helper for adding an empty CreateCrudServices instance, i.e. it’s equivalent to:

Plugins.Add(new AutoQueryFeature {
    GenerateCrudServices = new GenerateCrudServices {
        CreateServices = {
            new CreateCrudServices()
        }
        //....
    }
});

Multiple Schema’s and RDBMS Connections

This instructs ServiceStack to generate Services for the default Database connection, i.e. all tables in the Database of the default Connection String.

But you could also generate Services for multiple Databases and RDBMS Schemas simultaneously:

Plugins.Add(new AutoQueryFeature {
    GenerateCrudServices = new GenerateCrudServices {
        CreateServices = {
            new CreateCrudServices(),
            new CreateCrudServices { Schema = "AltSchema" },
            new CreateCrudServices { NamedConnection = "Reporting" },
            new CreateCrudServices { NamedConnection = "Reporting", Schema = "AltSchema" },
        }
        //....
    }
});

These will generated Types with the Multitenancy NamedConnection & OrmLite [Schema] attribute required for routing AutoQuery Services to use the appropriate RDBMS connection of Schema.

With this you could have a single API Gateway with access to multiple RDBMS Tables & Schema’s although be mindful that it doesn’t work well when there are identical table names in each as it has to go back and rewrite the Metadata References to use a non-ambiguous name, first using the NamedConnection, then the schema then a combination when both exists, if it’s still ambiguous it gives up and ignores it.

Explore Time!

We now have all the features we need to quickly servicify an existing database that we can easily customize to apply custom App logic to further protect & validate access.

So you can quickly explore these new features locally, I’ve further built on the Northwind example to demonstrate the features above in the new github.com/NetCoreApps/NorthwindCrud project which you can download & run with:

$ x download NetCoreApps/NorthwindCrud
$ cd NorthwindCrud
$ dotnet run

The demo is also configured with other new features in incoming release including Crud Events in Startup.cs:

// Add support for auto capturing executable audit history for AutoCrud Services
container.AddSingleton<ICrudEvents>(c => new OrmLiteCrudEvents(c.Resolve<IDbConnectionFactory>()));
container.Resolve<ICrudEvents>().InitSchema();

As well as support for dynamically generated db rules in Configure.Validation.cs:

services.AddSingleton<IValidationSource>(c => 
    new OrmLiteValidationSource(c.Resolve<IDbConnectionFactory>()));

appHost.Resolve<IValidationSource>().InitSchema();

To be able to test the custom code generation the example is pre-populated with 3 users with different roles in Configure.Auth.cs, where you can find their Emails and Passwords published on the home page:

northwindcrud-home
northwindcrud-home1192×1230 85.6 KB

One more thing :slight_smile:

The Northwind Crud Home Page also reveals ServiceStack Studio, something I’m very excited about that adds even more value to your AutoQuery Services (and ServiceStack instances in general) which I’ll announce in a separate thread.

2 Likes