Postgres Array Types as params for function

Not really sure how to approach array params, e.g. int[] for a function. Maybe we’re using the wrong method. Tried SqlList<T> and Select<T>

Example:

SELECT * FROM f_service_stack_function_int_array( int[] );

If you:

var sql = "SELECT * FROM f_service_stack_function_int_array(@paramValue)";
db.SqlList<MyFuncModel>(sql, new { paramValue = { 123, 456 } });

OrmLite expands the @paramValue array into individual integers so function call is:

// wrong
SELECT * FROM f_service_stack_function_int_array( integer, integer )
// should be
SELECT * FROM f_service_stack_function_int_array( int[] )

You could ARRAY[@paramValue], e.g.

var sql = "SELECT * FROM f_service_stack_function_int_array(ARRAY[@paramValue])";
db.SqlList<MyFuncModel>(sql, new { paramValue = { 123, 456 } });

You could also cast as array ARRAY[@paramValue]::int[], e.g.

var sql = "SELECT * FROM f_service_stack_function_int_array(ARRAY[@paramValue]::int[])";
db.SqlList<MyFuncModel>(sql, new { paramValue = { 123, 456 } });

HOWEVER it doesn’t work when the array param is null.

See tests below.

private const string DropFunctionArray = "DROP FUNCTION IF EXISTS f_service_stack_function_{0}_array({0});";

private const string WhitelistFuncAndVarNamePattern = "[^0-9a-zA-Z]+";

public class ServiceStackTypeFunctionResultIntArray
{
    public int Id { get; set; }
    // [CustomField("int[]")]
    public int[] Val { get; set; }
}

[Test]
public void Can_execute_function_with_int_array_param()
{
    using (var db = OpenDbConnection())
    {
        const string pgTypeToTest = "int[]";
        // remove [] from type, for func name.
        var pgFuncName = Regex.Replace(pgTypeToTest, WhitelistFuncAndVarNamePattern, "");
        int[] testVal = { 123, 456 };

        // if function already exists drop before create (can't change result)
        db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));

        db.ExecuteSql(CreateFunctionArray.Fmt(pgFuncName, pgTypeToTest));
        db.GetLastSql().Print();

        // WORKS: ARRAY[@paramValue]::int[]
        // var sql = "SELECT * FROM f_service_stack_function_{0}_array(ARRAY[@paramValue]::int[]);".Fmt(pgFuncName);

        // WORKS 2: ARRAY[@paramValue] (no cast)
        // var sql = "SELECT * FROM f_service_stack_function_{0}_array(ARRAY[@paramValue]);".Fmt(pgFuncName);

        // Doesn't work: @paramValue
        // MessageText: Npgsql.PostgresException : 42883: function f_service_stack_function_int_array(integer, integer) does not exist
        var sql = "SELECT * FROM f_service_stack_function_{0}_array(@paramValue);".Fmt(pgFuncName);

        var rows = db.SqlList<ServiceStackTypeFunctionResultIntArray>(
            sql,
            new
            {
                paramValue = testVal
            });

        db.GetLastSql().Print();

        Assert.That(rows.Count, Is.EqualTo((10)));
        Assert.That(rows[0].Val, Is.EqualTo(testVal));

        db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));
    }
}

[Test]
public void Can_execute_function_with_int_array_param_as_null()
{
    using (var db = OpenDbConnection())
    {
        const string pgTypeToTest = "int[]";
        // remove [] from type, for func name.
        var pgFuncName = Regex.Replace(pgTypeToTest, WhitelistFuncAndVarNamePattern, "");
        int[] testVal = null;

        // if function already exists drop before create (can't change result)
        db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));

        db.ExecuteSql(CreateFunctionArray.Fmt(pgFuncName, pgTypeToTest));
        db.GetLastSql().Print();

        var sql = "SELECT * FROM f_service_stack_function_{0}_array(@paramValue);".Fmt(pgFuncName);

        var rows = db.SqlList<ServiceStackTypeFunctionResultIntArray>(
            sql,
            new
            {
                paramValue = testVal
            });

        db.GetLastSql().Print();

        Assert.That(rows.Count, Is.EqualTo((10)));
        Assert.That(rows[0].Val, Is.Null);

        db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));
    }
}

There is no standard array type in RDBMS’s so OrmLite expands them as documented in Parameterized IN Values.

For PostgreSQL Arrays you can just expand them in the SQL, see the Custom SQL using PostgreSQL Arrays examples.

We understand… however EFCore using Npgsql can understand it. Array Type Mapping | Npgsql Documentation

While probably rare, SQL Server (via custom type) can use arrays for tables/functions. Using custom types as arrays in T-SQL | Microsoft Learn

We can inconveniently use NpgsqlCommand and add a parameter and use arrays.

int[] testVal = { 123, 456 };
var command = new Npgsql.NpgsqlCommand(sql, (NpgsqlConnection)db.ToDbConnection());
command.Parameters.AddWithValue("paramValue", testVal);
// Question: is there a way to translate that npgsql command to a data Model???

Sql Injection issue
If dealing with string[] string arrays, the potential for sql injection is high and ARRAY[@myArrayParam] with array expansion could spell disaster.

Null value issue
Plus we have to be able to deal with null values as ARRAY[null] is not the same as null. This would require us to build out logic in case of nulls for every time we need to use an array param.

How would that even work, b/c ARRAY[@param] doesn’t work for null values so you would almost need 2 versions of a sql statement to handle the null.

-- if not null
SELECT * from f_my_func(@param1, @param2, ARRAY[@param3]);
-- if null
SELECT * from f_my_func(@param1, @param2, NULL::int[]);

That would then have to be duplicated every time to handle null possibly causing lots of duplicated code. Maybe SS string.Format() (or SS’s .Fmt()) could be used in some situations.

We understand the need for OrmLite to keep backwards compatibility, but there should be a way to hiddenly/not-visibly use Npgsql params indirectly thru a different extension method, attribute, parameter or something else.

Currently, we cannot specify [CustomField("int[]")] to bypass the array expansion, e.g. myArray.Join(",").

Note: db.ExecuteProcedure(...) allows us to use the [CustomField(...)] attribute for arrays, but doesn’t provide results table and it sets CommandType.StoredProcedure which isn’t ideal when db changes happen as it makes param naming strict and tightly coupled to the db.

Note 2: db.ExecuteProcedure(...) isn’t helpful for Postgres as they’re functions and if you want to use the new stored procedures (pg v11) you have to call a regular command anyways.

Example using existing [CustomField("int[]")] (doesn’t work):
We cannot do the following as it still expands the array:

        public class ServiceStackTypeFunctionParamIntArray
        {
            [CustomField("int[]")]
            public int[] paramValue { get; set; }
        }

        [Test]
        public void Can_execute_function_with_int_array_param_dto()
        {
            using (var db = OpenDbConnection())
            {
                const string pgTypeToTest = "int[]";
                // remove [] from type, for func name.
                var pgFuncName = Regex.Replace(pgTypeToTest, WhitelistFuncAndVarNamePattern, "");
                int[] testVal = { 123, 456 };

                // if function already exists drop before create (can't change result)
                db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));

                db.ExecuteSql(CreateFunctionArray.Fmt(pgFuncName, pgTypeToTest));
                db.GetLastSql().Print();

                var sql = "SELECT * FROM f_service_stack_function_{0}_array(@paramValue);".Fmt(pgFuncName);

                var rows = db.SqlList<ServiceStackTypeFunctionResultIntArray>(
                    sql,
                    new ServiceStackTypeFunctionParamIntArray
                    {
                        paramValue = testVal
                    });

                db.GetLastSql().Print();

                Assert.That(rows.Count, Is.EqualTo((10)));
                Assert.That(rows[0].Val, Is.Null);

                db.ExecuteSql(DropFunctionArray.Fmt(pgFuncName, pgTypeToTest));
            }
        }

We see the following as possible solutions.

  1. Turn off array expansion in the SqlList<T>, Select<T> and etc. with an additional parameter/overload and use the correct NpgsqlType instead.
  2. Maybe allow [CustomField(...)] attribute in all sql queries to bypass array expansion, e.g. SqlList<T>(...), Select<T>(...), etc… This would work similar to ExecuteProcedure.
  3. and/or add an overload for ExecuteProcedure(...) that acts similar to SqlList<T>(...) or Select<T>(...) but doesn’t expand array params that can have custom sql and return a result dto and doesn’t use CommandType.StoredProcedure.
  4. and/or check for a new ArrayParam<T>() as a param, and if it’s param is ArrayParam then don’t expand the array and use the corresponding array NpgsqlType. (not ideal)

We think option 1 (turn off array expansion) and/or option 2 (allow [CustomField(...)]) may be the better options. We doubt [CustomField(...)] attribute is widely used on param objects currently and most probably use anonymous param objects.

What are your thoughts?

The cleanest way might be the [CustomField()] attribute.

Also, by doing this could allow json and jsonb parameters to work (if they don’t work… I haven’t tried yet as a param.)

That attribute would go on the param dto of the prop you want to skip array expansion.

Example:

        public class ServiceStackTypeFunctionParamIntArray
        {
            [CustomField("int[]")]
            public int[] paramValue { get; set; }
        }

Check for that attribute on SetParameters:

Skip to the else {} if that attribute exists

Make sure to lookup the correct param type from:

Note: I didn’t see a Dictionary to lookup Postgres types based upon .NET type. That might be better and then you don’t need to lookup by string "int[]". Could then use a different attribute then.

This would be similar to ExecuteProcedure(...) use SetParameter.

Please look at the example in the docs, there is no SQL injection issue embedding any value types like int[] and for string[] you can use SqlInValues to escape the string Params.

There is also no null param issues as empty arrays expand to empty ARRAY[].

AFAIC there’s no issue using SQL expansion and changing the default behavior that’s been in OrmLite for years is not an option.

If you want different behavior you can also look at adding your own custom extension methods locally that does what you want for PostgreSQL as I don’t want the unnecessary added confusion and complexity added to OrmLite’s surface area.

We did. It doesn’t work when the value is null.

int[] testVal = null;
var sql = "SELECT * FROM f_my_func(ARRAY[@paramValue]::int[]);"
var rows = db.Select<ServiceStackTypeFunctionResultIntArray>(
                    sql,
                    new
                    {
                        paramValue = testVal
                    });
Expected: null
  But was:  < 0 >

This is still using a param not SQL expansion, please Refer to the example in the docs.

Here’s an example we use in TechStacks of both int[] and string[] Arrays in PostgreSQL:

I can look into making the API nicer for strings, something like:

$”ARRAY[{db.Dialect().SqlSpread(slugs)}]”

And for nicer UX make the same syntax support other arrays as well.

Maybe I’m still misunderstanding but empty array returns empty, not null. Empty and null are different and if we need to pass null to the function we can’t.

var sql = "SELECT * FROM f_my_func(ARRAY[]::int[]);"

Expected: null
But was: <empty>

Which is exactly the desired behavior, an empty collection is expected an empty collection not null.

If you want null you can use your own API and elide away the ARRAY[] syntax in favor of null:

$”{PgSql.ToArray(array)}”

Edit: since it results in nicer typed code the new PgSql.Array() API is now available from v5.8.1 where you can use the same API for value type or strings alike, e.g:

$”{PgSql.Array(1,2,3)}”
var strings = new[]{ "A","B","C" };
$”{PgSql.Array(strings)}”

Which will make the TechStacks code above nicer, string[]:

q.And($"{PgSql.Array(labelSlugs)} && labels");  //equivalent to:
q.And($"ARRAY[{new SqlInValues(labelSlugs).ToSqlInString()}] && labels"); 

And for int[]:

q.And($"{PgSql.Array(request.AnyTechnologyIds)} && technology_ids") //equivalent
var techIds = request.AnyTechnologyIds.Join(",")
q.And($"ARRAY[{techIds}] && technology_ids")

If you want null instead of an empty array you can use the overload:

PgSql.Array(new string[0], nullIfEmpty:true)      //= null
PgSql.Array(new[]{"A","B","C"}, nullIfEmpty:true) //= ARRAY['A','B','C']

We’ll have to make our own API to use null then. In SQL null is preferred rather than empty when inserting records and filtering. At least for us.

Filtering on empty would be a pain.

select * from ss_test_empty where array_length(intarray, 1) is null;

Is there already a SS extension that would take a npgsql command and convert it to a data Model?

int[] testVal = { 123, 456 };
var command = new Npgsql.NpgsqlCommand(sql, (NpgsqlConnection)db.ToDbConnection());
command.Parameters.AddWithValue("paramValue", testVal);
// Question: is there a way to translate that npgsql command to a data Model???

From 9.4 you can use cardinality():

cardinality(anyarray)

There are APIs that accept IEnumerable<IDbDataParameter> I’ve not tested it with ARRAY[] Params though.

I should have been more clear.

Is there a SS extension way to take a NpgsqlCommand and execute it and return a data Model? e.g. ExecuteAndReturnDataModel<T> see below.

int[] testVal = { 123, 456 };
var command = new Npgsql.NpgsqlCommand(sql, (NpgsqlConnection)db.ToDbConnection());
command.Parameters.AddWithValue("paramValue", testVal);
// below is what I'm looking for
T dataModel = command.ExecuteAndReturnDataModel<T>();

Kind of like .Select<T> or .SqlList<T>

Have a look at the docs stored procedures examples, e.g. you can convert it into a list from a command with:

var results = cmd.ConvertToList<LetterFrequency>();

Thanks for all the help!!!

I think we’ll create a extension / API that would wrap up the following using NpgsqlParameter.

var p = new NpgsqlParameter("paramValue", NpgsqlDbType.Array | NpgsqlDbType.Integer);
p.Value = testVal;
var sql = "SELECT * FROM f_my_func(@paramValue)";
var rows = db.Select<ServiceStackTypeFunctionResultIntArray>(sql, new [] { p });

This seems to work in the small amount of testing done so far with Select<T>() and SqlList<T>()

1 Like

Cool, I’ve also checked in a new Dialect Spread API in this commit which provides a nicer UX for SQL expansion, e.g:

var dialect = db.Dialect();
dialect.SqlSpread(1, 2, 3)         //= 1,2,3
dialect.SqlSpread("A", "B", "C")   //= 'A','B','C'
dialect.SqlSpread("A'B", "C\"D")   //= 'A''B','C"D'
dialect.SqlSpread(new int[0])      //= ""
dialect.SqlSpread((string[])null)  //= ""
1 Like

I’ve also added a new PgSql.Param API in this commit which will make it easier to create NpgsqlParameter’s, e.g:

var p = PgSql.Param("paramValue", testVal);
var sql = "SELECT * FROM f_my_func(@paramValue)";
var rows = db.Select<ServiceStackTypeFunctionResultIntArray>(sql, new [] { p });

This change is now available from the latest v5.8.1 that’s now available on MyGet.

1 Like

Love it! :heart: Makes it much easier! THANKS @mythz!!!

FYI as it results in a nicer typed API I’ve also implemented the PgSql.Array() API floated earlier, see:

This is now my preferred approach for ARRAY params, there’s also null for empty collection option PgSql.Array(array, nullIfEmpty:true).

1 Like

One thing to note, is that if there are two function signatures for the same named function:

-- 1
f_my_func(int, int, text[])
-- 2
f_my_func(int, int, int[])

the null will need to be cast with the type, e.g. null::int[] or null::text[].

ERROR: function f_my_func(int, int, unknown) is not unique
Hint: Could not choose a best candidate function. You might need to add explicit type casts.

This scenario is probably small and may not need to be addressed in the API. Probably could be derived from the array type tho.