PocoDynamo + QueryAsync + Paging

1

Hi all,

I m trying to understand how paging works with query async on PocoDynamo. Here is what I m doing:

var q = _pocoDynamo.FromQuery()
.KeyCondition(x => x.PartitionKey == partitionKey);

var queryResultPublic = await _pocoDynamo.QueryAsync(q, token);

Any ideas?

2

Update:

For .NET Core / .NET Framework v4.72+ the QueryAsync & ScanAsync APIs now return an IAsyncEnumerable<T> which pages transparently behind the scenes as you’re iterating through the stream:

await foreach (var result in pocoDynamo.QueryAsync(q, token)) 
{
    //process(result)
}

The sync APIs return an IEnumerable<T> and will transparently page & lazily perform multiple queries behind the scenes as you’re iterating through the IEnumerable resultset, for .NET v4.5 the QueryAsync APIs need to return a concrete List<T> so it executes & fetches the entire list results before returning, there’s only an option to limit how many results should be returned with an optional limit param, e.g:

var results = await _pocoDynamo.QueryAsync(q, 100, token);

To be able to fetch the next batch of results you’d need to get the LastEvaluatedKey of the last response, the only QueryAsync API that lets you access DynamoDB’s QueryResponse to be able to fetch the LastEvaluatedKey is the converter API, so you should be able to page in 100 result batches with something like:

string lastKey = null;
q.Limit = 100;
do {
    q.ExclusiveStartKey = lastKey;
    var nextResults = await _pocoDynamo.QueryAsync(q, r => {
        lastKey = r.LastEvaluatedKey;
        return r.ConvertAll<T>();
    }, token);

    //... process(nextResults);

} while (!string.IsNullOrEmpty(lastKey));
3

That’s great. I ll use the sync API. Thank you.

Just to avoid creating another question. Can you also show me how I could do a Query where partitionKey (String) = XYZ (done) and rowKey (String) between two values?

4

I don’t understand the question, but the only query options available are on the QueryRequest DTO.

5

Assume we have a table with three columns: PK (String - our Partition Key), SK (String - our Sort Key) and Age (Int - some other column).

I m looking to Query efficiently (so not scan) this table given a specific PK and a min / max SK. So get a list of DTOs back for everything that matches the given PK and any SK between two values.

6

If you’re only needing to query the Hash & Range keys you should be able to do both in the Key Condition expression, something like:

var q = db.FromQuery<T>(x => x.Id == id && Dynamo.Between(x.RangeKey, min, max));

Otherwise you could try the querying the between expression in the FilterExpression, see Querying docs for examples.

I’m assuming that will work as it’s only querying the Hash & Range Keys but ultimately it’s up to DynamoDB what queries it allows, the typed expression is just a convenience wrapper in populating the QueryRequest DTO, it doesn’t enable performing queries DynamoDB doesn’t allow.

7

Perfect - That Dynamo.Between was exactly what I was after.

Thank you.

1 Like
8

FYI I’ve just upgraded the QueryAsync/ScanAsync APIs to use IAsyncEnumerable<T> for .NET Framework v4.72+ or .NET Core so you can lazily query them like their corresponding sync IEnumerable<T> APIs.

This change is available from the latest v5.9.3 that’s now available on MyGet.

9

Thank you @mythz - Just a qq.

Should all the [Alias] [HashKey] etc attributes work with Query and QueryAsync?

Basically I have a bunch of aliases on my DTOs. Puts and GetItem / GetItems work fine.Query and QueryAsync though don’t seem to take these aliases into account.

For example

[Alias("test_dto")]
public class TestDTO
{
    [RangeKey]
    [Alias("row_key")]
    public String RowKey { get; set; }

    [HashKey]
    [Alias("partition_key")]
    public String PartitionKey { get; set; }
}
  • PutItemAsync works fine
  • GetItemAsync works fine
  • GetItemsAsync works fine

This however doesn’t work

        var q = _pocoDynamo.FromQuery<TestDTO>(x => x.PartitionKey == partitionKey);

        var queryResultPublic = _pocoDynamo
            .Query<TestDTO>(q)
            .Skip(10)
            .Take(10).ToList();

I checked fiddler and it’s sending PartitionKey as PartitionKey and not partition_key therefore resulting in an exception as this is a required field in the query.

Hope this makes sense?

10

Query & Scans now use aliases in the latest v5.9.3 that’s now available on MyGet.

1 Like
11

Thank you! When do you guys generally make nuget releases? i.e. when will 5.9.3 end up on nuget?

Thanks

12

There’s no set release dates for releases, but aiming for next release towards end of month.