DynamoDbEnhancedClient limit paginated result

[tomoaki3284]

Referencing from this issue: #3226
And have further clarifying question.

From what I understand, querying list of item on DynamoDbTable from DynamoDbEnhancedClient, it will automatically paginate the result and fetch ALL the items from the table. I am concern about the performance.

I am still not clear of "PageIterable will send a new request to obtain more items if asked for more pages" from the issue I reference. Assuming ddbTable has 1k records, users want to get 10 items for one page. By simply setting limit on PageIterable, it will prevent fetching 1k items? e.g.:

DynamoDbTable<Object> table = this.ddbEnhancedClient.table(TABLE_NAME, TABLE_SCHEMA);
QueryEnhancedRequest request = QueryEnhancedRequest.builder().queryConditional(
                        QueryConditional.sortBeginsWith(
                                Key.builder()
                                        .partitionValue(pv)
                                        .sortValue(sv)
                                        .build()))
                .consistentRead(true)
                .scanIndexForward(false)
                .limit(10)
                .exclusiveStartKey(token)
                .build();
PageIterable<Object> result = table.query(request);
List<Object> page1 = result.iterator().next().items();

or

QueryEnhancedRequest request = QueryEnhancedRequest.builder().queryConditional(
                        QueryConditional.sortBeginsWith(
                                Key.builder()
                                        .partitionValue(pv)
                                        .sortValue(sv)
                                        .build()))
                .consistentRead(true)
                .scanIndexForward(false)
                .limit(10)
                .exclusiveStartKey(token)
                .build();
PageIterable<Object> result = table.query(request);
List<Object> page1 = result.stream().limit(1).forEach(o -> o.items());

I am concern about the performance too (fetching 1k items is too much), and hoping this will only fetch 10 data from the table.

[debora-ito]

Hi @tomoaki3284,

To be clear, even if no limit parameter is provided, a single query operation will read up to a maximum of 1 MB of data, according to the Query API Reference.

By simply setting limit on PageIterable, it will prevent fetching 1k items?

Yes. I'll use my code example in the issue you referenced to try to clarify:

    QueryEnhancedRequest request = QueryEnhancedRequest.builder()
                .queryConditional(QueryConditional.keyEqualTo(k -> k.partitionValue("001")))
                .limit(10)  // 10 items per response
                .scanIndexForward(true)
                .build();

    PageIterable<Customer> result = customerTable.query(request);

    result.stream()
          .limit(1)   // number of paginated calls
          .forEach(p -> p.items().forEach(customer -> System.out.println(customer.getName())));

In the example above, 10 is the limit of items in the DynamoDB Query response, and 1 is the number of times PageIterable stream will paginate, meaning number of times it will make query calls. In this example, for a table with 1k items, result will bring 10 items.
Removing the stream limit means the SDK will automatically paginate through all the 1k table items.

If you still have questions, I recommend you to run some local test experiments by enabling the verbose wirelogs (instructions here), change the query limit and the pagination limit and see in the logs how many query requests the Java SDK will do and how many items each response contains.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.