DynamoDB Query considerably slow when creating resource Table after each query call

[GuillermoOvejeroSanchez]

Problem Description

DynamoDB Query is considerably slow when creating resource Table after each query call. We have tested creating the Table only once before the query calls and it improves performance significantly when making thousands of requests

Version used from boto3 is latest as today (15 March 2023) boto3==1.26.91

Tested with: @EduardoHerreraM

Code

docker run -d --rm -p 8042:8000  --name dynamodb-local amazon/dynamodb-local

import uuid
import time
from typing import Dict, Any

import boto3
from boto3.dynamodb.conditions import Key

def save_message_in_table(
    dynamodb_client, table: str, message: Dict[str, Any]
) -> None:
    with dynamodb_client.Table(table).batch_writer() as batch:
        batch.put_item(Item=message)


dynamodb = boto3.resource(
                "dynamodb",
                region_name="eu-west-1",
                endpoint_url="http://localhost:8042/",
            )

# Assuring the table doesn't exist
try:
    table = dynamodb.Table("Test")
    table.delete()
except Exception:
    pass

dynamodb.create_table(
        TableName="Test",
        KeySchema=[
            {"AttributeName": "key", "KeyType": "HASH"},
        ],
        AttributeDefinitions=[
            {"AttributeName": "key", "AttributeType": "S"},
        ],
        ProvisionedThroughput={"ReadCapacityUnits": 10, "WriteCapacityUnits": 10}
    )

number_of_records = 3000


random_records = [
    {
        "key": str(x),
        "value": {
            "a": str(uuid.uuid4()),
            "b": str(uuid.uuid4()),
            "c": str(uuid.uuid4()),
        }
    }
    for x in range(number_of_records)
]

for record in random_records:
    save_message_in_table(
        dynamodb_client=dynamodb,
        table="Test",
        message=record
    )

# Test creating the table each time it's required
for t in range(2):
    t1 = time.time()
    for x in range(number_of_records):
        table = dynamodb.Table("Test")
        records = table.query(KeyConditionExpression=Key("key").eq(str(x)))
        record = records["Items"][0]
    print(f"Time for query by creating the table each time:\t{time.time() - t1}")


# Test creating the table only once
for t in range(2):
    t1 = time.time()
    table = dynamodb.Table("Test")
    for x in range(number_of_records):
        records = table.query(KeyConditionExpression=Key("key").eq(str(x)))
        record = records["Items"][0]
    print(f"Time for query by creating the table only once:\t{time.time() - t1}")

docker stop dynamodb-local

Results

Tested on different laptops:

Laptop 1:

Time for query by creating the table each time:	18.486364603042603
Time for query by creating the table each time:	19.328848838806152

Time for query by creating the table only once:	14.717013359069824
Time for query by creating the table only once:	14.453866004943848

Laptop 2:

Time for query by creating the table each time:    10.726242542266846
Time for query by creating the table each time:    10.750989437103271

Time for query by creating the table only once:    7.240025758743286
Time for query by creating the table only once:    7.214364767074585

Questions

1. Is there any problem to only instantiate once the Table resource?
2. This form of usage should be on the docs?

[tim-finnigan]

Here is the boto3 DynamoDB resources documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb/table/index.html. (Also general resources documentation for reference).

I'm not sure if I understand your use case here — can you just create the table resource once and reuse it?

[GuillermoOvejeroSanchez]

@tim-finnigan Yes, we can reuse the table resource. But that's the question, are there any drawbacks doing that? What's the correct way of doing it.