Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migration creates performance issue in case of huge records in LineAttribute table #4202

Open
awais786 opened this issue Nov 6, 2023 · 0 comments

Comments

@awais786
Copy link

awais786 commented Nov 6, 2023

django-oscar==3.2.2

Issue Summary

This migration brings all records and then update them 1/1.

Proposed solution

Instead doing some thing like this can make it more reliable.

LineAttribute = apps.get_model("basket", "LineAttribute")

batch_size = 100  # Set your desired batch size
update_batch = []

# Use iterator() to avoid loading the entire queryset into memory
for at in LineAttribute.objects.all().iterator():
    try:  # if the value is already valid JSON, continue
        json.loads(at.value)
        continue
    except json.JSONDecodeError:
        pass

    try:  # to parse the string as Python, then convert to JSON, then continue
        val = literal_eval(at.value)
        at.value = json.dumps(val, cls=DjangoJSONEncoder)
        update_batch.append(at)
        continue
    except ValueError:
        pass

 # Perform a bulk update 
    if len(update_batch) >= batch_size:
        LineAttribute.objects.bulk_update(update_batch, ['value'])
        update_batch = []

# Perform a final bulk update for less than 100
if update_batch:
    LineAttribute.objects.bulk_update(update_batch, ['value'])

@awais786 awais786 changed the title This migration 0010_convert_to_valid_json.py creates performance issue in case of huge records in LineAttribute table. migration creates performance issue in case of huge records in LineAttribute table Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant