Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process deleting projects as background task #1060

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jayanth-kumar-morem
Copy link
Contributor

  1. Fix Default Filtering in ProjectFilterSet:

    • Modified the ProjectFilterSet to include is_marked_for_deletion in the default filtering when is_archived is not provided.
    • This ensures that projects marked for deletion are excluded from default views.
  2. Migration for is_marked_for_deletion:

    • Added a migration (0052_project_is_marked_for_deletion.py) to introduce the is_marked_for_deletion field in the Project model.
  3. Updated Project Model:

    • Added is_marked_for_deletion field to the Project model with a default value of False.
  4. Modified ProjectListView to Exclude Marked Projects:

    • Modified the get_queryset method in ProjectListView to only fetch projects with is_marked_for_deletion=False.
  5. Introduced mark_for_deletion Method:

    • Added a mark_for_deletion method in the Project model to set the is_marked_for_deletion flag and save the model.
  6. Changed delete Method in Project Model:

    • Renamed the delete method to delete_action in the Project model to avoid conflicts.
    • Introduced a new delete method that marks the project for deletion and enqueues a background deletion task.
  7. Background Deletion Task:

    • Created a background deletion task using django_rq to handle project deletion asynchronously.
    • Checks if the project is still marked for deletion before proceeding.
    • If an error occurs during deletion, updates project status and logs an error.
  8. Updated ProjectActionView Success Message:

    • Modified the success message in ProjectActionView for the "delete" action to provide a more informative message based on the count.

Fixes: #1002

1. Fix Default Filtering in ProjectFilterSet:
   - Modified the `ProjectFilterSet` to include `is_marked_for_deletion` in the default filtering when `is_archived` is not provided.
   - This ensures that projects marked for deletion are excluded from default views.

2. Migration for `is_marked_for_deletion`:
   - Added a migration (`0052_project_is_marked_for_deletion.py`) to introduce the `is_marked_for_deletion` field in the `Project` model.

3. Updated `Project` Model:
   - Added `is_marked_for_deletion` field to the `Project` model with a default value of `False`.

4. Modified `ProjectListView` to Exclude Marked Projects:
   - Modified the `get_queryset` method in `ProjectListView` to only fetch projects with `is_marked_for_deletion=False`.

5. Introduced `mark_for_deletion` Method:
   - Added a `mark_for_deletion` method in the `Project` model to set the `is_marked_for_deletion` flag and save the model.

6. Add `delete_in_background` Method in `Project` Model:
   - Introduced a new `delete_in_background` method that marks the project for deletion and enqueues a background deletion task.

7. Background Deletion Task:
   - Created a background deletion task using `django_rq` to handle project deletion asynchronously.
   - Checks if the project is still marked for deletion before proceeding.
   - If an error occurs during deletion, updates project status and logs an error.

8. Updated `ProjectActionView` Success Message:
   - Modified the success message in `ProjectActionView` for the "delete" action to provide a more informative message based on the count.

Fixes: nexB#1002
Signed-off-by: Jayanth Kumar <jknani111@gmail.com>
Copy link
Member

@tdruez tdruez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty good start. See my comments for improvements.

scanpipe/filters.py Outdated Show resolved Hide resolved
scanpipe/models.py Outdated Show resolved Hide resolved
scanpipe/models.py Outdated Show resolved Hide resolved
scanpipe/models.py Outdated Show resolved Hide resolved
scanpipe/tasks.py Outdated Show resolved Hide resolved
scanpipe/tasks.py Outdated Show resolved Hide resolved
scanpipe/tasks.py Outdated Show resolved Hide resolved
scanpipe/views.py Outdated Show resolved Hide resolved
@pombredanne
Copy link
Member

This is a good approach and well explained... it is likely missing not much to get this merged... @jayanth-kumar-morem do you mind looking into some of the comments?

- Remove comments
- Use django rq
- Use update instead of save

Signed-off-by: Jayanth Kumar <jknani111@gmail.com>

active_count = Project.objects.filter(is_archived=False).count()
active_count = Project.objects.filter(
is_archived=False, is_marked_for_deletion=False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use active() here as well, no?


def delete_in_background(self):
# Mark the project for deletion and enqueue background deletion task
self.mark_for_deletion()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can put self.update(is_marked_for_deletion=True) directly here, no need for the mark_for_deletion method yet since it's a one-liner that is only used in one place.

self.update(is_marked_for_deletion=True)

def delete_in_background(self):
# Mark the project for deletion and enqueue background deletion task
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A docstring would be better than a comment.

project.delete()
except Exception as e:
info(f"Deletion failed: {str(e)}", project.pk)
project.update(is_marked_for_deletion=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering is this line is of any use, the is_marked_for_deletion must be already True to reach this line, no?

Merge remote-tracking branch 'upstream/main' into 1002
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failure to delete large project
3 participants