Added support for Overload attack #2337

CNOCycle · 2023-11-28T12:56:14Z

Description

This pull request adds the support of the Overload Attack proposed in [1].

[1] Overload: Latency Attacks on Object Detection for Edge Devices. [Paper]

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

Unit Test
Notebook Example

Test Configuration:

OS: Ubuntu 20.04
Python version: 3.8.12
ART version or commit number: 0400813
PyTorch version: 1.13.1+cu116
cudnn version: 8302

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
My changes have been tested using only GPU devices

beat-buesser · 2023-11-30T15:41:22Z

Hi @CNOCycle Thank you very much for your pull request! Could you please change the target and rebase on to dev branch dev_1.18.0?

codecov-commenter · 2023-11-30T15:58:33Z

Codecov Report

Merging #2337 (39f7587) into dev_1.18.0 (0400813) will decrease coverage by 15.73%.
The diff coverage is 15.38%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@               Coverage Diff               @@
##           dev_1.18.0    #2337       +/-   ##
===============================================
- Coverage       85.60%   69.87%   -15.73%     
===============================================
  Files             324      325        +1     
  Lines           29326    29443      +117     
  Branches         5407     5426       +19     
===============================================
- Hits            25104    20574     -4530     
- Misses           2840     7674     +4834     
+ Partials         1382     1195      -187

Files	Coverage Δ
art/attacks/evasion/__init__.py	`98.24% <100.00%> (+0.03%)`	⬆️
art/attacks/evasion/overload.py	`14.65% <14.65%> (ø)`

... and 83 files with indirect coverage changes

tests/attacks/evasion/test_overload_attack.py

+    try:
+        import torch
+        model = torch.hub.load('ultralytics/yolov5:v7.0',  model='yolov5s')
+        x = np.random(0.0, 1.0, size=(100, 3, 640, 640))


art/attacks/evasion/overload.py

+
+                idx_min = torch.argmin(scores)
+                grid_min = grid_box[idx_min]
+                x1, y1, x2, y2 = grid_min.int()


art/attacks/evasion/overload.py

+
+                idx_min = torch.argmin(scores)
+                grid_min = grid_box[idx_min]
+                x1, y1, x2, y2 = grid_min.int()


art/attacks/evasion/overload.py

+
+                idx_min = torch.argmin(scores)
+                grid_min = grid_box[idx_min]
+                x1, y1, x2, y2 = grid_min.int()


art/attacks/evasion/overload.py

+
+                idx_min = torch.argmin(scores)
+                grid_min = grid_box[idx_min]
+                x1, y1, x2, y2 = grid_min.int()


beat-buesser

Hi @CNOCycle Thank you very mich for your pull request! I have added a few review requests, please let me know what you think.

beat-buesser · 2024-05-27T14:38:06Z

art/attacks/evasion/overload.py

@@ -0,0 +1,260 @@
+# MIT License
+#
+# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2018


Suggested change

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2018

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2024

beat-buesser · 2024-05-27T14:39:20Z

art/attacks/evasion/overload.py

+        "batch_size",
+    ]
+
+    _estimator_requirements = ()


Please add parent classes for object detection estimators for PyTorch-specific models, similar to #2440.

beat-buesser · 2024-05-27T14:39:51Z

art/attacks/evasion/overload.py

+
+    def __init__(
+        self,
+        estimator: "torch.nn.Module",


Please add type for object detection models in PyTorch using the types to be defined in #2440 for the same purpose.

beat-buesser · 2024-05-27T14:41:37Z

art/attacks/evasion/overload.py

+        """
+        Create a overload attack instance.
+
+        :param estimator: A trained YOLO5 model.


Suggested change

:param estimator: A trained YOLO5 model.

:param estimator: A PyTorch object detection estimator for a YOLO5 model.

beat-buesser · 2024-05-27T14:44:53Z

art/attacks/evasion/overload.py

+        """
+
+        import torch


Suggested change

"""

import torch

"""

import torch

beat-buesser · 2024-05-27T20:05:46Z

tests/attacks/evasion/test_overload_attack.py

+    try:
+        import torch
+        model = torch.hub.load('ultralytics/yolov5:v7.0',  model='yolov5s')
+        x = np.random(0.0, 1.0, size=(100, 3, 640, 640))


Could you please check this line, should it not be np.random.normal?

beat-buesser · 2024-05-27T20:06:31Z

tests/attacks/evasion/test_overload_attack.py

+
+
+@pytest.mark.only_with_platform("pytorch")
+def test_check_params(art_warning):


This test seems to be failing.

beat-buesser · 2024-05-27T20:07:08Z

tests/attacks/evasion/test_overload_attack.py

+
+        x_adv = attack.generate(x)
+
+        assert x.shape == x_adv.shape


Could you please add additional asserts to check the expected values in x_adv?

beat-buesser · 2024-05-27T20:12:22Z

art/attacks/evasion/overload.py

+        """
+
+        import torch


Suggested change

"""

import torch

"""

import torch

beat-buesser · 2024-05-27T20:36:19Z

art/attacks/evasion/overload.py

+        # IoU = inter / (area1 + area2 - inter)
+        return inter / ((a2 - a1).prod(2) + (b2 - b1).prod(2) - inter + eps)
+
+    def _check_params(self) -> None:


I think it would be very helpful if we could add a check if the provided model is a Yolo v5 model.

Support Overload attack

39f7587

beat-buesser self-requested a review November 30, 2023 15:39

beat-buesser self-assigned this Nov 30, 2023

beat-buesser added the enhancement New feature or request label Nov 30, 2023

beat-buesser added this to the ART 1.18.0 milestone Nov 30, 2023

CNOCycle changed the base branch from main to dev_1.18.0 November 30, 2023 16:17

github-advanced-security bot found potential problems Nov 30, 2023

View reviewed changes

beat-buesser requested changes May 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Overload attack #2337

Added support for Overload attack #2337

CNOCycle commented Nov 28, 2023

beat-buesser commented Nov 30, 2023

codecov-commenter commented Nov 30, 2023 •

edited

beat-buesser left a comment

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

beat-buesser May 27, 2024

	# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2018
	# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2024

	:param estimator: A trained YOLO5 model.
	:param estimator: A PyTorch object detection estimator for a YOLO5 model.



		@pytest.mark.only_with_platform("pytorch")
		def test_check_params(art_warning):

Added support for Overload attack #2337

Are you sure you want to change the base?

Added support for Overload attack #2337

Conversation

CNOCycle commented Nov 28, 2023

Description

Type of change

Testing

Checklist

beat-buesser commented Nov 30, 2023

codecov-commenter commented Nov 30, 2023 • edited

Codecov Report

beat-buesser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Nov 30, 2023 •

edited