Number of times model was queried by black-box evasion attacks such as Boundary or HopSkipJump?

[amir1m]

Hello,
I am looking for a way to find out how many times model was queried/evaluated by blackbox evasion attacks such as Boundary or HopSkipJump?

Thanks!

[beat-buesser]

Hi @amir1m Thank you very much for your question! I have moved it to our new Discussions forum.

There are currently no specific counter attributes/properties for model evaluations in black-box attacks other than the query budget allocations following the original paper's algorithms.

I think it could be a useful feature for a future release to wrap the estimator's predict method in black-box attack to add a counter for actual model evaluations. Maybe something like:

def _predict_and_count(self, x):
    self._number_estimator_predictions += x.shape[0]
    return self.estimator.predict(x=x)

Would you be interested to implement this feature and contribute it to ART?

[amir1m]

Sure @beat-buesser. Sounds interesting to me. I will give it a try as this would be useful methods to have.

[FaramirHurin]

Hello,

I am working on the same problem. For Boundary I think it should be possible to formulate it as a function of the parameters:
init_size, max_iter, num_trial, sample_size .
I am currently trying to double-check the formula I reconstructed from the code.

A counter could be added to verify that no operation is repeated and that the effective number of queries to the model matches the one the algorithm should theoretically do.

[beat-buesser]

Hi @FaramirHurin Do you have a proposal for the formula, I would be happy to take a look?

[FaramirHurin]

Hi @beat-buesser ,

I will share my findings so far. Please keep in mind that it may not be correct yet. There is still a bit of confusion, but I think the road may be correct. From now on, I will refer to the number of queries for each data to attack, as the total number of queries scales linearly with the number of targets.

I will start with Boundary and discuss the code and the paper idea.
In the code, the line that queries the model is self.model.predict(). This is called three times in the method generate, twice in attack, and twice in init_sample. We can ignore the use inside the generate method for now, as it is only called to get the label for the initial image.

Then, _generate calls the method _perturb, which is applied once per observation. First, it runs _init_sample, which produces the initial adversarial point. To do so, it samples self.init_size possible initial perturbations and evaluates them, leading to self.init_size queries per observation.
Then, the _attack method iterates it self.max_iter times. Each time it performs num_trial times a prediction on an array potential_adv of size self.sample_size. Then it is called again in line 407 in the trust region method to adjust epsilon
All of the above should lead to a total number of queries in the order of
N = O(1) + self.init_size + 2 * self.max_iter * self.num_trial * self.sample_size

From the paper, the initialization is straightforward, and the O(1) + self.init_size is intuitive. Then, it is clear that it iterates max_iter times to get the observation closer to the original by moving along the border. I am still working on matching the final part with the code, but I understand that multiple trials are necessary to draw valid perturbations at each step and adjust the hyperparameters.

A similar analysis can be done for HopSkipJump, where the problem is the presence of binary search (and consequently of while statements in the code), which does not allow for setting a hard limit on the total number of queries. However, even here, a clear bound can be made by inserting a max number of iterations in the while statement (and hence in the binary search).

To double-check the formulas, I think even a simple global counter should do the work, provided it does not count only the number of times self.model.predict() is called but also the number of observations it is called to analyze every time.

[FaramirHurin]

I will probably implement the counter locally for a paper I am writing. I hope it will match the theoretical analysis.
If it works and it seems interesting, I can then collaborate to implement it also here