Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IntermediateLayerGetter #47908

Merged
merged 18 commits into from Dec 6, 2022
Merged
63 changes: 63 additions & 0 deletions python/paddle/vision/models/utils.py
Expand Up @@ -12,6 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import paddle.nn as nn
from collections import OrderedDict
from typing import Dict

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#43611 可以参考这个pr增加单测

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

def _make_divisible(v, divisor=8, min_value=None):
"""
Expand All @@ -30,3 +33,63 @@ def _make_divisible(v, divisor=8, min_value=None):
if new_v < 0.9 * v:
new_v += divisor
return new_v


class IntermediateLayerGetter(nn.LayerDict):
"""
Layer wrapper that returns intermediate layers from a model.

It has a strong assumption that the layers have been registered into the model in the
same order as they are used. This means that one should **not** reuse the same nn.Layer
twice in the forward if you want this to work.

Additionally, it is only able to query sublayer that are directly assigned to the model.
So if `model` is passed, `model.feature1` can be returned, but not `model.feature1.layer2`.

Args:
model (nn.Layer): model on which we will extract the features
return_layers (Dict[name, new_name]): a dict containing the names of the layers for
which the activations will be returned as the key of the dict, and the value of the
dict is the name of the returned activation (which the user can specify).

Examples:
.. code-block:: python

>>> m = paddle.vision.models.resnet18(pretrained=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

测试代码需要加import paddle,保证代码可以单独运行

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2022-11-29 14:26:01 subprocess return code: 1
2022-11-29 14:26:01 Error Raised from Sample Code:
2022-11-29 14:26:01 stderr:   File "samplecode_temp/paddle.vision.models.utils.IntermediateLayerGetter_example.py", line 4
2022-11-29 14:26:01     >>> import paddle
2022-11-29 14:26:01      ^
2022-11-29 14:26:01 SyntaxError: invalid syntax
2022-11-29 14:26:01 
2022-11-29 14:26:01 stdout: 

https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/7275600/job/20709077
示例代码跑不过,把>>>都去掉吧

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 😆

>>> # extract layer1 and layer3, giving as names `feat1` and feat2`
>>> new_m = paddle.vision.models.utils.IntermediateLayerGetter(m,
>>> {'layer1': 'feat1', 'layer3': 'feat2'})
>>> out = new_m(paddle.rand([1, 3, 224, 224]))
>>> print([(k, v.shape) for k, v in out.items()])
>>> [('feat1', [1, 64, 56, 56]),
>>> ('feat2', [1, 256, 14, 14])]
"""

__annotations__ = {
"return_layers": Dict[str, str],
}

def __init__(self, model: nn.Layer, return_layers: Dict[str, str]) -> None:
if not set(return_layers).issubset([name for name, _ in model.named_children()]):
raise ValueError("return_layers are not present in model")
orig_return_layers = return_layers
return_layers = {str(k): str(v) for k, v in return_layers.items()}
layers = OrderedDict()
for name, module in model.named_children():
layers[name] = module
if name in return_layers:
del return_layers[name]
if not return_layers:
break
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的layers表示return layer前所有layer?能否给出示例代码

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是的

IntermediateLayerGetter 的目的就是运行 model 的前向传播,并将中间的某些层的输出拿出来

layers 之所以包括传入 model 的所有 sublayer,是因为要通过传入model 的前向传播

之后执行:

super(IntermediateLayerGetter, self).__init__(layers)

由于继承自 nn.LayerDict 则它包含的子层将被注册和添加

所以在 self.forward 中,能通过 for name, module in self.items(): 来迭代每一层操作

如果某一层的输出,是我们想要的,即if name in self.return_layers,则添加到返回out = OrderedDict()

self.return_layers 字典变量用来指示哪些层的输出需要返回,返回之后的名字叫什么


super(IntermediateLayerGetter, self).__init__(layers)
self.return_layers = orig_return_layers

def forward(self, x):
out = OrderedDict()
for name, module in self.items():
x = module(x)
if name in self.return_layers:
out_name = self.return_layers[name]
out[out_name] = x
return out