Does torch.nn.Linear check the weights shape before assignment?

[lesjie-wen]

🐛 Describe the bug

Recently, l found that when l directly attribute weights to torch.nn.Linear, the shape of weights dismatch the in_features and out_features, but it still works.

import torch

linear1 = torch.nn.Linear(in_features=70, out_features=100, bias=False)
custom_weights = torch.ones((56, 90))
linear1.weight = torch.nn.Parameter(custom_weights)
print(linear1.weight.shape) 
# torch.Size([56, 90])

input = torch.ones((16, 90))
output = linear1(input)
print(output.shape)
# torch.Size([16, 56])

However, when l set the output shape to [100, 70] which is the original shape l want. Errors come.
So, l want to know whether the Linear check the weights shape before assignment? Or, there is a more correct way to attribute weights?

Versions

[pip3] numpy==1.24.4
[pip3] torch==2.0.1
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] numpy 1.24.4 pypi_0 pypi
[conda] torch 2.0.1 pypi_0 pypi
[conda] torchvision 0.15.2 pypi_0 pypi
[conda] triton 2.0.0 pypi_0 pypi

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki

[ayush0x00]

Hey @lesjie-wen , I am not able to understand your query. Can you provide code snippets where are you getting errors? The snippet you have provided is absolutely fine and works as expected. Please elaborate when l set the output shape to [100, 70] which is the original shape l want this statement.

[lesjie-wen]

Sorry, there is a mistake, what l want to say is as follows, when l initialized input with the shape of [16,70], error comes:

import torch

linear1 = torch.nn.Linear(in_features=70, out_features=100, bias=False)
custom_weights = torch.ones((56, 90))
linear1.weight = torch.nn.Parameter(custom_weights)
print(linear1.weight.shape) 
# torch.Size([56, 90])

input = torch.ones((16, 70))
output = linear1(input)
Traceback (most recent call last):
  File "/Users/lesjie/PycharmProjects/test/test.py", line 10, in <module>
    output = linear1(input)
  File "/Users/lesjie/anaconda3/envs/vima/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/lesjie/anaconda3/envs/vima/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x70 and 90x56)

The original shape of input tensor is [N, 70] due to the in_features=70

[ayush0x00]

inear1.weight = torch.nn.Parameter(custom_weights) You are changing the weight matrix of the linear layer over here so the weight of linear1 is no longer [N,70]. It's [56,90] that won't allow multiplication by [16,70].