Breaking Changes from Variable and Tensor merge
Suraj Subramanian edited this page Jul 24, 2023
·
1 revision
The Tensor and Variable classes had slightly different semantics. There are some breaking changes from combining the two classes.
- Indexing a one-dim Tensor returns a zero-dim Tensor instead of a Python number. The zero-dim Tensor shares the storage of the indexed Tensor. For example:
>>> x = torch.Tensor([1, 1, 1])
>>> v = x[0]
>>> x += 1
>>> print(v) # was 1
2
[torch.FloatTensor of size ()]
- The
type()
of a Tensor no longer reflects the data type. Useisinstance()
orx.type()
instead:
>>> x = torch.DoubleTensor([1, 1, 1])
>>> print(type(x)) # was torch.DoubleTensor
<class 'torch.Tensor'>
>>> print(x.type()) # OK: 'torch.DoubleTensor'
'torch.DoubleTensor'
>>> print(isinstance(x, torch.DoubleTensor)) # OK: True
True
- Tensor.copy_() always obeys broadcasting rules. Tensor assignment (
x[idx] = value
) also follows broadcasting rules more closely:
>>> x = torch.randn(3, 3)
>>> y = torch.randn(9)
>>> x.copy_(y, broadcast=False) # ERROR
>>> x.copy_(y.view_as(x)) # OK
Minor:
-
matmul
,ones_like
, andzeros_like
no longer have anout
parameter. - Functions that construct tensors based on sizes now interpret empty tuples/sizes differently; previous a empty tuple was treated to mean the size of an empty tensor, now an empty tuple is treated to mean the size of a scalar (0-dimensional tensor).
New Issues:
-
masked_scatter(...)
andmasked_fill(...)
follow in place broadcasting rules.
- If you have code that accumulates losses across iterations (e.g. to compute an
average at the end of an epoch), such as
total_loss += loss
whereloss
is your per-iteration loss, you may find increased memory usage in your program. This is becauseloss
probably used to be a Python float (such as when it is the result of.sum()
), while it is now a zero-dim Tensor.total_loss
is thus accumulatingTensor
s and their gradient history, which may keep around large autograd graphs for much longer than necessary. To fix this, be sure to convert the per-iteration loss to a Python number as soon as possible, for example withtotal_loss += float(loss)
.
- Install Prerequisites
- Fork, clone, and checkout the PyTorch source
- Install Dependencies
- Build PyTorch from source
- Tips for developing PyTorch
- PyTorch Workflow Git cheatsheet
- Overview of the Pull Request Lifecycle
- Finding Or Reporting Issues
- Pre Commit Checks
- Create a Pull Request
- Typical Pull Request Workflow
- Pull Request FAQs
- Getting Help
- Codebase structure
- Tensors, Operators, and Testing
- Autograd
- Dispatcher, Structured Kernels, and Codegen
- torch.nn
- CUDA basics
- Data (Optional)
- function transforms (Optional)