[v.1.10.0] Release Tracker #65438

malfet · 2021-09-21T23:45:39Z

We cut a release branch for the 1.10.0 release.

Our plan from this point from this point is roughly:

Phase 1 (until 10/7): work on finalizing the release branch
Phase 2 (after 10/7): perform extended integration/stability/performance testing based on Release Candidate builds.

This issue is for tracking cherry-picks to the release branch.

Cherry-Pick Criteria

Phase 1 (until 10/7/21):

Only low-risk changes may be cherry-picked from master:

Fixes to regressions against the most recent minor release (e.g. 1.9 for 1.10 release; see module: regression issue list)
Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
Fixes to new features introduced in the most recent minor release (e.g. 1.9.1 for 1.10 release)
Test/CI fixes
Documentation improvements
Compilation fixes or ifdefs required for different versions of the compilers or third-party libraries
Release branch specific changes (e.g. change version identifiers)

Any other change requires special dispensation from the release managers (currently @malfet, @seemethere, @gchanan, @zhouzhuojie ). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 10/7/21):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

Create (but do not land) a PR against the release branch.

# Find the hash of the commit you want to cherry pick
# (for example, abcdef12345)
git log

git fetch origin release/1.10
git checkout release/1.10
git cherry-pick abcdef12345

# Submit a PR based against 'release/1.10' either:
# via the GitHub UI
git push my-fork

# via the GitHub CLI
gh pr create --base release/1.10

Make a request below with the following format:

Link to landed master PR (if applicable):
Link to release branch PR:
Criteria Category:

Someone from the release team will reply with approved / denied or ask for more information.
If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

See HUD 1.10 or #65434 for release branch CI status

The text was updated successfully, but these errors were encountered:

malfet · 2021-09-21T23:51:12Z

Link to landed master PR (if applicable): N/A
Link to release branch PR: #65433
Criteria Category: Releng

@malfet: merged

zou3519 · 2021-09-22T21:14:16Z

Link to landed master PR (if applicable): n/a
Link to release branch PR: #65496
Criteria Category: Release branch specific changes (removing prototype features not ready for release)

@malfet: merged

malfet · 2021-09-24T17:30:38Z

1.10.0-rc1 build is here
0.11.0-rc1 build of torchvision is here

prabhat00155 · 2021-09-28T17:49:49Z

Link to landed master PR (if applicable): #65495 + #65921
Link to release branch PR: #65755
Criteria Category: 2

@malfet: [edit] After further discussion, reverted as feature work by #66308

zhouzhuojie · 2021-09-28T23:06:41Z

Link to landed master PR: N/A
Link to release branch PR: #65787
Criteria Category: 4

@zhouzhuojie: merged

prabhat00155 · 2021-09-29T11:35:43Z

Link to landed master PR (if applicable): #65495 Link to release branch PR: #65755 Criteria Category: 2

@malfet: This does not look like a regression/backward compatibility fix, but a bit more like a feature work: as PR adds new argument to torch.optim.AveragedModel class

AveragedModel's update_parameter() doesn't update the parameters correctly in certain cases. This PR fixes that. Look at #65495 (comment) and #65495 (review) for more details.

malfet · 2021-09-29T18:39:35Z

Link to landed master PR: #65721
Link to release branch PR: #65835
Criteria Category: Performance regression from previous PR

@malfet: merged

ejguan · 2021-09-30T16:08:56Z

Link to landed master PR: #65220
Link to release branch PR: #65924
Criteria Category: Fixes to new features introduced in the most recent minor release (e.g. 1.9.1 for 1.10 release) #64951

@malfet: merged

ejguan · 2021-09-30T16:38:38Z

Link to landed master PR: #63646
Link to release branch PR: #65926
Criteria Category: Fixes to regressions against the most recent minor release (#63609)

Gentle ping to get this PR landed into release before branch cut: @malfet

@malfet: merged

NivekT · 2021-09-30T17:07:58Z

Link to landed master PR: #65424, #65827, #65877
Link to release branch PR: #65932
Criteria Category: Fixes to regressions against the most recent minor release

@malfet: merged

garymm · 2021-10-01T00:54:00Z

Link to landed master PR: #64370 #64371 #64373 #64380 #64382
Link to release branch PR: #65962
Criteria Category: Special Dispensation for ONNX-only changes

@malfet: merged

malfet · 2021-10-01T01:55:13Z

Link to landed master PR: #65934
Link to release branch PR: #65979
Criteria Category: CI fixes

@malfet: merged

n-v-k · 2021-10-05T09:15:23Z

Link to landed master PR: #66031
Link to release branch PR: #66117
Criteria Category: 6

@malfet: merged

malfet · 2021-10-05T19:00:41Z

Link to the landed master PR: #65444
Link to the release branch PR: #66155
Criteria Category: Regressions against the most recent minor release

@malfet: merged

stas00 · 2021-10-05T19:30:18Z

a likely regression - not triaged - #66086

Jianhui-Li · 2021-10-05T23:17:27Z

Hi, we would like to get Intel Extension to Pytorch tutorial into 1.10 release. Could you please merge it in?

@vitaly-fedyunin and @gottbrath give me this link.

Thanks. @malfet

malfet · 2021-10-06T02:50:21Z

a likely regression - not triaged - #66086

This issue is already have 1.10 milestone

malfet · 2021-10-06T02:52:14Z

Hi, we would like to get Intel Extension to Pytorch tutorial into 1.10 release. Could you please merge it in?

@vitaly-fedyunin and @gottbrath give me this link.

Thanks. @malfet

@Jianhui-Li can you please share link to the PR? Has it been merged into the main branch yet?

malfet · 2021-10-06T03:04:50Z

Link to the landed master PR: N/A
Link to the release branch PR: #66186
Criteria category: Release specific CI fixes

@malfet: merged

malfet · 2021-10-06T03:11:27Z

Link to the landed master PR: #66004
Link to the release branch PR: #66188
Criteria category: CI fixes

@malfet: merged

Jianhui-Li · 2021-10-06T05:13:08Z

Hi, we would like to get Intel Extension to Pytorch tutorial into 1.10 release. Could you please merge it in?
@vitaly-fedyunin and @gottbrath give me this link.
Thanks. @malfet

@Jianhui-Li can you please share link to the PR? Has it been merged into the main branch yet?

Sorry to forget the link pytorch/tutorials#1702

garymm · 2021-10-11T17:27:36Z

@garymm I’m a bit confused, are you asking to cherry pick anything?

No. @malfet told me this would be an appropriate place to provide the release notes. If another place is better let me know.

ngimel · 2021-10-12T23:00:03Z

release branch PR #66524
master branch PR #66463,
criteria: regression, fixes #65709

@malfet: merged

anjali411 · 2021-10-13T21:04:15Z

Master branch PR
#63602, #66572
Release branch PR #66667
Criteria: bug fixes for conjugate view (beta in 1.10)

@malfet: merged

anjali411 · 2021-10-13T21:21:55Z

Master branch PR: #66082 (commit hash: a82fcd3)

Release branch PR: #66642
Criteria: bug fixes for conjugate view (beta in 1.10)

@malfet: merged

malfet · 2021-10-14T16:46:21Z

Master branch PR: #66433
Release branch PR: #66629
Criteria category: crashes, regression from previous release, see #66433

@malfet: merged

kiukchung · 2021-10-14T22:20:46Z

Link to landed master PR (if applicable): #66182
Link to release branch PR: #66662
Criteria Category: special dispensation (documentation improvements, fixes to regressions against the most recent minor release)

@malfet: merged

malfet · 2021-10-15T04:26:51Z

pytorch v1.10.0-rc3 https://app.circleci.com/pipelines/github/pytorch/pytorch/394756/workflows/b23151b1-e3f1-4ddd-82f4-5afc61a66364
torchaudio v0.10.0-rc3 https://app.circleci.com/pipelines/github/pytorch/audio/7859/workflows/30e00570-2725-4496-b6a5-a295c6a8acb3
torchvision v0.11.0-rc3 https://app.circleci.com/pipelines/github/pytorch/vision/11356/workflows/308a35e3-ba62-482a-87e9-08f2d1a0a5f8
torchtext v0.11.0-rc3 https://app.circleci.com/pipelines/github/pytorch/text/4000/workflows/3ec5a632-4db8-4160-afc9-7cb9c195e156

imaginary-person · 2021-10-15T19:21:00Z

Sorry, but #66712 might be a blocking issue for the release. I had introduced a bug that I fixed in #66703. 😞
Essentially, PyTorch builder seems to be using gcc 7.3 on CentOS for building release binaries, such as the nightly releases.
However, gcc versions below 9 don't support some AVX512 intrinsics present in the codebase, so AVX512 ATen kernels aren't being compiled, but due to the aforementioned bug, the CPU Capability in torch config is being displayed as AVX512 on machines that support AVX512 instruction sets being used by ATen kernels.

malfet · 2021-10-15T19:29:23Z

Sorry, but #66712 might be a blocking issue for the release. I had introduced a bug that I fixed in #66703. 😞 Essentially, PyTorch builder seems to be using gcc 7.3 on CentOS for building release binaries, such as the nightly releases. However, gcc versions below 9 don't support some AVX512 intrinsics present in the codebase, so AVX512 ATen kernels aren't being compiled, but due to the aforementioned bug, the CPU Capability in torch config is being displayed as AVX512 on machines that support AVX512 instruction sets being used by ATen kernels.

Hmm, this is indeed unfortunate, but what's the performance impact? Are AVX2 or default kernels will be selected in that case?

imaginary-person · 2021-10-15T19:30:36Z

Hmm, this is indeed unfortunate, but what's the performance impact? Are AVX2 or default kernels will be selected in that case?

AVX2

malfet · 2021-10-15T19:36:34Z

AVX2

In that case it does not sound like a release blocker: i.e. users would not benefit from AVX512 perf boost, but CPU performance would be no different (and sometimes faster), than in PyTorch-1.9.0

imaginary-person · 2021-10-15T19:44:50Z

Thanks, @malfet! Yes, the primary issue is that without fixing the aforementioned bug, many users might be misled about the ATen CPU capability.

BTW, please confirm if it's possible to release another binary that also has AVX512 kernels.
If that's not an option, is it possible to release the next version with gcc version >= 9, so that ATen AVX512 kernels are included in binaries (by then, the community would've tested it on nightly releases)?
Thanks!

jgong5 · 2021-10-19T08:35:06Z

hi @malfet : about the AVX-512 support, since it is a major feature added after 1.9 by @imaginary-person , will you highlight it for the 1.10 release? If so, having it enabled by default in the 1.10 binary release sounds more reasonable?

randolf-scholz · 2021-10-21T08:09:46Z

Did you guys just accidentally release 1.10? It can be installed through conda:

pytorch                       1.10.0     py3.6_cpu_0  pytorch             
pytorch                       1.10.0 py3.6_cuda10.2_cudnn7.6.5_0  pytorch             
pytorch                       1.10.0 py3.6_cuda11.1_cudnn8.0.5_0  pytorch             
pytorch                       1.10.0 py3.6_cuda11.3_cudnn8.2.0_0  pytorch             
pytorch                       1.10.0     py3.7_cpu_0  pytorch             
pytorch                       1.10.0 py3.7_cuda10.2_cudnn7.6.5_0  pytorch             
pytorch                       1.10.0 py3.7_cuda11.1_cudnn8.0.5_0  pytorch             
pytorch                       1.10.0 py3.7_cuda11.3_cudnn8.2.0_0  pytorch             
pytorch                       1.10.0     py3.8_cpu_0  pytorch             
pytorch                       1.10.0 py3.8_cuda10.2_cudnn7.6.5_0  pytorch             
pytorch                       1.10.0 py3.8_cuda11.1_cudnn8.0.5_0  pytorch             
pytorch                       1.10.0 py3.8_cuda11.3_cudnn8.2.0_0  pytorch             
pytorch                       1.10.0     py3.9_cpu_0  pytorch             
pytorch                       1.10.0 py3.9_cuda10.2_cudnn7.6.5_0  pytorch             
pytorch                       1.10.0 py3.9_cuda11.1_cudnn8.0.5_0  pytorch             
pytorch                       1.10.0 py3.9_cuda11.3_cudnn8.2.0_0  pytorch

However, neither the releases page nor the 1.10.0 milestone nor the 1.10 release tracker nor pypi indicate the update shipped yet.

seemethere · 2021-10-25T17:34:22Z

Closing this issue since 1.10.0 has been released!

Thanks everyone in this thread for all of the contributions that made 1.10.0 happen!

malfet · 2021-10-25T19:35:50Z

Started another torchaudio build to account for missing cuda-113 windows binaries: https://app.circleci.com/pipelines/github/pytorch/audio/8064/workflows/3c737d4b-72fa-4bb1-b5c0-90fa0c82c21a

malfet · 2021-10-26T16:40:58Z

Started torchvision build to fix packaging problem for cuda-113 linux wheels:
https://app.circleci.com/pipelines/github/pytorch/vision/11853/workflows/8521dddf-27bd-4276-a42b-331bac97521b

malfet added oncall: releng In support of CI and Release Engineering triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Sep 21, 2021

malfet added this to the 1.10.0 milestone Sep 21, 2021

zhouzhuojie pinned this issue Sep 22, 2021

xuhdev unpinned this issue Sep 24, 2021

xuhdev pinned this issue Sep 24, 2021

rohan-varma mentioned this issue Sep 28, 2021

_object_to_tensor is much slower for large objects since 1.9 #65696

Closed

ejguan mentioned this issue Sep 30, 2021

[DataPipe] Fixes an issue where TarArchiveReader closes stream when read into a buffer #65877

Closed

NivekT mentioned this issue Sep 30, 2021

[DataPipe] DataPipe Fix and Deprecation Warnings for Release 1.10 #65932

Merged

mberr mentioned this issue Oct 13, 2021

🎃 🔥 Updates for PyTorch 1.9 pykeen/pykeen#489

Merged

3 tasks

abhiskk mentioned this issue Oct 14, 2021

default branch name breaking torch.hub facebookresearch/WSL-Images#19

Closed

seemethere modified the milestones: 1.10.0, 1.10.1 Oct 25, 2021

pytorch locked as resolved and limited conversation to collaborators Oct 25, 2021

seemethere modified the milestones: 1.10.1, 1.11.0, 1.10.0 Oct 25, 2021

seemethere closed this as completed Oct 25, 2021

seemethere unpinned this issue Oct 25, 2021

seemethere pinned this issue Feb 3, 2022

seemethere unpinned this issue Feb 3, 2022

[v.1.10.0] Release Tracker #65438

[v.1.10.0] Release Tracker #65438

Comments

malfet commented Sep 21, 2021

Cherry-Pick Criteria

Cherry-Pick Process

malfet commented Sep 21, 2021 • edited

zou3519 commented Sep 22, 2021 • edited by malfet

malfet commented Sep 24, 2021 • edited

prabhat00155 commented Sep 28, 2021 • edited by malfet

zhouzhuojie commented Sep 28, 2021

prabhat00155 commented Sep 29, 2021

malfet commented Sep 29, 2021 • edited

ejguan commented Sep 30, 2021 • edited by malfet

ejguan commented Sep 30, 2021 • edited by malfet

NivekT commented Sep 30, 2021 • edited by malfet

garymm commented Oct 1, 2021 • edited by malfet

malfet commented Oct 1, 2021 • edited

n-v-k commented Oct 5, 2021 • edited by malfet

malfet commented Oct 5, 2021 • edited

stas00 commented Oct 5, 2021

Jianhui-Li commented Oct 5, 2021 • edited

malfet commented Oct 6, 2021

malfet commented Oct 6, 2021

malfet commented Oct 6, 2021 • edited

malfet commented Oct 6, 2021 • edited

Jianhui-Li commented Oct 6, 2021

garymm commented Oct 11, 2021

ngimel commented Oct 12, 2021 • edited by malfet

anjali411 commented Oct 13, 2021 • edited by malfet

anjali411 commented Oct 13, 2021 • edited

malfet commented Oct 14, 2021 • edited

kiukchung commented Oct 14, 2021 • edited by malfet

malfet commented Oct 15, 2021 • edited

imaginary-person commented Oct 15, 2021

malfet commented Oct 15, 2021

imaginary-person commented Oct 15, 2021

malfet commented Oct 15, 2021 • edited

imaginary-person commented Oct 15, 2021 • edited

jgong5 commented Oct 19, 2021

randolf-scholz commented Oct 21, 2021

seemethere commented Oct 25, 2021

malfet commented Oct 25, 2021

malfet commented Oct 26, 2021

malfet commented Sep 21, 2021 •

edited

zou3519 commented Sep 22, 2021 •

edited by malfet

malfet commented Sep 24, 2021 •

edited

prabhat00155 commented Sep 28, 2021 •

edited by malfet

malfet commented Sep 29, 2021 •

edited

ejguan commented Sep 30, 2021 •

edited by malfet

ejguan commented Sep 30, 2021 •

edited by malfet

NivekT commented Sep 30, 2021 •

edited by malfet

garymm commented Oct 1, 2021 •

edited by malfet

malfet commented Oct 1, 2021 •

edited

n-v-k commented Oct 5, 2021 •

edited by malfet

malfet commented Oct 5, 2021 •

edited

Jianhui-Li commented Oct 5, 2021 •

edited

malfet commented Oct 6, 2021 •

edited

malfet commented Oct 6, 2021 •

edited

ngimel commented Oct 12, 2021 •

edited by malfet

anjali411 commented Oct 13, 2021 •

edited by malfet

anjali411 commented Oct 13, 2021 •

edited

malfet commented Oct 14, 2021 •

edited

kiukchung commented Oct 14, 2021 •

edited by malfet

malfet commented Oct 15, 2021 •

edited

malfet commented Oct 15, 2021 •

edited

imaginary-person commented Oct 15, 2021 •

edited