[optim] Merge the pyi files into py files of optimizer #125452

david20571015 · 2024-05-03T01:07:38Z

Continue the work of #125153

pytorch-bot · 2024-05-03T01:07:41Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/125452

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3a87b3b with merge base 3759676 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

david20571015 · 2024-05-04T03:07:32Z

@pytorchbot drci

janeyx99

Started reviewing, but it’d be easier to review if you could split the lr_scheduler changes from the optims

janeyx99 · 2024-05-05T20:01:53Z

torch/optim/adam.py

-    lr_dict = (
-        {lr.device: lr} if isinstance(lr, Tensor) and str(lr.device) != "cpu" else None
-    )
+    lr = torch.tensor(lr)


why is this necessary? we don’t want to wrap lr into a Tensor normally

I think my modification here is incorrect, I will fix this.

janeyx99 · 2024-05-05T20:02:34Z

torch/optim/adamw.py

@@ -27,10 +29,10 @@ class AdamW(Optimizer):
    def __init__(
        self,
        params: ParamsT,
-        lr: Union[float, Tensor] = 1e-3,
+        lr=1e-3,


why the removal of types here?

reverted
I accidently removed it.

janeyx99 · 2024-05-05T20:04:08Z

torch/optim/asgd.py

@@ -376,6 +377,7 @@ def _multi_tensor_asgd(
            torch._foreach_add_(grouped_state_steps, 1)

        # intermediate = grad + param * lambd
+        intermediate: Union[Tuple[Tensor, ...], List[Tensor]]


what’s the diff between defining this here vs the first time it’s instantiated?

without this, mypy will raise an error Incompatible types in assignment (expression has type "tuple[Tensor, ...]", variable has type "list[Tensor]") in line 386, 392 and 411 because intermediate was assigned as list[Tensor] in line 384. I think it is more concise to assign a union type here than add 3 type ignores.

david20571015 · 2024-05-05T21:51:52Z

thanks for your review @janeyx99, I splited the changes about lr_scheduler into #125556.

janeyx99

Almost there!

janeyx99 · 2024-05-06T19:09:24Z

torch/optim/adam.py

@@ -27,10 +29,10 @@ class Adam(Optimizer):
    def __init__(
        self,
        params: ParamsT,
-        lr: Union[float, Tensor] = 1e-3,
+        lr=1e-3,


looks like there are more places you accidentally removed

fixed. sorry for this.

janeyx99 · 2024-05-06T19:10:55Z

torch/optim/adamw.py

-    lr_dict = (
-        {lr.device: lr} if isinstance(lr, Tensor) and str(lr.device) != "cpu" else None
-    )
+    lr = torch.tensor(lr)


janeyx99 · 2024-05-06T19:11:05Z

torch/optim/adamw.py

    weight_decay: float,
    eps: float,
    maximize: bool,
-    capturable: bool,  # Needed for consistency.


why delete the comment?

I thought this is unnecessary because has_complex didn't have this comment. But I add it back and add a same comment for has_complex.

janeyx99 · 2024-05-06T19:14:59Z

torch/optim/swa_utils.py

-                        Tuple[List[List[Tensor]], Indices],
-                    ],
-                    grouped_tensors,
+                grouped_tensors = Optimizer._group_tensors_by_device_and_dtype(


Why the switch to use the optimizer one here?

The optimizer one internally calls the original function and also supports compiling. It doesn't need to perform a lot of type casting because it is ignored in the optimizer one. Is this not suitable here?

eh...the one in optimizer is a special case where it gets skipped in compile world..I'd rather not use it here if not necessary yet.

Okay, I'll roll it back. Thanks for explaining this.

janeyx99 · 2024-05-06T21:48:01Z

torch/optim/adam.py

+        eps=1e-8,
+        weight_decay=0,


though I see other places where this wasn't necessary--do you know the difference between defining vs not here?

If it's fine to not have a type, I am happy with not having one.

Sorry, could you please explain the difference to me? I think mypy interprets weight_decay as an int, so should I specify a float here or change 0 to 0.0 (but there won't be any mypy error if using int weight_decay as a float)?

It does look like there's no error and it will allow an int or a float or anything (it might just infer it as Any). From my lil endeavor, it still feels the best to include the type. For example:

class Foo: def __init__(self, arg, kwarg=0): self.arg = arg self.kwarg = kwarg a = Foo(1, "l") print(5 + a.kwarg)

This will not error and will do the type conversion automatically.

But the intention is closer to:

class Foo: def __init__(self, arg, kwarg: int = 0): self.arg = arg self.kwarg = kwarg a = Foo(1, "l") print(5 + a.kwarg)

And here, with the type, mypy will error properly at the instantiation of a

Okay, understood completely. I'll go ahead and add the types.

janeyx99 · 2024-05-06T21:48:24Z

torch/optim/adamw.py

+        eps=1e-8,
+        weight_decay=1e-2,


janeyx99 · 2024-05-06T21:50:55Z

torch/optim/swa_utils.py

-                        Tuple[List[List[Tensor]], Indices],
-                    ],
-                    grouped_tensors,
+                grouped_tensors = Optimizer._group_tensors_by_device_and_dtype(


eh...the one in optimizer is a special case where it gets skipped in compile world..I'd rather not use it here if not necessary yet.

david20571015 · 2024-05-10T01:25:38Z

@pytorchbot merge

pytorch-bot · 2024-05-10T01:25:42Z

Pull workflow has not been scheduled for the PR yet. It could be because author doesn't have permissions to run those or skip-checks keywords were added to PR/commits, aborting merge. Please get/give approval for the workflows and/or remove skip ci decorators before next merge attempt. If you think this is a mistake, please contact PyTorch Dev Infra.

janeyx99 · 2024-05-10T04:29:17Z

torch/optim/adamw.py

@@ -318,92 +318,6 @@ def step(self, closure=None):
 )


-def adamw(


wait is this deletion intended?

I merged main branch to this because github says there are some merge conflict, and 0f02e0a moved the function to the end of the file.

david20571015 · 2024-05-11T02:03:39Z

@pytorchbot merge

pytorch-bot · 2024-05-11T02:03:43Z

Pull workflow has not been scheduled for the PR yet. It could be because author doesn't have permissions to run those or skip-checks keywords were added to PR/commits, aborting merge. Please get/give approval for the workflows and/or remove skip ci decorators before next merge attempt. If you think this is a mistake, please contact PyTorch Dev Infra.

janeyx99 · 2024-05-11T03:21:25Z

@pytorchbot merge

pytorchmergebot · 2024-05-11T03:23:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-11T03:23:23Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

BC Lint / bc_linter

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

david20571015 · 2024-05-11T04:04:26Z

@pytorchbot rebase -b main

pytorchmergebot · 2024-05-11T04:06:07Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2024-05-11T04:06:09Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/main pull/125452/head returned non-zero exit code 1

Rebasing (1/28)
dropping b3b18e394d1915b6b885d77344b7cc99070ba6d4 fix: type of `TensorListList` -- patch contents already upstream
Rebasing (2/28)
Auto-merging torch/optim/adadelta.py
CONFLICT (content): Merge conflict in torch/optim/adadelta.py
error: could not apply 4f707fdcad9... merge pyi into py
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 4f707fdcad9... merge pyi into py

Raised by https://github.com/pytorch/pytorch/actions/runs/9040977794

pytorchmergebot · 2024-05-11T05:40:09Z

Merge failed

Reason: 3 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

david20571015 · 2024-05-11T05:45:37Z

Sorry @janeyx99, I mistakenly thought the failures might have been fixed, so I rebased onto the main branch. However, after investigating the error message, I found that it was caused by my code, and I fixed it in 165fed5. Could you please approve the CI again?

david20571015 · 2024-05-13T22:06:16Z

Should I keep the parameter name d_p_list? None of the affected functions are exposed to external users.

janeyx99 · 2024-05-13T22:13:49Z

Should I keep the parameter name d_p_list? None of the affected functions are exposed to external users.

Yes, we should keep the name. sgd is publicly exposed and the argument is sadly not positional only.

david20571015 · 2024-05-13T22:22:26Z

Should I keep the parameter name d_p_list? None of the affected functions are exposed to external users.

Yes, we should keep the name. sgd is publicly exposed and the argument is sadly not positional only.

Okay, I will fix it.
But wasn't sgd removed in torch/optim/init.py? Also, its only usage seems to be in SGD.step() (search for sgd( using vscode) and only torch/optim/sgd.py use the name d_p_list (search for d_p_list).

janeyx99 · 2024-05-13T22:31:03Z

The removal in optimizer __init__.py is the torch.optim.sgd module, not the function, which is offered so people can manage their own state. (I have yet to figure out why we do the dels there but that is not relevant for this discussion.)

david20571015 · 2024-05-14T03:25:41Z

Should I keep the parameter name d_p_list? None of the affected functions are exposed to external users.

Yes, we should keep the name. sgd is publicly exposed and the argument is sadly not positional only.

done.

janeyx99 · 2024-05-14T11:25:59Z

looks like there are merge conflicts now :/

david20571015 · 2024-05-14T12:45:58Z

looks like there are merge conflicts now :/

fixed

janeyx99 · 2024-05-14T13:44:55Z

@pytorchbot merge

pytorchmergebot · 2024-05-14T13:47:45Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-14T14:04:08Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / lintrunner-noclang / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

janeyx99 · 2024-05-14T14:12:09Z

Please run lintrunner locally to ensure lint passes for the next commit (and usually before pushing). Getting lintrunner locally is pretty simple: just

pip install lintrunner
lintrunner init
lintrunner -a

david20571015 · 2024-05-14T14:21:29Z

Please run lintrunner locally to ensure lint passes for the next commit (and usually before pushing). Getting lintrunner locally is pretty simple: just
pip install lintrunner
lintrunner init
lintrunner -a

ok! fixed

david20571015 · 2024-05-14T14:21:49Z

@pytorchbot merge

pytorchmergebot · 2024-05-14T15:23:49Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added the release notes: optim label May 3, 2024

pytorchbot added the open source label May 3, 2024

david20571015 marked this pull request as ready for review May 4, 2024 03:06

david20571015 requested review from albanD and janeyx99 as code owners May 4, 2024 03:06

janeyx99 reviewed May 5, 2024

View reviewed changes

david20571015 force-pushed the patch branch from 59b76b3 to 354001c Compare May 5, 2024 21:48

david20571015 requested a review from janeyx99 May 5, 2024 22:31

janeyx99 reviewed May 6, 2024

View reviewed changes

albanD removed their request for review May 6, 2024 20:10

janeyx99 approved these changes May 6, 2024

View reviewed changes

david20571015 requested a review from janeyx99 May 10, 2024 01:26

janeyx99 reviewed May 10, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 11, 2024

pytorchmergebot added the merging label May 11, 2024

pytorchmergebot removed the merging label May 11, 2024

typing: adam

7104597

pytorchmergebot removed the merging label May 11, 2024

david20571015 changed the title ~~Merge the pyi files into py files of optimizer~~ [optim] Merge the pyi files into py files of optimizer May 13, 2024

david20571015 requested a review from janeyx99 May 13, 2024 21:45

revert d_p_list for BC

c0d84f7

Merge branch main into patch

9ebcf14

pytorchmergebot added the merging label May 14, 2024

pytorchmergebot removed the merging label May 14, 2024

fix: lint

3a87b3b

pytorchmergebot added the merging label May 14, 2024

pytorchmergebot added the Merged label May 14, 2024

pytorchmergebot closed this in 1a28f73 May 14, 2024

pytorchmergebot removed the merging label May 14, 2024

david20571015 deleted the patch branch May 14, 2024 18:58

		@@ -318,92 +318,6 @@ def step(self, closure=None):
		)


		def adamw(

[optim] Merge the pyi files into py files of optimizer #125452

[optim] Merge the pyi files into py files of optimizer #125452

Conversation

david20571015 commented May 3, 2024

pytorch-bot bot commented May 3, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/125452

✅ No Failures

david20571015 commented May 4, 2024

janeyx99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david20571015 May 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david20571015 May 5, 2024 • edited

Choose a reason for hiding this comment

david20571015 commented May 5, 2024

janeyx99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david20571015 May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david20571015 May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david20571015 commented May 10, 2024

pytorch-bot bot commented May 10, 2024

Choose a reason for hiding this comment

david20571015 May 10, 2024 • edited

Choose a reason for hiding this comment

david20571015 commented May 11, 2024

pytorch-bot bot commented May 11, 2024

janeyx99 commented May 11, 2024

pytorchmergebot commented May 11, 2024

Merge started

pytorchmergebot commented May 11, 2024

Merge failed

david20571015 commented May 11, 2024

pytorchmergebot commented May 11, 2024

pytorchmergebot commented May 11, 2024

pytorchmergebot commented May 11, 2024

Merge failed

david20571015 commented May 11, 2024 • edited

david20571015 commented May 13, 2024

janeyx99 commented May 13, 2024 • edited

david20571015 commented May 13, 2024 • edited

janeyx99 commented May 13, 2024

david20571015 commented May 14, 2024

janeyx99 commented May 14, 2024

david20571015 commented May 14, 2024

janeyx99 commented May 14, 2024

pytorchmergebot commented May 14, 2024

Merge started

pytorchmergebot commented May 14, 2024

Merge failed

janeyx99 commented May 14, 2024 • edited

david20571015 commented May 14, 2024

david20571015 commented May 14, 2024

pytorchmergebot commented May 14, 2024

Merge started

pytorch-bot bot commented May 3, 2024 •

edited

david20571015 May 5, 2024 •

edited

david20571015 May 5, 2024 •

edited

david20571015 May 6, 2024 •

edited

david20571015 May 6, 2024 •

edited

david20571015 May 10, 2024 •

edited

david20571015 commented May 11, 2024 •

edited

janeyx99 commented May 13, 2024 •

edited

david20571015 commented May 13, 2024 •

edited

janeyx99 commented May 14, 2024 •

edited