Difference between pytorch lightning and lightning

[sudarshan85]

Hi,

I was working with Pytorch lightning a year ago. Since then I had to work on other projects that didn't require Pytorch. Now I'm back and I'm kind of confused with so many different versions of Lightning/Pytorch Lightning around. While the README says Pytorch Ligthning has been renamed to Lightning, most of the documentation still uses Pytorch Lightning. I think we also install the package as pytorch_lightning. There's also Pytorch Ligthning bolts, pytorch lightning transformers, Lightning Flash, torch metrics. I'm really getting confused with so many different versions and am not sure where to start. Could someone explain the difference between these packages and what to use when? Also, if I wanted to start using Ligthning now, how would I install and import it as (pl or L)?

Thanks.

[awaelchli]

Hey @sudarshan85

This is an important question that probably many users like you have.

TLDR:

Also, if I wanted to start using Ligthning now, how would I install and import it as (pl or L)?

pip install lightning, import lightning as L

Explanation:

Correct, PyTorch Lightning got renamed to Lightning. And the company behind it is called Lightning AI.
However, renaming an open source package that is so widely used can't be done over night. The pytorch-lightning package still exits, gets updates and can be downloaded because that's what many users and companies rely on today. We are currently in this transition phase where we are updating the code in our repositories, updating examples and, as you mentioned, the docs also need updates for all imports. The effort is large but we're making good progress.

In the future, our Readme, docs and example will show

pip install lightning

and

import lightning as L

L.Trainer
L.LightningModule
L.LightningApp

etc.

All of this already works, it just isn't visible everywhere in our docs and examples. This is the experience new users will have.

Existing users who have projects relying on pytorch_lightning are still able to work with the old imports and package, but we won't show it anymore in our new docs.

I hope this clears things up a bit 😃

Just a heads up: For future questions, consider posting them in our new Forum over at lightning.ai/forums. We are slowly beginning the migration away from GH discussions.

[offchan42]

does it mean that pytorch_lightning and lightning packages are identical in terms of functionality? can we import any and it will work fine (as long as we know pytorch_lightning is deprecated)?

[awaelchli]

The lightning package contains more, but yes, lightning.pytorch is equivalent to pytorch_lightning. Once Lightning 2.0 comes out (in a few days), there will also be an equivalent pytorch-lightning 2.0 package :) We are not really deprecating it, but we will only show documentation, examples, etc. using the lightning imports going forward.

[sudarshan85]

Thanks for the reply. I thought GH discussions was the result of migrating away from a disource forum?

[iamlockelightning]

👀

[zjost]

I'm also very confused. I first started with pip install lightning, and it gave me some version from 10.2022 that failed upon import. Once I cleaned up and did pip install pytorch-lightning, things seem to work. More confusing still, the main Lightning webpage currently says pytorch-lightning, but if you click on "Docs", the first thing you see is lightning. When I click "Try Lightning", it takes me to a webpage that doesn't load at all.

I'm sure it's very complicated to change names this late in the game. Updating the main page to reflect the new way is high ROI on effort, I would think. Maybe add a note to the docs indicating the team is in the process of making this switch, along with a link to a detail page that explains the situation in a little more depth.

[sudarshan85]

That is indeed very confusing. I've still not had a chance to start working with Pytorch Ligthning and seeing this naming conflicts makes me nervous.

[Jsontorch]

Yep. I'm a new user and when I see pytorch_lightning and lightning was confused too, add an extra explanation will much better.

[jxtngx]

here is the correct main page: https://lightning.ai/

[davidgilbertson]

A mistake I made (coming back after a few months of not using lightning):

1. Install lightning
2. Assume pytorch-lightning is now redundant, so uninstall that.

Re-running the install of lightning fixed things.

A quick note on the front page for people who currently have pytorch-lightning installed and want to update to lightning would be a good thing.

[surak]

This is VERY confusing and there's absolutely no clarification on the Pipy pages about this.

What I noticed is that if I install pytorch-lightning with a running pytorch installation, I get this:

Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.3 frozenlist-1.4.0 lightning-utilities-0.9.0 multidict-6.0.4 pip-23.2.1 pytorch-lightning-2.0.7 torchmetrics-1.0.3 yarl-1.9.2

And if I install lightning against an identical clean install, I get:

Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 arrow-1.2.3 async-timeout-4.0.3 backoff-2.2.1 blessed-1.20.0 croniter-1.4.1 dat
eutils-0.6.12 deepdiff-6.3.1 fastapi-0.101.1 frozenlist-1.4.0 fsspec-2023.6.0 inquirer-3.1.3 lightning-2.0.7 lightning-cloud-0.5.37 lightning-utilities-0.9.0 markdown-it-py-3.0.0 mdurl-0.1.2 multidict-6.0.4 ordered-set-4.1.0 pip-23.2.1 pygments-2.16.1 python-editor-1.0.4 python-multipart-0.0.6 pytorch-lightning-2.0.7 readchar-4.0.5 rich-13.5.2 starlette-0.27.0 starsessions-1.3.0 torchmetrics-1.0.3 typing-extensions-4.7.1 uvicorn-0.23.2 websocket-client-1.6.1 websockets-11.0.3 yarl-1.9.2

Which by itself install pytorch-lightning!

So is one the superset of the other?

[sudarshan85]

I'm surprised this hasn't been categorically resolved by now. Its been nearly 6 months since I originally opened the issue. I've not had a need to use Pytorch Lightning during this time so I haven't run into the versions problems. But this needs to be made a priority and resolved both in the packages and in the documentation.

[jxtngx]

Hi @sudarshan85 👋 let's see if I can help clarify this. Also, thank you (and @surak) for being persistent in this thread.

Note

we are going to address import conventions in issue 18327

I started using the Lightning ecosystem last year before joining the team a couple of months ago. I've experienced the changes as a community member, and now as a developer advocate. Before I get into the weeds, here is a high level explanation.

• Lightning Flash is for deep learning beginners.
• Lightning Bolts is for practitioners who need a foundational model to finetune.
• PyTorch Lightning is for practitioners who are implementing a model "from-scratch" with PyTorch and want a robust trainer.
• We are fine to install PyTorch Lightning as a standalone dependency with pip install pytorch-lightning

Diving In

Note

this is my framing and not an official position

Lightning is a mono package that contains 1) PyTorch Lightning 2) Lightning Fabric and 3) Lightning Apps. Using pip install lightning installs all three and TorchMetrics. I'll circle back to this after briefly covering Flash, Bolts, and TorchMetrics.

I think of Flash and Bolts as the scikit-learn of the Lightning AI ecosystem in that each is high-level, and contains model implementations. Whereas without Flash or Bolts we would create our own models "from-scratch" with PyTorch, and then train with PyTorch Lightning or Lightning Fabric – Lightning Fabric is a lower level trainer solution that gives more control back to experienced practitioners.

On Flash. Flash is high level and is for getting started with Deep Learning. Flash offers common deep learning tasks you can use out-of-the-box in a few lines of code, and it is built on top of PyTorch Lightning.

On Bolts. Bolts is also high level and built on top of PyTorch Lightning. We would want to use bolts for its in-built models, to finetune on our own data, and to extend the in-built models for additional research.

With respect to Flash and Bolts, this means the leveling looks something like:

flowchart LR;
    A[PyTorch] --> B[PyTorch Lightning] --> C[Bolts];
    B --> D[Flash]

TorchMetrics is a collection of 100+ metrics for several domains, some of which are: audio, image, text, classification, and regression. TorchMetrics can be thought of as the equivalent of sklearn.metrics. However, TorchMetrics implementations are capable of being used on CPU or GPU, and in distributed training environments.

On Lightning and PyTorch Lightning

Last year the team rolled out Lightning Apps and with that came a decision to unify PyTorch Lightning and Lightning Apps into a single repo and framework – Lightning. Around that time Lighting Fabric – a lower level trainer – was also created and placed into the Lightning repo. Meaning, all of (PyTorch Lightning, Lightning Fabric, and Lightning Apps) exist in Lightning.

We can still install PyTorch Lightning as a standalone framework for backwards compatibility with Lightning 2.0.

With respect to PyTorch Lightning and Lightning Fabric, the levels are:

flowchart LR;
    A[PyTorch] --> B[Lightning Fabric] --> C[PyTorch Lightning];

And the flow for deciding between a PyTorch Lightning and custom Lightning Fabric Trainer is something like:

flowchart LR;
    A[PyTorch] --> B{{Prefers using a pre-built Trainer}} --> D[PyTorch Lightning];
    A --> C{{Prefers creating a custom Trainer}} --> E[Lightning Fabric]
    D --> F[your trained or tuned model checkpoint]
    E --> F

Where it is implied that either a PyTorch Lighting Trainer or a custom Trainer built with Lightning Fabric enjoys out of the box distributed training capabilities by setting common arguments such as devices, num_nodes, and strategy, and training on either CPU or GPU by setting the accelerator argument.

[jxtngx]

to install any of (PyTorch Lightning, Lightning Fabric, or Lightning Apps), we would do:

PyTorch Lightning

Note

enables backwards compatibility between Lightning 2.0 and PyTorch Lightning 2.0

pip install pytorch-lightning

To install PyTorch Lightning, Lightning Fabric, and Lightning Apps (i.e, the Lightning Ecosystem):

pip install lightning

[sudarshan85]

Thank you so much for such a detailed reply. This makes so much sense and this should go somewhere in the documentation. What I can understand from this is, Pytorch lightning can be used the SAME way as it was used a year and half ago. From install (pytorch-lightning) to import (import pytorch_lightning as pl) to instantiation (pl.LightningModule). Am I right?

[jxtngx]

I've been using Lightning (instead of PyTorch Lightning) since last year so, I actually need to check this out for you before giving a solid yes.

[jxtngx]

@sudarshan85 the answer is – yes.

[PierFio]

Hello,
Thank you so much @JustinGoheen for this detailed presentation.
I have found this nice thread because I get caught by an apparent incompatibility between bolt and lightning:

Because no versions of lightning-bolts match >0.7.0,<0.8.0
 and lightning-bolts (0.7.0) depends on pytorch-lightning (>1.7.0,<2.0.0), lightning-bolts (>=0.7.0,<0.8.0) requires pytorch-lightning (>1.7.0,<2.0.0).
So, because MyModule depends on both pytorch-lightning (^2.0.6) and lightning-bolts (^0.7.0), version solving failed.

Regressing to pytorch-lightning 1.9.5 solved the problem, but raises the question about the coordination between these different developments. As far as I understand what is happening, bolts is an independent development from lightning.
Now I am quite uncertain where I should stand in term of version choices and unclear why some exception occurs (is it me or some mismatch of versions ?). I could certainly hack my way around but this kind of defeat the purpose of using Lighning in the first place. I would very much like your opinion and advice on this point.

[svnv-svsv-jm]

And yet, if you do:

import lightning.pytorch as pl
from pytorch_lightning import LightningModule

class MyModel(LightningModule):
    pass

model = MyModel()
trainer = pl.Trainer()
trainer.fit(model)

this still won't work, you will get:

TypeError: `model` must be a `LightningModule` or `torch._dynamo.OptimizedModule`, got `LightningModule`

because the Trainer was imported from lightning.pytorch, while the model is a LightningModule from pytorch_lightning. Are they the same or not?

[jovan-absci]

@JustinGoheen could you clarify? I ran into the same issue

[svnv-svsv-jm]

You can't mix them. You either always use lightning.pytorch or pytorch_lightning.