Saving and loading HF transformer model fine tuned with PL?

[brijow]

I am fine-tuning hugging face transformer models, essentially exactly as shown in the following example found in the pytorch lightning docs:

https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/text-transformers.html

Where we instantiate the LightningModule doing something like this:

class GLUETransformer(LightningModule):

    def __init__(self, ... ):
        super().__init__()
        self.config = AutoConfig.from_pretrained(model_name_or_path, num_labels=num_labels)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            model_name_or_path, config=self.config
        )

But I have been confused about how I should be saving and loading checkpoints.

When saving checkpoints, should I be using
mymodel.model.save_pretrained("model_save_dir"),
and reloading from this checkpoint using
AutoModelForSequenceClassification.from_pretrained("model_save_dir"),

or saving with
trainer.save_checkpoint("model_save_dir/checkpoint.ckpt"),
and reloading with
GLUETransformer.load_from_checkpoint("model_save_dir/checkpoint.ckpt")?

[tchaton]

Dear @brijow,

You should be using the second approach. An even better one would be to rely on ModelCheckpoint to save the checkpoints and provide Trainer(resume_from_checkpoint=...) for reloading all the states.

Best,
T.C

[brijow]

Thank you again, @tchaton

[vblagoje]

@brijow Is there a way to unpack HF PL checkpoints into constituents (e.g. pytorch_model.bin, config.json, tokenizer.json etc.) usually found on the HF hub hosted models. Most importantly, how would I extract just the pytorch_model.bin from the PL checkpoint?