[v3] Training refactor - MultiGPU, loss logging, bf16, etc.
#2449
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
These are the remaining experiments that did not fit in the original PR description due to the character limit. Example 4This example finetunes Training Scriptfrom collections import defaultdict
import datasets
from datasets import Dataset
from sentence_transformers import (
SentenceTransformerTrainingArguments,
SentenceTransformer,
SentenceTransformerTrainer,
losses,
evaluation,
)
from sentence_transformers.models import Transformer, Pooling
def to_triplets(dataset):
premises = defaultdict(dict)
for sample in dataset:
premises[sample["premise"]][sample["label"]] = sample["hypothesis"]
queries = []
positives = []
negatives = []
for premise, sentences in premises.items():
if 0 in sentences and 2 in sentences:
queries.append(premise)
positives.append(sentences[0]) # <- entailment
negatives.append(sentences[2]) # <- contradiction
return Dataset.from_dict({
"anchor": queries,
"positive": positives,
"negative": negatives,
})
snli_ds = datasets.load_dataset("snli")
snli_ds = datasets.DatasetDict({
"train": to_triplets(snli_ds["train"]),
"validation": to_triplets(snli_ds["validation"]),
"test": to_triplets(snli_ds["test"]),
})
multi_nli_ds = datasets.load_dataset("multi_nli")
multi_nli_ds = datasets.DatasetDict({
"train": to_triplets(multi_nli_ds["train"]),
"validation_matched": to_triplets(multi_nli_ds["validation_matched"]),
})
all_nli_train_ds = datasets.DatasetDict({
"multi_ds": multi_nli_ds["train"],
"snli_ds": snli_ds["train"],
})
all_nli_eval_ds = datasets.DatasetDict({
"multi_ds": multi_nli_ds["validation_matched"],
"snli_ds": snli_ds["validation"],
})
stsb_dev = datasets.load_dataset("mteb/stsbenchmark-sts", split="validation")
stsb_test = datasets.load_dataset("mteb/stsbenchmark-sts", split="test")
training_args = SentenceTransformerTrainingArguments(
output_dir="checkpoints",
num_train_epochs=1,
seed=33,
per_device_train_batch_size=64,
per_device_eval_batch_size=64,
learning_rate=2e-5,
warmup_ratio=0.1,
bf16=True,
logging_steps=100,
evaluation_strategy="steps",
eval_steps=400,
save_steps=400,
save_total_limit=2,
metric_for_best_model="spearman_cosine",
greater_is_better=True,
)
transformer = Transformer("microsoft/mpnet-base", max_seq_length=384)
pooling = Pooling(transformer.get_word_embedding_dimension(), pooling_mode="mean")
model = SentenceTransformer(modules=[transformer, pooling])
loss = losses.MultipleNegativesRankingLoss(model)
dev_evaluator = evaluation.EmbeddingSimilarityEvaluator(
stsb_dev["sentence1"],
stsb_dev["sentence2"],
[score / 5 for score in stsb_dev["score"]],
main_similarity=evaluation.SimilarityFunction.COSINE,
name="sts-dev",
)
trainer = SentenceTransformerTrainer(
model=model,
evaluator=dev_evaluator,
args=training_args,
train_dataset=all_nli_train_ds,
eval_dataset=all_nli_eval_ds,
loss=loss,
)
trainer.train()
test_evaluator = evaluation.EmbeddingSimilarityEvaluator(
stsb_test["sentence1"],
stsb_test["sentence2"],
[score / 5 for score in stsb_test["score"]],
main_similarity=evaluation.SimilarityFunction.COSINE,
name="sts-test",
)
results = test_evaluator(model)
print(results)Training LogsTraining NotesAs the logs show, we now run two bits of evaluation, producing a The loss goes as we had hoped, and the Spearman/Pearson performances are much jumpier, perhaps due to the much smaller batch size resulting in less "normalization"? Interestingly, if we set the The training loss keeps jumping back and forth, but the evaluation performance is nice and smooth. If we set the logging_steps a bit higher, the training loss would also start looking much smoother. This shows that unlike in experiment 3, the dataset samples are kept fully separate here, and it shows that SNLI is m uch harder than MultiNLI. Fascinating! Example 5This example finetunes The script evaluates on SNLI and MultiNLI via the Softmax loss, on STSBenchmark dev via the CosineSimilarityLoss, and on the STSBenchmark development set via the Training Scriptfrom collections import defaultdict
from typing import Dict
import datasets
from datasets import Dataset
from transformers import EvalPrediction
from sentence_transformers import (
SentenceTransformerTrainingArguments,
SentenceTransformer,
SentenceTransformerTrainer,
losses,
evaluation,
)
from sentence_transformers.models import Transformer, Pooling
snli_ds = datasets.load_dataset("snli")
snli_ds = datasets.DatasetDict({
"train": snli_ds["train"],
"validation": snli_ds["validation"],
"test": snli_ds["test"],
})
multi_nli_ds = datasets.load_dataset("multi_nli")
multi_nli_ds = datasets.DatasetDict({
"train": multi_nli_ds["train"].remove_columns(set(multi_nli_ds["train"].column_names) - {"premise", "hypothesis", "label"}),
"validation_matched": multi_nli_ds["validation_matched"].remove_columns(set(multi_nli_ds["validation_matched"].column_names) - {"premise", "hypothesis", "label"}),
})
def normalize_label(sample):
sample["label"] = sample["label"] / 5
return sample
stsb_train = datasets.load_dataset("mteb/stsbenchmark-sts", split="train").select_columns(["sentence1", "sentence2", "score"]).rename_column("score", "label").map(normalize_label)
stsb_dev = datasets.load_dataset("mteb/stsbenchmark-sts", split="validation").select_columns(["sentence1", "sentence2", "score"]).rename_column("score", "label").map(normalize_label)
stsb_test = datasets.load_dataset("mteb/stsbenchmark-sts", split="test").select_columns(["sentence1", "sentence2", "score"]).rename_column("score", "label").map(normalize_label)
train_dataset = datasets.DatasetDict({
"multi_nli": multi_nli_ds["train"],
"snli": snli_ds["train"].rename_column("premise", "snli_premise").filter(lambda x: x["label"] != -1),
"stsb": stsb_train,
})
eval_dataset = datasets.DatasetDict({
"multi_nli": multi_nli_ds["validation_matched"].select(range(100)),
"snli": snli_ds["validation"].rename_column("premise", "snli_premise").filter(lambda x: x["label"] != -1),
"stsb": stsb_dev,
})
training_args = SentenceTransformerTrainingArguments(
output_dir="checkpoints",
num_train_epochs=1,
seed=33,
per_device_train_batch_size=128,
per_device_eval_batch_size=128,
learning_rate=2e-5,
warmup_ratio=0.1,
bf16=True,
logging_steps=1,
evaluation_strategy="steps",
eval_steps=10,
save_steps=10,
save_total_limit=2,
metric_for_best_model="spearman_cosine",
greater_is_better=True,
)
transformer = Transformer("microsoft/mpnet-base", max_seq_length=384)
pooling = Pooling(transformer.get_word_embedding_dimension(), pooling_mode="mean")
model = SentenceTransformer(modules=[transformer, pooling])
nli_loss = losses.SoftmaxLoss(model=model, sentence_embedding_dimension=model.get_sentence_embedding_dimension(), num_labels=3)
cosine_loss = losses.CosineSimilarityLoss(model)
dev_evaluator = evaluation.EmbeddingSimilarityEvaluator(
stsb_dev["sentence1"],
stsb_dev["sentence2"],
stsb_dev["label"],
main_similarity=evaluation.SimilarityFunction.COSINE,
name="sts-dev",
)
trainer = SentenceTransformerTrainer(
model=model,
evaluator=dev_evaluator,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss={
"multi_nli": nli_loss,
"snli": nli_loss,
"stsb": cosine_loss,
},
)
trainer.train()
test_evaluator = evaluation.EmbeddingSimilarityEvaluator(
stsb_test["sentence1"],
stsb_test["sentence2"],
stsb_test["label"],
main_similarity=evaluation.SimilarityFunction.COSINE,
name="sts-test",
)
results = test_evaluator(model)
print(results)Training LogsTraining NotesAlthough this model is not as good as the previous ones, that was to be expected given the less useful loss functions. That said, the experiment was primarily to train a model with multiple loss functions effectively. Because the logging_steps is just 1, we end up with a very jumpy training loss: there's no averaging after all: It's hard to tell what's going on there, but luckily the evaluation losses are much clearer: Also note that training concluded fairly quickly, because in the default round-robin behaviour that
|
|
@b5y you expressed an interest in training Sentence Transformer models using the transformers Trainer in #2446 - this PR should work for you. If you'd like, you're free to experiment with it! You can install it (for now) with:
|
|
Thanks a lot for mentioning! |
|
Hey @tomaarsen ! Thanks for your work again! I looked carefully and didn't find any examples for asymmetric semantic search using MSMARCO-based models and dot-product score. Are you planning to add this kind of example or you expect others to try out? |
|
Heya! You might be able to use any of the MSMarco examples from here, e.g. this one: https://github.com/tomaarsen/sentence-transformers/blob/v3/trainer_refactor/examples/training/ms_marco/train_bi-encoder_mnrl.py I haven't yet tested if this one works, but I would like for it to work before I could merge this PR.
|
|
If it's using the |
|
Yes, it should be supported. |
|
Using this PR, is it now doable to use multiuple GPUs for training? I've read several places that users actually get a decrease in performance when using distirbuted training (although that is some ago i.e not using this version). |
|
我已收到您的邮件，我会尽快查看的。
来自 张惠玲
|
Yes. Consider checking out the training examples in the v3.0 pre-release branch. Each of those scripts should work with multi-GPU (although you will likely need to wrap the entire script in
|
|
@tomaarsen Aah sorry, I missed that benchmark (thanks a bunch!). Is the branch "v3.0-pre-release" the most "stable 3.0" branch or this there another branch which would be better to clone if I wan't to use it? |
|
No worries, there's a lot of comments/commits here! Even I scrolled past the benchmark 😄
In the coming days, #2632 will have more documentation on how training with v3 will work, but for now the best way is to look at the PR description for this merged PR and the updated training examples. Feel free to let me know if you have any issues/feedback.
|
|
I'm currently looking at it now and I struggle a bit on how to define the Before (v2.7) the training script was training_data = [InputExample(texts=[t1, t2]) for t1, t2 in zip(data["Texts"], data["MoreText"])]
model = SentenceTransformer("intfloat/multilingual-e5-small", device="cuda")
dataloader = DataLoader(training_data)
loss = losses.MultipleNegativesRankingLoss(model)
model.fit([(dataloader,loss)])It seems like the column/key names etc. doesn't matter using |
|
2 comments:
from datasets import Dataset
anchors = []
positives = []
for ...
anchors.append(...)
positives.append(...)
dataset = Dataset.from_dict({"anchor": anchors, "positive": positives})There are a few rules for the dataset:
So, the dataset from my previous snippet corresponds with I have prepared a few datasets in https://huggingface.co/datasets?other=sentence-transformers that can be used directly with Sentence Transformers. You can load these e.g. with: from datasets import load_dataset
dataset = load_dataset("sentence-transformers/all-nli", "pair")Lastly, here is a complete training script containing essentially all training features: from datasets import load_dataset
from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments
from sentence_transformers.losses import MultipleNegativesRankingLoss
from sentence_transformers.training_args import BatchSamplers
from sentence_transformers.evaluation import TripletEvaluator
# 1. Load a model to finetune
model = SentenceTransformer("microsoft/mpnet-base")
# 2. Load a dataset to finetune on
dataset = load_dataset("sentence-transformers/all-nli", "triplet")
train_dataset = dataset["train"]
eval_dataset = dataset["dev"]
test_dataset = dataset["test"]
# 3. Define a loss function
loss = MultipleNegativesRankingLoss(model)
# 4. (Optional) Specify training arguments
args = SentenceTransformerTrainingArguments(
# Required parameter:
output_dir="models/mpnet-base-all-nli-triplet",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
warmup_ratio=0.1,
fp16=True, # Set to False if you get an error that your GPU can't run on FP16
bf16=False, # Set to True if you have a GPU that supports BF16
batch_sampler=BatchSamplers.NO_DUPLICATES, # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
# Optional tracking/debugging parameters:
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
logging_steps=100,
run_name="mpnet-base-all-nli-triplet", # Will be used in W&B if `wandb` is installed
)
# 5. (Optional) Create an evaluator & evaluate the base model
dev_evaluator = TripletEvaluator(
anchors=eval_dataset["anchor"],
positives=eval_dataset["positive"],
negatives=eval_dataset["negative"],
name="all-nli-dev",
)
dev_evaluator(model)
# 6. Create a trainer & train
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
)
trainer.train()
# Evaluate the trained model on the test set
test_evaluator = TripletEvaluator(
anchors=test_dataset["anchor"],
positives=test_dataset["positive"],
negatives=test_dataset["negative"],
name="all-nli-test",
)
test_evaluator(model)
# Save the trained model and/or push it to the Hugging Face Hub
model.save_pretrained("models/mpnet-base-all-nli-triplet")
# model.push_to_hub("mpnet-base-all-nli-triplet")
|
Yeah, that is why I wanted to use the "new" approach.
If a column is named "score" or "label", then it corresponds with the "Labels" from the Loss Overview. Nice, thanks!
I'm getting a "no keyword argument 'eval_strategy'" ? |
|
|
Hi, how to set max_seq_length. |
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("bert-base-uncased")
model.max_seq_length = 256
print(model.max_seq_length)
# => 256Works both for Sentence Transformer models (e.g. "all-mpnet-base-v2") and non-Sentence Transformer models (e.g. "bert-base-cased")
|
|
Hello @tomaarsen def train_model():
model = SentenceTransformer("intfloat/multilingual-e5-small", device="cuda")
loss = losses.MultipleNegativesRankingLoss(model)
training_args = SentenceTransformerTrainingArguments(
# Required parameter:
output_dir="./sbert_fitted/",
# Optional training parameters:
num_train_epochs=1,
per_device_train_batch_size=4,
per_device_eval_batch_size=32,
warmup_ratio=0.1,
fp16=False, # Set to False if you get an error that your GPU can't run on FP16
bf16=False, # Set to True if you have a GPU that supports BF16
# Optional tracking/debugging parameters:
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
logging_steps=100,
run_name="sts", # Will be used in W&B if `wandb` is installed,
report_to="none"
)
data = get_main_data()
train_dataset = Dataset.from_pandas(data[["Text1","Text2"]])
trainer = SentenceTransformerTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
loss=loss,
)
print("Training ...")
trainer.train()
if __name__=="__main__":
train_model()it is run in a Docker (on google cloud) using the following command
which should use 2 GPUs. I'm getting an error though Is that something you have seen before? On another note; it seems like |
|
Hello! Thanks for testing this; I have seen something similar, yes. I experienced a similar error with #2647. Let me quickly try to create a fix, then you can give that a try. Edit: FYI, your script looks good.
|
This was referenced May 16, 2024
|
Shall it also work with the CrossEncoder training? It looks like cross encoder training is still using an old training backend. |
|
I'm afraid that CrossEncoder training will still use the old recipe, but updating CrossEncoder training is high on the TODO list after v3.
|
|
Hello @tomaarsen, I have used your training script code from example, wrap up training and model initialization in main() and called the main function. It works!!! Thanks But, label is required for training this script. I am planning to work on sentence-transformer for semantic search where I am fine-tuning my model using sbert. For that this script is giving me an error that label is needed. How to run sentence-transformer for semantic search usecase using multi-gpu with label score(float) and without label score? |
|
Hello! It depends on your data and your loss functions. Some loss functions require a label, and some don't. There's a few common formats for your data, and you can read about them here: https://sbert.net/docs/training/loss_overview.html#loss-overview So, if you for example have
Then you can use all of these loss functions: (A popular one for semantic search is There's a lot of examples of training datasets that fit this type here: https://huggingface.co/collections/sentence-transformers/embedding-model-datasets-6644d7a3673a511914aa7552 Also, here's a bit of related work-in-progress documentation: Hope this helps a bit
|
|
Does the below code, support multi-gpu? in V3? Code : train_examples = []
train_data = data['text']
for text, pos in zip(data['text'], data['positive']):
train_examples.append(InputExample(texts=[text, pos]))
train_dataloader = DataLoader(train_examples, shuffle=True)
word_embedding_model = models.Transformer('mixedbread-ai/mxbai-embed-large-v1')
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension())
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])
train_loss = losses.AnglELoss(model=model)
num_epochs=30
warmup_steps=8
model.fit(train_objectives=[(train_dataloader, train_loss)],
epochs=num_epochs,
warmup_steps=warmup_steps,
show_progress_bar=True)```
I ran this code but it is using only single gpu for now! |
|
I believe this should already use multi-GPU by default. But the AnglELoss requires float similarity score labels, which you don't seem to have. You have 2 texts (text and a positive text), so you can use these options without any preprocessing: Maybe then it'll work on Multi-GPU out of the box, too (although maybe you have to wrap your code in
|
|
Yes, got it. I am using label_score now so using AnglELoss how to run the file? using accelerate/torchrun or just python? I ran with python and saw wandb project it is showing multi-gpu! |
|
|
tomaarsen
added a commit
that referenced
this pull request
May 28, 2024
) * [`v3`] Training refactor - MultiGPU, loss logging, bf16, etc. (#2449) * See #1638: Adds huggingface trainer for sentence transformers * Fix type of tokenizer * Get the trainer using the feature collation * Update the docstring to reflect changes * Initial draft for refactoring training usig the Transformers Trainer * Separate 'fit' functionality (new and old) into a mixin * Resolve test issues * Reformat * Update the imports * Add TODO regarding custom label columns * Remove dead code * Don't provide the trainer to the eval sampler * Introduce datasets as a dependency * Introduce "accelerate" as a dependency * Avoid use_amp on CPU tests * Specify that SentenceTransformer is a class, not a module * Avoid circular import * Remove | used as an "or" operator in typing * Use test evaluator after training, as intended * Use tokenize function instead of tokenizer; Add EvaluatorCallback which calls the evaluator on every epoch (for BC); Stop saving "do_lower_case" from Transformer; * Reformat * Revert Transformer tokenizer changes * Add support for the tokenizer to return more than just input_ids & attention_masks Required for LSTM * Use the test evaluators after training the examples * Use pure torch for BoW tokenization * Use dev evaluator for BiLSTM - test fails * Add Trainer support for BoW-based models * Pass epoch to evaluator in every-epoch callback For fit backwards compatibility * Run formatting * Use steps_per_epoch to set max_steps if possible * Ignore extracting dataloader arguments for now * Remove dead code * Allow both "label" and "score" columns for labels * Reformatting * Improve errors if datasets don't match with loss dictionary well * Made tests more consistent; list instead of set * Simplify trainer with DatasetDict * Implement a proportional sampler in addition to round robin * Add CLIP finetuning support to the Trainer * Start updating evaluators to return dictionaries * Reformat * Hackishly insert the DataParallel model into the loss function * Allow for fsdp=["full_shard", "auto_wrap"] with fsdp_config={"transformer_layer_cls_to_wrap": "BertLayer"} * Re-add support for DataParallel * Use 'ParallelMode.NOT_PARALLEL' * Prevent crash with DDP & an evaluation set * When training with multiple datasets, add "dataset_name" column Rather than relying on some Batch Sampler hacking (which fails with some distributed training approaches) * Update type hints: make loss & evaluator optional Co-authored-by: Wang Bo <kingbolanda@live.com> * Set correct superclasses for samplers * Override 'accelerator.even_batches' as it's incompatible with multi-dataset * Throw exception if "return_loss" or "dataset_name" columns are used * Set min. version for accelerate * Heavily extend model card generation * Remove some dead code * Fix evaluator type hints * Ensure that 'model_card_template.md' is included in the built package * Rephrase comments slightly * Heavily refactor samplers; add no duplicates/group by label samplers * Ensure that data_loader.dataset exists in FitMixin * Adopt 8 as the default batch * Fix logging error in example * Remove the deprecated correct_bias * Simplify with walrus operator * Fix some bugs in set_widget_examples with short datasets * Improve docstring slightly * Add edge case in case training data has an unrecognized format * Fix extracting dataset metadata * Remove moot TYPE_CHECKING * Set base model when loading a ST model also * Add test_dataloader, add prefetch_factor to dataloaders * Resolve predict_example fix; fix newlines in text * Fix bug in compute_dataset_metrics examples * Add call to action in ValueError * Reuse original model card if no training is done * Also collect nested losses (e.g. MatryoshkaLoss) and make losses in tags * Remove generated tag; keep loss: prefix on tags * Remove unused arguments * Add support for "best model step" in model card * Make hyperparameters code-formatted * Fix load_best_model for Transformers models, prevent for non-Transformers * Store base_model_revision in model_card_data * Prevent crash when loading a local model * Allow for bfloat16 inference --------- Co-authored-by: Matthew Franglen <matthew@franglen.org> Co-authored-by: Wang Bo <kingbolanda@live.com> * [`v3`] Add `similarity` and `similarity_pairwise` methods to Sentence Transformers (#2615) * Add similarity function to model configuration * Add more tests * Replace util.cos_sim with model.similarity in some examples * Reintroduce evaluation.SimilarityFunction * Remove last references of score function in ST class * Add similarity_fn_name to model card * Add save_pretrained alias for save * Introduce DOT alias for DOT_PRODUCT * [`v3`] Fix various model card errors (#2616) * Prevent model card save failure * Print exceptions in more detail when they occur * Fix edge case if dataset language is None * [`v3`] Fix trainer `compute_loss` when evaluating/predicting if the `loss` updated the inputs in-place (#2617) * Recompute the features if return_output * Add SimilarityFunction to __init__, increment dev version * Never return None in infer_datasets (#2620) * Implement resume_from_checkpoint (#2621) * [`v3`] Update example scripts to the new v3 training format (#2622) * Update example scripts to the new v3 training format * Add distillation training examples * Add Matryoshka training examples * Add NLI training examples * Add STS training scripts * Fix accidentally overriding eval set * Update paraphrases multi-dataset training script * Convert regular dicts to DatasetDict on Trainer init * Update Quora duplicate training scripts * Update "other" training scripts * Update multilingual conversion script * Add example scripts to Evaluators * Add example to ST class itself * Update docs formatting slightly * Fix model card snippet * Add short docstring for similarity_fn_name property * Remove "return_outputs" as it's not strictly necessary. Avoids OOM & speeds up training (#2633) * Fix crash from inferring the dataset_id from a local dataset (#2636) See #2635 * Fix multilingual conversion script; extend MSELoss to multi-column (#2641) And remove the now-unnecessary make_multilingual_sys.py * Update evaluation scripts to use HF Datasets (#2642) * Increment the version in setup.py (as well) * Fix resume_from_checkpoint by also updating the loss (#2648) I'm not very sure if updating the potential wrapped model like this will also work; it seems a bit risky, but it's equally risky to not do it. * Fix an issue with in-place variable overriding preventing backwards passes on MSELoss (#2647) Only when there's multiple columns * Simplify load_from_checkpoint using load_state_dict (#2650) Overriding the model has several downsides, e.g. regarding the model card generation * Don't override the labels variable to avoid inplace operation (#2651) * Resolve "one of the variables needed for gradient computation has been modified by an inplace operation." (#2654) * [`v3`] Add hyperparameter optimization support by letting `loss` be a Callable that accepts a `model` (#2655) * Add HPO support by letting the 'loss' be a function * Only add "dataset_name" column if required by the loss function * Add tag hinting at the number of training samples (#2660) * [`v3`] For the Cached losses; ignore gradients if grad is disabled (e.g. eval) (#2668) * For the Cached losses; ignore gradients if grad is disabled (e.g. eval) * Warn that Matryoshka/AdaptiveLayer losses are not compatible with Cached * [`docs`] Rewrite the https://sbert.net documentation for v3.0 (#2632) * Start restructuring/rewriting the docs * Update Pretrained Models section for ST * Update & add many docstrings * Completely overhaul "Training Overview" docs page for ST * Update dataset overview * Remove kwargs from paraphrase_mining signature * Add "aka sbert" * Remove Hugging Face docs page * Update ST Usages * Fix some links * Use the training examples corresponding to that model type * Add hyperparameter optimization example script + docs * Add distributed training docs * Complete rewrite for the Sentence Transformer docs portion * Update the CE part of the docs * Specify if __name__ == "__main__" & dataloader_drop_last with DDP * Update the entire project to Google-style docstring * Remove contact page * Update README with updated links, etc. * Update the loss examples * Fix formatting * Add remove_columns/select_columns tip to dataset overview * [`v3`] Chore - include import sorting in ruff (#2672) * Include import sorting in ruff * Remove deprecated ignore-init-module-imports * Remove --select I from ruff.toml again after CI issues * [`v3`] Prevent warning with 'model.fit' with transformers >= 4.41.0 due to evaluation_strategy (#2673) * Prevent warning with 'model.fit' with transformers >= 4.41.0 due to evaluation_strategy * Reformat * [`v3`] Add various useful Sphinx packages (copy code, link to code, nicer tabs) (#2674) * No longer hide toctrees in API Reference * Add linkcode support It's not perfect, as it'll always link to 'master', but it'll do pretty nicely for the most part. * Add copy button to all code blocks * Add nicer tabs * Reformatted * [`v3`] Make the "primary_metric" for evaluators a bit more robust (#2675) * Make the "primary_metric" for evaluators a bit more robust * Also remove some other TODOs that are not very important or already done * Set `broadcast_buffers = False` when training with DDP (#2663) * [`v3`] Warn about using DP instead of DDP + set dataloader_drop_last with DDP (#2677) * Warn about using DP instead of DDP + set dataloader_drop_last with DDP * Prevent duplicate warnings * Remove note, done automatically now * Avoid inequality comparison to True * [`v3`] Add warning that Evaluators only run on 1 GPU when multi-GPU training (#2678) * Add warning that Evaluators only run on 1 GPU when multi-GPU training * Also add a note in the distributed training docs * [`v3`] Move training dependencies into a "train" extra (#2676) * Move training dependencies into a "train" extra * Install the train extra with the CI tests * Simplify dev install: also include train deps there * Implement is_..._available in ST instead; add is_training_available * Update references to the API ref (#2679) * [`v3`] Add "dataset_size:" to the tag denoting the number of training samples (#2680) * Prepend "dataset_size:" instead. I can always change the look of this later On the HF side * Fix formatting of Python modules * Docs: pairwise_cosine_similarity -> pairwise_similarity * Link to the yet-to-be-released release notes instead * Update phrasing on local_files_only docstring * Link directly to the 2DMSE preprint * Add missing subset in quora-duplicates * Add missing docstrings arguments for Cached... losses * Update training overview docs based on the blogpost reviews --------- Co-authored-by: Matthew Franglen <matthew@franglen.org> Co-authored-by: Wang Bo <kingbolanda@live.com>
This was referenced Jun 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #2436, Resolves #1446
Hello!
Pull Request overview
transformersTrainer rather than a manual training loop.fitbackwards compatibility.SentenceTransformerTrainerwould become the recommended training approach. It features:Details
This PR expands on work from @matthewfranglen & updates it to work in much more situations, e.g. with different dataset formats. Notably, training now centers around:
DatasetorDatasetDict. This class is much more suited for sharing & efficient modifications than lists/DataLoaders ofInputExampleinstances. ADatasetcan contain multiple text columns that will be fed in order to the corresponding loss function. So, if the loss expects (anchor, positive, negative) triplets, then your dataset should also have 3 columns. The names of these columns are irrelevant at this time. If there is a "label" column, it is treated separately, and used as the labels during training.A
DatasetDictcan be used to train with multiple datasets at once, e.g.:DatasetDictis used, thelossparameter to theSentenceTransformerTrainermust also be a dictionary with these dataset keys, e.g.:{ 'multi_nli': SoftmaxLoss(...), 'snli': SoftmaxLoss(...), 'stsb': CosineSimilarityLoss(...), }SentenceEvaluatorinstance. I am considering some breaking changes here, as these evaluators currently return floats on__call__, but I would like them to return a dictionary with all values that they've computed instead.After this PR, these instances either return a float, or a dictionary with metric keys and values. If the latter, the class must also defined
evaluator.primary_metricso e.g. the "best model" checkpointing can be based on an evaluator score.Models can now be evaluated both on an evaluation dataset with some loss function and/or a
SentenceEvaluatorinstance.SentenceTransformersTrainerinstance. This instance is provided with a SentenceTransformer model, a SentenceTransformerTrainingArguments class, a SentenceEvaluator, a training and evaluation Dataset/DatasetDict and a loss function/dict of loss functions. Most of these parameters are optional. Once provided, all you have to do is calltrain().Let's have a look at some example scripts of what training could look like in v3:
Example usage
Note: The old training loop (with
DataLoaders,InputExamples andmodel.fit) still works, you're free to keep using it. But, the new training loop has more arguments, etc.Example 1
This example finetunes
nli-distilroberta-base-v2using in-batch negatives on SICK, while evaluating on both the SICK evaluation set via loss and via theEmbeddingSimilarityEvaluatorTraining Script
Training Logs
Training Notes
W&B correctly logged the Spearman & Pearson correlation coefficients, as well as the evaluation loss
Example 2
This example finetunes
nli-distilroberta-base-v2using in-batch negatives on SNLI, while evaluating on both the SNLI development set via loss and SICK evaluation set via theEmbeddingSimilarityEvaluatorTraining Script
Training Logs
Training Notes
W&B captures the training & evaluation loss lowering. It also tracked runtime, learning rates, steps, & all
evaluatorvalues regarding the Spearman/Pearson correlation coefficient.Example 3
This example finetunes
microsoft/mpnet-baseusing in-batch negatives on AllNLI, a combination of SNLI and MultiNLI, while evaluating on both the AllNLI evaluation set via loss and on STSBenchmark development set via theEmbeddingSimilarityEvaluator. It uses a cached variant of the MultiNegativeRankingLoss (in-batch negatives loss), allowing us to set the batch size really high while using much smaller mini-batches. This gives much superior performance to the final model. After training, the model performance is tested against the STSBenchmark test set.Training Script
Training Logs
Training Notes
This Spearman Cosine of 84.56 on the test set is actually superior to
all-mpnet-base-v2, which scores a 83.42. We can attribute this increase to the higher batch size from the cached MNRL. Big thanks to @kwang2049 for implementing this.W&B captures the training & evaluation loss on every step as you'd hope:
Additionally, the evaluator results are also captured:
For the interested, this also tracks training speed, disk usage, GPU usage, memory usage, etc.
See the first comment for more examples! I ran out of characters in the PR description.
Backwards compatibility
I want to point out that the existing examples should still work, e.g. the ones that train using:
Notes
At a glance, it feels like the training time & memory usage is equivalent to the old manual training loop.
TODO's
Many TODOs still remain:
fitis used)Trainer.evaluateworks.cc @bwanglzu @ir2718 @johneckberg @aamir-s18 as I know you're interested in my TODO list.
cc @osanseviero @LysandreJik