ignitenlp

Source code on GitHub.

README

Transformers Example with Ignite

In this example, we show how to use _Ignite_ to finetune a transformer model:

• on 1 or more GPUs or TPUs
• compute training/validation metrics
• log learning rate, metrics etc
• save the best model weights

Configurations:

• [x] single GPU
• [x] multi GPUs on a single node
• [x] TPUs on Colab

Requirements:

• pytorch-ignite: pip install pytorch-ignite
• transformers: pip install transformers
• datasets: pip install datasets
• tqdm: pip install tqdm
• tensorboardx: pip install tensorboardX
• python-fire: pip install fire
• Optional: clearml: pip install clearml

Alternatively, install the all requirements using pip install -r requirements.txt.

Usage:

Run the example on a single GPU:

``bash python main.py run ` If needed, please, adjust the batch size to your GPU device with --batch_size argument.

The default model is bert-base-uncased incase you need to change that use the --model argument, for details on which models can be used refer here

Example: `bash #Using DistilBERT which has 40% less parameters than bert-base-uncased python main.py run --model="distilbert-base-uncased" `

For details on accepted arguments:

`bash python main.py run -- --help `

Distributed training

#### Single node, multiple GPUs

Let's start training on a single node with 2 gpus:

`bash

using torch.distributed.launch

`bash

using function spawn inside the code

`b...