This notebook has been automatically translated to make it accessible to more people, please let me know if you see any typos.

The increasing size of language models makes it more and more expensive to train them because more and more VRAM is needed to store all their parameters and the gradients derived from training

In the paper LoRA – Low rank adaption of large language models they propose to freeze the model weights and train two matrices called A and B greatly reducing the number of parameters to be trained.

Let’s see how this is done

To understand how LoRA works, we first have to remember what happens when we train a model. Let’s go back to the most basic part of deep learning, we have a dense layer of a neural network that is defined as:

y = Wx + b

Where W is the weights matrix and b is the bias vector.

For the sake of simplicity we will assume that there is no bias, so it would look like this

y = Wx

Suppose that for an input x we want it to have an output ŷ.

- First what we do is to calculate the output we get with our current value of pesos W, i.e. we get the value y.
- Next we calculate the error that exists between the value of y that we have obtained and the value that we wanted to obtain ŷ. We call this error loss, and we calculate it with some mathematical function, now it does not matter which one.
- We compute the gardient (the derivative) of the error loss with respect to the weights matrix W, i.e. ΔW = dloss/dW.
- We update the weights W by subtracting from each of their values the value of the gradient multiplied by a learning factor α, i.e. W = W – α·ΔW.

The authors of LoRA propose that the weights matrix W can be decomposed into

W ~ W + ΔW

So, by freezing the W matrix and training only the ΔW matrix, it is possible to obtain a model that fits new data without having to retrain the whole model.

But you may think that ΔW is a matrix of size equal to W so nothing has been gained, but here the authors rely on

, a paper in which they showed that although the language models are large and their parameters are matrices with very large dimensions, to adapt them to new tasks it is not necessary to change all the values of the matrices, but changing a few values is enough, which in technical terms, is called Low Rank Adaptation. Hence the name LoRA (Low Rank Adaptation).**Aghajanyan et al. (2020)**

We have frozen the model and now we want to train the ΔW matrix, let’s assume that both W and ΔW are matrices of size 20×10, so we have 200 trainable parameters

Now suppose that the matrix ΔW can be decomposed into the product of two matrices A and B, i.e.

ΔW = A · B

For this multiplication to occur the sizes of the matrices A and B have to be 20xn and 10×10 respectively. Suppose n=5, so A would be of size 20×5, i.e. 100 parameters, and B of size 5×10, i.e. 50 parameters, so we would have 100+50=150 trainable parameters. We already have less trainable parameters than before

Now let’s suppose that W is actually a matrix of size 10.000×10.000, so we would have 100.000.000 trainable parameters, but if we decompose ΔW in A and B with n=5, we would have a matrix of size 10.000×5 and another one of size 5×10.000, so we would have 50.000 parameters of one and another 50.000 parameters of the other, in total 100.000 trainable parameters, that is to say we have reduced the number of parameters 1000 times.

You can already see the power of LoRA, when you have very large models, the number of trainable parameters can be greatly reduced.

If we look again at the image of the LoRA architecture, we will understand it better.

But it looks even better, the savings in number of trainable parameters with this image

Since language models are implementations of transformers, let’s see how LoRA is implemented in transformers. In the transformer architecture there are linear layers in the Q, K and V attention matrices, and in the feedforward layers, so LoRA can be applied to all these linear layers. In the paper they say that for simplicity they apply it only to the linear layers of the Q, K and V attention matrices.

These layers have a size d_model x d_model, where d_model is the embedding dimension of the model.

In order to have these benefits, the size of the range r have to be smaller than the size of the linear layers. Since we have said that they only implemented it in the linear layers of attention, which have a size d_model x d_model, the rank size r has to be smaller than d_model.

The matrices A and B are initialized with a random Gaussian distribution for A and zero for B, so the product of both matrices will be zero at the beginning, i.e.

ΔW = A · B = 0

Finally, in the LoRA implementation, a α parameter is added to establish the degree of influence of LoRA on training. It is similar to the learning rate in normal fine tuning, but in this case it is used to establish the influence of LoRA on the training. Thus the LoRA formula would look like this

W = W + α ΔW = W + α(A · B)

We are going to repeat the training code of the post Fine tuning SLMs, specifically the training for text classification with the Hugging Face libraries, but this time we are going to do it with LoRA. In the previous post we used a batch size of 28 for the training loop and 40 for the evaluation loop, however, as now we are not going to train all the weights of the model, but only the LoRA matrices, we will be able to use a bigger batch size

We log in to upload the model to the Hub

from huggingface_hub import notebook_login

notebook_login()

We download the dataset we are going to use, which is a dataset of reviews from Amazon

from datasets import load_dataset

dataset = load_dataset("mteb/amazon_reviews_multi", "en")

dataset

We create a subset in case you want to test the code with a smaller dataset. In my case I will use 100% of the dataset

percentage = 1

subset_dataset_train = dataset['train'].select(range(int(len(dataset['train']) * percentage)))

subset_dataset_validation = dataset['validation'].select(range(int(len(dataset['validation']) * percentage)))

subset_dataset_test = dataset['test'].select(range(int(len(dataset['test']) * percentage)))

subset_dataset_train, subset_dataset_validation, subset_dataset_test

We see a sample

from random import randint

idx = randint(0, len(subset_dataset_train))

subset_dataset_train[idx]

We obtain the number of classes, to obtain the number of classes we use

and not **dataset['train']**

because if the subset is too small it is possible that there are no examples with all the possible classes of the original dataset.**subset_dataset_train**

num_classes = len(dataset['train'].unique('label'))

num_classes

We create a function to create the

field in the dataset. The downloaded dataset has the **label**

field but the **labels**

library needs the field to be called **transformers**

and not **label**

.**labels**

def set_labels(example):

example['labels'] = example['label']

return example

We apply the function to the dataset

subset_dataset_train = subset_dataset_train.map(set_labels)

subset_dataset_validation = subset_dataset_validation.map(set_labels)

subset_dataset_test = subset_dataset_test.map(set_labels)

subset_dataset_train, subset_dataset_validation, subset_dataset_test

Here is a sample again

subset_dataset_train[idx]

We implement the tokenizer. To avoid errors, we assign the end of string token to the padding token.

from transformers import AutoTokenizer

checkpoint = "openai-community/gpt2"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

tokenizer.pad_token = tokenizer.eos_token

We create a function for tokenizing the dataset

def tokenize_function(examples):

return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=768, return_tensors="pt")

We apply the function to the dataset and remove the columns that we do not need

subset_dataset_train = subset_dataset_train.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_validation = subset_dataset_validation.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_test = subset_dataset_test.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_train, subset_dataset_validation, subset_dataset_test

We see again a sample, but in this case we only see the

.**keys**

subset_dataset_train[idx].keys()

We instantiate the model. Also, in order to avoid errors, we assign the end of string token to the padding token.

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=num_classes)

model.config.pad_token_id = model.config.eos_token_id

As we have already seen in the post Fine tuning SLMs we get a warning that some layers have not been initialized. This is because in this case, as it is a classification problem and when we have instantiated the model we have told it that we want it to be a classification model with 5 classes, the library has removed the last layer and replaced it with a 5 neuron one at the output. If you do not understand this well go to the post that I quote that is better eplicado

Before implementing LoRA, we look at the number of trainable parameters that the model has

total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f"Total trainable parameters before: {total_params:,}")

We see that it has 124M trainable parameters. Now let’s freeze them

for param in model.parameters():

param.requires_grad = False

total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f"Total trainable parameters after: {total_params:,}")

After freezing there are no more trainable parameters

Let’s see what the model looks like before applying LoRA

model

First we create the LoRA layer.

It has to inherit from

so that it can act as a layer of a neural network.**torch.nn.Module**

In the

method we create the **_init_**

and **A**

matrices initialized as explained before, the **B**

matrix with a random Gaussian distribution and the **A**

matrix with zeros. We also create the parameters **B**

and **rank**

.**alpha**

In the

method we calculate LoRA as explained above.**forward**

import torch

class LoRALayer(torch.nn.Module):

def __init__(self, in_dim, out_dim, rank, alpha):

super().__init__()

self.A = torch.nn.Parameter(torch.empty(in_dim, rank))

torch.nn.init.kaiming_uniform_(self.A, a=torch.sqrt(torch.tensor(5.)).item()) # similar to standard weight initialization

self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))

self.alpha = alpha

def forward(self, x):

x = self.alpha * (x @ self.A @ self.B)

return x

Now we create a linear class with LoRA.

As before, it inherits from

so that it can act as a layer of a neural network.**torch.nn.Module**

In the

method we create a variable with the original linear layer of the network and we create another variable with the new LoRA layer that we had implemented before**_init_**

In the

method we add the outputs of the original linear layer and the LoRA layer.**forward**

class LoRALinear(torch.nn.Module):

def __init__(self, linear, rank, alpha):

super().__init__()

self.linear = linear

self.lora = LoRALayer(

linear.in_features, linear.out_features, rank, alpha

)

def forward(self, x):

return self.linear(x) + self.lora(x)

Finally we create a function that replaces the linear layers by the new linear layer with LoRA that we have created. What it does is that if it finds a linear layer in the model, it replaces it with the linear layer with LoRA, if not, it applies the function within the sublayers of the layer.

def replace_linear_with_lora(model, rank, alpha):

for name, module in model.named_children():

if isinstance(module, torch.nn.Linear):

# Replace the Linear layer with LinearWithLoRA

setattr(model, name, LoRALinear(module, rank, alpha))

else:

# Recursively apply the same function to child modules

replace_linear_with_lora(module, rank, alpha)

We apply the function to the model to replace the linear layers of the model by the new linear layer with LoRA

rank = 16

alpha = 16

replace_linear_with_lora(model, rank=rank, alpha=alpha)

We now see the number of trainable parameters

total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f"Total trainable LoRA parameters: {total_params:,}")

We have gone from 124M trainable parameters to 12k trainable parameters, i.e. we have reduced the number of trainable parameters 10,000 times!

We see the model again

model

Let’s compare them layer by layer

Original Model | Model with LoRA |
---|---|

GPT2ForSequenceClassification( | GPT2ForSequenceClassification( |

(transformer): GPT2Model( | (transformer): GPT2Model( |

(wte): Embedding(50257, 768) | (wte): Embedding(50257, 768) |

(wpe): Embedding(1024, 768) | (wpe): Embedding(1024, 768) |

(drop): Dropout(p=0.1, inplace=False) | (drop): Dropout(p=0.1, inplace=False) |

(h): ModuleList( | (h): ModuleList( |

(0-11): 12 x GPT2Block( | (0-11): 12 x GPT2Block( |

(ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) | (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) |

(attn): GPT2Attention( | (attn): GPT2Attention( |

(c_attn): Conv1D() | (c_attn): Conv1D() |

(c_proj): Conv1D() | (c_proj): Conv1D() |

(attn_dropout): Dropout(p=0.1, inplace=False) | (attn_dropout): Dropout(p=0.1, inplace=False) |

(resid_dropout): Dropout(p=0.1, inplace=False) | (resid_dropout): Dropout(p=0.1, inplace=False) |

) | ) |

(ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) | (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) |

(mlp): GPT2MLP( | (mlp): GPT2MLP( |

(c_fc): Conv1D() | (c_fc): Conv1D() |

(c_proj): Conv1D() | (c_proj): Conv1D() |

(act): NewGELUActivation() | (act): NewGELUActivation() |

(dropout): Dropout(p=0.1, inplace=False) | (dropout): Dropout(p=0.1, inplace=False) |

) | ) |

) | ) |

) | ) |

(ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True) | (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True) |

) | ) |

(score): LoRALinear() | |

(score): Linear(in_features=768, out_features=5, bias=False) | (linear): Linear(in_features=768, out_features=5, bias=False) |

(lora): LoRALayer() | |

) | |

) | ) |

We see that they are the same except at the end, where in the original model there was a normal linear layer and in the model with LoRA there is a

layer that inside has the linear layer of the original model and a **LoRALinear**

layer.**LoRALayer**

Once the model has been instantiated with LoRA, let’s train it as usual

As we have said, in the post Fine tuning SLMs we used a batch size of 28 for the training loop and 40 for the evaluation loop, while now that there are fewer trainable parameters we can use a larger batch size.

Why does this happen? When training a model, the model and its gradients must be saved in the GPU memory, so both with LoRA and without LoRA the model must be saved, but in the case of LoRA only the gradients of 12k parameters are saved, while with LoRA the gradients of 128M parameters are saved, so with LoRA less GPU memory is needed, so a larger batch size can be used.

from transformers import TrainingArguments

metric_name = "accuracy"

model_name = "GPT2-small-LoRA-finetuned-amazon-reviews-en-classification"

LR = 2e-5

BS_TRAIN = 400

BS_EVAL = 400

EPOCHS = 3

WEIGHT_DECAY = 0.01

training_args = TrainingArguments(

model_name,

eval_strategy="epoch",

save_strategy="epoch",

learning_rate=LR,

per_device_train_batch_size=BS_TRAIN,

per_device_eval_batch_size=BS_EVAL,

num_train_epochs=EPOCHS,

weight_decay=WEIGHT_DECAY,

lr_scheduler_type="cosine",

warmup_ratio = 0.1,

fp16=True,

load_best_model_at_end=True,

metric_for_best_model=metric_name,

push_to_hub=True,

logging_dir="./runs",

)

import numpy as np

from evaluate import load

metric = load("accuracy")

def compute_metrics(eval_pred):

print(eval_pred)

predictions, labels = eval_pred

predictions = np.argmax(predictions, axis=1)

return metric.compute(predictions=predictions, references=labels)

from transformers import Trainer

trainer = Trainer(

model,

training_args,

train_dataset=subset_dataset_train,

eval_dataset=subset_dataset_validation,

tokenizer=tokenizer,

compute_metrics=compute_metrics,

)

trainer.train()

Once trained we evaluate on the test dataset

trainer.evaluate(eval_dataset=subset_dataset_test)

Now that we have our model trained, we can share it with the world, so first we create a model card.

trainer.create_model_card()

And now we can publish it. As the first thing we have done is to log in with the huggingface hub, we can upload it to our hub without any problem.

trainer.push_to_hub()

We clean as much as possible

import torch

import gc

def clear_hardwares():

torch.clear_autocast_cache()

torch.cuda.ipc_collect()

torch.cuda.empty_cache()

gc.collect()

clear_hardwares()

clear_hardwares()

As we have uploaded the model to our hub we can download it and use it.

from transformers import pipeline

user = "maximofn"

checkpoints = f"{user}/{model_name}"

task = "text-classification"

classifier = pipeline(task, model=checkpoints, tokenizer=checkpoints)

Now if we want to return the probability of all classes, we simply use the classifier we just instantiated, with the parameter

.**top_k=None**

labels = classifier("I love this product", top_k=None)

labels

If we only want the class with the highest probability we do the same but with the parameter

.**top_k=1**

label = classifier("I love this product", top_k=1)

label

And if we want n classes we do the same but with the parameter

.**top_k=n**

two_labels = classifier("I love this product", top_k=2)

two_labels

We can also test the model with Automodel and AutoTokenizer.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

import torch

model_name = "GPT2-small-finetuned-amazon-reviews-en-classification"

user = "maximofn"

checkpoint = f"{user}/{model_name}"

num_classes = num_classes

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=num_classes).half().eval().to("cuda")

tokens = tokenizer.encode("I love this product", return_tensors="pt").to(model.device)

with torch.no_grad():

output = model(tokens)

logits = output.logits

lables = torch.softmax(logits, dim=1).cpu().numpy().tolist()

lables[0]

If you want to test the model further you can see it in Maximofn/GPT2-small-LoRA-finetuned-amazon-reviews-en-classification

We can do the same with the

library of Hugging Face. Let’s take a look at it**PEFT**

We log in to upload the model to the Hub

from huggingface_hub import notebook_login

notebook_login()

We re-download the dataset

from datasets import load_dataset

dataset = load_dataset("mteb/amazon_reviews_multi", "en")

dataset

We create a subset in case you want to test the code with a smaller dataset. In my case I will use 100% of the dataset

percentage = 1

subset_dataset_train = dataset['train'].select(range(int(len(dataset['train']) * percentage)))

subset_dataset_validation = dataset['validation'].select(range(int(len(dataset['validation']) * percentage)))

subset_dataset_test = dataset['test'].select(range(int(len(dataset['test']) * percentage)))

subset_dataset_train, subset_dataset_validation, subset_dataset_test

We obtain the number of classes, to obtain the number of classes we use

and not **dataset['train']**

because if the subset is too small it is possible that there are no examples with all the possible classes of the original dataset.**subset_dataset_train**

num_classes = len(dataset['train'].unique('label'))

num_classes

We create a function to create the

field in the dataset. The downloaded dataset has the **label**

field but the **labels**

library needs the field to be called **transformers**

and not **label**

.**labels**

def set_labels(example):

example['labels'] = example['label']

return example

We apply the function to the dataset

subset_dataset_train = subset_dataset_train.map(set_labels)

subset_dataset_validation = subset_dataset_validation.map(set_labels)

subset_dataset_test = subset_dataset_test.map(set_labels)

subset_dataset_train, subset_dataset_validation, subset_dataset_test

We instantiate the tokenizer. To avoid errors, we assign the token of end of string to the token of padding

from transformers import AutoTokenizer

checkpoint = "openai-community/gpt2"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

tokenizer.pad_token = tokenizer.eos_token

We create a function for tokenizing the dataset

def tokenize_function(examples):

return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=768, return_tensors="pt")

We apply the function to the dataset and remove the columns that we do not need

subset_dataset_train = subset_dataset_train.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_validation = subset_dataset_validation.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_test = subset_dataset_test.map(tokenize_function, batched=True, remove_columns=['text', 'label', 'id', 'label_text'])

subset_dataset_train, subset_dataset_validation, subset_dataset_test

We instantiate the model. Also, in order to avoid errors, we assign the end of string token to the padding token.

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=num_classes)

model.config.pad_token_id = model.config.eos_token_id

Before creating the model with LoRA, let’s take a look at its layers

model

As we can see there is only one

layer, which is **Linear**

and that is the one we are going to replace.**score**

We can create a LoRA configuration with the PEFT library and then apply LoRA to the mo

from peft import LoraConfig, TaskType

peft_config = LoraConfig(

r=16,

lora_alpha=32,

lora_dropout=0.1,

task_type=TaskType.SEQ_CLS,

target_modules=["score"],

)

With this configuration we have configured a rank of 16 and an alpha of 32. In addition we have added a dropout to the lora layers of 0.1. We have to indicate the task to the LoRA configuration, in this case it is a sequence classification task. Finally we indicate which layers we want to replace, in this case the

layer.**score**

We now apply LoRA to the model

from peft import get_peft_model

model = get_peft_model(model, peft_config)

Let’s see how many trainable parameters the model has now

model.print_trainable_parameters()

We obtain the same trainable parameters as before

Once the model has been instantiated with LoRA, let’s train it as usual

from transformers import TrainingArguments

metric_name = "accuracy"

model_name = "GPT2-small-PEFT-LoRA-finetuned-amazon-reviews-en-classification"

LR = 2e-5

BS_TRAIN = 400

BS_EVAL = 400

EPOCHS = 3

WEIGHT_DECAY = 0.01

training_args = TrainingArguments(

model_name,

eval_strategy="epoch",

save_strategy="epoch",

learning_rate=LR,

per_device_train_batch_size=BS_TRAIN,

per_device_eval_batch_size=BS_EVAL,

num_train_epochs=EPOCHS,

weight_decay=WEIGHT_DECAY,

lr_scheduler_type="cosine",

warmup_ratio = 0.1,

fp16=True,

load_best_model_at_end=True,

metric_for_best_model=metric_name,

push_to_hub=True,

logging_dir="./runs",

)

import numpy as np

from evaluate import load

metric = load("accuracy")

def compute_metrics(eval_pred):

print(eval_pred)

predictions, labels = eval_pred

predictions = np.argmax(predictions, axis=1)

return metric.compute(predictions=predictions, references=labels)

from transformers import Trainer

trainer = Trainer(

model,

training_args,

train_dataset=subset_dataset_train,

eval_dataset=subset_dataset_validation,

tokenizer=tokenizer,

compute_metrics=compute_metrics,

)

trainer.train()

Once trained we evaluate on the test dataset

trainer.evaluate(eval_dataset=subset_dataset_test)

We create a model card

trainer.create_model_card()

We publish it

trainer.push_to_hub()

We clean as much as possible

import torch

import gc

def clear_hardwares():

torch.clear_autocast_cache()

torch.cuda.ipc_collect()

torch.cuda.empty_cache()

gc.collect()

clear_hardwares()

clear_hardwares()

As we have uploaded the model to our hub we can download it and use it.

from transformers import pipeline

user = "maximofn"

checkpoints = f"{user}/{model_name}"

task = "text-classification"

classifier = pipeline(task, model=checkpoints, tokenizer=checkpoints)

Now if we want to return the probability of all classes, we simply use the classifier we just instantiated, with the parameter

.**top_k=None**

labels = classifier("I love this product", top_k=None)

labels

If we only want the class with the highest probability we do the same but with the parameter

.**top_k=1**

label = classifier("I love this product", top_k=1)

label

And if we want n classes we do the same but with the parameter

.**top_k=n**

two_labels = classifier("I love this product", top_k=2)

two_labels

If you want to test the model further you can see it in Maximofn/GPT2-small-PEFT-LoRA-finetuned-amazon-reviews-en-classification