Microsoft researchers publish LoRA
Freezing pretrained weights and training small added matrices instead cut GPT-3's trainable parameter count by a factor the authors put at 10,000, with no extra inference cost.
Ideas & essays · Open weights & ecosystem