Despite the complexity zoo of LoRA variants, I do sort of feel like the only formal theory for modern models that we’re going to get is going to come from formalizing LoRA (via, e.g., Kac-Rice) and MoE (via convex optimization) Physics-y informal arguments suggest this strongly
TuringPost
TuringPost13.7. klo 20.34
13 new types of LoRA ▪️ T-LoRA ▪️ SingLoRA ▪️ LiON-LoRA ▪️ LoRA-Mixer ▪️ QR-LoRA ▪️ FreeLoRA ▪️ LoRA-Augmented Generation (LAG) ▪️ ARD-LoRA (Adaptive Rank Dynamic LoRA) ▪️ WaRA ▪️ BayesLoRA ▪️ Dual LoRA Learning (DLoRAL) ▪️ Safe Pruning LoRA (SPLoRA) ▪️ PLoP (Precise LoRA Placement) Save the list! Check this out to learn what they are and get the links to the papers →
4,26K