Skip Navigation

LocalLLaMA @sh.itjust.works

noneabove1182 @sh.itjust.works

2y ago

Llama-2, Mo’ Lora (proof of concept MOE of LoRAs)

crumbly.medium.com

Llama-2, Mo’ Lora

https://twitter.com/aicrumb/status/1681846805959528448?t=sG6Xn4p0hodDoB-g7gmuJQ

https://colab.research.google.com/#fileId=https%3A//huggingface.co/datasets/crumb/Wizard-EvolInstruct70k-k4/blob/main/MoLora_7b_(PROOF_OF_CONCEPT).ipynb

Very interesting concept, excited to see where it goes

2 comments

What's the context for the Moe/Simpson's pic?
- Probably just because of the MOE (mixture of experts) lol