Educational Implementation · Python
Low-Rank-Pretraining
LoRPt (Low-Rank Pretraining) is a novel technique that applies LoRA-style low-rank matrix factorization directly to model pretraining, enabling dramatically reduced memory consumption and faster training times for large language models.
What it is
The repository provides a low-rank factorized linear layer (LoRPtLinear) for memory-efficient pretraining, accompanied by extensive documentation describing its motivation, benchmark results, and ablation studies.
Merit describes the code at this commit. Later pushes count once the owner rescans. How merit is scored
Where it counts
- NLP & LLMs90%
- Machine Learning85%
How much of the project is about each field, as scoring counts it. It ranks in the field it is most about.
Around it on the NLP & LLMs board
Whole board- 89diffwhispererAI Developer Tool · Python46.5ImplementationemmjayhNLP & LLMs
- 90langflow-personalAI Developer Tool · TypeScript46.3ImplementationPetr SovadinaNLP & LLMs
- 91Low-Rank-PretrainingEducational Implementation · Python46.0ImplementationFlameF0XNLP & LLMs
- 92llamahtmlML Application · HTML44.1Implementationjerry howellNLP & LLMs
- 93ling-3.0-tiny-cu130-torch213-py312-sm89Inference System · Python43.8ImplementationRickNLP & LLMs
