AI Developer Tool · Python
Single-GPU decoder-only LLM pretraining in PyTorch. Built for study and experimentation with modular architectures and Hydra configs: components are written from scratch to modify in source, with toggles for BF16 AMP, SDPA, compile, and Liger FLCE for speed.
mini-llm-lab is a configurable framework for pretraining decoder-only language models from scratch, featuring custom implementations of model components, a two-stage warmup-stable-decay training schedule, bit-exact resume capability, and an integrated chat interface for interacting with trained checkpoints.
Merit describes the code at this commit. Later pushes count once the owner rescans. How merit is scored
How much of the project is about each field, as scoring counts it. It ranks in the field it is most about.
Read from the project's code at its scored commit.
The owner keeps this project's limitations private.
