ColiZig

Back to projects
LanguageOther
LicenseMIT
PlatformGitHub
Added19 days ago

A from-scratch Zig inference engine for large MoE models - stream experts from SSD to RAM to VRAM and keep the resident working set small. ColiZig explores a different way to run Mixture-of-Experts models: instead of loading the entire model into memory, it treats expert weights as a tiered working set and streams them between SSD, RAM and opt. VRAM