3 Commits

Include renames

Author	SHA1	Message	Date
dela	49aede9cb2	Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE ... Whole-mixer checkpoint plus T×T MLA scores OOM'd a 31GB GPU on backward. Checkpoint each AttnRes block, run absorbed MLA through SDPA, and compute CE in vocab chunks so [B,T,V] logits are never materialized. --max-tokens is now the training budget; default --steps 2000 no longer caps a 1B-token run at 250 optimizer steps.	2026-08-25 20:09:27 +08:00
dela	7a12f61de1	LatentMoE: K3 sigmoid routing and Switch aux/z-loss ... Route with σ(W_r x), Top-k(s+b), then L1-normalize over the selected set. Add Switch/GShard aux and router z-loss into train_k3 and train_sft. Wiki parquet URLs honor HF_ENDPOINT for mirrored downloads.	2026-08-25 19:50:07 +08:00
dela	584f7e9e73	Initial K3 snapshot: 0.5B KDA/MLA/MoE train path ... Standalone tree split from LLMRL/projects/kda. Includes Triton dt_bias backward fix, train_k3 --preset 0.5b, SFT, Docker runtime, and tests.	2026-08-25 14:43:17 +08:00