carrydela/K3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17 Commits 1 Branch 0 Tags
a2c4217dae5541e0280de6471363a9324651a4f7
Commit Graph
2 Commits
Author SHA1 Message Date
dela 49aede9cb2 Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE 
Whole-mixer checkpoint plus T×T MLA scores OOM'd a 31GB GPU on backward.
Checkpoint each AttnRes block, run absorbed MLA through SDPA, and compute
CE in vocab chunks so [B,T,V] logits are never materialized.

--max-tokens is now the training budget; default --steps 2000 no longer
caps a 1B-token run at 250 optimizer steps.
 2026-08-25 20:09:27 +08:00
dela 584f7e9e73 Initial K3 snapshot: 0.5B KDA/MLA/MoE train path 
Standalone tree split from LLMRL/projects/kda. Includes Triton dt_bias
backward fix, train_k3 --preset 0.5b, SFT, Docker runtime, and tests.
 2026-08-25 14:43:17 +08:00