dela 49aede9cb2 Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE ...

Whole-mixer checkpoint plus T×T MLA scores OOM'd a 31GB GPU on backward.
Checkpoint each AttnRes block, run absorbed MLA through SDPA, and compute
CE in vocab chunks so [B,T,V] logits are never materialized.

--max-tokens is now the training budget; default --steps 2000 no longer
caps a 1B-token run at 250 optimizer steps.

2026-08-25 20:09:27 +08:00

..

__init__.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

causal_lm.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

config.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

k3_config.py

LatentMoE: K3 sigmoid routing and Switch aux/z-loss

2026-08-25 19:50:07 +08:00