dela 8442f92c58 Keep LatentMoE routed bmm in activation dtype under bf16 autocast ...

Python float scales and fp32 expert weights promoted SiTU outputs to
fp32, so index_add mixed BFloat16 dest with Float source and crashed
the 0.5b run. Cast packed weights and gate scalars to z.dtype.

2026-08-25 20:22:51 +08:00

..

__init__.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

attn_res.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

block.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

kda_attn.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

latent_moe.py

Keep LatentMoE routed bmm in activation dtype under bf16 autocast

2026-08-25 20:22:51 +08:00

mla.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

rmsnorm.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

swiglu.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00