dela 8442f92c58 Keep LatentMoE routed bmm in activation dtype under bf16 autocast ...

Python float scales and fp32 expert weights promoted SiTU outputs to
fp32, so index_add mixed BFloat16 dest with Float source and crashed
the 0.5b run. Cast packed weights and gate scalars to z.dtype.

2026-08-25 20:22:51 +08:00

..

__init__.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_attn_res.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_causal_checkpoint.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

test_causal_lm.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_ckpt_compat.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_eval_files.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_eval_mt.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_k3_arch.py

Keep LatentMoE routed bmm in activation dtype under bf16 autocast

2026-08-25 20:22:51 +08:00

test_pretrain_data.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_schedule.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

test_sft_data.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_tensorlens.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_torchlens.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

test_train_overfit.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00