dela 7a12f61de1 LatentMoE: K3 sigmoid routing and Switch aux/z-loss ...

Route with σ(W_r x), Top-k(s+b), then L1-normalize over the selected set.
Add Switch/GShard aux and router z-loss into train_k3 and train_sft.
Wiki parquet URLs honor HF_ENDPOINT for mirrored downloads.

2026-08-25 19:50:07 +08:00

..

sec-01.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-02.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-03.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-04.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-05.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-06.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-07.tex

LatentMoE: K3 sigmoid routing and Switch aux/z-loss

2026-08-25 19:50:07 +08:00

sec-08.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-09.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-10.tex

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

sec-11.tex

LatentMoE: K3 sigmoid routing and Switch aux/z-loss

2026-08-25 19:50:07 +08:00