dela 7a12f61de1 LatentMoE: K3 sigmoid routing and Switch aux/z-loss ...

Route with σ(W_r x), Top-k(s+b), then L1-normalize over the selected set.
Add Switch/GShard aux and router z-loss into train_k3 and train_sft.
Wiki parquet URLs honor HF_ENDPOINT for mirrored downloads.

2026-08-25 19:50:07 +08:00

..

__init__.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

attn_res.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

block.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

kda_attn.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

latent_moe.py

LatentMoE: K3 sigmoid routing and Switch aux/z-loss

2026-08-25 19:50:07 +08:00

mla.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

rmsnorm.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

swiglu.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00