dela 94d0f2ff6a Do not let HF tokenizers truncate wiki articles at 4096 ...

Yi-6B sets model_max_length=4096. encode() would clip long Wikipedia
pages before we pack seq_len chunks. Raise the cap so only our
chunker limits context.

2026-08-26 14:31:07 +08:00

..

__init__.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

data.py

Do not let HF tokenizers truncate wiki articles at 4096

2026-08-26 14:31:07 +08:00

eval_mt.py

Fix chrF fallback so English wiki garbage fails translation_success

2026-08-26 14:23:22 +08:00

prompts.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

schedule.py

Fit 0.5b training on 32GB: SDPA MLA, block checkpoint, chunked CE

2026-08-25 20:09:27 +08:00

success.py

Fix chrF fallback so English wiki garbage fails translation_success

2026-08-26 14:23:22 +08:00

swanlab_env.py

Sanitize SwanLab env before login so 0.9 nested project does not crash

2026-08-26 10:00:06 +08:00

toy.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00

train_tokenizer.py

Initial K3 snapshot: 0.5B KDA/MLA/MoE train path

2026-08-25 14:43:17 +08:00