feat: add superderive child and open phase-2 routing
Attach the stepwise derivation toolkit as a submodule and default SUPERPAPER_PHASE to 2 so rewrite-figure can use superderive. Notes still keep align for short expansions without cancel or subst.
This commit is contained in:
@@ -0,0 +1 @@
|
||||
$QK^\top \to QK^\top/\sqrt{d_k}$。缩放压方差,避免 softmax 饱和。
|
||||
@@ -0,0 +1,14 @@
|
||||
# Figure request F3
|
||||
toolkit: superderive
|
||||
language: cjk
|
||||
claim: 1/√d_k 是改写,不是新算子
|
||||
grammar: derivation
|
||||
work_rel_dir: figures/F3
|
||||
|
||||
## Flow
|
||||
stage: {name: D1, text: "缩放来自方差,不是装饰"}
|
||||
row: {name: R1}
|
||||
in_row:
|
||||
- {macro: sdstep, name: s1, role: rewrite,
|
||||
lhs: "QK^{\\top}", rhs: "QK^{\\top}/\\sqrt{d_k}"}
|
||||
- {macro: sdreason, name: s1, text: "§3.2.1,避免 softmax 饱和"}
|
||||
@@ -0,0 +1,20 @@
|
||||
% Golden example -- a scale rewrite that must be seen, not just aligned.
|
||||
% ../scripts/build.sh rewrite-cancel.tex
|
||||
\documentclass[border=10pt]{standalone}
|
||||
\usepackage[cjk]{superderive}
|
||||
|
||||
\begin{document}
|
||||
\begin{tikzpicture}
|
||||
\sdstage{D1}{缩放来自方差,不是装饰}
|
||||
\sdrow{R1}
|
||||
\sdstep[role=rewrite]{s1}{QK^{\top}}{QK^{\top}/\sqrt{d_k}}
|
||||
\sdreason{s1}{§3.2.1,避免 softmax 饱和}
|
||||
\sdrowend
|
||||
\sdbbox{all}
|
||||
\sdtopformula{F}{$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(QK^{\top}/\sqrt{d_k})V$}
|
||||
\sdmeaningbox{mb}{120mm}{all}
|
||||
{点积方差随 $d_k$ 线性涨}
|
||||
{缩放是改写,不是新算子}
|
||||
{没有缩放,softmax 进饱和区,梯度消失}
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 109 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 105 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 238 KiB |
@@ -0,0 +1,14 @@
|
||||
\documentclass[a4paper]{article}
|
||||
\input{notes-macros}
|
||||
\renewcommand{\notetitle}{缩放来自方差}
|
||||
\renewcommand{\notepaper}{Scaled attention}
|
||||
\begin{document}
|
||||
\tableofcontents
|
||||
\newpage
|
||||
\input{sections/sec-01.tex}
|
||||
\input{sections/sec-02.tex}
|
||||
\input{sections/sec-03.tex}
|
||||
\appendix
|
||||
\section{符号表}
|
||||
\input{sections/symbols.tex}
|
||||
\end{document}
|
||||
@@ -0,0 +1,2 @@
|
||||
\section{这篇论文在问什么}
|
||||
注意力里的 $1/\sqrt{d_k}$ 是装饰,还是一次必须看见的改写?
|
||||
@@ -0,0 +1,8 @@
|
||||
\section{主张与贡献}
|
||||
\splabel{C1}
|
||||
点积方差随 $d_k$ 涨。除掉 $\sqrt{d_k}$ 是改写,不是新算子。
|
||||
|
||||
\spfig{F3}{缩放来自方差。}{fixture;toolkit: \texttt{superderive};ledger id: F3。}
|
||||
|
||||
\subsection{本章小结}
|
||||
这一步必须被看见,所以走 superderive,而不是笔记里的 \texttt{align}。
|
||||
@@ -0,0 +1,2 @@
|
||||
\section{总结与延伸}
|
||||
四步以内、没有消项或代入要画出来时,仍然用 \texttt{align}。
|
||||
Reference in New Issue
Block a user