附录 A:PyTorch 诊断野战指南
把真实 PyTorch 症状路由到能解决它的最小诊断仪器:野战表、证据包、事故工作单、AI prompt、升级规则,以及从张量跟踪到回归、编译执行的完整栈。
这个附录不引入新 PyTorch 机制。
这里的一切都是本书前面挣来的:建小东西,故意弄坏,测变了什么,定位现实与意图算子第一次分手处。
这个附录的目的不同。
真模型挂时,通常不需要另一篇 autograd、广播、注意力、CUDA 讲解。需要回答更窄的问题:
先伸手拿哪台仪器?
本书反复的动作是:
预测
↓
观察
↓
找到第一个分歧
↓
确认机制
↓
只改一处
↓
验证修复
那是默认流程。下面按常见症状路由到产出有用证据的最小仪器。
中心规则不变:
猜之前先让隐藏结构可见。
从这里开始:症状路由
| 你看到的 | 先问 | 第一台仪器 | 章 |
|---|---|---|---|
| 张量错出现在下游很远 | 张量第一次在哪错了? | 张量跟踪 / 显式形状契约 | 2 |
grad 缺失 |
损失到参数的通路第一次消失在哪? | autograd 可达 / 图检查 | 3 |
| 梯度在,raw 网还挂 | 哪个断言真没支撑? | 六断言审计 | 4 |
层在 forward() 里,检查点优化里消失 |
哪套结构看得见这个对象? | 四结构审计 | 5 |
| GPU 利用率低或 batch 到得不规律 | 消费者真在等输入吗? | 生产者-消费者拆分 | 6 |
| 训成了,模型看到的表示错了 | 哪个变换边界含义变了? | 一样本阶段跟踪 + 语义探针 | 7 |
| CNN 维度对不上了 | 这层该产出什么形状? | 推导再跟踪形状台账 | 8 |
| 线性层分数太多,或特征空间数被误读 | 一向量表示什么,解释的是哪个量? | 特征空间检查器 | 9 |
| 注意力形状对行为错 | 哪个正确性层先挂? | 注意力台账 + 干预 | 10 |
前向反向 optimizer.step() 全跑,模型不学 |
学习哪个期待后果先缺席? | 学习台账 | 11 |
| 程序对但慢或内存多 | 哪个资源 limiting workload? | 测量契约 + 性能台账 | 12 |
torch.compile 对但重编、停顿、加速没了 |
编译器哪个假设不再成立? | 图断裂 / 守卫 / 重编译证据 | 13 |
| 新跑看着比旧跑差 | 有值得解释的差吗? | 基线 variation + 配对比较 + run 记录 | 14 |
| 完整 Transformer 行为怪 | 症状归哪个子系统? | 路由到上面恰当仪器 | 15 |
别把这张表当每 bug 必走的严格序列。
它是路由表。
一失败能跨几系统。变换 bug 先表现为训练失败。优化器归属 bug 先表现为平损失。编译器问题先表现为性能回归。
工作是从可见症状走到最早无支撑的断言。
最小证据包
改代码前,捕获足够 state,让失败在第一次试图修复后活下来。
训练问题,最小有用包常常是:
one known batch
input shape / dtype / device / range
target shape / dtype / meaning
model mode
loss value
loss.requires_grad
loss.grad_fn
expected trainable parameter names
grad state for those parameters
optimizer parameter membership
parameter deltas after one controlled step
current learning rate for every parameter group
性能问题:
workload
input shapes
dtype
device
train/eval/inference mode
eager/compiled
metric
warmup
measurement window
repetitions
synchronization boundary
peak allocated memory
peak reserved memory
回归:
code version
configuration
data version
seed
environment
evaluation contract
baseline variation
paired deltas if pairing is valid
exact 包随问题变。
原则不变:
干预毁掉失败 state 前保留观测。
1. 训练循环:分开前向、反向、更新
系统还小,先用第 1 章的分解:
forward compute prediction and loss
backward compute derivatives and accumulate them into .grad
update change parameter values
三独立机制。
loss.backward() 不更新参数。
更新不清梯度。
下降的损失不证明意图任务在学。
问
What value produced this loss?
What gradient was produced by this loss?
What parameter value existed before the update?
What parameter value exists after the update?
Was the old gradient cleared before the next backward pass?
Is this still the same parameter object?
高价值测量
before = p.detach().clone()
loss.backward()
grad = None if p.grad is None else p.grad.detach().clone()
optimizer.step()
delta = p.detach() - before
受控朴素 SGD 步(无动量、无 weight decay)下:
delta ≈ -learning_rate * gradient
deliberate 受限实验下等式不成立,更新机制不是你以为的那个。
成立,别盲目 generalize 到 Adam、动量、weight decay 等有 state 更新规则。
停于
分别证明:
the forward calculation is the intended one
backward produced the expected derivative
the intended parameter object moved
old gradients are not contaminating the next step
再往系统的下一层。
2. 张量:找第一个错的张量,不是第一个不合法的
张量不只有值。
有用的描述是:
shape
dtype
device
stride
contiguity
axis meaning
value meaning
前五张量上可查。
后两属于程序的契约。
第一步
算子跑前写下期待的张量。
不是:
I expect something like [32, ?]
而是:
[B, T, D]
B = examples
T = positions
D = feature coordinates
再每边界期待对观测比。
布局问题用确定性值
随机数藏重排。
reshape、转置、头拆分、置换 bug,用:
x = torch.arange(...)
小到能打印。
两算子产出同样形状,看值去了哪。
常见 false 修复
reshape until the exception disappears
change in_features to match the tensor that arrived
change in_channels to match an NHWC batch
squeeze dimensions without naming what they represent
这些能把上游错误变成合法的下游结构。
停于
找到观测张量事实或轴语义第一次不再对上期待契约的第一边界。
在那里修。
3. Autograd:找依赖通路停止存在处
参数没收到期待梯度,别从终 .grad 开始。
从问题开始:
跑出的图里这个损失真依赖这个参数吗?
autograd 记录这次前向跑过的算子。它不记录你想跑的模型。
证据梯子
parameter.requires_grad
parameter.is_leaf
loss.requires_grad
loss.grad_fn
parameter.grad after a controlled backward pass
reachability from loss back to expected leaves
健康的 loss.grad_fn 证明损失有可微历史。
它不证明每意图参数都在那历史上。
区分
grad is None
与:
grad is a tensor of zeros
.grad = None 开始的受控 pass 下:
None意思是该参数没累积梯度;- 零张量意思是 autograd 物化了累积导数为零的梯度。
不同机制。
通路缺席的常见病因
detach()
torch.no_grad() around part of the forward computation
a branch that did not execute
a parameter that was not used
rebinding to a new non-tracked tensor
停于
指出意图依赖不再存在的第一算子或分支。
4. Raw 神经网络:六断言里哪个挂了
第 4 章确立“梯度看起来 fine”只回答一个问题。
机械跑着的网,六断言分类失败:
| Claim | Question | Useful evidence |
|---|---|---|
| Representation | 结构表达得了要的函数吗? | 代数、证伪探针、更简单基线 |
| Computation | 前向算的是意图结构吗? | 中间张量、形状/值对比、独立实现 |
| Objective | 损失编码了意图问题吗? | 手工损失检查、可信参照、logits/目标契约 |
| Differentiation | 每意图参数到损失有通路吗? | autograd 可达、.grad |
| Update | 意图参数真变了吗? | 参数快照与 delta |
| Evaluation | 度量的是真在乎的行为吗? | 留出数据、有意义基线 |
为什么重要
这些失败看起来像:
missing nonlinearity
symmetric initialization
zero initialization
detached parameter
healthy gradient on an un-updated parameter
wrong objective
单梯度统计分不开所有。
停于
说得出证据支撑哪个断言、哪个断言仍无支撑。
别拿一断言的证据当另一断言的证明。
5. nn.Module:比四套结构
东西好似消失,停止把“模型”当一东西想。
有四重叠结构:
1. Python object graph
2. registered module tree
3. autograd graph
4. optimizer parameter groups
回答不同问题。
| Operation | Structure it consults |
|---|---|
forward() through normal Python attribute access |
Python objects |
named_modules() |
registered module tree |
named_parameters() |
registered parameters |
named_buffers() |
registered buffers |
state_dict() |
registered parameters + persistent buffers |
.to(device) |
registered parameters + buffers |
train() / eval() |
registered modules |
loss.backward() |
autograd graph created by execution |
optimizer.step() |
that optimizer’s parameter groups |
诊断问题
出故障的操作遍历哪套结构,我的对象真在里面吗?
张量能:
require gradients
receive gradients
participate in forward
还不:
registered
saved
moved by model.to(...)
owned by the optimizer
对象同一性重要
优化器构造后 head 被换,优化器可能仍拥有旧参数对象。
名字形状看着对。
同一性错了。
停于
reconcile:
what Python can reach
what PyTorch registered
what autograd reached
what the optimizer owns
失败相关的每对象。
6. DataLoader 与输入供给:生产者消费者拆开
“GPU 利用率低”不是 DataLoader 诊断。
先问训练循环是不是在等 batch。
最小有用测量是:
t0 = perf_counter()
batch = next(iterator)
t1 = perf_counter()
train_step(batch)
t2 = perf_counter()
现在有两不同观测:
wait = t1 - t0
work = t2 - t1
建模流水线
producers
↓
queue / buffering
↓
consumer
再分开:
producer throughput
consumer throughput
buffering
end-to-end throughput
更多 worker 重叠等待。
变不出已饱和的 CPU 资源。
尾也要记,不只平均
batch 等待,捕获:
median
p95
maximum
mean 低尾大,消费者照样反复饿。
一次 sweep 一 control
比如:
num_workers = 0, 1, 2, 4, 8
记:
examples/s
batch-wait distribution
post-batch work time
等待降而消费者 work 涨、端到端吞吐不变,worker 在和消费者争同样资源。
停于
说得出实际 limiting 的资源或阶段:
read
decode
transform
collate
process startup
CPU contention
host-to-device transfer
consumer compute
再修那个机制。
7. 变换与预处理:跟踪一样本
预处理流水线能产出合法而含义错的张量。
第 7 章把输入契约分成五类:
1. STRUCTURE
2. NUMERIC INTERPRETATION
3. SEMANTICS
4. RELATIONSHIPS
5. PROVENANCE
要不同仪器。
| Category | Typical evidence |
|---|---|
| Structure | type, rank, shape |
| Numeric interpretation | dtype, range, scale, normalization state |
| Semantics | known-sample probe, direct inspection |
| Relationships | image-mask/box alignment, input-target assertion |
| Provenance | recorded split, fitted statistics, checkpoint/preprocessing version |
方法
找到样本第一次不再满足意图输入契约的变换边界。
从:
one sample
one process
no batching
controlled randomness
开始逐阶段走。
重要局限
全绿 metadata 报告不是健康证明。
张量说不出:
its RGB channels were interpreted as BGR
the augmentation invalidated the label
the mask no longer aligns with the image
normalization statistics were fitted on validation data
the serving tokenizer differs from training
那些要显式探针或记录。
停于
认出五契约类里最早失败的阶段。
到那时再引入 batch、worker、性能复杂度。
8. CNN 几何:跑前推导
卷积网两问题分开:
CHANNEL QUESTION
C_in → C_out
SPATIAL QUESTION
H_in,W_in → H_out,W_out
C_out 声明的。
空间尺寸推导的。
方法
层跑之前推导期待形状。与观测形状比。调查第一次不一致。
用台账:
stage
operation
observed input
expected output
observed output
first divergence
终 Linear 永远别先修
对:
mat1 and mat2 shapes cannot be multiplied
倒推:
Where did the flattened feature count come from?
Which earlier layer changed C, H or W?
Was that change intended?
上游几何变化是故意的,改 in_features 才对。
记住语义极限
形状台账证明组合。
它不证明高宽没互换、预处理保住含义、结构适合任务。
停于
每改形状的算子处推导与观测几何一致,直到失败边界。
9. 特征空间:解释数前先说出被表示对象
张量形状:
[B, T, D]
问:
What does one D-vector represent?
How many such vectors are present?
Which axis contains its coordinates?
nn.Linear(D_in, D_out) 变换末轴并保留前导轴。
所以 [B,T,D] → [B,T,1] 意思是每(样本,位置)一分,不是一样本一分。
解释标量前先命名
dot product
cosine similarity
Euclidean distance
classifier score
signed distance to a hyperplane
不可互换。
检查不变式
问什么在下面不变:
one vector is rescaled
w and b are scaled together
one feature coordinate changes units
不该影响解释的东西动了量,多半量的名字错了。
检查器
表示的有用行包括:
vector norms
per-coordinate scales
non-finite values
pairwise cosine
collapse toward zero
collapse toward one direction
dominant coordinates
停于
说得出:
what one vector represents
which axis holds its coordinates
what operation acts on those coordinates
what numerical quantity came out
which invariances that quantity has
10. 注意力:四层正确性
注意力要的不只形状检查。
用四层:
LEVEL 1 SHAPE
Does the tensor have the dimensions I derived?
LEVEL 2 AXIS SEMANTICS
Do those dimensions contain the objects I intended?
LEVEL 3 NUMERICAL INVARIANTS
Do valid query rows normalize over keys?
Are forbidden weights zero?
Are values finite?
LEVEL 4 BEHAVIOR
Does a future-token intervention leave the prefix unchanged?
Does the implementation agree with a trusted reference?
方法
derive the axes
↓
execute one stage
↓
verify the invariant introduced by that stage
↓
stop at the first failure
高价值检查
merge(split(x)) == x
scores shape == [B, Nh, Tq, Tk]
softmax normalizes over Tk
forbidden weights == 0
no fully blocked query unless explicitly supported
merged layout restores [B, Tq, E]
future-token intervention preserves the causal prefix within tolerance
Level 4 为什么存在
缺 1/sqrt(Dh) 缩放能保留:
shape
axis semantics
finite values
row sums
还与意图注意力算子实质不一致。
停于
知道挂的最早正确性层与阶段。
11. 跑而不学的训练:走学习链
学习链是:
TASK
↓
OBJECTIVE
↓
DEPENDENCY
↓
GRADIENT
↓
OPTIMIZER OWNERSHIP
↓
PARAMETER UPDATE
↓
CAPABILITY
这个工具适合:
forward runs
loss is finite
backward runs
gradients exist
optimizer.step() runs
and the model is still useless
每 link 问什么
TASK
Does the target still describe the input?
OBJECTIVE
Are the outputs in the representation the loss expects?
Can one example reproduce the loss manually?
DEPENDENCY
Does the loss depend on the intended parameters?
GRADIENT
Are expected gradients present?
None, zero, finite, non-finite?
OPTIMIZER OWNERSHIP
Does the optimizer own those exact parameter objects?
Does it own stale objects that are no longer in the model?
PARAMETER UPDATE
Did the intended objects actually move?
CAPABILITY
Can the controlled system overfit a tiny fixed batch?
重要警告
调 backward() 与 optimizer.step() 的学习台账是改 state 的探针。
拷贝/检查点上跑,或 deliberate 接受牺牲步。
停于
找到第一个没出现的期待后果。
在那里调。
早 link 无支撑时别调学习率、结构、精度、正则。
12. 性能:优化前先定义比较
性能不是一行代码的属性。
它是契约下的结果:
workload
hardware
software/runtime
execution mode
metric
measurement boundary
先选问题
latency
throughput
peak memory
cold-start cost
steady-state cost
这些能反向走。
方法
Measure → localize → intervene → remeasure.
性能台账
捕获:
WORKLOAD
model / operation
input shape
dtype
device
mode
eager / compiled
MEASUREMENT
warmup
repeats
synchronization
metric
TIME
input wait
transfer
forward
loss
backward
optimizer
logging / synchronization
MEMORY
allocated
reserved
peak allocated
peak reserved
CUDA 计时
设备异步时,无完成边界的主机 wall-clock 计时测的可能是提交而不是完成的设备 work。
知道在测:
host-visible latency
device elapsed time
end-to-end workload time
内存
分开:
allocated live PyTorch-managed allocations
reserved allocator-managed pool
peak transient high-water mark
memory_allocated() 在相同迭代 climb,活 PyTorch 分配集在长大。跟踪归属。
单凭保留内存高别叫活张量泄漏。
Profiler 规则
profiler 管定位。
无 profile benchmark 管最终性能比较。
停于
说得出:
the metric that matters
the phase/resource that dominates it
the mechanism hypothesis
the one intervention that tests that hypothesis
the correctness property that must remain unchanged
the uninstrumented before/after result
13. torch.compile:检查编译器的 hidden state
返回正确数的编译调用证明了正确。
它没证明:
the hot region was captured
the graph was reused
guards remained valid
recompilation stopped
steady-state performance improved
cold cost amortized
有用的 progression 是:
RUNS
↓
CORRECT
↓
CAPTURED
↓
STABLE
↓
WORTH IT
从 eager 开始
eager 已错,编译不是第一问题。
再比正确性
用 workload 恰当的陈述 tolerance。
再看捕获
有用证据包括:
graph-break logs
fullgraph=True as a capture test for the exact call
structured compiler summaries
版本里没有一仪器自动 ground truth。工具分歧时三角验证。
再看稳定
捕获重编译证据。
问:
Which guard failed?
shape?
dtype?
device?
Python value?
Did the workload legitimately vary on that dimension?
Did a cached specialization get reused?
Did a new specialization get created?
Did the cache limit force an unmatched call to eager?
再问值不值
分开:
cold first call
warm-cache first call
steady-state call
recurring recompilation
启动代价重要时算摊销。
停于
说得出不再成立的编译器假设,或确立编译稳定、剩下是普通 workload 经济学。
14. 回归:解释前先确立差
新指标差了。
那是观测。
还不是回归机制。
前三可能
NOTHING MEANINGFUL CHANGED
ordinary run-to-run variation
MEASUREMENT CHANGED
different evaluation path / split / preprocessing / mode
SYSTEM CHANGED
code, configuration, data, environment, compiler state
解释第三前排除前两。
基线 variation
没改配置多种子跑。
报告分布,不只一数。
观测 min/max spread 是 context。
它不是通用统计检测阈值。
配对 valid 时配对
共享种子在两跑保留足够共享随机让种子 effect 抵消时,subtle 比较更敏感。
看:
delta(seed_i) = B(seed_i) - A(seed_i)
不只比两独立范围。
验证评估契约
记到回答:
same validation examples?
same preprocessing?
same aggregation?
same model.eval() state?
same checkpoint-selection rule?
fingerprint 只证明 hash 进去的字段。
记录 run
至少:
code version
config
data identity
seed
environment
evaluation contract
compiler state if relevant
result
小心二分
二分 assumed 好/坏 predicate 行为合适。
变化能交互。
某变化被边界 implicates 后,候选单独施加确认 effect。
停于
重复证据确立值得解释的差,测量契约已知可比。
到那时再调查机制。
15. 完整模型:路由,不发明新仪器
capstone 的课是单诊断 certify 不了大模型。
完整系统同时含前面每系统:
text
↓
tokenization
↓
batch/target alignment
↓
embeddings
↓
attention
↓
residual blocks
↓
logits
↓
objective
↓
autograd
↓
optimizer
↓
performance / compilation
↓
generation
↓
evaluation and comparison
全模型挂,路由症状。
例子:
Generated text is nonsense but training loss is excellent
→ TASK / target alignment first
A future token changes an earlier logit
→ attention Level 4 intervention
Initial loss is wildly inconsistent with the declared initialization
→ OBJECTIVE / initialization contract
Gradients are finite but the head never moves
→ optimizer ownership + parameter delta
Training is correct but throughput is low
→ performance ledger
Compiled model pauses on new sequence lengths
→ guards / recompilation
New configuration improves one run by 0.001
→ baseline variation + paired comparison
大模型不让前面仪器 obsolete。
它让选对仪器更重要。
症状到证据野战表
这张表故意与逐章指南冗余。反方向用:从观测出发。
| Symptom | Tempting explanation | Evidence to collect first |
|---|---|---|
mat1 and mat2 shapes cannot be multiplied |
“Linear 层的 in_features 错了。” |
推导每上游形状,找到第一次几何分歧。 |
expected ... channels, got ... |
“Conv2d.in_channels 错了。” |
说出进轴含义。改层前检查 NHWC vs NCHW。 |
.grad is None |
“忘了 requires_grad。” |
grad=None 开始,跑一次反向,再看图可达。 |
.grad 有限但参数不动 |
“学习率太小。” | 查优化器对象同一性与测得的参数 delta。 |
| 损失降但任务行为错 | “模型容量不够。” | 已知样本验证输入目标对齐与目标。 |
| 小 batch 过拟合过了 | “整个训练系统对了。” | 记住它没证明的:任务语义、验证契约、泛化。 |
| GPU 利用率低 | “加 num_workers。” |
输入等待、传输、前向、反向、优化器时间分开测。 |
nvidia-smi 内存高 |
“内存泄漏了。” | 固定形状重复迭代比已分配、保留、峰值。 |
empty_cache() 让可见内存降了 |
“泄漏修好了。” | 活已分配内存变没变。 |
| 注意力行形状对 | “注意力对了。” | 查头往返、key 轴归一化、forbidden 权重、干预行为。 |
torch.compile 返回正确输出 |
“编译 work 了。” | 捕获、守卫、复用/重编译、workload 经济学分开测。 |
| 编译代码更慢 | “torch.compile 对这个模型不好。” |
冷代价、热代价、recurring 重编译、稳态分开。 |
| commit 间验证变了 | “代码改动导致的。” | 先确立基线 variation,验证评估契约。 |
| 同种子跑出不同 | “PyTorch 无视种子。” | 比线程/设备/版本/RNG 来源与确定性配置。 |
| 两跑终指标相同 | “等价。” | 比轨迹与预先声明的检查点/评估政策。 |
一 bug 规则
几失败同时,手修一边界重跑。
别一次改:
input layout
layer dimensions
learning rate
optimizer
normalization
在一次 edit。
多改修复毁掉因果信息。
本书反复用的序列:
find first divergence
fix one thing
rerun
allow the next divergence to reveal itself
每 edit 慢,诊断快。
观测与干预是不同算子
仪器能改变系统。
例子:
hooks can alter lifetime and add overhead
profiler instrumentation changes timing
.item() can synchronize CUDA
printing values inside compiled code can create graph breaks
a diagnostic optimizer step mutates optimizer state
switching train/eval changes module behavior
seeding changes stochastic execution
gradient clipping changes the gradient you were trying to inspect
叫观测前问:
这个仪器仅 present 就改了什么?
答案重要,终验证去掉仪器再跑。
已知好参照 disproportionately 值钱
难推理的算子,把参数 reduce。
用:
manual arithmetic versus PyTorch
manual loss versus F.cross_entropy
manual attention versus scaled_dot_product_attention
eager versus compiled
healthy preprocessing versus candidate preprocessing
healthy run versus candidate run at shared seeds
参照对比在每没控变量 hold 固定时最强。
输出不一致,按顺序比中间阶段。
第一次不一致通常比终误差大小有用。
区分两解释的最小实验
好的调试实验不只让一假设看起来 plausible。
它让两解释预测不同结局。
例子:
缺失梯度
假设:
A. graph path is broken
B. path exists but local derivative is zero
实验:
clear grad to None
run one backward pass
inspect whether grad remains None or becomes a zero tensor
GPU 利用率低
假设:
A. input pipeline is starving the device
B. model/host execution is the bottleneck
实验:
measure batch wait separately from post-batch work
编译慢
假设:
A. steady-state kernel is slower
B. recurring recompilation dominates
实验:
measure a repeated fixed-shape steady state
capture recompile logs on the real varying workload
Apparent 回归
假设:
A. model behavior changed
B. ordinary seed variation explains the observation
实验:
run both versions on shared seeds and inspect paired deltas
最好的实验通常不是仪器最多的。
是结局迫使假设分开的那个。
可复用事故工作单
抄进 issue、notebook、调试 session。
PYTORCH INCIDENT
SYMPTOM
What exactly was observed?
EXPECTED
What specific value, shape, behavior or metric was expected?
REPRODUCTION
What is the smallest input / batch / configuration that reproduces it?
FIRST CONTRACT
Which contract should be checked first?
tensor
graph
ownership
input representation
geometry
attention
learning chain
performance
compiler
run comparison
OBSERVATIONS
What did the instrument actually report?
FIRST DIVERGENCE
Where does observed behavior first disagree with the expectation?
HYPOTHESES
What are the smallest two or three mechanisms consistent with that evidence?
DISTINGUISHING EXPERIMENT
What one experiment makes those hypotheses predict different outcomes?
INTERVENTION
Change exactly one mechanism.
MUST CHANGE
Which measurement should change if the diagnosis is correct?
MUST NOT CHANGE
Which correctness property must remain unchanged?
VERIFICATION
Did the original failure disappear for the predicted reason?
REGRESSION TEST
What small test should remain so this exact failure cannot silently return?
MUST CHANGE 与 MUST NOT CHANGE 填不出,proposed 修复大概还太 vague。
AI 当诊断伙伴
本书最有用的通用 prompt 不是“修 PyTorch 模型”。
用接近这个的:
Here is a failing PyTorch system.
Do not rewrite it and do not propose fixes yet.
1. State the expected contract at each relevant boundary.
2. Tell me which facts I can inspect directly and which require a semantic,
behavioral or provenance probe.
3. Identify the smallest instrument from this list that can localize the failure:
tensor trace
autograd reachability
six-claim audit
four-structure audit
producer-consumer timing
preprocessing stage trace
CNN shape ledger
feature-space inspector
attention four-level check
learning ledger
performance ledger
compiler guard/recompile evidence
paired run comparison
4. Ask for the missing evidence.
5. Identify the FIRST boundary whose evidence contradicts the intended system.
6. Give at most three mechanisms consistent with that boundary.
7. Propose one experiment that distinguishes those mechanisms.
8. State what should change if the diagnosis is correct.
9. State what must remain unchanged.
Do not propose a repair until the first divergence is supported by evidence.
prompt 的目的不是让助手谨慎。
是让助手可证伪。
何时升级
从便宜开始。
便宜仪器过了再升级。
print / assert tensor facts
↓
known-sample contract
↓
graph / ownership inspection
↓
controlled one-step probe
↓
tiny-batch capability test
↓
hooks / detailed activation inspection
↓
profiler / allocator snapshot / compiler logs
↓
multi-seed comparison
这不是通用代价排序。小 workload 种子 sweep 能便宜,一步 profiler 能便宜。
原则是:
用区分当前假设的 least invasive 仪器。
何时停调
四句话全 true 时停:
1. You can name the mechanism that produced the symptom.
2. The repair changed the measurement the mechanism predicted.
3. An independent correctness property remained unchanged.
4. The failure can now be expressed as a test, assertion, contract or recorded
comparison that would catch it if it returned.
“好似 work 了”的模型没达标。
诊断达标了。
完整诊断栈
本书从一标量参数开始,到装不进脑子的模型结束。仪器按同样顺序累积。
TRAINING LOOP
forward / backward / update
TENSOR
first wrong tensor
AUTOGRAD
first broken dependency edge
RAW NETWORK
representation / computation / objective /
differentiation / update / evaluation
MODULE STATE
Python graph / registered tree /
autograd graph / optimizer groups
INPUT DELIVERY
producer / queue / consumer
PREPROCESSING
structure / numeric interpretation /
semantics / relationships / provenance
CNN GEOMETRY
derive / trace / first shape divergence
FEATURE SPACE
represented object / feature axis /
quantity / invariance
ATTENTION
shape / axis semantics /
numerical invariants / behavior
LEARNING
TASK → OBJECTIVE → DEPENDENCY → GRADIENT →
OPTIMIZER OWNERSHIP → PARAMETER UPDATE → CAPABILITY
PERFORMANCE
measurement contract → phase/resource →
intervention → uninstrumented remeasurement
COMPILATION
RUNS → CORRECT → CAPTURED → STABLE → WORTH IT
REGRESSION
establish the difference →
verify the measurement →
localize what changed
CAPSTONE
choose the instrument that owns the symptom
底没有终极通用检查器。
这就是要点。
不同失败违反不同契约,危险的常常保住下面每更简单的契约。
张量能形状对含义错。
注意力块能外部维度全保留而比错对象。
参数能梯度健康从不动。
模型能损失最小化学错任务。
benchmark 能为错测量返回精确数。
编译模型能数值正确、灾难性慢。
新跑能变差而有意义的东西什么都没回归。
common 技能不是背修复。
是知道当前症状还没给出什么证据。
让隐藏结构可见。找到第一个分歧。修复造成它的机制。验证系统真变好了。