Dropout
Dropout 是一种在神经网络训练过程中常用的正则化技术，其核心思想是随机“丢弃”部分神经元（即将其输出置为零），从而防止模型过度依赖某些特定特征，增强泛化能力，有效抑制过拟合。
传入Dropout的数据：
词嵌入+位置嵌入后:tensor([[[-0.0364,0.0127,0.0204,-0.0125,-0.0314,-0.0665],[-0.0411,-0.0429,-0.0012,-0.0266,-0.0179,0.0581],[-0.0073,0.0003,0.0268,-0.0152,-0.0243,-0.0015],[-0.0009,-0.0107,-0.0140,-0.0276,-0.0088,-0.0626]],[[-0.0374,0.0206,-0.0100,0.0124,-0.0243,-0.0383],[0.0432,0.0088,0.0093,-0.0341,-0.0013,0.1053],[0.0359,0.0300,0.0243,0.0110,-0.0061,0.0303],[-0.0443,0.0572,-0.0235,0.0327,-0.0307,-0.0337]]],grad_fn=<AddBackward0>)词嵌入+位置嵌入后形状:torch.Size([2,4,6])随机失活后：
Dropout后:tensor([[[-0.0404,0.0141,0.0226,-0.0139,-0.0348,-0.0739],[-0.0457,-0.0476,-0.0013,-0.0296,-0.0199,0.0646],[-0.0000,0.0003,0.0298,-0.0169,-0.0270,-0.0000],[-0.0010,-0.0119,-0.0000,-0.0306,-0.0000,-0.0695]],[[-0.0416,0.0229,-0.0111,0.0137,-0.0270,-0.0426],[0.0480,0.0098,0.0104,-0.0379,-0.0000,0.0000],[0.0000,0.0334,0.0270,0.0123,-0.0068,0.0336],[-0.0492,0.0635,-0.0261,0.0364,-0.0000,-0.0374]]],grad_fn=<MulBackward0>)Dropout后形状:torch.Size([2,4,6]) 第1句第1字的意思为：[ -0.0364, 0.0127, 0.0204, -0.0125, -0.0314, -0.0665 ],随机失活后得到[-0.0404, 0.0141, 0.0226, -0.0139, -0.0348, -0.0739]。
可以看到第1句第1个字并没有被失活，但是每个数值依然都乘了1/0.9,比如-0.0364 × 1.1111 ≈ -0.0404。
再观察第1条句子第3个字的失活后结果：[-0.0000, 0.0003, 0.0298, -0.0169, -0.0270, -0.0000], 其原始向量为：[-0.0073, 0.0003, 0.0268, -0.0152, -0.0243, -0.0015]。
这里第1维和第6维被丢弃（输出为 0），而其他维度则同样乘以 1/0.91/0.9，即：
0.0003×1.1111≈0.0003,0.0268×1.1111≈0.0298,−0.0152×1.1111≈−0.0169,−0.0243×1.1111≈−0.0270 Dropout的作用是抑制训练过程中某些神经元或注意力头过度激活，从而提升模型在新数据上的表现。
——谙弆悕
