## Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution

<h3 id="background">Background</h3> <p>$\textcircled{\small 1}$ Embedding $\xrightarrow{\text{实体 (User, Item) 的向量表示}}$ 现代推荐系统的基础</p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619102807.png" alt="Image" style="max-width: 65%; height: auto;margin: 0 auto;"> </div>

<h3 id="background-1">Background</h3> <p>$\textcircled{\small 2}$ 多元信息 $\xrightarrow{\text{交互信息, 类别相似性}}$ 潜在的结构性约束</p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619110409.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>❓Embedding 学习如何高效融入这些结构性先验</p>

<h3 id="background-2">Background</h3> <p>$\textcircled{\small 3}$ 图结构先验 $\underset{\text{超大规模 Embedding table}}{\xrightarrow{\text{训练随机性: 数据采样, dropout}}}$ 😞低效的信息融合</p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619112212.png" alt="Image" style="max-width: 90%; height: auto;margin: 0 auto;"> </div> <p>$\mathbf{E}$: <span style="color:gray">Embedding</span>       $\mathcal{L}$: <span style="color:gray">Loss</span>       $\nabla_{\mathbf{E}} \mathcal{L}$: <span style="color:gray">Gradient</span></p>

<h3 id="background-3">Background</h3> <p>$\textcircled{\small 4}$ 图+序列模型: 过于依赖<strong>特定场景</strong>, 高昂的<strong>训练/推理代价</strong></p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250711202256.png" alt="Image" style="max-width: 90%; height: auto;margin: 0 auto;"> </div> <p><u>LightGCN:</u> <span style="color:gray">GNN-only</span>       <u>SASRec:</u> <span style="color:gray">Transformer-only</span></p> <p><u>SR-GNN/LESSR/MAERec:</u> <span style="color:gray">GNN-based sequence models</span></p>

<h3 id="weighted-adjacency-matrix">Weighted Adjacency Matrix $\mathbf{A}$</h3> <p>🤔 如何形式化定义图结构先验?</p> $$ \mathcal{G} = (\mathcal{V}, \mathbf{A} = [w_{ij}]), \quad w_{ij} \textcolor{green}{\uparrow} \longrightarrow \text{两个节点越相似} \textcolor{green}{\uparrow}. $$<p>🤔 如何刻画 $\mathbf{X} \in \mathbb{R}^{|\mathcal{V}| \times d}$ 与邻接矩阵 $\mathbf{A} \in \mathbb{R}^{n \times n}$ 所刻画节点相似度的一致性?</p> $$ \mathcal{J}_{smoothness} (\mathbf{X}; \mathcal{G}) := \sum_{i, j \in \mathcal{V}} w_{ij} \left \| \frac{\mathbf{x}_i}{d_i} - \frac{\mathbf{x}_j}{d_j} \right \|. $$<p>💡 $\mathcal{J}_{smoothness}\downarrow$ $\longrightarrow$ 越相似的两个节点的表示越接近</p> <div class="slide-ref"> <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div> <p style="margin: 2px 0;">Zhou D., et al. Learning With Local and Global Consistency. NeurIPS, 2003.</p> <p style="margin: 2px 0;">Chen S., et al. Signal denoising on graphs via graph filtering. GlobalSIP, 2014.</p> </div>

<h3 id="structure-aware-embedding-evolution">Structure-aware Embedding Evolution</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619151431.png" alt="Image" style="max-width: 90%; height: auto;margin: 0 auto;"> </div> $$ \Delta \mathbf{E}_{t-1} \xrightarrow{\textcolor{blue}{\psi}(\cdot)} \psi(\mathbf{E}_{t-1}). $$<p>$\textcircled{\small 1}$ <span style="color: blue"><strong>Structure-aware:</strong></span> $\small \mathcal{J}_{smoothness} \left (\textcolor{blue}{\psi} (\Delta \mathbf{E}) \right) \le \mathcal{J}_{smoothness} \left (\Delta \mathbf{E} \right)$</p> <p>$\textcircled{\small 2}$ <span style="color: blue"><strong>Direction-aware:</strong></span> $\small \left\langle \textcolor{blue}{\psi} (\Delta \mathbf{E}), \Delta \mathbf{E} \right\rangle > 0, \quad \forall \Delta \mathbf{E} \not= \bm{0}$</p>

<h3 id="smoothness-vs-convergence">Smoothness vs. Convergence</h3> <ul> <li>理想的 $\psi$ 应当是平滑性和收敛性的平衡:</li> </ul> $$ \psi^* \left( \Delta \mathbf{E}; \beta \right) = \underset{\Delta}{\text{argmin}} \: (1 - \beta) \| \Delta - \Delta \mathbf{E}\| + \beta \mathcal{J}_{smoothness}(\Delta). $$<ul> <li>直觉上,</li> </ul> $$ \text{Smoothness} \xleftarrow{\beta \rightarrow 1} \psi^*(\Delta \mathbf{E}) \xrightarrow{\beta \rightarrow 0} \text{Convergence} $$<ul> <li>闭式解: $\psi^*(\Delta \mathbf{E}; \beta) = (1 - \beta) \left(\mathbf{I} - \beta \mathbf{\tilde{A}}\right)^{-1} \Delta \mathbf{E}$ <ul> <li>😞矩阵逆难以精确求解</li> </ul> </li> </ul>

<h3 id="近似解">近似解</h3> <p>$\textcircled{\small 1}$ <strong>$L$-layer iterative approximation:</strong></p> $$ \hat{\psi}_{iter} (\Delta \mathbf{E}) := \hat{\psi}_L (\Delta \mathbf{E}), \: \hat{\psi}_l (\Delta \mathbf{E}) = \beta \mathbf{\tilde{A}} \hat{\psi}_{l-1} (\Delta \mathbf{E}) + (1 - \beta) \Delta \mathbf{E}. $$<p>$\textcircled{\small 2}$ <strong>$L$-layer Neumann series approximation:</strong></p> $$ \hat{\psi}_{nsa} (\Delta \mathbf{E}) := (1 - \beta) \sum_{l=1}^L \beta^l \mathbf{\tilde{A}}^l \Delta \mathbf{E}. $$<div class="slide-ref"> <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div> <p style="margin: 2px 0;">Klicpera J., et al. Predict Then Propagate: Graph Neural Networks Meet Personalized Pagerank. ICLR, 2019.</p> <p style="margin: 2px 0;">Huang Q., et al. Combining Label Propagation and Simple Models Out-performs Graph Neural Networks. ICLR, 2021.</p> </div>

<h3 id="近似解的缺陷">近似解的缺陷</h3> <p>$\textcircled{\small 1}$ $\psi_{iter}$ 的受限平滑性:</p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619161557.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>$\textcircled{\small 2}$ $\psi_{nsa}$ 的次优收敛性:</p> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619161810.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>😄 <strong>SEvo:</strong></p> $$ \hat{\psi} (\Delta \mathbf{E}; \beta) = \frac{1 - \beta}{\textcolor{orange}{1 - \beta^{L+1}}} \sum_{l=0}^L \beta^l \mathbf{\tilde{A}}\Delta \mathbf{E}. $$

<h3 id="sgd--sevo">SGD + SEvo</h3> <ul> <li>SEvo 可以直接应用在其它优化器所衍生的更新量</li> </ul> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619163020.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div>

<h3 id="adam--sevo">Adam + SEvo</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619163150.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div>

<h3 id="adamw--sevo">AdamW + SEvo</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619163339.png" alt="Image" style="max-width: 95%; height: auto;margin: 0 auto;"> </div>

<h3 id="overall-comparisons">Overall Comparisons</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619164021.png" alt="Image" style="max-width: 95%; height: auto;margin: 0 auto;"> </div> <p>✅ <strong>Accuracy:</strong> 平均 10+% 的提升</p> <p>✅ <strong>Efficiency:</strong> 略微训练消耗 & <strong>零</strong>推理成本增加</p>

<h3 id="empirical-analysis">Empirical Analysis</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619164506.png" alt="Image" style="max-width: 95%; height: auto;margin: 0 auto;"> </div> <p>✅ <strong>收敛性:</strong> $\textcolor{orange}{1 / (1 - \beta^{L+1})}$ 加快收敛</p> <p>✅ <strong>平滑性:</strong> $\beta \rightarrow 1$ 愈加平滑</p>

<h3 id="ablation-study">Ablation Study</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619164907.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>✅ <strong>泛化性:</strong> 适用于 SGD/Adam/AdamW</p> <p>✅ <strong>Theorem 1:</strong> 简单的迭代近似仅在较小的 $\beta$ 可行</p> <p>✅ <strong>AdamW correction:</strong> 稀疏梯度矫正的必要性</p>

<h3 id="类别一致先验">类别一致先验</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619165639.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>✅ <strong>类别一致性:</strong> 类别一致的表示更加接近</p>

<h3 id="教师模型先验">教师模型先验</h3> <div class="slide-img"> <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250619165855.png" alt="Image" style="max-width: 100%; height: auto;margin: 0 auto;"> </div> <p>✅ <strong>知识迁移性:</strong> SEvo 本身就有较强的知识蒸馏能力</p> <p>✅ <strong>泛化性:</strong> SEvo 可以和其它知识蒸馏方法结合</p>

<div style=" display: flex; justify-content: center; align-items: center; height: 40%; font-size: 10rem; "> Thanks! </div>