Is Vector Quantization the Future of Recommendation?

## Is Vector Quantization the Future of Recommendation?

<h3 id="solutions">Solutions</h3>
<ul>
<li>
<p><strong>Undesirable Gradient Estimator:</strong></p>
<ol>
<li>Gumbel-softmax estimator${}^{\text{[1]}}$;</li>
<li>Rotation-trick estimator${}^{\text{[2]}}$</li>
</ol>
</li>
<li>
<p><strong>Codebook Collapse:</strong></p>
<ol>
<li>对于 codebook 采用 K-means ++ 初始化${}^{\text{[3]}}$;</li>
<li>Fixed Codebook${}^{\text{[4]}}$;</li>
<li>Fixed Codebook + Trainable linear transformation${}^{\text{[5]}}$</li>
</ol>
</li>
</ul>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">[1] Takida Y., et al. SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization. ICML, 2022.</p>
    <p style="margin: 2px 0;">[2] Fifty C., et al. Restructuring Vector Quantization with the Rotation Trick. ICLR, 2025.</p>
    <p style="margin: 2px 0;">[3] Lancucki A., et al. Robust Training of Vector Quantized Bottleneck Models. 2020.</p>
    <p style="margin: 2px 0;">[4] Mentzer F., et al. Finite Scalar Quantization: VQ-VAE Made Simple. 2023.</p>
    <p style="margin: 2px 0;">[5] Zhu Y., et al. Addressing Representation Collapse in Vector Quantized Models with One Linear Layer. 2024.</p>
</div>

<h3 id="residual-quantization-rq-vae">Residual Quantization (RQ-VAE)</h3>
<p>😞 $\text{Size}\textcolor{red}{\downarrow} \longrightarrow$ 表达能力$\textcolor{red}{\downarrow}$   <strong>vs</strong>   $\text{Size}\textcolor{green}{\uparrow} \longrightarrow$ Collapse$\textcolor{red}{\uparrow}$</p>
<ul>
<li>
<p><strong>RQ-VAE:</strong></p>
$$
    \bm{z} 
    \overset{\phi}{\rightarrow} \textcolor{red}{\bm{c}_{k_1}}
    \overset{\bm{z} - \bm{c}_{k_1}}{\longrightarrow} \bm{r}_1
    \overset{\phi}{\rightarrow} \textcolor{red}{\bm{c}_{k_2}}
    \overset{\bm{r}_1 - \bm{c}_{k_2}}{\longrightarrow} \bm{r}_2
    \rightarrow \cdots
    $$</li>
<li>
<p><strong>连续近似:</strong></p>
$$
    \bm{q} = \bm{z} + \text{sg}\Big(\sum_{i=1}^{N} \bm{c}_{k_i} - \bm{z} \Big)
    $$</li>
<li>
<p><strong>离散编码:</strong> $(k_1, k_2, \ldots, k_N)$</p>
</li>
</ul>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">Lee D., et al. Autoregressive Image Generation using Residual Quantization. CVPR, 2022.</p>
</div>

<h3 id="allocation-conflict">Allocation Conflict❓</h3>
<div class="slide-cols">

<div class="slide-col-6">
<div class="slide-img">
  <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250707105044.png" 
  alt="Image" 
  style="max-width: 100%; height: auto;margin: 0 auto;">
</div>
</div>

<div class="slide-col-4">
<ul>
<li>
<p>Amazon2014Beauty_1000_LOU</p>
</li>
<li>
<p>#Users: 12,595 #Items: 75,253</p>
</li>
<li>
<p><strong>Encoder:</strong> All-MiniLM-L12-V2</p>
</li>
<li>
<p><strong>Attributes:</strong> (title, categories, brand)</p>
</li>
<li>
<p>#Blocks$\textcolor{green}{\uparrow}$ $\longrightarrow$ #Invalids $\textcolor{green}{\downarrow}$</p>
</li>
</ul>
</div>
</div>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">Zheng B., et al. Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation. ICDE, 2024.</p>
</div>

<h3 id="cold-start-item-recommendation">Cold-Start Item Recommendation❓</h3>
<ul>
<li>Cold-start items 可直接编码, 但</li>
</ul>
<div class="slide-img">
  <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250327143851.png" 
  alt="Image" 
  style="max-width: 80%; height: auto;margin: 0 auto;">
</div>
<div class="slide-img">
  <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250327145319.png" 
  alt="Image" 
  style="max-width: 65%; height: auto;margin: 0 auto;">
</div>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">Yang L., et al. Unifying Generative and Dense Retrieval for Sequential Recommendation. 2024.</p>
    <p style="margin: 2px 0;">Yang Y., et al. Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations. 2025.</p>
</div>
<p>Note:
虽然应用 VQ 可以很好地支持冷启动d的商品 (可以相当方便地进行编码), 但是 LIGER 发现, 利用 VQ 训练的非常容易过拟合到出现过的组合中去, 反而冷启动的效果特别差.</p>

<h3 id="rq-vae-vs-hierarchicalresidual-kmeans">RQ-VAE vs (Hierarchical/Residual) KMeans❓</h3>
<ul>
<li>RQ-VAE 相较于 (Hierarchical/Residual) KMeans 的优势?</li>
</ul>
<table>
  <thead>
      <tr>
          <th style="text-align: center"></th>
          <th style="text-align: center">HR@1</th>
          <th style="text-align: center">HR@5</th>
          <th style="text-align: center">HR@10</th>
          <th style="text-align: center">NDCG@5</th>
          <th style="text-align: center">NDCG@10</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td style="text-align: center">Random</td>
          <td style="text-align: center">0.0025</td>
          <td style="text-align: center">0.0080</td>
          <td style="text-align: center">0.0114</td>
          <td style="text-align: center">0.0052</td>
          <td style="text-align: center">0.0063</td>
      </tr>
      <tr>
          <td style="text-align: center">KMeans</td>
          <td style="text-align: center">0.0038</td>
          <td style="text-align: center"><strong>0.0154</strong></td>
          <td style="text-align: center"><strong>0.0246</strong></td>
          <td style="text-align: center"><strong>0.0096</strong></td>
          <td style="text-align: center"><strong>0.0126</strong></td>
      </tr>
      <tr>
          <td style="text-align: center">STE</td>
          <td style="text-align: center">0.0023</td>
          <td style="text-align: center">0.0111</td>
          <td style="text-align: center">0.0188</td>
          <td style="text-align: center">0.0067</td>
          <td style="text-align: center">0.0091</td>
      </tr>
      <tr>
          <td style="text-align: center">Rotation</td>
          <td style="text-align: center"><strong>0.0041</strong></td>
          <td style="text-align: center">0.0122</td>
          <td style="text-align: center">0.0195</td>
          <td style="text-align: center">0.0083</td>
          <td style="text-align: center">0.0106</td>
      </tr>
      <tr>
          <td style="text-align: center">SimVQ</td>
          <td style="text-align: center">0.0029</td>
          <td style="text-align: center">0.0092</td>
          <td style="text-align: center">0.0164</td>
          <td style="text-align: center">0.0060</td>
          <td style="text-align: center">0.0083</td>
      </tr>
  </tbody>
</table>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">Wang Y., et al. EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration. 2024.</p>
    <p style="margin: 2px 0;">OneRec Team. OneRec Technical Report. 2025.</p>
</div>
<p>Note:
参数还没有细调.</p>

<h3 id="semantic-features--collaborative-signals">Semantic Features + Collaborative Signals❓</h3>
<ul>
<li>微调 Encoder:</li>
</ul>
<div class="slide-img">
  <img src="https://raw.githubusercontent.com/MTandHJ/blog_source/master/images/20250624153117.png" 
  alt="Image" 
  style="max-width: 100%; height: auto;margin: 0 auto;">
</div>
<div class="slide-ref">
    <div style="width: 100px; height: 1px; background: black; margin-bottom: 5px;"></div>
    <p style="margin: 2px 0;">Luo X., et al. OneRec Team. OneRec Technical Report. 2024.</p>
    <p style="margin: 2px 0;">OneRec Team. OneRec Technical Report. 2025.</p>
</div>
<p>Note:</p>
<ol>
<li>通过 miniCPM-V-8B 将多模态信息整合为 $\mathbf{M} \in \mathbb{R}^{N_M \times d_t}$ 大小的 token vectors (per item).</li>
<li>通过 QFormer 进一步融合得到 $\mathbf{\tilde{M}} \in \mathbb{R}^{N_{\tilde{M}} \times d_t}$, 通常 $N_{\tilde{M}} = 4$ (而 $N_M = 1280$).</li>
<li>通过 item-item 间的<strong>相似度</strong>构建高质量的 item-pair dataset $\mathcal{D}_{pair}$, 然后通过 item-item 间的对比学习来促使 item features 融合进这部分信息.</li>
<li>此外, 额外引入 Caption loss, 即通过 LLaMA3 来预测 Caption, 保证 features 不会丢失内容信息.</li>
</ol>

<h3 id="decoder-encoder-xxx-vector-quantization-1">Decoder-Encoder-XXX Vector Quantization</h3>
<ul>
<li>实验结果:</li>
</ul>
<table>
  <thead>
      <tr>
          <th style="text-align: center"></th>
          <th style="text-align: center">HR@1</th>
          <th style="text-align: center">HR@5</th>
          <th style="text-align: center">HR@10</th>
          <th style="text-align: center">NDCG@5</th>
          <th style="text-align: center">NDCG@10</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td style="text-align: center">Random</td>
          <td style="text-align: center">0.0025</td>
          <td style="text-align: center">0.0080</td>
          <td style="text-align: center">0.0114</td>
          <td style="text-align: center">0.0052</td>
          <td style="text-align: center">0.0063</td>
      </tr>
      <tr>
          <td style="text-align: center">KMeans</td>
          <td style="text-align: center">0.0038</td>
          <td style="text-align: center"><strong>0.0154</strong></td>
          <td style="text-align: center"><strong>0.0246</strong></td>
          <td style="text-align: center"><strong>0.0096</strong></td>
          <td style="text-align: center"><strong>0.0126</strong></td>
      </tr>
      <tr>
          <td style="text-align: center">Rotation</td>
          <td style="text-align: center"><strong>0.0041</strong></td>
          <td style="text-align: center">0.0122</td>
          <td style="text-align: center">0.0195</td>
          <td style="text-align: center">0.0083</td>
          <td style="text-align: center">0.0106</td>
      </tr>
      <tr>
          <td style="text-align: center">DEX-VQ</td>
          <td style="text-align: center">0.0033</td>
          <td style="text-align: center">0.0126</td>
          <td style="text-align: center">0.0216</td>
          <td style="text-align: center">0.0079</td>
          <td style="text-align: center">0.0107</td>
      </tr>
  </tbody>
</table>