Addressing Representation Collapse in Vector Quantized Models with One Linear Layer
SimVQ, 坐标变换替代可学习 Codebook
2024
Let’s Verify Step by Step
来自 OpenAI 的 process supervision
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
来自 DeepSeek 的 process supervision
Base of RoPE Bounds Context Length
讨论 RoPE base 对于相似 Tokens 感知能力的影响
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
GaLore, 低秩空间中的梯度投影以及权重更新
MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
MicroAdam, 通过梯度稀疏化以及 error compensation 实现轻量的优化器
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
Q-GaLore, 对 GaLore 进一步施加低精度量化
Scaling Laws for Online Advertisement Retrieval
快手, 广告场景下的 Scaling Laws
EMR-MERGING: Tuning-Free High-Performance Model Merging
EMR-MERGING, 多任务权重合并技术