Round and Round We Go! What makes Rotary Positional Encodings useful?
理解 RoPE 的高低频
Blogs
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
Data Augmentation as Free Lunch: Exploring the Test-Time Augmentation for Sequential Recommendation
TTA, Test-Time Augmentation
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Adam 预训练的 1-bit SGD 优化方法
CPT: Efficient Deep Neural Network Training via Cyclic Precision
CPT, 类似 CosineAnnealingWarmRestarts 的 Precision 循环机制
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
GaLore, 低秩空间中的梯度投影以及权重更新
MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
MicroAdam, 通过梯度稀疏化以及 error compensation 实现轻量的优化器
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
Q-GaLore, 对 GaLore 进一步施加低精度量化
SWALP: Stochastic Weight Averaging in Low-Precision Training
SWALP, 通过 SWA 稳定低精度训练
Scaling Laws for Online Advertisement Retrieval
快手, 广告场景下的 Scaling Laws