Base of RoPE Bounds Context Length
讨论 RoPE base 对于相似 Tokens 感知能力的影响
Blogs
Round and Round We Go! What makes Rotary Positional Encodings useful?
理解 RoPE 的高低频
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
Data Augmentation as Free Lunch: Exploring the Test-Time Augmentation for Sequential Recommendation
TTA, Test-Time Augmentation
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Adam 预训练的 1-bit SGD 优化方法
CPT: Efficient Deep Neural Network Training via Cyclic Precision
CPT, 类似 CosineAnnealingWarmRestarts 的 Precision 循环机制
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
GaLore, 低秩空间中的梯度投影以及权重更新
MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
MicroAdam, 通过梯度稀疏化以及 error compensation 实现轻量的优化器
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
Q-GaLore, 对 GaLore 进一步施加低精度量化
SWALP: Stochastic Weight Averaging in Low-Precision Training
SWALP, 通过 SWA 稳定低精度训练