An Image is Worth 32 Tokens for Reconstruction and Generation
非结构化的一维离散 Visual Tokens 生成策略
2024
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
自回归图像生成: Multi-scale Quantization & Next-scale Prediction
SpaceByte: Towards Deleting Tokenization from Large Language Modeling
探究非 Subword Tokenizer 的可能性
Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation
LC-Rec, LLM + RQ-VAE + 丰富的多任务训练
Addressing Representation Collapse in Vector Quantized Models with One Linear Layer
SimVQ, 坐标变换替代可学习 Codebook
Let’s Verify Step by Step
来自 OpenAI 的 process supervision
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
来自 DeepSeek 的 process supervision
Base of RoPE Bounds Context Length
讨论 RoPE base 对于相似 Tokens 感知能力的影响
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
GaLore, 低秩空间中的梯度投影以及权重更新