Base of RoPE Bounds Context Length
讨论 RoPE base 对于相似 Tokens 感知能力的影响
LLM
Round and Round We Go! What makes Rotary Positional Encodings useful?
理解 RoPE 的高低频
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
Language Representations Can be What Recommenders Need: Findings and Potentials
Next-token embedding 之于协同过滤
Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
探究 LLM 如何记忆和提取知识的实验性文章