Positional Encoding

Connection Bottleneck in Attention

Base of RoPE Bounds Context Length

讨论 RoPE base 对于相似 Tokens 感知能力的影响

Round and Round We Go! What makes Rotary Positional Encodings useful?

理解 RoPE 的高低频