transformers

Rotary position embeddings add order information to attention

RoPE introduces position through rotations in attention vectors. Understand the relative-position idea without mistaking it for unlimited context.