The Evolution of Position Encoding in the Transformer

In this blog, I will go through different position encoding mechanisms used in Transformer-based neural network architectures, from sinusoidal to RoPE, and extend to the visual domain.