Parallax: A Parameterized Local Linear Attention That Keeps Softmax and Adds a Learned Covariance Correction Branch
The Transformer’s consideration mechanism has barely modified since 2017. Most effectivity work has tried to interchange softmax consideration outright. A new paper takes a completely different route. It retains softmax consideration and bolts on a correction department. A staff of researchers from Northwestern University, Tilde Research, and University of Washington introduce a parameterized Local Linear…
