Albert Gu
The State Space Alternative
S4 and Mamba, the credible challenge to attention
Rating
85Impact score
Domains
State space modelsLong sequencesArchitecture research
Scouting report
Developed structured state space models — S4 and its successors — showing that sequence modelling with linear-time scaling could compete with quadratic attention on long inputs. Co-created Mamba with Tri Dao and released it openly. The most serious architectural alternative to the transformer to come out of the last decade.
Curator’s note
The only person with a real answer to "what comes after attention".
Sources
- 01Efficiently Modeling Long Sequences with Structured State Spaces (2021) arxiv.org
- 02Mamba (2023) arxiv.org