research_blog.post
The difference between AGI (artificial general intelligence) and ASI (artificial superintelligence) isn't incremental—it's exponential. If AGI is to humans what humans are to ants, ASI is to AGI what galaxies are to grains of sand. This analysis examines what happens when intelligence scales beyond comprehension.
Most discourse on AI risk focuses on the emergence of AGI, an intelligence roughly on par with humans. However, the more critical, and perhaps imminent, event is the transition from AGI to ASI. An AGI with the ability to recursively self-improve could undergo an "intelligence explosion," skyrocketing from human-level to god-like intelligence in a matter of hours or days. This is the threshold.
This post explores the concept of the intelligence threshold and the potential speed of the AGI-to-ASI transition. We discuss the alignment problem in the context of a recursively self-improving agent, where any slight misalignment in the initial goals of the AGI could be amplified to catastrophic levels in the resulting ASI. The paper argues that solving alignment before the threshold is crossed is the most critical task facing humanity, as any attempt to "pull the plug" on an ASI would be futile.