The Autoregressive Nature of LLM Operations

LLMs are AUTOREGRESSIVE generative models.

An autoregression is a regression of a variable against its own lagged values.

For example, an AR(1) predicts the current value based on the immediately preceding value, AR(n) uses the n most recent values.  

One remarkable feature of these models is in-context learning, which has been hypothesised as being Bayesian in nature.