The Statistical Logic of LLM Operations

LLMs are autoregressive generative models. An autoregression is a regression of a variable against its own lagged values, for example, an AR(1) predicts the current value based on the immediately preceding value, AR(n) uses the n most recent values.  One remarkable feature of these models is in-context learning, which has been hypothesised as being Bayesian in nature.