Revisiting The Legendary Doob

Pick up a book on stochastic processes and you will invariably see references to the legendary Doob.

Joseph Doob was an American mathematician who developed the theory of martingales.

He was born in Cincinnati, Ohio and the family later moved to New York City.

He studied at Harvard, where is advisor as Joseph Walsh, known for the Walsh function.

Who First Conceived of Linear Programming and How did it Become Mainstream

Linear programming (LP) was first conceived in 1939 by Kantarovich, who began studies at Leningrad University (now known as the Saint Petersburg State University) at the age of 14.

He developed LP in the context of the real-world problem of optimizing production in the plywood industry.

(Quiz question - what was Kantarovich's middle name? Answer - Vitalievich).

Kantarovich published his findings in a 1939 book called "The Mathematical Method of Production Planning and Organization". He received the Nobel Prize as well as the Soviet Union's Lenin Prize (1965) and the Order of Lenin (1967), the highest civilian decoration bestowed by the Soviet Union.

Portland-Oregon born George Dantzig, whose parents were linguists (his father Tobias was also a mathematician) took Kantarovich's ideas further and developed an algorithm to address LP problems.

The so-called "simplex method" in 1948 made LP practical for use in solving real world problems. The name "simplex" refers to the shape of the search space for the feasible set.

The related theory of duality was later developed by John von Neumann.

What Pythonistas Ought to Know About NEGs

Euclidean geometry (we will call it EG) is very familiar thanks to Euclid's Elements. EG possesses the interesting fact that the sum of the angles of a triangle at up to 180 degrees, or PI radians. This property is not shared by non Euclidean geometries (we shall call it NEGs) including spherical and hyperbolic geometries.

Complex Functions need L-Series, Like Real functions need T-Series - thanks PAL!

Any mathematical programmer needs nodding acquaintance with Laurent series.

So what is a "Laurent Series"?

These super-important series are a way of expressing a complex function as a power series when a Taylor series is not possible. They are interesting in that negative exponents are part and parcel of the series.

Pierre Alphonse Laurent ("PAL") was the discoverer of the Laurent series.

He was an engineer directing operations for the enlargement of the port of Le Havre. He applied deep theoretical knowledge to practical use.

His name is also given to the polynomial known as the Laurent polynomial (characterised by terms of negative degree).

Category Theory for Pythonistas

Category theory is a language composed of "objects" and "arrows". It is a general mathematical theory of structures. It can be regarded as philosophy as much as mathematics or theoretical computer science.

Mathematics as the Path to Progress in Computer Programming

Mathematics enables a wide range of computer algorithms to be defined and created. But pure mathematics (study of logical structures, patterns and extraction of general principles and axioms) is the path to progress in many areas of computers - it is a "fuel" that can help push the boundaries of computers and increases our capacity for invention. Many computing advances were created by mathematicians who chose to apply themselves to computers.

PyPI - The Python Package Index

PyPI is the Python Package Index.  Newish stuff on there include neural network toolkits, IoT utilities such as mbed-flasher and math language interfaces e.g. amplpy.

PyTorch

PyTorch is a way to do DL/neural nets in Python. Anaconda is the recommended package manager (used by Bloomberg, BMW, PIMCO). Latest PIP (replacement for easy_install) and numPy packages are required.

"Deep learning" (DL) As Subfield of "Machine Learning" (ML) - The Chris Manning View

Chris Manning at the Departments of Computer Science and Linguistics at Stanford University describes deep learning as a subfield of machine learning - which is a form of computational statistics. He emphasises the human-computer partnership in successful machine learning, in the sense that ML methods shown to work well have done so due to "human-designed features or representations". 

Examples given are SIFT or HoG (Histogram of Oriented Gradients) features for vision and MFCC (mel-frequency cepstral coefficients) or LPC (linear predictive coding) features for speech. ML then becomes a weighting scheme optimization process to make the best prediction.

OK, but so how does deep learning (DL) differentiate itself from more "conventional" machine learning (ML)? What are the key characteristics of this much touted subfield?

One element is representation learning (also known as "feature" learning) to learn good features and representations,with DL learning multiple levels of these representations. Neural networks are currently the tool of choice for this.  One could almost claim that "DL" is the new marketing spin on Neural Networks. "Differentiable programming" is another trendy name for this.

Why now for DL - the large amounts of training data, modern multi-core CPUs/GPUs, and just maybe, some progress in algorithmic science along the way?

Who coined the term "machine learning" anyhow?

The term "machine learning" was coined by American AI pioneer Arthur Lee Samuel in 1959 in the context of an auto-learning checkers program. He developed one of the earliest implementations of hash tables.

"The Most Important Noog of Our Time" - Gilbert Strang

Gilbert Strang described the Fast Fourier Transform as "the most important numerical algorithm of our time".

The FFT samples a signal over time and divides it into its frequency components.

Cooley and Tukey are credited with the development of the FFT in its modern form although Gauss had developed his own methods in an unpublished work from 1805 (using it to interpolate the orbit of asteroids).

John Tukey was an American statistician who coined the term "bit"(a short form for "binary digit"). He and Cooley developed the FFT algo in 1965 while working at IBM.

Manchester-born Frank Yates (graduate of St John's College Cambridge and one of the pioneers of 20th Century statistics, known to A-level students for Yates Continuity Correction) also created something similar in 1932 which he called the interaction algorithm.

Modern FFT was developed in the context of processing sensor data and has become a leading algorithm in the art of digital signal processing.

Mathematician Marcus du Sautoy (Fellow of New College Oxford) also talks about it in his podcast on Fourier, on the BBC's podcast series A Brief History of Mathematics.

Introduction to Classification - The Spam Filter

A spam filter is a simple example of statistical classification, putting data into known categories.

Classification is a case of supervised learning - where a training set is provided with correctly identified observations. An algorithm that performs classification is called a classifier.

A simple class of classification algorithms is called Naive Bayes, which has been studied extensively since the 1950s.

Naive Bayes (NB) is a popular choice for text classification e.g. is this a novel versus is this a poem or an essay. It is also used in word sense disambiguation.

Discriminative Models (aka Conditional Models)

Discriminative models are also known as conditional models.  They are used in Machine Learning to model the dependence of unobserved variables on observed variables, modeled probabilistically using P(y|x), where y is the unobserved variable vector, and x is the observed variable vector.

What is Nonparametric Statistics?

"Conventional" statistics uses distributions and parameters like mean and variance.

"Nonparametric" statistics relies on being "distribution free" or using unspecified distribution parameters.

"Support Vector Machines" are a form of nonparametric statistics useful in machine learning. It is a "discriminative" classifier. 

Analytic Number Theory

Ah, so you've heard of the Riemann zeta function, have you not, and no doubt you will want to start programming with it, yes?  If so, read on!

To know it in its contemporary form, you should have a basic knowledge of complex analysis, including Cauchy's theorem and contour integration.   (Note that Euler studied this function earlier without complex analysis but his analysis was limited to the R-z as a function of real variable, rather than a complex variable).

A flavour of the subject can be found here and here (the latter addresses analytic number theory under the headline of multiplicative number theory). Prepare to be dazzled by the spectral theory of automorphic forms.

The Los Alamos Background to the GSL

The GSL project was initiated in 1996 by Los Alamos physicists Mark Galassi and James Theiler. 

GNU Cim is Open Source Simula

Want to do some Simula? Try the GNU Cim compiler. It is written in C and produces C which is passed to a C compiler to be translated to machine code. (Simula was developed in the 1960s as a simulation language at the Norwegian Computing Center (Norsk Regnesentral) in Oslo. It had two incarnations, Simula I and Simula 67).

The Python Hackers Guide to Probability from the True Pioneer Himself and an Ode to Lesbesgue

A Python Hacker wanting to know probability from the founder of modern probability should read Foundations of the Theory of Probability by Kolmogorov. It is a monograph written "to give an axiomatic foundation for the theory of probability". Kolmogorov acknowledges a debt of gratitude to the Beauvais-born Monsieur Henri Lesbegue, specifically to "Lebesgue's theories of measure and integration" (due to the connection between measure of a set, and probability of an event).