Introduction

Introduction#

Famous Recent Examples of Scientific Machine Learning#

With scientific machine learning becoming an ever larger mainstay at machine learning conferences, and ever more venues and research centres at the intersection of machine learning and the natural sciences / engineering appearing there exist ever more impressive examples of algorithms which connect the very best of machine learning with deep scientific insight into the respective underlying problem to advance the field.

Below are a few prime examples of recent flagship algorithms in scientific machine learning, of which every single one of them personifies the very best algorithmic approaches we have available to us today.

AlphaFold - predicts 3D protein structure given its sequence:#

_images/alphafold.png — Fig. 1 AlphaFold model. (Source: Jumper et al., 2021)#

GNS - capable of simulating the motion of water particles:#

_images/gns.png — Fig. 2 GNS model. (Source: Sanchez-Gonzalez et al., 2020)#

Codex - translating natural language to code:#

_images/codex.gif — Fig. 3 Codex demo (Source: openai.com)#

Geometric Deep Learning #

Geometric deep learning aims to generalize neural network models to non-Euclidean domains such as graphs and manifolds. Good examples of this line of research include:

SFCNN - steerable rotation equivariant CNN, e.g. for image segmentation#

_images/sfcnn.png — Fig. 4 SFCNN model. (Source: Weiler et al., 2018)#

SEGNN - molecular property prediction model#

_images/segnn.png — Fig. 5 SEGNN model. (Source: Brandstetter et al., 2022)#

Stable Diffusion - generating images from natural text description#

_images/stable_diffusion_art.jpg — Fig. 6 Stable Diffusion art. (Source: stability.ai)#

Stable Diffusion reconstructing visual experiences from human brain activity #

_images/stable_diffusion_brain.jpg — Fig. 7 Stable Diffusion brain signal reconstruction. (Source: Takagi & Nishimito, 2023)#

ImageBind - Holistic AI learning across six modalities#

_images/imagebind.gif — Fig. 8 ImageBind modalities. (Source: ai.meta.com)#

Definition#

Machine learning at the intersection of engineering, physics, chemistry, computational biology etc. and core machine learning to improve existing scientific workflows, derive new scientific insight, or bridge the gap between scientific data and our current state of knowledge.

Important here to recall is the difference in approaches between engineering & physics, and machine learning on the other side:

Engineering & Physics

Models are derived from conservation laws, observations, and established physical principles.

Machine Learning

Models are derived from data with imprinted priors on the model space either through the data itself, or through the design of the machine learning algorithm.

Supervised vs Unsupervised#

There exist 3 main types of modern day machine learning:

Supervised Learning
Unsupervised Learning
Reinforcement Learning

Supervised Learning#

In supervised learning we have a mapping \(f: X \rightarrow Y\), where the inputs \(x \in X\) are also called features, covariates, or predictors. The outputs \(y \in Y\) are often also called the labels, targets, or responses. The correct mapping is then learned from a labeled training set

(1)#\[\mathcal{D}_{N} = \left\{ \left( x_{n}, y_{n} \right) \right\}_{n=1:N}\]

with \(N\) the number of observations. Depending on the type of the response vector \(y\), we can then perform either regression, or classification

Some also call it “glorified curve-fitting”

Regression#

In regression, the target \(y\) is real-valued, i.e. \(y \in \mathbb{R}\)

Example of a response surface being fitted to a number of data points in 3 dimensions, where in this instance the x- and y-axes are a two-dimensional space, and the z-axis is the temperature in the two-dimensional space.

Classification#

In classification the labels \(y\) are categorical i.e. \(y \in \mathcal{C}\), where \(\mathcal{C}\) defines a set of classes.

Example of flower classification, where we aim to find the decision boundaries which will sort each individual node into the respective class.

Unsupervised Learning#

In unsupervised learning, we only receive a dataset of inputs

(2)#\[\mathcal{D}_{N} = \left\{ x_{n} \right\}_{n=1:N}\]

without the respective outputs \(y_{n}\), i.e. we only have unlabelled data.

The implicit goal here is to describe the system, and identify features in the high-dimensional inputs.

Two famous examples of unsupervised learning are clustering (e.g. k-means) and especially dimensionality reduction (e.g. principal component analysis) which is commonly used in engineering and scientific applications.

Clustering of Principal Components#

_images/pca_clustering.png — Fig. 11 Clustering based on principal components. (Source: [Brunton and Kutz, 2019], Section 1.5)#

Combining clustering with principal component analysis to show the samples which have cancer in the first three principal component coordinates.

Supervised vs Unsupervised, the tl;dr in Probabilistic Terms#

The difference can furthermore be expressed in probabilistic terms, i.e., in supervised learning we are fitting a model over the outputs conditioned on the inputs \(p(y|x)\), whereas in unsupervised learning we are fitting an unconditional model \(p(x)\).

Reinforcement Learning#

In reinforcement learning, an agent sequentially interact with an unknown environment to obtain an interaction trajectory \(T\), or a batch thereof. Reinforcement learning then seeks to optimize the way the agent interacts with the environment through its actions \(a_{t}\) to maximize for a (cumulative) reward function to obtain an optimal strategy.

_images/rl.png — Fig. 12 Reinforcement learning overview. (Source: lilianweng)#

Polynomial Curve Fitting#

Let’s presume we have a simple regression problem, e.g.

_images/lin_reg_1d.png — Fig. 13 Linear regression example. (Source: [Murphy, 2022], Introduction)#

Then we have a number of scalar observations \({\bf{x}} = (x_{1}, \ldots, x_{N})\) and targets \({\bf{y}} = (y_{1}, \ldots, y_{N})\). Then the tool we have probably seen before in the mechanical engineering curriculum is the simple approach to fit a polynomial function

(3)#\[y(x, \mathbf{w}) = \omega_{0} + \omega_{1}x + \omega_{2} x^{2} + \ldots + \omega_{M}x^{M} = \sum_{j=0}^{M}\omega_{j}x^{j}\]

Then a crucial choice is the degree of the polynomial function.

This class of models is called Linear Models because we want to learn only the linear scaling coefficients \(w_i\), given any choice of basis for the variable \(x\) like the polinomial basis shown here.

We can then construct an error function with the sum of squares approach in which we are computing the distance of every target data point to our polynomial

(4)#\[E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^{N} \{ y(x_{n}, \mathbf{w}) - y_{n} \}^{2}\]

in which we are then optimizing for the value of \(w\).

_images/lin_reg_1d_distances.png — Fig. 14 Linear regression error computation. (Source: [Murphy, 2022], Introduction)#

To minimize this we then have to take the derivative with respect to the coefficients \(\omega_{i}\), i.e.

(5)#\[\frac{\partial E(w)}{\partial \omega_{i}}=\sum_{n=1}^{N}\{ y(x_{n}, w) - y_{n} \}x_{n}^{i}=\sum_{n=1}^{N}\{ \sum_{j=0}^{M}\omega_{j} x_{n}^{j} - y_{n} \}x_{n}^{i}\]

which we are optimizing for and by setting to 0, we can then find the minimum

(6)#\[\sum_{n=1}^{N}\sum_{j=0}^{M}\omega_{j}x_{n}^{i}x_{n}^{j}=\sum_{n=1}^{N}y_{n}x_{n}^{i}\]

This can be solved by the trusty old Gaussian elimination. A general problem with this approach is that the degree of the polynomial is a decisive factor which often leads to over-fitting and hence makes this a less desirable approach. Gaussian elimination, or a matrix inversion approach when implemented on a computer can also be a highly expensive computational operation for large datasets.

This is a special case of the Maximum Likelihood method.

Bayesian Curve Fitting#

Recap: Bayes Theorem

(7)#\[\mathbb{P}(A|B) = \frac{\mathbb{P}(B|A)\mathbb{P}(A)}{\mathbb{P}(B)}\]

If we now seek to reformulate the curve-fitting in probabilistic terms, then we have to begin by expressing our uncertainty over the target \(y\) with a probability distribution. For this we presume a Gaussian distribution over each target where the mean is the point value we previously considered, i.e.

(8)#\[p(y|x, \mathbf{w}, \beta)=\mathcal{N}(y|y(x, \mathbf{w}), \beta^{-1}),\]

where \(\beta\) corresponds to the inverse variance of the normal distribution \(\mathcal{N}\). We can then apply the maximum likelihood principle to find the optimal parameters \(\mathbf{w}\) with our new likelihood function

(9)#\[p(y|x, \mathbf{w}, \beta)=\prod^{N}_{n=1}\mathcal{N}(y_{n}|y(x_{n},\mathbf{w}), \beta^{-1}).\]

_images/bayesian_reg_1d.png — Fig. 15 Bayesian regression example. (Source: [Bishop, 2006], Section 1.2)#

Taking the log likelihood we are then able to find the definitions of the optimal parameters

(10)#\[\text{ln } p(y|x, \mathbf{w}, \beta) = - \frac{\beta}{2} \sum^{N}_{2} \{ y(x_{n}, \mathbf{w}) - y_{n} \}^{2} + \frac{N}{2} \text{ln } \beta - \frac{N}{2} \text{ln }(2 \pi)\]

Which we can then optimize for the \(\mathbf{w}\).

If we consider the special case of \(\frac{\beta}{2}=\frac{1}{2}\), and instead of maximizing, minimizing the negative log-likelihood, then this is equivalent to the sum-of-squares error function.

The herein obtained optimal maximum likelihood parameters \(\mathbf{w}_{ML}\) and \(\beta_{ML}\) can then be resubstituted to obtain the predictive distribution for the targets \(y\).

(11)#\[p(y|x, \mathbf{w}_{ML}, \beta_{ML})=\mathcal{N}(y|y(x, \mathbf{w}_{ML}),\beta_{ML}^{-1})\]

To arrive at the full Bayesian curve-fitting approach we now have to apply the sum and product rules of probability

Recap: Sum Rules of (disjoint) Probability

(12)#\[\mathbb{P}(A \cap B) = \mathbb{P}(A) + \mathbb{P}(B)\]

Recap: Product Rules of Probability - for independent events \(A,B\).

(13)#\[\mathbb{P}(A, B) = \mathbb{P}(A) \cdot \mathbb{P}(B)\]

The Bayesian curve fitting formula is hence

(14)#\[p(y|x, {\bf{x}}, {\bf{y}}) = \int p(y|x, \mathbf{w})p(\mathbf{w}|{\bf{x}}, {\bf{y}})d\mathbf{w}\]

with the dependence on \(\beta\) omitted for legibility reasons. This integral can be solved analytically hence giving the following predictive distribution

(15)#\[p(y|x, {\bf{x}}, {\bf{y}})=\mathcal{N}(y|m(x), s^{2}(x))\]

with mean and variance

(16)#\[m(x) = \beta \phi(x)^{T} {\bf{S}} \sum_{n=1}^{N}\phi(x_{n})y_{n},\]

(17)#\[s^{2}(x) = \beta^{-1} + \phi(x)^{T} {\bf{S}} \phi(x),\]

and \({\bf{S}}\) defined as

(18)#\[{\bf{S}}^{-1} = \alpha {\bf{I}} + \beta \sum_{n=1}^{N} \phi(x_{n}) \phi(x)^{T},\]

and \({\bf{I}}\) the unit matrix, while \(\phi(x)\) is defined by \(\phi_{i}(x) = x^{i}\). Examining the variance \(s^{2}(x)\), then the benefits of the Bayesian approach become readily apparent:

\(\beta^{-1}\) represents the prediction uncertainty
The second term of the variance is caused by the uncertainty in the parameters \(w\).

Maximum Likelihood & Log-Likelihood#

Formalizing the principle of maximum likelihood estimation, we seek to act in a similar fashion to function extrema calculation in high school in that we seek to differentiate our likelihood function, and then find the maximum value of it by setting the derivative equal to zero.

(19)#\[\hat{\theta} = \underset{\theta \in \Theta}{\arg \max} \mathcal{L}_{n}(\theta; y),\]

where \(\mathcal{L}\) is the likelihood function. In the derivation of the Bayesian Curve Fitting approach we have already utilized this principle by exploiting the independence between data points and then taking log of the likelihood to then utilize the often nicer properties of the log-likelihood

(20)#\[l(\theta;y) = \ln \mathcal{L}_{n}(\theta;y).\]

Recap#

Supervised Learning: Mapping from inputs \(x\) to outputs \(y\)
Unsupervised Learning: Only receives the datapoints \(x\) with no access to the true labels \(y\)
Maximum Likelihood Principle
- Polynomial Curve Fitting a special case
- Wasteful of training data, and tends to overfit
- Bayesian approach less prone to overfitting

Further References#

Using AI to Accelerate Scientific Discovery - inspirational video by Demis Hassabis