Capsa Documentation

Capsa is a Python library that can automatically transform any machine learning model into one that can estimate its own uncertainty. Different uncertainty methods are packaged into easy-to-use “wrappers” which can be applied and scaled across any model, without needing to change your model or implement the uncertainty estimation method manually yourself.

What is a Wrapper?

In Capsa, a wrapper is an easy-to-use, and automated way of applying arbitrary model modifications. Often these modifications are made to enhance the capabilities of the model (e.g., learn faster, increase accuracy, uncover new insights). Capsa focuses on a subset of model modifications which specifically allow the model to estimate different forms of uncertainty. While many of the algorithms to perform uncertainty estimation require precise and low-level model modification with deep mathematical understanding, Capsa automates the entire process for you!

Types of Uncertainty

Uncertainty

Measure

Cause

Epistemic

What the model doesn’t know well

Due to limited training data or bias

Aleatoric

What the model can’t understand

Due to randomness and irreducible noise

Vacuitic

What the model hasn’t seen before

Due to missing training data or anomalies

Supported Wrappers

Note

Vacuitic uncertainty wrapper to be released September 2024!

Wrapper

Uncertainty

Memory

Speed

Uncertainty Quality

Prediction Accuracy

Train from scratch

Sculpt

Aleatoric

Small

Fast

High

Medium

Yes

Sample

Epistemic

Small

Slow

Low

High

Optional

Vote

Epistemic

Medium

Medium

Medium

High

Optional

An Example

cat dog classifier

Let’s consider a simple example to grasp the importance and differences between these different uncertainties.

Consider the problem of training a cat vs dog classifier model on a dataset of their images. When shown a new image of either a cat or dog we would expect the model to not only be accurate but also confident in its prediction. However, predicted class probabilities do not convey confidence and there are several scenarios where it is critical for uncertainty in the data and model to be reliably detected. Consider the following scenarios:

Suppose there are some dog images that confuse human labelers (perhaps they are blurry or taken from far away); some of these images may be mistakenly labelled as cats in the training set. It would be impossible for the classifier to reliably output the correct label for these kinds of images as there is random noise present. Aleatoric uncertainty would be high whenever this model sees such images due to the irreducible noise in the data.

Next, what if our classifier is shown images of a breed of dogs which it has never seen before? Some features common to all kinds of dogs will likely still be detected but, with a less clear signal than normal, the epistemic uncertainty will be higher for these images.

Finally, what if our cat-dog classifier is shown a photo of a horse? Not only will it still classify the horse as either cat or dog - it may do so very confidently since it has never seen any ambiguous data of anything beyond a cat or dog. Vacuitic uncertainty would increase when seeing the horse to reflect that no data like this has been seen before during training.