Capsa Documentation

Capsa is a Python library that can automatically transform any machine learning model into one that can estimate its own uncertainty. Different uncertainty methods are packaged into easy-to-use “wrappers” which can be applied and scaled across any model, without needing to change your model or implement the uncertainty estimation method manually yourself.

What is a Wrapper?

In Capsa, a wrapper is an easy-to-use, and automated way of applying arbitrary model modifications. Often these modifications are made to enhance the capabilities of the model (e.g., learn faster, increase accuracy, uncover new insights). Capsa focuses on a subset of model modifications which specifically allow the model to estimate different forms of uncertainty. While many of the algorithms to perform uncertainty estimation require precise and low-level model modification with deep mathematical understanding, Capsa automates the entire process for you!

Types of Uncertainty

Uncertainty	Measure	Cause
Epistemic	What the model doesn’t know well	Due to limited training data or bias
Aleatoric	What the model can’t understand	Due to randomness and irreducible noise
Vacuitic	What the model hasn’t seen before	Due to missing training data or anomalies

Supported Wrappers

Note

Vacuitic uncertainty wrapper to be released September 2024!

Wrapper	Uncertainty	Memory	Speed	Uncertainty Quality	Prediction Accuracy	Train from scratch
Sculpt	Aleatoric	Small	Fast	High	Medium	Yes
Sample	Epistemic	Small	Slow	Low	High	Optional
Vote	Epistemic	Medium	Medium	Medium	High	Optional

An Example

Let’s consider a simple example to grasp the importance and differences between these different uncertainties.

Consider the problem of training a cat vs dog classifier model on a dataset of their images. When shown a new image of either a cat or dog we would expect the model to not only be accurate but also confident in its prediction. However, predicted class probabilities do not convey confidence and there are several scenarios where it is critical for uncertainty in the data and model to be reliably detected. Consider the following scenarios:

Suppose there are some dog images that confuse human labelers (perhaps they are blurry or taken from far away); some of these images may be mistakenly labelled as cats in the training set. It would be impossible for the classifier to reliably output the correct label for these kinds of images as there is random noise present. Aleatoric uncertainty would be high whenever this model sees such images due to the irreducible noise in the data.

Next, what if our classifier is shown images of a breed of dogs which it has never seen before? Some features common to all kinds of dogs will likely still be detected but, with a less clear signal than normal, the epistemic uncertainty will be higher for these images.

Finally, what if our cat-dog classifier is shown a photo of a horse? Not only will it still classify the horse as either cat or dog - it may do so very confidently since it has never seen any ambiguous data of anything beyond a cat or dog. Vacuitic uncertainty would increase when seeing the horse to reflect that no data like this has been seen before during training.