Capsa Documentation

Capsa is a Python library that can automatically transform any machine learning model into one that can estimate its own uncertainty. Different uncertainty methods are packaged into easy-to-use “wrappers” which can be applied and scaled across any model, without needing to change your model or implement the uncertainty estimation method manually yourself.

What Is a Wrapper?

In Capsa, a wrapper is an easy-to-use, automated way of applying flexible model modifications. Often these modifications are made to enhance the capabilities of the model (e.g., learn faster, increase accuracy, uncover new insights). Capsa focuses on a subset of model modifications which specifically allow the model to estimate different forms of uncertainty. While many of the algorithms to perform uncertainty estimation require precise and low-level model modification with deep mathematical understanding, Capsa automates this process for you!

Types of Uncertainty

Uncertainty	Measure	Cause
Aleatoric	What the model can’t understand	Due to randomness and irreducible noise
Epistemic	What the model doesn’t know well	Due to limited training data or gaps
Vacuitic	What the model hasn’t seen before	Due to missing training data or anomalies

Supported Wrappers

Wrapper	Uncertainty	Memory	Speed	Uncertainty Quality	Pre-training*	Finetuning*
Sculpt	Aleatoric	Medium	Fast	High	Not Recommended	Required
Sample	Epistemic	Low	Slow	Low	Optional	Recommended
Vote	Epistemic	Medium	Medium	High	Optional	Required
Neo	Vacuitic	Medium	Fast	Medium	Required	Required

*“Pre-training” refers to training the model weights before wrapping with Capsa. “Finetuning” refers to training the model weights after wrapping with Capsa; either from scratch, or from a pretrained starting point, respectively.

An Example

Let’s consider a simple example to grasp the importance and differences between these different uncertainties.

Consider the problem of training a cat vs dog classifier model on a dataset of their images. When shown a new image of either a cat or dog we would expect the model to not only be accurate but also confident in its prediction. However, predicted class probabilities do not convey confidence and there are several scenarios where it is critical for uncertainty in the data and model to be reliably detected. Three scenarios illustrate this:

Suppose there are some dog images that confuse human labelers (perhaps they are blurry or taken from far away); some of these images may be mistakenly labeled as cats in the training set. It would be impossible for the classifier to reliably label such images due to this irreducible noise, resulting in high aleatoric uncertainty.

Next, what if our classifier is shown images of a breed of dogs for which it has seen few, if any, examples? Features common to all kinds of dogs may still be detected but with a weaker or less certain signal than normal, so the epistemic uncertainty will be higher for these images.

Finally, what if our cat-dog classifier is shown a photo of a horse? Not only will it still classify the horse as either cat or dog–it may do so very confidently since it has never seen any data beyond cats and dogs. This is because, when given a random input unlike any of the training data, the classifier scores will also tend to be random. Vacuitic uncertainty would increase when seeing the horse to reflect that this input lies outside the training distribution, thereby enabling detection of anomalies without relying on the classification score.