AFISP Package

Module contents

class afisp.WorstSubsetFinder(*args: Any, **kwargs: Any)

Bases: BaseEstimator

This is a class for performing a Stability Analysis of a trained machine learning (ML) model. Given a test dataset, a WorstSubsetFinder object identifies a data subset of a particular sample size that produces the worst performance (i.e., a worst-case data subset of a given size). A worst case data subset corresponds to an adversarial covariate shift in the distribution of user-specified features.

Parameters:

• subset_fractions (List[float]) – A list of subset fractions between 0 and 1. For each subset fraction, the WorstSubsetFinder identifies the worst performing subset of approximately that subset fraction size (i.e., for a subset fraction f and a dataset of size N, the subset will be size approximately f*N, defaults to [0.1, 0.15, 0.2, 0.4, 0.6, 0.8, 1.0].

• conditional_loss_model – A supervised learning model with sklearn interface for estimating the expected conditional loss of the ML model to be evaluated, defaults to None which will fit an ExplainableBoostingRegressor.

• cv (int) – Number of folds for cross validation to predict the loss for each data sample, defaults to 10.

• verbose (bool) – If True, then prints information during calls to self.fit, defaults to False.

• eps (float) – Corresponds to the max amount of noise to add to conditional loss estimates. It is used to break ties between samples with the same expected loss. It should be set to a small positive value if many subgroup features are discrete, defaults to 0.

__annotations__ = {}

__init__(subset_fractions=[0.1, 0.15, 0.2, 0.4, 0.6, 0.8, 1.0], conditional_loss_model=None, cv=10, verbose=False, eps=0.0)

Constructor method

__module__ = 'afisp.stability_analysis'

__orig_bases__ = (sklearn.base.BaseEstimator,)

check_subset_sizes(plot=True, ax=None)

A visual diagnostic for checking subset sizes are correct.

Parameters:

• plot (bool) – Whether or not to plot the results, defaults to True

• ax (Matplotlib axis) – Matplotlib axis to plot on if plot is True, defaults to None, optional

Returns:

Two lists, the first containing the subset fractions and the second containing the identified fractions. When plotted against each other the relationship should match the line y=x.

Return type:

(List[float], List[float])

compute_effect_sizes(plot=False, ax=None)

Compute the effect size (Cohen’s d) of the difference in the average loss for each worst-case subset and the full test dataset. Can also plot the effect sizes vs subset fraction.

Parameters:

• plot (bool) – Whether or not to plot the results, defaults to False

• ax (Matplotlib axis) – Matplotlib axis to plot on if plot is True, defaults to None, optional

Returns:

Two lists, the first containing the subset fractions and the second containing the effect size of the difference in the average loss in the worst subset and the full test dataset.

Return type:

(List[float], List[float])

confidence_intervals()

Returns analytical confidence intervals for the worst-case loss estimates for each subset fraction size.

Returns:

Array with first column containing lower confidence interval and second column containing upper confidence interval.

Return type:

((# of subset fraction sizes, 2),ndarray)

find_max_effect_size()

Finds the index of the worst subset producing the max effect size for the difference in the average loss of the subset and full dataset.

Returns:

The index of the subset producing the max effect size and the max effect size.

Return type:

(int, float)

fit(subgroup_feature_data, samplewise_losses, feature_names=None)

Computes the worst-case data subsets for each subset fraction and returns the average loss on each subset.

Parameters:

• subgroup_features (((nsamples, nfeatures), ndarray)) – A numpy array of dim (nsamples, nfeatures) containing the subgroup defining feature values for each sample.

• samplewise_losses (((nsamples,), array)) – A numpy array of the per-sample observed loss

• feature_names (List[string], optional) – A list of names for the subgroup characterizing features, optional

Returns:

An array containing the average loss on each worst-case subset

Return type:

(# of subset_fractions, array)

subset_masks()

Computes a boolean mask for indexing worst-case data subsets.

Returns:

List of boolean masks for indexing the worst-case data subsets corresponding to each subset fraction

Return type:

List[(# of samples, boolean ndarray)]

class afisp.SubgroupPhenotyper(*args: Any, **kwargs: Any)

Bases: BaseEstimator

This class performs subgroup phenotyping. After using stability analysis to identify a poorly performing data subset, the SubgroupPhenotyper is used to find specific data phenotypes or subgroups that are present within the data subset.

__annotations__ = {}

__init__()

Constructor method

__module__ = 'afisp.subgroup_phenotyping'

__orig_bases__ = (sklearn.base.BaseEstimator,)

fit(subgroup_feature_data, subset_labels, test_loss, method='DecisionList', depth=2, cv=False, rule_max=50, p0=0.025, input_fname='data_for_sirus.csv', output_fname='sirus_rules.txt', verbose=0)

Computes the subgroup phenotypes using an interpretable classifier. The subgroup phenotyper expects categorical features to encoded as binary dummy variables.

Parameters:

• subgroup_feature_data – Array containing the subgroup feature data.

• subset_labels – Binary labels for whether each sample is in the worst data subset.

• test_loss – The observed loss for each sample. Used for filtering rules based on statistical significance and effect size.

• method – Selects the interpretable classification method used for extracting the subgroup phenotypes. “SIRUS” uses the ‘Stable and Interpretable RUle Set’ method implemented in R. It requires a working R distribution with the ‘sirus’ package installed. This is recommended. As a python alternative, the “DecisionList” will use the SkopeRules DecisionListClassifier, defaults to “DecisionList’.

• cv – For method SIRUS, whether or not to use cross-validation to select the rule selection threshold. If True, then p0 is ignored. Using cross-validation can be very slow, but results are often better.

• p0 (Float in (0, 1)) – For method SIRUS, the threshold (between 0 and 1) for rule selection. Ignored if cv is True, since cv is used to determine a good value for p0. The higher the value of p0, the fewer rules that will be selected. Recommended to choose a value < 0.1, defaults to 0.025.

• rule_max (int > 0) – The max number of rules that SIRUS will consider, defaults to 50.

• depth –

Returns:

The candidate rules extracted by the Subgroup Phenotyper

Return type:

List[string]

generate_subgroup_table(y_test, test_preds, loss_fn=sklearn.metrics.brier_score_loss)

Generates a summary table reporting the subgroup phenotypes identified to have poor performance by the SubgroupPhenotyper. The table also reports the sample size and the performance (including a 95% bootstrap confidence interval)

Parameters:

• y_test – The true labels for the test dataset

• test_preds – Test set predictions from model being evaluated.

• loss_fn – Loss function for computing average performance on test dataset.

Submodules

afisp.utils module

afisp.utils.bootstrap_ci(y_true, y_pred, n_bootstrap=100, confidence=0.95, loss=sklearn.metrics.roc_auc_score, return_samples=False)

Computes non-parametric bootstrap confidence interval for model performance.

Parameters:

• y_true – True target labels

• y_pred – Model predictions (be it regression predictions, probability predictions, or classification predictions).

• n_bootstrap (int) – Number of bootstrap resamples to perform, defaults to 100.

• confidence (Float, between 0 and 1) – The confidence level for the interval as a decimal, defaults to 0.95

• loss – Loss function for computing average model performance. Should have signature ‘loss(y_true, y_pred)’, defaults to sklearn.metrics.roc_auc_score

Returns:

The mean performance, the lower interval, and the upper interval from the bootstrap samples.

afisp.utils.brier(y, y_pred)

Samplewise brier score for probabilistic classification

Parameters:

• y (Numpy array with values in {0, 1}) – Array of true binary classification labels

• y_pred (Numpy array with values in (0, 1)) – Array of probabilistic predictions (between 0 and 1)

Returns:

Array of per-sample brier scores

Return type:

Array[float]

afisp.utils.clip_predictions(preds, upper_bound=0.99, lower_bound=0.01)

Clip probability predictions to be in the (0, 1) open interval.

Parameters:

• preds ((num samples,) np array) – Array of sample predictions

• upper_bound (float) – Upper bound of clipped predictions, defaults to 0.99

• lower_bound (float) – Lower bound of clipped predictions, defaults to 0.01

Returns:

Predictions clipped to be in [lower_bound, upper_bound] interval

Return type:

List[float]

afisp.utils.cohens_d(x, y)

Computes effect size as measured by Cohen’s d. The effect size is a scaled difference in the means between two groups.

Parameters:

• x – The measurements for group 1.

• y – The measurements for group 2.

Returns:

Cohen’s d, the effect size for the difference in measurements between the two groups.

Return type:

Float

afisp.utils.cross_entropy(y, y_pred)

Samplewise cross entropy loss for probabilistic classification

Parameters:

• y (Numpy array with values in {0, 1}) – Array of true binary classification labels

• y_pred (Numpy array with values in (0, 1)) – Array of probabilistic predictions (between 0 and 1)

Returns:

Array of per-sample cross entropy losses

Return type:

Array[float]

afisp.utils.entropy(y, y_pred)

afisp.utils.hinge_surrogate(labels, logits)

afisp.utils.logit(p)

afisp.utils.mse(y, y_pred)

Samplewise mean squared error for regression

Parameters:

• y – Array of true regression labels

• y_pred – Array of regressin predictions

Returns:

Array of samplewise mean squared errors

afisp.utils.roc_auc_surrogate(y, y_pred, surrogate='xent')

afisp.utils.torch_roc_auc_surrogate(y, y_pred, surrogate='xent')

PyTorch computation of a surrogate samplewise AUROC loss.

Parameters:

• y (Numpy array with values in {0, 1}) – Array of true binary classification labels

• y_pred (Numpy array with values in (0, 1)) – Array of probabilistic predictions (between 0 and 1)

• surrogate – String specifying which surrogate loss function to use, defaults to ‘xent’. ‘xent’ Cross-entropy surrogate. ‘hinge’ Hinge loss surrogate.

Returns:

Array of samplewise surrogate AUROC losses.

afisp.utils.xent_surrogate(labels, logits)

afisp.utils.zero_one_loss(y, y_pred)

Samplewise Zero-One Loss for binary classification

Parameters:

• y (Numpy array with values in {0, 1}) – Array of true binary classification labels

• y_pred (Numpy array with values in {0, 1}) – Array of binary classification predictions {0, 1}

Returns:

Array of per-sample zero-one losses

Return type:

Array[float]