Some system performance metrics, which indicate obtained benefits like correct or favorable operation, were found unevenly distributed across the population. These biases occurred in at least one prediction class and at least one way of aggregating the comparison among multiple groups. Expert assessment is needed to help understand which biases may be considered unfair. The biased metrics are:

the true negative rate/specificity (tnr)
the true rejection ratio (true negatives compared to all) (trr)
the Matthews correlation coefficient (mcc)
the true acceptance ratio (true positives compared to all) (tar)
the positive rate (pr)
the accuracy (acc)
the geometric mean of tpr and tnr - accounts for class imbalance (gmi)
the f1 score (f1)
the Cohen's Kappa score (kappa)
the lift ratio (tpr divided by pr) (lift)
the true positive rate/recall/sensitivity/hit rate (tpr)
the positive predictive value/precision (ppv)

Fairness does not end after producing AI outputs.

💡 Continue interacting with stakeholders to assert that their idea of fairness is correctly implemented.
💡 Monitor the outputs of deployed systems by rerunning the analysis on updated models and datasets.
💡 Test model and dataset variations for multiple sensitive characteristics and parameters.

Groups were compared pairwise. Values deviating more than 0.050 from their ideal target were counted as problematic. These deviations guide where deeper inspection is needed. The result is considered biased if it lays 0.050 away from its ideal target that would indicate fairness. For example, the ideal target is 0 for differences between measure values, and 1 for values that should be large (e.g., the minimum accuracy across all groups). Some metrics have no known ideal values.

The analysis considered 7 protected groups:
education unknown
education tertiary
education secondary
education primary
marital married
marital single
marital divorced

tabular data with custom formatting

path: /home/maniospas/Documents/mammoth-commons/data/bank/bank.csv
delimiter: ;
numeric: age, duration, campaign, pdays, previous
categorical: marital, job, education, default, housing, contact, loan, poutcome
label: y
skip invalid lines: True
Uses pandas to load a CSV file that contains custom specification of numeric, categorical, and predictive data columns. Each row corresponds to a different data sample, with the first one sometimes holding column names (this is automatically detected).

a serialized machine learning model

path: /home/maniospas/Documents/mammoth-commons/data/model.onnx
trained with sensitive: True
Loads an inference model stored in the ONNx format, which is a generic cross-platform way of representing AI models with a common set of operations. The loaded model should be compatible with the dataset being analysed, for example having been trained on the same tabular data columns.

Technical details and how to export a model to this format.

Several machine learning frameworks can export to ONNx. The latter supports several different runtimes, but this module's implementation selects the `CPUExecutionProvider` runtime to run on to maintain compatibility with most machines. For inference in GPUs, prefer storing and loading models in formats that are guaranteed to maintain all features that could be included in the architectures of respective frameworks; this can be achieved with other model loaders. Here are some quick links on how to export ONNx models from popular frameworks:

• scikit-learn
• PyTorch
• TensorFlow

Summary of measures.

Name	Description
acc	the accuracy
tpr	the true positive rate/recall/sensitivity/hit rate
ppv	the positive predictive value/precision
f1	the f1 score
gmi	the geometric mean of tpr and tnr - accounts for class imbalance
mcc	the Matthews correlation coefficient
kappa	the Cohen's Kappa score
pr	the positive rate
tar	the true acceptance ratio (true positives compared to all)
trr	the true rejection ratio (true negatives compared to all)
lift	the lift ratio (tpr divided by pr)
tnr	the true negative rate/specificity

Summary of reductions.

Name	Description
min	the minimum
maxerror	the maximum deviation from the ideal value
wmean	the weighted average
mean	the average
gm	the geometric mean
maxrel	the maximum relative difference
maxdiff	the maximum difference
gini	the gini coefficient
stdx2	the standard deviation x2