Social norm bias: residual harms of fairness-aware algorithms

Abstract

Many modern machine learning algorithms mitigate bias by enforcing fairness constraints across coarsely-defined groups related to a sensitive attribute like gender or race. However, these algorithms seldom account for within-group heterogeneity and biases that may disproportionately affect some members of a group. In this work, we characterize Social Norm Bias (SNoB), a subtle but consequential type of algorithmic discrimination that may be exhibited by machine learning models, even when these systems achieve group fairness objectives. We study this issue through the lens of gender bias in occupation classification. We quantify SNoB by measuring how an algorithm’s predictions are associated with conformity to inferred gender norms. When predicting if an individual belongs to a male-dominated occupation, this framework reveals that “fair” classifiers still favor biographies written in ways that align with inferred masculine norms. We compare SNoB across algorithmic fairness techniques and show that it is frequently a residual bias, and post-processing approaches do not mitigate this type of bias at all.

Appendices

Appendix A: Gendered words used in classifiers

We provide insight into some of the differences across the classifiers that may be driving the SNoB  described in preceding sections. We define $\beta _w$ as the weight of a word w based on the value of the classifiers’ coefficients. We focus on the logistic regression classifiers using the BOW and WE representations since the BERT representations are contextualized, so each word does not have a fixed weight to the model that is easily interpretable.

For the BOW representation of a biography x, each feature in the input vector $v_x$ corresponds to a word w in the vocabulary. We define $\beta _w$ as the value of the corresponding coefficient in the logistic regression classifier. The magnitude of $\beta _w$ is a measure of the importance of w to the occupation classification, while the sign (positive or negative) of $\beta$ indicates whether w is correlated or anti-correlated with the positive class of the classifier.

For the WE representation, we compute the weight of each word as

$$\begin{aligned} \beta _w = \frac{e_w\cdot W_c}{|e_w||W_c|}, \end{aligned}$$

i.e. the cosine similarity between each word’s fastText word embedding $e_w$ and the coefficient weight vector $W_c$ of the WE-representation classifier. Like in the BOW representation, the magnitude of $\beta _w$ quantifies the word’s importance, while the sign indicates the direction of the association.

If a word w has positive/negative weight for classifier $Y_c$, then adding w to a biography x increases/decreases the predicted probability $Y_c(x)$ respectively.

Let $\beta _w(Y_c)$ be the weights for approach $Y_c.$ We examine the words whose weights $\beta _w$ satisfy

• 1. $|\beta _w(Y_c)| > T$,

• 2. $|\beta _w(G)| > T$,

• 3. $|\beta _w(Y_c)| > T'\cdot |\beta _w(Y_c')|$,

where $T, T'$ are significance thresholds and $Y_c, Y_c'$ are two different occupation classification approaches.

Words that satisfy these conditions are not only associated with either masculinity or femininity but also weighted more highly in approach $Y_c$ compared to $Y_c'$. Thus, including these gendered words in a biography influences $Y_c$’s classification more strongly than that of $Y_c'$. This suggests that they may contribute more strongly to the $\rho ({\textbf {p}}_C, {\textbf {r}}_C)$ in one approach than the other. For example, we examined these words for the occupations of surgeon, software engineer, composer, nurse, dietitian, and yoga teacher, which are the six most gender-imbalanced occupations, with $Y_c =$ {BOW, post-processing}, $Y_c' =$ {BOW, decoupled}, $T = 0.5$ and $T' = 0.7$. The words that satisfy these conditions are “miss”, “mom”, “wife”, “mother”, and “husband.” Conversely, with $Y_c =$ {BOW, DE} and $Y_c' =$ {BOW, PO}, the words are “girls”, “women”, “gender”, “loves”, “mother”, “romance”, “daughter”, “sister”, and “female.”

These gendered words illustrate the multiplicity of gender present in the biographies beyond categorical labels, which standard group fairness interventions do not consider.

Our analysis is limited by the fact that we only consider the individual influence of each word conditioned on the remaining words, while the joint influence of two or more words may also be of relevance.

Appendix B: Analysis on nonbinary dataset

Table 3 Correlation $r^{{\textsc {nb}}}_{{\textsc {professor}}}$ (first three columns) and $r_{{\textsc {professor}}}$  (latter three columns) across pre-processing (pre-proc), post-processing (post-proc), and decoupled approaches

We aim to consider how algorithmic fairness approaches affect nonbinary individuals, who are overlooked by group fairness approaches (Keyes et al. 2021). Using the same regular expression as De-Arteaga et al. (2019) to identify biography-format strings, we collected a dataset of biographies that use nonbinary pronouns such as “they”, “xe”, and “hir.” Since “they” frequently refers to plural people, we manually inspected a sample of 2000 biographies using“they” to identify those biographies that refer to individuals. professor is the only occupation title with more than 20 such biographies; the other occupations have too few biographies to perform meaningful statistical analysis. We computed $r^{{\textsc {nb}}}_{{\textsc {professor}}}$, which is analogous to $r_{{\textsc {professor}}}$, the measure of SNoB for an individual occupation classifier introduced in Sect. 4. While $r_{{\textsc {professor}}}$ is Spearman’s correlation computed across the biographies in $S_c$, $r^{{\textsc {nb}}}_{{\textsc {professor}}}$  is the correlation across the nonbinary biographies in the profession. The results are reported in Table 3. We find that $r^{{\textsc {nb}}}_{{\textsc {professor}}}$ is positive across different approaches. However, the associated p values are quite large $(>0.1)$, so it is challenging to analyze these associations. This is likely due to the small sample size; while $r_{{\textsc {professor}}}$  is computed across the 10677 professor biographies that use “she” pronouns, $r^{{\textsc {nb}}}_{{\textsc {professor}}}$  is across only 21 biographies.

Appendix C: Word weights

In Fig. 6, we plot the weight of each word in the BOW vocabulary in the occupation classifiers and gender classifiers. These weights illuminate some of the mechanisms behind the predictions. Ideally, without SNoB, every point would have small magnitude in either the occupation or gender classifier, i.e. lie on either the $x-$ or $y-$axis of Fig. 6. We observe that in the DE approach, words are closer to the $y-$axis compared to the post-processing approach. This corresponds to the smaller value of $\rho ({\textbf {p}}_C, {\textbf {r}}_C)$ exhibited by the decoupled approach compared to the post-processing one in Table 2. Note that the post-processed classifier is trained on all of the biographies, while the decoupled classifier is trained on only biographies that use the same pronoun.

Fig. 6

Words’ weights in the occupation and gender classifiers for different approaches in the surgeon occupation. Each point represents a word; its x-position and y-position represents its weight in $\hat{Y}_c$ and G respectively. Each point is colored based on its quadrant in the post-processing approach. Many points are closer to the $y-$axis in the decoupled approach