How bias enters artificial intelligence systems, who it affects, and whether fairness can truly be achieved.
Introduction: Are Algorithms Really Neutral?
We often think of computers as objective. Unlike humans, they do not have personal opinions, emotions, or prejudices. So, when an algorithm makes a decision, shouldn't it be fair?
Not necessarily.
Artificial intelligence systems learn from data created by people and shaped by society. If that data contains historical inequalities or underrepresents certain groups, an AI system can learn and reproduce those patterns.
This is known as algorithmic bias.
The concern is not simply that an algorithm might make a mistake. It is that its mistakes may consistently disadvantage particular individuals or communities.
1. Where Does Algorithmic Bias Come From?
Algorithmic bias refers to systematic patterns in an AI system that produce unfair or unequal outcomes for particular individuals or groups.
Bias can enter an AI system at several stages of its development and deployment.
1.1 Historical bias in training data
Machine learning models learn from examples. If historical data reflect unequal opportunities or discriminatory practices, a model may reproduce those patterns.
Consider a recruitment system trained using records from an organisation that historically hired more men than women for technical positions. The system may learn associations between previous hiring decisions and successful candidates.
Rather than identifying professional ability alone, it may reproduce preferences embedded in the historical records.
The problem is not necessarily that the algorithm was deliberately designed to discriminate. It is that the examples used to train it may contain patterns that should not be treated as desirable.
1.2 Representation bias
A dataset may not adequately represent the population on which the model will be used.
For example, an image recognition system trained predominantly on images of people from certain demographic groups may perform less reliably for groups that are underrepresented in its training data.
In their influential study, Gender Shades, Joy Buolamwini and Timnit Gebru (2018) examined commercial gender classification systems and identified substantial differences in classification error rates across skin type and gender groups. Their findings demonstrated why evaluating overall accuracy alone can conceal important performance disparities.
1.3 Measurement and labelling bias
Sometimes the problem lies in how information is collected or labelled.
Consider a dataset in which human evaluators assign scores to student assignments. If evaluators apply inconsistent expectations or allow personal assumptions to influence their assessments, the resulting labels may carry those biases into a machine learning model.
Similarly, using an indirect measurement as a substitute for the actual outcome of interest can produce misleading decisions.
1.4 Bias in model design and deployment
Bias does not originate exclusively in training data.
Choices about which features to include, which prediction objective to optimise, what threshold to apply, and how model outputs are used can also influence fairness.
A model may achieve high overall accuracy while performing poorly for a smaller population group. Furthermore, an apparently neutral feature may act as a proxy for a sensitive characteristic.
This shows that fairness must be considered throughout the AI lifecycle, rather than treated as a final testing activity.
2. When Algorithms Discriminate: Real-World Examples
Algorithmic bias becomes particularly concerning when automated systems influence decisions involving opportunities, resources, and essential services.
2.1 Recruitment and employment
Automated recruitment tools may screen applications, rank candidates, or identify applicants considered suitable for a position.
If these systems learn from historically unbalanced recruitment decisions, they may favour patterns associated with previously selected candidates.
For instance, an algorithm might associate particular educational backgrounds, career histories, or professional keywords with successful applicants, potentially disadvantaging qualified candidates whose profiles differ from historical examples.
The broader lesson is that historical success is not necessarily an appropriate definition of future potential.
2.2 Healthcare
An important example comes from research by Obermeyer and colleagues (2019), published in Science.
The researchers examined a widely used healthcare risk-prediction algorithm in the United States. The system used healthcare costs as a proxy for patients' healthcare needs.
However, because healthcare expenditure did not accurately represent medical need across different racial groups, the algorithm underestimated the health needs of many Black patients.
The study demonstrated how an apparently reasonable measurement can produce unequal outcomes when the chosen proxy does not represent the actual objective.
This example highlights an essential principle: selecting the right target for prediction is just as important as selecting the right algorithm.
2.3 Education
AI-driven educational systems increasingly support student assessment, personalised learning, feedback generation, and recommendations.
However, an educational model trained on limited or unrepresentative student data may not perform equally well for learners with different language backgrounds, learning needs, or educational experiences.
For example, an automated assessment system may interpret certain writing styles as less capable, even when the student demonstrates an equivalent level of conceptual understanding.
In adaptive learning environments, such differences can influence the difficulty of recommended exercises, the quality of feedback, and the learning opportunities offered to students.
Therefore, fairness in educational AI must consider more than prediction accuracy. It must also examine whether learners receive equivalent opportunities to understand, participate, and progress.
3. Can Fairness Be Measured?
Identifying bias is only the beginning. Researchers and practitioners also need methods to evaluate whether an AI system produces fair outcomes.
However, fairness is not represented by one universal mathematical formula. Different applications may require different definitions depending on their objectives, populations, and consequences.
Some commonly discussed fairness measures include:
3.1 Demographic parity
Demographic parity examines whether different demographic groups receive positive outcomes at similar rates.
For example, in a recruitment system, it compares the proportion of candidates selected from different groups.
This measure can reveal differences in selection rates, although equal selection rates alone do not establish that every decision is fair.
3.2 Equal opportunity
Equal opportunity focuses on whether qualified individuals from different groups have comparable chances of receiving a positive prediction.
In recruitment, this could involve comparing the selection rates of genuinely qualified applicants across groups.
The objective is to examine whether equally qualified individuals have similar opportunities.
3.3 Equalised odds
Equalised odds considers both true positive rates and false positive rates across groups.
It examines whether an algorithm makes comparable types of prediction errors for different populations.
This is particularly relevant in applications where both missed positive cases and incorrect positive predictions can have significant consequences.
3.4 Individual fairness
Individual fairness focuses on treating similar individuals similarly.
For example, two students demonstrating comparable knowledge should ideally receive comparable assessment outcomes, even if their writing styles or backgrounds differ.
The challenge is determining what it means for individuals to be sufficiently similar and which differences should legitimately influence a decision.
An important consideration is that these fairness definitions are not always simultaneously achievable. A system may satisfy one fairness criterion while violating another, depending on the data and underlying conditions.
Consequently, choosing a fairness metric is not merely a mathematical decision. It requires understanding the application, the affected population, and the ethical consequences of different outcomes.
4. Making Algorithms Fairer
Although eliminating every form of bias may be difficult, several approaches can help identify, evaluate, and reduce unfair outcomes.
4.1 Improve dataset quality
Training datasets should be examined for representation gaps, missing information, inconsistent labels, and historical inequalities.
Data collection should consider the diversity of the population in which the system will operate.
4.2 Conduct fairness audits
Fairness audits evaluate model performance across relevant demographic and other population groups.
Instead of relying exclusively on overall accuracy, developers should examine group-level performance, error rates, and the distribution of model outcomes.
4.3 Apply bias mitigation techniques
Fairness interventions are commonly organised into three broad categories:
- Pre-processing: Modifying or rebalancing training data before model training.
- In-processing: Incorporating fairness-related constraints or objectives during model training.
- Post-processing: Adjusting model predictions or decision thresholds after training.
The appropriate approach depends on the application, the available data, and the fairness objective.
4.4 Maintain human oversight
High-impact decisions should not rely entirely on automated predictions.
Human oversight provides an opportunity to examine unusual outcomes, challenge incorrect decisions, and consider contextual information that a model may overlook.
However, human involvement alone does not guarantee fairness. Reviewers also need appropriate training, accountability, and the authority to question automated recommendations.
4.5 Ensure transparency and accountability
Organisations should document how AI systems are developed, what data they use, what limitations they have, and how their outputs influence decisions.
People affected by automated decisions should have meaningful ways to seek explanations, raise concerns, and request appropriate review.
Fairness is not achieved simply by developing a model and deploying it. It requires continuous evaluation and responsibility throughout its use.
5. Is a Completely Fair Algorithm Possible?
Perhaps one of the most difficult questions in AI ethics is whether an algorithm can ever be completely fair.
There is no straightforward answer.
Different communities may have different expectations of fairness. A decision that appears fair under one mathematical definition may be considered unfair under another. Furthermore, fairness requirements may change depending on whether an algorithm is used for recruitment, healthcare, education, or financial services.
This does not mean that fairness is impossible or that organisations should abandon the effort.
Instead, it suggests that fairness must be treated as an ongoing objective rather than a one-time technical achievement.
Developers need to understand the social context in which their systems operate. Organisations must take responsibility for the decisions they automate. Researchers must continue investigating methods that identify and mitigate unfair outcomes.
Most importantly, the people affected by AI systems should not be excluded from discussions about how these systems are designed and evaluated.
Conclusion: Beyond Accuracy, Towards Responsibility
Artificial intelligence has the potential to improve decision-making, expand access to services, and support innovation across different sectors. Yet, its benefits cannot be evaluated through accuracy and efficiency alone.
An algorithm may perform exceptionally well according to its technical objectives while producing outcomes that disadvantage particular groups.
The challenge, therefore, is not simply to build more powerful algorithms, but to build systems that recognise human diversity, respect individual rights, and remain accountable for their consequences.
Fairness should not be an optional feature added after a model has been developed. It should be considered from the earliest stages of data collection and system design through evaluation, deployment, and continuous monitoring.
Ultimately, the question is not whether an algorithm can make decisions faster than a human.
It is whether the decisions it makes are worthy of our trust.
Because an algorithm can learn from the past, but it should not blindly reproduce its inequalities.
References and Further Reading
- Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research, 81, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
- Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys, 54(6), Article 115. https://doi.org/10.1145/3457607
- Pessach, D., & Shmueli, E. (2022). A Review on Fairness in Machine Learning. ACM Computing Surveys, 55(3), Article 51. https://doi.org/10.1145/3494672
- Ntoutsi, E., et al. (2020). Bias in data-driven artificial intelligence systems—An introductory survey. WIREs Data Mining and Knowledge Discovery, 10(3), e1356. https://doi.org/10.1002/widm.1356
Leave a Reply