AI Bias is Costing You Money & Here’s How to Fix It
Part 2 of the AI for Product Manager Series where we explore different concepts surrounding AI
Have you ever watched the not-so-popular gem called “Better Off Ted”? It’s a pretty clever show that never got deserved praise for highlighting the latest technology and how it affects people’s lives. So basically, it's a way lighter approach than Black Mirror. Why did I mention it? Well, this clip highlights something that even AI struggles with today. The inability to remove biases because, well… biases are everywhere, and they were trained with everything.
Last week, we talked about hallucinations in AI (read more about it here). We learned how AI confidently generates incorrect yet plausible-sounding information and how product managers can mitigate that by building checks and balances in their products. This week, we’ll examine biases, another critical issue that can be as damaging, if not more.
Bias is about AI getting things unfairly wrong. You can spot it when the system produces outputs that systematically favor or disadvantage certain groups, perspectives, or scenarios. Now why do these biases happen? Today, we’ll take a look at 2 main reasons: Imbalance Training Data that often lack representation and Human Biases.
Then we discuss why it’s bad. But not in the obvious way. I’ll take a stab at reasoning out why these biases are bad for business, even if they do favour you. And like always, we’ll apply a product management lens on this, and how can you mitigate this problem from your product.
Imbalanced Training Data
AIs are trained. They’re trained with tons (and tons) of data being fed into them and learning from them. So if your data skews in one direction that underrepresents or overrepresents a particular group, concepts, etc., you’ll have biases.
How does this look in real life?
Well, we can look at this NIH scientific article describing different types of biases in healthcare and AI. According to them, they highlight the Data-Driven Bias based on how the training datasets often lack diversity, leading to poor performance for underrepresented groups (e.g., racial minorities, women, and older adults).
And how this led to a number of misdiagnoses, unequal treatment, and worsening health disparities. For example: Pulse oximeters overestimate oxygen levels in Black patients, leading to undetected hypoxemia. Or how Cardiovascular disease prediction models trained on male-dominated datasets perform poorly for women. Another one is that sleep-scoring algorithms trained on young, healthy individuals fail to diagnose sleep disorders in older adults accurately.
Reflection of Human Biases
While imbalanced training data may come from a lack of representation of certain groups, human biases are more complex as it’s often in the subconscious that holds implicit or explicit biases:
Implicit Biases: Humans have unconscious preferences and stereotypes that seep into the data. For example, if historical hiring data shows a preference for male candidates in tech roles, the AI will learn to favor men.
Explicit Biases: Sometimes, the data itself is flawed because of deliberate human decisions. For instance, if a dataset labels certain groups as “high risk” based on biased criteria, the AI will replicate that bias.
Think of it this way: AI is neutral, but it will soak up all the biases that are fed into it, the good, the bad, and the ugly.
How does this look in real life?
In 2018, Amazon had to scrap its hiring AI tool because it showed biases against women when it came to hiring. The algorithm was trained on resumes submitted to Amazon over a 10-year period, most of which came from men (a reflection of the male-dominated tech industry). The AI learned to penalize resumes that included words like “women’s” (e.g., “women’s chess club captain”) and downgraded resumes from all-women colleges. And the result? A tool that systematically favored male candidates, perpetuating the very bias it was supposed to eliminate.
Why should you care?
I’m going to skip the whole ethical implications and discrimination that’s being highlighted that may not be in your favour when it comes to dealing with future opportunities. Not because I don’t believe it’s not important (I certainly do!). However, I can also say that at the end of the day, these things will continue to happen until they start to affect people who are safe from these biases and have the power to change everything.
So, let’s assume you’re the lucky few who actually get prioritized because of all these biases. Why would you want to change it? Well, let’s take a lesson from economics: everything is finite. You can only market to a certain group of non-discriminated segments for so long. Not only that, when other people are left out of consideration, it would cost lost revenue, missed opportunities, and damaged reputations.
Losing Top Talent = Losing Lots of Money
According to McKinsey, Companies in the top quartile for gender diversity on executive teams were 21% more likely to outperform on profitability and 27% more likely value creation. And companies in the top quartile for ethnic/cultural diversity on executive teams are 33% more likely to get industry-leading profitability.
Wow, that’s a pretty significant margin, especially regarding profits. And while I can talk more about “What about the culture? We need to have a cohesive culture!” Well, the point is not to worry about that because you’re too busy making more money, and if you keep leaning on that, then you’re missing out on billions in potential revenue. A homogeneous workforce limits your ability to innovate, connect with diverse customers, and solve complex problems.PR Disasters = Brand Damage
No company wants to deal with a public backlash. I get it. Implicit biases are often done subconsciously, so it’s not like you intended for it to happen. However, it does. And if it continues long enough, it can damage your brand, business, or place of work. That can lead to a backlash that affects your reputation and profit.
Remember Microsoft had an AI bot long ago before every other company had theirs? Neither do I. This is because within 24 hours after launching Microsoft Tay, it began spewing racist, sexist, and offensive tweets after being manipulated by users. Tay’s offensive tweets went viral, leading to widespread condemnation and ridicule. Microsoft, a company known for its professionalism, was forced to apologize and shut down Tay within a day of its launch. Now, this is an extreme case but you never hear Microsoft being the leader of AI or recovering from it.
Legal and Regulatory Risks = $$$
Now, I’ve been trying really hard not to make this about ethical arguments. But in this instance, it directly correlates with not just you losing profits but also with you costing the company lots of money by getting your legal team busy battling cases. You might not care as long as it doesn’t affect you, but when you’re caught with it, biased AI can lead to lawsuits, fines, and compliance costs.
Just take a look at Facebook and how it battled five discrimination lawsuits for 2 years. Facebook is paying nearly $5 million to settle several lawsuits that alleged its advertising platform allowed for discrimination in housing, employment, and credit ads. These lawsuits came from civil rights and labor organizations, workers, and individuals.
Now that you care, how can you improve your AI product and reduce biases?
A lot of the cases above happened before AI was even as powerful as it is today. But if you don’t control your biases now, imagine how many of these things can slip from your business. So, taking my product management hat back on, these are the types of epics and features that I’d start initiating if I were to build an AI or AI-tangental product.
1. Build a Diverse and Representative Dataset
Since most of these problems originate from the training dataset, focusing on it would be the biggest game-changer. If your dataset lacks diversity, your AI will too. The reverse is also true.
What to Do:
Audit Your Data: Analyze your training data for representation across key demographics (e.g., race, gender, age, geography).
Expand Data Sources: Collect data from a variety of sources to ensure a balanced representation. For example, if you’re building a facial recognition system, include faces from all skin tones, ages, and genders.
Use Synthetic Data: If certain groups are underrepresented, consider using synthetic data to fill the gaps.
Product Feature Idea:
Data Diversity Dashboard: Build a tool that visualizes the diversity of your training data and flags underrepresented groups.
2. Implement Bias Detection and Mitigation Tools
Test, test, test! But the last thing you want is to disrupt your flow and add time to your tied hand trying to mitigate biases too. So what you can do is just add a checker into your product development flow. I would say, it’s best around the QA stage but feel free to change it according to your development cycle.
What to Do:
Use Fairness Metrics: Incorporate metrics like demographic parity, equalized odds, or disparate impact into your model evaluation process.
Leverage Bias Detection Tools: Use tools like IBM’s AI Fairness 360, Google’s What-If Tool, or Microsoft’s Fairlearn to identify and mitigate bias.
Conduct Regular Audits: Test your AI system in real-world scenarios to ensure it performs well for all user groups.
Product Feature Idea:
Bias Audit Module: Integrate a bias detection tool into your product development pipeline, allowing teams to test for bias at every stage.
3. Design for Transparency and Explainability
Similar to the suggested tools to eliminate AI hallucination (Seriously, it’s worth the read if you haven’t yet), you can build tools that regularly test for bias, ensuring your product remains fair and inclusive.
What to Do:
Show Confidence Scores: Instead of presenting AI outputs as absolute truth, display a confidence level (e.g., “80% certain based on available data”).
Provide Source References: Whenever possible, cite the sources behind AI-generated content. For example, “This summary is based on [Source X] data.”
Use Explainable AI Models: Implement models like decision trees or saliency maps to highlight the factors driving a decision.
Product Feature Idea:
Explainability Panel: Add a feature that explains how the AI arrived at a particular decision, including the key factors and confidence levels.
4. Add Human Oversight for High-Stakes Decisions
Similar to the above, we’ve discussed this to help reduce AI hallucinations. Look, sometimes it’s hard to automate, or there’s no way to automate without human oversight.
What to Do:
Require Human Review: Build workflows where experts review critical AI outputs before they reach users.
Enable Active Learning Loops: Use human corrections to train the AI over time. Every time a human fixes an error, the AI should learn from it.
Crowdsource Verification: For community-driven products, let users flag or correct AI outputs.
Product Feature Idea:
Human-in-the-Loop Workflow: Design a system where high-stakes AI outputs are automatically flagged for human review.
AI biases are a reality. As much (or little) as we want to believe that AI will be the end-all-be-all, AI isn’t infallible, especially in high-stakes scenarios like hiring, healthcare, or financial services. But with the right strategies and tools, we can build AI systems that are fair, inclusive, and trustworthy—systems that not only drive revenue but also make the world a better place.
Let me know your thoughts. If you have any war stories about AI gone wrong, I’d love to hear them.