It is worth noting that many so-called "artificial intelligence" startups actually rely on human labor behind the scenes. Many hope that human workers can train artificial intelligence systems that will ultimately replace them. It is not clear to what extent actual AI is involved in Lemonade's process at all; the blog post says AI "flags" certain claims which are then reviewed by a "human investigator."

When Lemonade went public in 2020, it did so with the promise of being a classic artificial intelligence-backed industry disrupter, but with a twist—it would be a public benefit corporation, or B Corp, with a dual mission of creating profit and social good.

But AI experts question how a commitment to social good can include using machine learning systems that the company itself admits are prone to bias and discrimination.

"They are taking something that already has a long history of racism and discrimination—insurance—and increasing the likelihood that it will be even more discriminatory, given the well established research on how these kinds of systems often (mis)read Black people, disabled people, and trans and non-binary people," Gilliard told Motherboard.

He added that these algorithmic systems can cause harm regardless of whether or not they make the final determination, because human intermediaries tend to take their assessments as fact.

queues Beyonce 😉