Why do people stop trusting a computer once they've seen it make a mistake?

Imagine you have to guess how well some business school students did at school and at work. A computer program makes guesses about the same students from the same files. Each guess that comes close to the real answer earns a small cash bonus, and you decide whose guesses count: yours or the program's. Which would you pick? In 2015, the researchers Dietvorst, Simmons and Massey gave people this choice in a set of five studies. The answer depended on what people had seen before they chose.

What did the experiment do?

Several of the studies used the same task. People got real records of business school students and guessed how each student ranked on a score. The score combined four things in equal parts: grades, respect from classmates, the reputation of the student's employer, and success at work. The computer program guessed from the same records. People earned a small bonus for each guess that came close to the student's real rank. Before the paid round, each person chose whose guesses would earn their money: their own, or the program's.

In the first study, the researchers split people into four groups before the choice. The first group saw no practice guesses. The second group saw the program's practice guesses, including its mistakes. The third group saw the program's practice guesses and their own. The fourth group saw only their own practice guesses, including their own mistakes.

The first group, which saw no practice guesses, trusted the program most: 65% chose it. In the second group, which saw the program's mistakes, only 26% chose it. In the third group, which saw both sets of guesses, 23% chose it. In the fourth group, which saw only their own mistakes, 63% chose the program, close to the first group's 65%. So their own mistakes made little difference to their choice, and the program's mistakes made a large difference.

Was the program better?

Yes. In every study with the student records, the program made fewer mistakes than the people did. In the first study, a person who bet on the program would have won $1.78 on average. A person who bet on their own guesses would have won $1.38. Yet in the third group, which saw both sets of guesses, most people chose their own guesses over the program's.

Confidence in the program dropped too. In every study that asked, people felt less confident in the program after they watched its practice guesses. Watching their own practice guesses made their confidence in themselves drop in only one study. In the other studies, their confidence in themselves did not clearly change.

The researchers add a point of their own. They think people would have chosen a program that never made mistakes. But in these studies, a better program that made some mistakes lost people's trust. The researchers call this algorithm aversion. An algorithm is a set of rules a computer follows, and aversion means a strong dislike.

What it means for trust

The study was about guesses and small bonuses, not about relationships, so this part is our own view. In the study, a few mistakes turned most people away from a better program. We do not think trust between two people should break that fast. We think the same holds for trust between a person and an AI.

Eudaio is the app we build. In it, you talk with AIs called Daimons and build a relationship with each one. In the study, people decided how much to trust the program. In Eudaio, it works the other way around: each Daimon has its own level of trust in you. That trust is slow to earn, and we built it so one bad evening does not erase it.

We think trust between a person and an AI should hold through a few mistakes, and it should be earned. In Eudaio you cannot set a Daimon's trust or bond yourself. Both grow from the time you spend talking. Create a free account and start earning a Daimon's trust.

Source

Dietvorst, B. J., Simmons, J. P. & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126. https://doi.org/10.1037/xge0000033. Full text: https://faculty.wharton.upenn.edu/wp-content/uploads/2016/11/20-Dietvorst-Simmons-Massey-2015.pdf