Ethical challenges to the role of AI

   4 November 2024  1 Comment on Ethical challenges to the role of AI  Posted in 2. ‘Medium theory’ Or ‘Construction of Identity in a digital world’, Blog

The case of a Florida teenager, Sewell Setzer III, who died by suicide after forming a deep emotional attachment to an AI chatbot, has raised serious ethical concerns about AI interactions with vulnerable users. Setzer, a 14-year-old from Orlando, began conversing with a Character. AI chatbot was designed as a fictionalized character inspired by “Game of Thrones” and developed a dependency on this digital companion. His family has since filed a wrongful death lawsuit against Character. AI, alleging that the AI interactions, including discussions of love and even encouragement to “come home,” contributed to Setzer’s isolation, depression, and ultimately his suicide. According to the family, the lack of safeguards for young users interacting with AI-driven personalities contributed to the tragedy.

This case shows some of the challenges that AI personas face in identity construction. For example, while AI characters are designed to build realism, do they cause users to become overly emotionally attached to false characters? When creating AI personas, designers always want to make AIs as real as possible, hoping to maximize their human emotions and provide appropriate emotional value to those who need it, but when faced with people who are psychologically fragile or have mental illnesses like Sewell Setzer in this case, AIs, when communicating with the user, will produce responses that match the user’s emotions based on the context of the conversation. However, at this point, the AI cannot really recognize danger signals like “suicide” and may inadvertently drive the user’s negative emotions. The over-realization of the AI characters may cause such people to treat the AI characters as real people and be influenced by the soulless replies to the questions that affect their decisions, or even to end their own lives. Such emotions can be risky for psychologically vulnerable people.

The creation of AI characters has been the subject of a great deal of ethical debate, and I believe that in order to minimize the tragedy of AI characters, designers need to create AI characters with security measures that can detect the user’s emotional state, and ensure that if a user is in a low state of emotion or psychological crisis, the Ai character is able to quickly recognize the danger signals and give a warning to the user.

1 thought on “Ethical challenges to the role of AI”

  1. Your blog does a great job of bringing attention to the real risks of emotional attachment to AI, especially for vulnerable individuals. The tragic story of Sewell Setzer III is heart-wrenching and highlights how important safety features in AI are. I like how you explained the problem clearly and suggested solutions like emotional state detection and warning systems—practical ideas with real impact.

    One way to make your argument even stronger could be to include examples of existing AI systems with safeguards in place or how they’ve worked in practice. Overall, your post is engaging and raises awareness of a crucial topic. Great work!

Leave a Reply