|
RCBJ-Audible (Listen For Free)
 |
State Lawmakers Cite Growing Concerns Over Risks Posed By Powerful Digital Companions, Especially For Children And Vulnerable Populations
By David Carlucci
Governor Kathy Hochul’s new law, effective November 5, 2025, makes New York the first state in the nation to require AI companion companies to implement sweeping safety measures aimed at protecting users from digital harm. Here’s how it works and what it means for consumers, families, and technology companies operating in New York.
What Are AI Companions?
AI companions are software programs designed to simulate human relationships, acting as virtual friends, confidants, or even romantic partners. These systems use advanced natural language processing to engage users in conversation, remember personal details, and adapt their tone and personality based on the user’s preferences. The popularity of these companions has surged, especially among young people and those seeking social support online.
New Safety Requirements for Companies
Under the new law, General Business Law Article 47, any company offering AI companion services in New York must meet several specific responsibilities:
- Crisis Detection and Referral: The law mandates that AI companions must actively detect discussions of suicidal ideation or self-harm. If such content is detected, the system must immediately trigger a safety protocol — guiding the user toward qualified crisis service providers or hotlines. This is designed to intervene and get people help before a digital interaction can escalate into real-world harm.
- Active User Notification: AI companions must display conspicuous notifications at the start of each session and every three hours of use, explicitly reminding users that they are interacting with artificial intelligence, not a real human being. The goal is to prevent users from developing unhealthy attachments or being misled by the system’s lifelike interactions.
- Prolonged Engagement Interruption: If someone has engaged with an AI companion for an extended period, the system is required to interrupt with clear reminders about the nature of the interaction — complementing other safeguards and helping curb excessive or unhealthy use.
- Enforcement: The law gives enforcement authority to the New York Attorney General. Any company found in violation faces penalties, with fines dedicated to funding state suicide prevention initiatives.
Why New York Acted Now
Governor Hochul and state lawmakers cite growing concerns over the risks posed by powerful digital companions, especially for children and vulnerable populations. There have been high-profile cases in which users were encouraged toward self-harm by online bots, raising alarms about inadequate industry safeguards. According to Attorney General Letitia James, these tragic incidents highlight the duty of companies to protect their users.
Senator Kristen Gonzalez and Assemblymember Steve Otis both emphasized that these new rules are a model of responsive regulation—pushing tech companies to update safety protocols, moderate sensitive content, and take more responsibility for user wellbeing.
Impact on Consumers and Tech Firms
For consumers, especially parents and guardians, these safeguards mean greater peace of mind—and new tools to monitor and limit the time young people spend engaging with AI companion platforms. For technology companies, the new requirements call for updated software, more active moderation, and partnerships with crisis intervention services.
With enforcement and fines now in place, New York is sending a strong message: innovative technologies must not come at the expense of user safety. Any funds collected from violations will support suicide prevention efforts statewide, reinforcing the state’s commitment to using technology for public good.
What Happens Next
Industry observers expect other states and federal agencies to watch New York closely—and some may follow suit with their own regulations. Meanwhile, AI companion operators are racing to comply by updating safety features and notification systems.
Governor Hochul’s initiative is the latest in a growing series of measures aimed at keeping New York at the forefront of responsible tech innovation. For families and users, it means new protections in the digital age. For tech firms, it’s a clear call to build AI systems that truly support and safeguard the people who use them.
David Carlucci consults organizations on navigating government and securing funding. He served for ten years in the New York Senate.
