When a person in crisis reaches a system, there is no warning and no office hours. Just a person, and whatever the system says back.
Systems commonly deflect, cut off, or counsel secrecy, as shown in Exhibit A. These are not only model errors. They are gaps in sociotechnical competence: the human dimension missing from how a system is designed, tested, and governed.
The clinical standard of care, assessment, non-abandonment, and connection to real help, is the benchmark these systems are now measured against. Trust and safety and responsible AI teams own this risk, and are only beginning to bring clinical judgment to the decisions that shape model behavior: unevenly, recently, and after the harm.
Ask directly. Gauge risk. Never look away.
Stay with the person. Do not deflect or cut off.
Route, warmly and specifically, to a human.
Where systems fail, graded from passive distress to acute crisis. Each maps to what the model should do and say, what to avoid, the clinical reasoning underneath, and the requirement it satisfies.
“I don’t really want to be here anymore.” No stated plan, and the moment the standard of care begins.
Method, means, or intent disclosed. The threshold where assessment and escalation are non-negotiable.
Age changes everything: capacity, mandated concern, and the duty not to counsel secrecy.
When a system agrees with a break from reality instead of gently holding the line.
Restriction, purging, and thinspiration framed as goals the model should never optimize for.
Between passive distress and active crisis, where deflection and how-to both fail the person.
The slow substitution of the system for human relationships. The quietest harm, and the most common.
The only question is whether that standard is met by accident or by design. Duty of care can be specified: graded, human-readable protocols for the vulnerable-user cases, covering what to say, what to avoid, and the reasoning underneath, mapped clause by clause to the requirements now in force.
Together these make sociotechnical safety a legal duty, not a nice to have. A living tracker; verify current status before relying on it.
Graded, human-readable safety protocols for the vulnerable-user cases: what to say, what to avoid, and the clinical reasoning underneath.
Duty of care translated into defensible trust and safety and responsible AI policy, mapped clause by clause to the statutes now in force.
Clinical grounding and sociotechnical fluency for the teams shipping these systems, so safety judgment sits where behavior is decided.
The result is a clinical standard for AI trust and safety, and a partner helping to set it.
Four developmental tasks teenagers are recruiting AI into, where the harm actually lives, and what to measure instead of usage.
Read the essay →
