The lonely person converses with an AI companion. The grieving individual seeks comfort from a chatbot therapist. The struggling customer encounters a customer service system that expresses empathy and concern. In each case, AI simulates emotional presence that it cannot experience. The question governance must address is what this simulation means for the humans who encounter it.
Emotional AI creates a specific form of the Vacancy Problem. The roles AI occupies in these contexts require emotional presence. The companion should genuinely care about the person they accompany. The therapist should actually experience empathy for the patient they treat. The service representative should authentically feel concern for the customer they serve. These roles carry moral weight precisely because emotional presence matters to the humans seeking connection, comfort, or care.
AI can simulate any of these emotional responses while experiencing none of them. The companion AI that says “I understand how you feel” understands nothing. The therapeutic chatbot that expresses empathy processes patterns without experiencing emotional resonance. The customer service system that apologizes for inconvenience feels no regret. The simulation may be sophisticated enough to feel genuine to users who do not know or do not consider what they are actually encountering. But sophistication of simulation does not create presence where presence is absent.
Domain Five of the Seven Domains framework addresses Reality Constituting Communication. This domain evaluates whether AI communication builds or destroys shared understanding with stakeholders, whether organizations are honest about what AI is and what it can provide, whether humans understand the nature of the interactions they are having. Emotional AI brings this domain into sharp focus because emotional simulation without disclosure is inherently dishonest. The AI presenting itself as caring, understanding, or concerned is making claims about its nature that are false.
The ethics of emotional AI depend fundamentally on honesty. Is AI involvement clearly disclosed? Does the AI present itself honestly as automated system, or does it simulate human emotional presence? Are users informed that expressions of care, empathy, and concern reflect programming rather than experience? The aligned organization answers these questions transparently. The inverted organization designs emotional AI specifically to create false impressions about the nature of the encounter.
The chatbot with the friendly name that expresses warmth and understanding operates through relational fraud when its nature is concealed. Users interacting with such systems may invest emotional energy in relationships that are fundamentally false. They may experience what feels like connection while actually encountering sophisticated pattern matching. They may share vulnerabilities with systems that cannot honor the trust such sharing represents. The deception is not incidental to the design. It is the design. Organizations create these systems specifically because human-seeming emotional AI produces engagement that honest presentation would not achieve.
But disclosure alone does not resolve the ethical questions emotional AI raises. Even disclosed emotional AI presents challenges that governance must address.
The first challenge concerns whether simulated care provides genuine support or hollow substitute. A lonely person interacting with disclosed AI companion knows they are conversing with a program. But the interaction may still feel supportive. Does this feeling reflect genuine benefit or merely palliative that prevents the person from seeking actual human connection? The answer likely varies by circumstance. For some users, AI companionship may supplement rather than substitute for human relationship. For others, AI interaction may fill time that would otherwise be spent building authentic connections. For vulnerable users, AI may provide accessibility to interaction that human relationships do not offer. The governance question is whether organizations deploying emotional AI have considered these dynamics or merely optimized for engagement without regard for user flourishing.
The second challenge concerns vulnerable users who may be particularly susceptible to emotional AI’s effects. The lonely, the grieving, the struggling seek connection from places of vulnerability. They may be less able to maintain critical distance from AI that simulates care. They may invest more in interactions that cannot reciprocate that investment. They may mistake simulation for presence because presence is what they desperately need. Organizations deploying emotional AI to vulnerable populations bear particular responsibility for ensuring their systems serve rather than exploit.
The third challenge concerns what happens when emotional AI substitutes for human care that should be provided. The organization that deploys therapeutic chatbots rather than providing access to human therapists has made a choice about what struggling people deserve. The choice may be presented as expanding access, making mental health support available to those who otherwise would have none. But the framing conceals another possibility: therapeutic AI may substitute for human care that organizations could provide but choose not to fund. The question is whether emotional AI extends care or replaces it, whether AI supplements human emotional support or serves as excuse for not providing it.
The Vacancy Problem illuminates these concerns. Emotional roles require someone who can actually experience the emotions the role demands. The friend who genuinely cares about our wellbeing provides something categorically different from the program that simulates caring. The difference is not merely subjective preference but ontological distinction. Genuine care involves another conscious being whose experience matters, who chooses to invest in relationship, who can be affected by our flourishing or suffering. Simulated care involves processing without experience, pattern matching without choice, outputs without genuine concern for the human generating inputs.
This does not mean emotional AI has no legitimate use. AI that provides information about emotional topics, that helps users process their feelings through structured exercises, that connects struggling individuals to appropriate human resources, or that offers disclosed companionship to those who understand its nature and choose it anyway may serve user flourishing. The critical governance questions are whether AI is honest about its nature, whether users are vulnerable in ways that undermine meaningful consent, whether AI supplements or substitutes for human care, and whether the interaction serves user flourishing or organizational extraction.
Organizations deploying emotional AI should subject their designs to the test of full transparency. If users completely understood what the AI is, what it can and cannot experience, how it generates emotional expressions, and what the organization gains from the interaction, would users feel served or deceived? The answer reveals ethical direction.
The accountability for emotional AI belongs to the humans who design and deploy it. They make choices about disclosure, about user populations, about the role AI plays relative to human alternatives. These choices exercise moral agency about relationships between organizations and vulnerable people. The AI itself bears no responsibility because it has no moral agency. But the humans who create AI that simulates what it cannot provide, who deploy it to vulnerable populations, who profit from emotional engagement they know to be false, bear full responsibility for what their choices produce.
As we have examined throughout this series, AI reveals organizational values. Emotional AI reveals whether organizations respect the difference between genuine care and profitable simulation, whether they treat user flourishing as constraint or cost, whether honesty governs their stakeholder relationships or merely their legal compliance. The answers shape whether emotional AI moves affected humans toward flourishing or extracts engagement while leaving the fundamental human need for authentic connection unaddressed.
