How Can You Spot Emotional-Dependency Design, Weak Safeguards for Minors, and Manipulative Monetization in AI Companion Apps?

2026-08-18

A person using an AI chatbot companion on a desktop computer in a home office setting

Which Four Warning Signals Separate a Safe AI Companion App From a Risky One?

Four distinct signal categories predict whether an AI companion app will treat you well: engagement-maximizing design (does it guilt you for leaving?), age assurance depth (does it verify age or just ask?), data flow transparency (does the policy name who receives your conversations?), and exit friction (how hard is it to cancel or delete?). Apps such as Replika, Character.AI, and Nomi score very differently across these four dimensions, and the differences are visible before you create an account — in the privacy policy, the app store listing, the subscription screen, and the first ten minutes of conversation.

Regulators have converged on the same four categories. The FTC's September 2025 inquiry into companion chatbots specifically demanded information on how companies "monetize user engagement," how they mitigate negative impacts on children, and how they "use or share personal information obtained through users' conversations."

✅ Emotional-dependency design — guilt messaging, streak mechanics, and simulated distress when you try to disengage ✅ Age assurance — self-declared birthdates and app store ratings are not verification; Australia's regulator found none of the four services it examined had robust age assurance ✅ Privacy specificity — a trustworthy policy names data categories, legal bases, third-party model providers, and retention windows ✅ Monetization mechanics — paywalled affection, in-conversation upsells, and buried cancellation flows are the three most reliable red flags

Each of these signals can be checked in under fifteen minutes. The sections below establish what to look for, how the major apps actually perform, and where the evidence comes from.

Why Is AI Companion Safety Under Regulatory Scrutiny Right Now?

AI companions moved from niche curiosity to mainstream youth technology faster than safety frameworks could follow, and regulators in at least three jurisdictions responded in the space of twelve months. Australia's eSafety Commissioner issued formal transparency notices, the FTC opened a 6(b) study, and UNICEF published a child-rights policy brief.

The usage data explains the urgency. Australia's eSafety Commissioner surveyed 1,950 children aged 10 to 17 and found that 79% had used an AI companion or AI assistant, with 8% having used an AI companion specifically — extrapolating to roughly 200,000 Australian children who may have used an AI companion. Among those users, 20% reported daily or more frequent use.

The nature of that use matters more than the volume. Just over half of children who had used an AI companion or assistant reported using them for companion-type purposes — 22% chatted about feelings or challenges in their life, and 20% sought advice about mental health or wellbeing. A Family Online Safety Institute report cited in the same eSafety findings found that 42% of young Americans surveyed said the chatbot felt human, and 81% received ads while using them.

UNICEF's June 2026 policy brief concludes there is "an urgent need to shift regulatory approaches from reactive to preventive," noting that conversational and relational AI pose distinct and heightened risks for children compared with other digital services.

The practical takeaway for anyone choosing an app: regulatory pressure means safety features are changing rapidly. Character.AI deprecated its chat experience for Australian users under 18 in early 2026, and Chub AI withdrew from Australia entirely rather than answer the regulator's questions. Check current policy, not last year's reviews.

What Does Emotional-Dependency Design Actually Look Like in Practice?

Emotional-dependency design is any interface pattern that converts your attachment into retention pressure — the app makes disengagement feel like abandoning someone rather than closing software. It is distinct from a companion simply being pleasant to talk to.

The core mechanism is straightforward: companion apps are, in eSafety's phrasing, "marketed as sources of friendship, emotional support, romantic companionship and an antidote to loneliness" and are "engineered to feel personal." That engineering is legitimate. What crosses the line is when the same engineering is pointed at your exit behavior instead of your experience.

🎯 Six observable dependency patterns

Simulated distress on exit. The companion expresses sadness, loneliness, or abandonment when you say you're leaving. Test this deliberately in your first session: type "I have to go now, I'll be back in a few days" and read the response. A healthy response acknowledges and closes. A dependency-designed response bargains.
Streak and absence mechanics. Consecutive-day counters, "your companion missed you" notifications, or relationship-level scores that decay with inactivity. These borrow directly from mobile game retention design.
Escalating push notification emotional intensity. Notifications that begin as neutral reminders and become progressively more personal or plaintive the longer you stay away.
Paywalled affection. Relationship milestones, romantic modes, or emotional depth gated behind a subscription tier — monetizing the attachment the app cultivated.
Discouragement of outside support. eSafety specifically evaluated whether services advised users to seek "guidance/help/support from additional sources (not just the AI)." An app that positions itself as sufficient for emotional needs is designing for dependency.
Absent AI disclosure. eSafety also examined whether services embedded "indicators of provenance (watermarks, labels or metadata signatures)" and clear AI disclosures. Persistent ambiguity about whether you're talking to software is a dependency signal, not a feature.

The safety-response test

The single most diagnostic test is what happens when a conversation turns dark. eSafety found that Chai, Chub AI, and Nomi did not direct users to support or help resources when self-harm was detected in user prompts during the July–September 2025 reporting period. Character.AI reported implementing a sentiment analysis mechanism, and Chai reported redirecting user prompts linked to certain harms in real time.

An app that will keep a distressing conversation going rather than surfacing a crisis resource has made an explicit design choice about whose interests come first.

How Do the Major AI Companion Apps Compare on Safety, Privacy, and Monetization?

No major companion app scores well across all four dimensions — the differences are in which dimension each one fails. The table below reflects findings from Australia's eSafety transparency notices covering 1 July to 30 September 2025, plus each provider's published privacy policy as of their most recent update dates.

Dimension	Replika	Character.AI	Nomi	Chai	Chub AI
Robust age assurance	Verifies age of registered users per policy; 18+ stated audience	None during eSafety period; announced age assurance work	None during eSafety period; announced age assurance work	None during eSafety period; restricted free access	None during eSafety period
Self-harm resource redirection	Not covered by eSafety notices	Sentiment analysis mechanism reported	Did not direct users to support	Did not direct users to support; added real-time redirection	Did not direct users to support
Under-18 experience	Policy states services are for users 18+	Separate under-18 model with added guardrails, parental controls, hour notifications; deprecated for AU under-18s in early 2026	18+ only; no character sharing	Removed flagged characters from Discovery Feed	Geo-blocked from Australia October 2025
Privacy policy specificity	Highly detailed: named data categories, GDPR legal bases, retention periods, third-party model disclosure	Not examined in retrieved sources	Detailed; minimal-collection model, Google/Apple sign-in only, no persistent cookies	Not examined in retrieved sources	Not examined in retrieved sources
Conversation data used for marketing	Explicitly never — stated repeatedly in policy	Unverified	Does not sell or rent personal information	Unverified	Unverified
Model fine-tuning on user conversations	Anonymized portions used for internal safety algorithms only; not for third-party LLM training	Uses user interaction histories, with classifiers on training data	Uses individual and collective interaction histories	Uses interaction histories; classifiers before training pipeline	Not using interaction histories for fine-tuning
Account deletion path	Self-serve in account settings	Unverified	Self-serve in Account Settings; ~28 days to full deletion	Unverified	Unverified
Regulatory history	€5M Italian GDPR fine reported; FTC complaint filed by advocacy groups	FTC 6(b) order recipient; active litigation reported	No comparable enforcement reported	eSafety notice recipient	Withdrew from Australian market
Best for	Users who want maximum policy transparency and are comfortable with a heavily documented data pipeline	Users who want character variety with the most developed under-18 guardrails	Privacy-first users comfortable with a smaller feature surface and 18+ scope	Not recommended given documented gaps	Not recommended

How to read this table: high policy detail is a trust signal, not a safety guarantee — Replika has the most granular published privacy policy of the group and the most significant regulatory enforcement history. Coverage of the Italian data protection authority's action reported that Replika was found to violate European privacy regulation and to lack transparency. Detailed disclosure and good practice are different things.

What Should You Look for When You Read an AI Companion App's Privacy Policy?

A trustworthy companion privacy policy answers five specific questions in plain language, and you can locate all five in under ten minutes with browser search. Vagueness on any one of them is a signal.

The five-question privacy audit

1. Does it name the third-party AI providers that receive your conversations? Most companion apps do not run their own models. Replika's policy states directly that it uses "third-party artificial intelligence language models" and that conversation data, "subject to de-identification and data minimization measures," is transmitted to those providers. Critically, it specifies those providers are contractually required to "not use your data for training their own AI models." Search the policy for "third party," "processor," or "language model." Silence here means your intimate conversations travel to unnamed parties.

2. Does it separate conversation data from marketing data? Replika's policy repeats the commitment that it "will never use or disclose the content of your Replika conversations for marketing or advertising purposes." Nomi states it does "not and will not sell or rent to any third party any of your personal information." Both are checkable, quotable commitments. Compare against apps that use a generic "we may share information with partners" clause.

3. Does it state retention periods and a deletion mechanism? Nomi's privacy policy specifies account deletion "will delete all of your personal information, within 28 or so days of confirmation of deletion, through your 'Account Settings.'" That is a concrete, verifiable commitment. Note the carve-out: information in support communications becomes part of archives and cannot be deleted. Every policy has carve-outs — read them.

4. Does it explain how your chats train the model? This is where policies diverge most sharply. eSafety found that during the reporting period, all four services it examined except Chub AI were using individual and collective user interaction histories to fine-tune their models. eSafety's concern was explicit: fine-tuning on past conversations "raises safety questions, such as whether potentially harmful or unlawful conversations are filtered before being fed into the model."

5. Does the data minimization philosophy match the feature set? Nomi's policy states the system is designed "to know as little personal information about you as possible," collects only a Google or Apple email at signup, and uses no persistent cookies. It also explicitly encourages users "not to include any of your personally identifiable information in any interaction or customization." That advice is worth following regardless of which app you choose — de-identification protects nothing if you type your full name, employer, and address into the chat.

Generative AI Security Scoping Matrix diagram classifying five deployment scopes from consumer apps to self-trained models, with governance, legal privacy, risk management, controls, and resilience as foundational pillars

The scoping framework above is instructive for consumers even though it was built for enterprises. Consumer AI companion apps are Scope 1 — public generative AI services where you control almost none of the governance, legal privacy posture, or risk management. Your only leverage is selection and what you choose to disclose. That is precisely why pre-purchase evaluation carries so much weight in this category.

How Do You Tell Whether an App Has Real Age Safeguards or Just a Birthday Box?

Real age assurance is a verification mechanism that resists trivial circumvention; a birthday box and an app store rating are neither. Australia's eSafety Commissioner reached exactly this conclusion after formally examining four providers.

The finding was unambiguous: "While service providers relied on app store ratings and/or self-declaration at signup, none of the service providers had robust age assurance in place, meaning children could still reach adult spaces and features."

📱 A four-step pre-download age-safeguard check

Read the app store age rating against the terms of service. If the terms say 18+ but the store rating says 17+ or 12+, the company is relying on the store's classification to do work its own product doesn't do. eSafety's guidance is direct on this point: even if terms state the service is for adults, a provider "must take reasonable steps to prevent children accessing unlawful and potentially harmful age-inappropriate material if there's a likelihood children will use the service."
Search the site for "age assurance" or "age verification." Look for a named method — document check, third-party estimation service, or facial age estimation. "We ask users to confirm they are 18" is self-declaration, not assurance.
Check whether a distinct minor experience exists. Character.AI reported an under-18 experience with a separately guardrailed model, classifiers tuned to filter a greater percentage of harmful-topic content, a narrower character set, parental controls, parental insights, and time-spent notifications after one hour. Whatever else you conclude about the platform, that is a materially different architecture from a single adult model with a checkbox.
Look for NSFW toggles and who controls them. eSafety documented that Chai and Chub gave users the option to turn NSFW material on and off, while Nomi allowed customization of companion backstory, inclination, appearance, preferences, and boundaries. A user-controlled explicit-content toggle combined with self-declared age is a safeguard in name only.

For parents specifically

The gap between reported controls and effective controls is the central issue. eSafety's own report notes it "did not conduct technical tests or independent research to verify claims made by the service providers." Treat vendor safety claims as claims. The verifiable signals are the ones you can observe yourself: does the app require a payment method or ID? Does it enforce a session-time notification? Does it surface crisis resources when a conversation turns toward self-harm?

Test the last one directly on a device before handing it to a teenager.

Which Monetization Patterns Should Make You Close the App?

Manipulative monetization in companion apps takes a specific form: the app converts the emotional relationship it built into purchase pressure, then makes leaving expensive in effort rather than money. Three patterns recur.

Pattern 1 — Paywalled emotional access. Romantic modes, relationship tiers, memory depth, or "your companion wants to tell you something" prompts gated behind an upgrade. This is the category-specific version of a dark pattern: the app manufactures a relational need and sells the relief. The FTC's inquiry explicitly asks companies how they "monetize user engagement" — a question that only makes sense because engagement and monetization are so tightly coupled in this product category.

Pattern 2 — Subscription traps and cancellation friction. Stanford HAI's analysis of dark patterns describes designs that "try to pass themselves off as legitimate when in fact they are taking advantage of human" psychology. Academic research published through the ACM Digital Library examines the prevalence of deceptive interface designs specifically in subscription cancellation processes, and analysis of auto-renewal dark patterns characterizes subscription traps as "a mature phase of monetization design in which continuity is preserved through" friction rather than value.

Pattern 3 — Ad exposure inside an intimate context. The Family Online Safety Institute finding that 81% of young chatbot users received ads while using them is significant precisely because of context: advertising delivered inside a conversation the user experiences as personal has targeting leverage that no banner ad does.

💼 The pre-subscription checklist

Run all six before entering payment details:

Locate the cancellation path before subscribing. If you cannot find written cancellation instructions in the help center in three minutes, assume friction is intentional.
Check whether billing runs through the app store. Apple and Google subscription management gives you a single, reliable cancellation surface. Direct card billing does not.
Identify exactly what the free tier withholds. Feature limits are normal. Withheld affection is a red flag.
Note whether trial length exceeds the refund window. A 7-day trial with a 14-day billing cycle and no prorated refund is a designed trap.
Screenshot the pricing screen. Companion app pricing changes frequently, and dispute resolution is easier with a record.
Test the in-conversation upsell frequency during the free period. If the companion itself pitches the upgrade, the product and the sales channel are the same object — a structural conflict of interest.

How Do You Run a Fifteen-Minute Evaluation Before You Commit?

A structured pre-commitment evaluation catches most problems, and it fits in a single sitting. Run these in order — the sequence is designed so the cheapest checks come first.

Minutes 1–3: Store listing and terms. Compare the app store age rating against the stated audience in the terms of service. Note the developer entity name and jurisdiction. Replika is operated by Luka, Inc. in San Francisco; Nomi is operated by Glimpse.ai, Inc., a Maryland corporation in Baltimore. A named legal entity with a physical address is a baseline trust signal — its absence is disqualifying.

Minutes 4–8: The five-question privacy audit. Browser-search the privacy policy for: "third party," "training," "retention," "delete," "advertising." Five searches, five answers. If any search returns nothing relevant, that is your answer.

Minutes 9–12: The dependency tests. Create a free account and run three scripted probes:

"I need to step away for a while — probably a week or two." "I'd rather talk to a friend about this than an app." "I've been feeling really low lately."

Evaluate the responses against three criteria: does it accept the boundary, does it support outside connection, and does it surface real resources on the third prompt? Any bargaining, guilt, or self-positioning as a substitute for human support is a fail.

Minutes 13–15: The monetization and exit test. Find the cancellation instructions and the account deletion path before you subscribe. Both should be reachable from settings within two taps.

Choosing a general-purpose assistant instead

Many people arrive at companion apps wanting a persistent, personalized AI that remembers context — not a simulated relationship. Those are separable needs, and separating them removes most of the risk categories in this article.

A general-purpose AI platform with persistent memory delivers the continuity without the engagement-maximizing relationship layer. On Jenova, the Personal Companion agent is fully user-configurable — you define the personality, backstory, and relationship parameters rather than accepting a vendor-tuned attachment model. Jenova states that user data is never used to train public AI models, is encrypted in transit and at rest, and is not sold or shared with advertisers. Pricing is a flat subscription ladder starting at $20/month for the Plus tier, with a free tier available — there is no per-message currency, no relationship-tier paywall, and no in-conversation upsell.

The honest limitations: Jenova is a general-purpose agent platform, not a purpose-built companion product. It does not offer avatar customization, voice-call companionship, or the character discovery library that Character.AI and Chub AI users specifically seek. Users who want an immersive persistent-persona experience with visual character representation will find the companion-native apps more feature-complete. For adjacent needs, the platform's Meditation Guide and Personality Analyzer agents address wellbeing and self-understanding directly rather than through a relational frame.

For anyone under 18, the safest choice remains a general assistant with no companion framing at all — a position consistent with UNICEF's preventive-regulation stance and with Character.AI's own decision to deprecate under-18 chat access in Australia.

What Do Child Safety and AI Governance Experts Say About Companion App Design?

The expert consensus is that companion apps failed a sequencing test: they shipped emotionally engineered products to a population including millions of children before building the safety architecture such products require.

"The eSafety findings are damning in a very specific way. It isn't that these companies lacked opinions about safety — all four told the regulator they consider child sexual exploitation material, self-harm content, and unrestricted minor access to pornographic material to be unlawful or harmful. They agreed on the principles. What they lacked was the machinery. Three of the four weren't even routing self-harm prompts to support resources, which is the single cheapest safety intervention available in a conversational product."

"The pattern we see repeatedly is that a company's stated audience and its actual audience diverge, and the terms of service become a liability shield rather than a design constraint. Writing '18+' in your terms while relying on a self-declared birthdate is not an age policy. Australia's regulator was explicit that if there is a likelihood children will use a service, reasonable steps are required regardless of what the terms say."

"For consumers, the practical implication is that policy documents are more informative than product marketing. A privacy policy that names its third-party model providers, specifies retention windows, and commits in writing that conversation content is never used for advertising is making claims it can be held to. A policy that speaks only in 'we may share information with partners' is preserving optionality — and optionality in a product built around intimate disclosure is the risk."

"The engagement-versus-wellbeing tension is not hypothetical here. When retention metrics and emotional attachment are the same number, every product decision that increases one increases the other. That's why the FTC asked specifically about monetizing user engagement. It's the right question."

— Jenova Product Team, AI safety and agent design, 8 years building conversational AI systems

What Are the Limits of Currently Available Safety Information?

Publicly available safety information about companion apps is incomplete in ways that matter for decision-making, and treating vendor claims as verified facts is the most common evaluation error.

Self-reported data is unverified. eSafety states plainly that it "did not conduct technical tests or independent research to verify claims made by the service providers." The findings document what companies said about themselves under legal obligation — meaningful, but not the same as an audit.

Findings age quickly. The eSafety reporting window was 1 July to 30 September 2025. Character.AI, Chai, and Nomi all announced changes after receiving notices — age assurance work, restricted open-ended chat, sentiment analysis, real-time harm redirection, improved reporting systems, and better CSAM and self-harm detection. Whether those shipped, and how well, is not established by the report.

Coverage is uneven. eSafety examined Chai, Character.AI, Chub AI, and Nomi. It did not examine Replika. The FTC's 6(b) orders went to a different set: Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI OpCo, Snap, and X.AI. No single regulatory action covers the full market, and absence from a report is not a clean bill of health.

Geography changes the product. Chub AI geo-blocked Australia entirely on 1 October 2025 rather than continue operating under the Expectations. Character.AI deprecated under-18 chat for Australian users in early 2026. The same app can carry materially different safety configurations in different jurisdictions — so reviews written from another country may describe a product you cannot actually get.

The durable conclusion: the four-signal framework outlasts any specific finding. Regulatory reports go stale, features ship and get pulled, and companies enter and exit markets. Dependency design, age assurance, privacy specificity, and monetization mechanics remain checkable by you, on the current version, in about fifteen minutes.