Apple and Google Hit With Demand to Pull AI ‘Nudify’ Apps [Updated]

A pivotal legal challenge has been launched against two of the world’s largest technology companies, Apple and Google, as the San Francisco City Attorney’s office has issued cease-and-desist letters demanding the immediate removal of 13 specific artificial intelligence (AI) "face-swap" applications. These apps, widely available on the App Store and Play Store, are accused of facilitating the creation and distribution of nonconsensual nude images, a practice deemed "illegal, harmful, and completely unacceptable" by city officials. The letters, dispatched on Thursday, July 17, 2026, according to a report by WIRED, mark an escalating effort to hold platform providers accountable for harmful content generated by AI tools hosted on their digital marketplaces.

The legal action targets eight applications found on Apple’s App Store and five on Google’s Play Store. While these apps often market themselves innocuously as "face-swapping" or image manipulation tools, investigations by the City Attorney’s office revealed their primary utility lies in generating explicit deepfake images by digitally "undressing" photos of real individuals without their consent. Beyond merely demanding the removal of these particular applications, the San Francisco City Attorney is pushing for a more profound systemic change, insisting that both Apple and Google sever all ties with the developers responsible and cease collecting any revenue generated from in-app purchases. The core of the argument is that by profiting from these transactions, the tech giants are effectively "aiding and abetting" the sale and proliferation of illicit content. The City Attorney’s office estimates that Apple and Google may have collectively reaped millions of dollars in fees from these morally and legally dubious applications.

The Rise of "Nudify" AI and Its Devastating Impact

The technology underpinning these "nudify" apps leverages sophisticated AI algorithms, particularly deep learning models, to manipulate images and videos. Initially, face-swapping technology gained notoriety through benign, often humorous, applications that allowed users to swap faces with friends or celebrities. However, the rapid advancement and democratization of AI tools have led to a darker application: the creation of nonconsensual intimate imagery (NCII), commonly known as deepfake pornography. These "nudify" apps allow users to upload an image of a person, and the AI then generates a realistic-looking nude version of that individual, often by mapping their face onto existing explicit images or synthesizing new ones.

The proliferation of such tools has had devastating consequences for victims, predominantly women and girls. Research from organizations like Sensity AI and the National Center for Missing and Exploited Children (NCMEC) consistently highlights that deepfake pornography constitutes a significant and growing portion of all nonconsensual intimate imagery. Studies have shown that over 90% of deepfake videos identified online are nonconsensual pornography, with the vast majority targeting women. The psychological impact on victims is severe, including profound emotional distress, reputational damage, social ostracization, and, in some cases, professional repercussions. Unlike traditional image manipulation, AI-generated deepfakes are increasingly difficult to detect and often indistinguishable from genuine photos or videos, further complicating efforts to discredit them and protect victims. The accessibility of these apps on mainstream platforms has exacerbated the problem, making it easier for individuals with malicious intent to create and distribute such harmful content.

A Chronology of Growing Concern and Regulatory Action

The San Francisco City Attorney’s demand is not an isolated incident but rather the latest development in a prolonged struggle against harmful AI-generated content. Concerns over deepfake technology have been mounting for several years:

• Early 2020s: Early iterations of deepfake technology become more accessible, leading to a surge in nonconsensual explicit content. Advocacy groups and cybersecurity researchers begin flagging the issue and calling for platform accountability.
• Throughout 2020-2025: Various reports detail the increasing sophistication and prevalence of deepfake pornography. Major tech platforms, including Apple and Google, face sporadic pressure to address specific apps or content. Both companies have long-standing developer rules prohibiting pornography and similar illicit content, and they have historically removed batches of "nudify" apps when flagged by researchers or media.
• June 2026: In a significant preemptive move, Apple reportedly tightened its App Store guideline language regarding developer responsibility for pornographic content. This update, which emerged publicly in June, aimed to clarify and strengthen the company’s stance, placing more explicit onus on developers to ensure their apps do not facilitate the creation or distribution of such material. This adjustment signaled an awareness within Apple of the escalating issue and a move towards reinforcing their existing policies.
• July 17, 2026 (Thursday): The San Francisco City Attorney sends formal cease-and-desist letters to Apple and Google, specifically naming 13 apps and outlining the legal arguments for their removal and the termination of developer accounts.

This timeline illustrates a gradual recognition and response to the threat posed by these technologies, culminating in the current aggressive legal action by a major municipal authority.

The Legal Framework: Aiding and Abetting Platform Liability

The legal argument put forth by the San Francisco City Attorney is particularly potent: that by hosting these apps and collecting revenue from them, Apple and Google are "aiding and abetting" illegal activity. This legal concept implies that a party, even if not directly committing the primary offense, can be held liable if they knowingly provide substantial assistance or encouragement to another party committing a crime.

In the context of technology platforms, the question of "aiding and abetting" or secondary liability is complex and often debated. Section 230 of the Communications Decency Act in the United States generally shields online platforms from liability for content posted by third-party users. However, this protection is not absolute, particularly when platforms are accused of actively facilitating or profiting from illegal activities, or when they are deemed to have publisher-like control over the content.

The City Attorney’s argument suggests that Apple and Google, through their curated app stores and revenue-sharing models, exercise sufficient control and derive direct financial benefit from these apps to potentially overcome Section 230 protections in this specific context. By providing the infrastructure for distribution, promoting the apps (even implicitly by listing them), and taking a percentage of in-app purchases, they are argued to be more than passive conduits. The "illegal" nature of creating nonconsensual intimate imagery is well-established in many jurisdictions, including California, which has specific laws against revenge porn and the unlawful distribution of explicit deepfakes. The City Attorney’s office is leveraging these statutes to argue that the platforms are complicit in facilitating these violations.

Official Responses from Tech Giants and Broader Industry Implications

Following the demand, both Apple and Google have issued statements confirming their commitment to combating such illicit content.

A Google spokesperson informed WIRED that the company has already taken significant action. "We have deleted ‘hundreds’ of apps with nudifying features for policy violations, including the five named in the letters," the spokesperson stated. This suggests a proactive approach, albeit one that may require external flagging to identify all problematic applications. Google’s ongoing efforts indicate a recognition of the scale of the problem and the necessity of continuous enforcement.

Apple’s response, provided to MacRumors, was more detailed and affirmed the company’s strict stance. "Apple takes reports of apps against its guidelines seriously," the company stated, emphasizing that its App Review Guidelines include specific provisions prohibiting overtly sexual or pornographic content. Apple reiterated that "nudification" apps are explicitly against these guidelines, and the company has "proactively rejected and removed many such apps, including when they have been flagged by users." The company further elaborated on its actions in response to the San Francisco City Attorney’s letters:

"The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are against our App Review Guidelines and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts from our program. We are in contact with four others that need to address policy violations or risk being removed as well."

Apple also directed users to reportaproblem.apple.com to report illegal, offensive, or abusive content, as well as scams or fraud on the App Store, highlighting the role of user vigilance in content moderation.

These responses underscore a common challenge faced by large platforms: the sheer volume of apps and content makes comprehensive, proactive moderation incredibly difficult, often necessitating a reactive approach once issues are identified. While both companies assert their commitment to upholding content policies, the San Francisco City Attorney’s action highlights that enforcement, particularly against rapidly evolving AI-powered tools, may still fall short of what regulators and the public expect.

The Broader Impact and Future Challenges

The San Francisco City Attorney’s aggressive stance against "nudify" apps carries significant implications for the tech industry and the future of digital safety.

• Heightened Platform Accountability: This action could set a precedent for increased legal pressure on platform providers to proactively police AI-generated content. If the "aiding and abetting" argument gains traction, it could lead to a re-evaluation of Section 230 protections and compel platforms to take more responsibility for the downstream effects of technologies they host and profit from.
• Evolving Content Moderation: The incident highlights the continuous arms race between malicious actors leveraging new technologies and platforms trying to moderate them. As AI capabilities advance, so too does the sophistication of harmful content, requiring platforms to invest more heavily in AI-driven detection tools and human review. The tightening of Apple’s guidelines in June 2026 suggests an internal recognition of this evolving threat.
• Consumer Trust and Safety: Ensuring app stores are safe environments is crucial for maintaining consumer trust. Incidents involving harmful deepfake apps erode this trust and necessitate stronger measures to protect users from exploitation and harassment.
• Regulatory Scrutiny: This local action by San Francisco could inspire similar moves by other cities, states, or even federal regulators. The global nature of tech companies means that a patchwork of regulations could emerge, potentially pushing for more harmonized international standards against nonconsensual AI-generated content.
• The Future of AI Ethics: The "nudify" app controversy is a stark reminder of the ethical dilemmas inherent in AI development. While AI offers immense potential for good, its misuse for harmful purposes demands robust ethical frameworks, responsible development practices, and clear legal boundaries. The accessibility of powerful AI models to the public necessitates a focus not just on what AI can do, but what it should do, and how its malicious applications can be effectively curtailed.

As technology continues to advance at an unprecedented pace, the battle against nonconsensual intimate imagery, particularly that generated by AI, will remain a critical front in the broader fight for digital privacy, safety, and ethical technological governance. The San Francisco City Attorney’s legal challenge serves as a potent reminder that platforms, regardless of their size or influence, are increasingly being held accountable for the real-world harms facilitated by the content they host.