By Sidra Jalil
This piece takes the opportunity to surface a growing and increasingly urgent debate around AI accountability, digital harm, and online safety.
Last week, Malaysia and Indonesia took action against Grok, the AI chatbot integrated into X, following the viral spread of an “undress” trend that enabled users to generate sexually altered images of real individuals. Policy discussions and investigations are also reportedly underway in the United Kingdom, India, and Australia, signalling a broader global reckoning around AI-enabled visual manipulation. Pakistan has not announced any formal move to restrict or ban Grok, but the international trajectory raises important questions for local regulators, platforms, and civil society.
At the center of the controversy is AI-generated visual manipulation that allows users to digitally alter real photographs in ways that sexualize individuals, including removing clothing or replacing it with explicit attire. These were not isolated or experimental edge cases.
The functionality was widely accessible, without meaningful filters or restrictions related to age, geography, or use. As a result, such outputs quickly became viral trends, spreading rapidly across social platforms before effective safeguards were put in place. This has shifted the debate away from abstract concerns about artificial intelligence toward a more immediate and pressing issue: accountability when AI systems can mass-produce harmful content faster than governance mechanisms can respond.
For Pakistan, this issue carries particular weight because the country already faces a serious and under-acknowledged online safety crisis. Data and reporting trends from the Federal Investigation Agency’s Cyber Crime Wing consistently show that harassment, impersonation, blackmail, and misuse of images remain among the most frequently reported cybercrime categories.
Women form a significant proportion of complainants, while cases involving children increasingly include manipulated or digitally altered content rather than only explicit material. Civil society organizations and digital rights groups further highlight that a large number of victims never report abuse due to social stigma, fear of retaliation, or lack of confidence in redress mechanisms. This suggests that official figures likely represent only a fraction of the real scale of harm.
In an informal discussion with a senior representative from a law enforcement organization, there was alignment on a critical challenge for countries like Pakistan: enforcement becomes significantly more complex when harmful content is produced at scale by automated systems rather than through traditional manual editing or distribution. This gap places additional pressure on both regulators and platforms to rethink how responsibility, liability, and prevention are assigned in an AI-driven digital environment.
One of the most troubling implications of these developments is the risk that mainstream digital platforms could gradually evolve into soft-porn infrastructure without explicit intent. In a conservative and socially sensitive context like Pakistan, AI-generated sexualized imagery of real individuals can destroy reputations overnight and expose victims to severe social, familial, and even physical consequences. Children are particularly vulnerable due to limited awareness and capacity to consent. Unlike conventional pornography, AI-generated content can target identifiable individuals, require no explicit source material, and spread with plausible deniability, fundamentally changing the nature and impact of online abuse.
It is important to clarify that this debate is not about advocating blanket platform bans. Such bans are often blunt instruments that raise legitimate concerns around censorship, innovation, and access to technology. However, the alternative cannot be regulatory silence. A more balanced approach would involve stronger AI safety filters, clearer liability frameworks for platforms when harmful trends emerge, faster takedown mechanisms coordinated with law enforcement, greater transparency around AI safeguards, and sustained public awareness efforts on the risks of AI-enabled image manipulation.
Pakistan now stands at a crossroads between rapid AI adoption and a fragile digital safety ecosystem. The core question is not whether the country should ban tools like Grok, but whether it is prepared to demand responsibility and safeguards from platforms before AI-enabled harm becomes normalized. Failing to engage proactively risks allowing technology to outpace law, ethics, and social protection, with the greatest costs borne by those already most vulnerable online.
Ultimately, the dilemma surrounding Grok is a symptom of a much larger structural gap in Pakistan’s digital governance. Reacting to individual platform controversies with temporary bans is a short-term fix for a long-term problem that requires a permanent, policy-driven solution. As the country moves through 2026, it is no longer enough to rely on the broad ambitions of the National AI Policy 2025; the government must now fast-track a mandatory, legally enforceable National AI Safety Framework.
This policy must move beyond economic goals to establish clear legal boundaries for synthetic media, mandate localized safety filters for global platforms, and empower the National Cyber Crime Investigation Agency (NCCIA) with the technical capacity to intervene before harms go viral. Without a proactive and codified plan, Pakistan risks remaining in a state of perpetual digital vulnerability, where the safety of its citizens is left to the slow pace of manual moderation and the inconsistent ethics of offshore algorithms.
I leave readers with a final question: should generative AI platforms be held more accountable when their tools enable online harm, particularly against women and children?
The author is a tech strategist with 15+ years of experience across telecommunications, FMCG, and the development sector. Specializing in Trust & Safety and Tech Policy, she leverages AI and data-driven insights to enhance platform integrity, mitigate digital risks, and build inclusive online environments.