Hawker News

Iranian campaign planted fake articles in real U.S. publications using ChatGPT

washingtonpost.com

187 pointsby cainxinth167 comments

https://archive.is/EFOhL
chvid[5 comments hidden]
"But ChatGPT maker OpenAI said Thursday that Hoskins’s purported work was generated using its chatbot by users based in Iran as part of an “influence campaign.”"

Sock puppet accounts and fake opinion pieces are nothing new.

The new thing is a super centralized "tool" that besides doing your writing for you, logs everything, judges it, and reports to authorities and news outlets as it sees fit.

estearum[2 comments hidden]
... well that's one thing that's new.

The other thing that's new (and more structurally relevant, since obviously not all LLMs do what you describe) is the dramatic reduction in cost affiliated with this strategy.

It's (literally) unbelievable that this many allegedly sophisticated people on HN have no concept of cost and what dramatic cost reductions can achieve.

intended[hidden]
Structural issues, the way LLMs and even social media behave at scale are the heart of the problems being faced.

There's new socio economic groups that are "profitable" to target with phishing attacks, because LLMs made it cheaper in 2024-25.

MSFT_Edging[2 comments hidden]
This is no different than Chatgpt picking up a RadioFree* think piece in the grand scheme.
stogot[hidden]
Radio free is not sanctioned
OgsyedIE[33 comments hidden]
Given the plurality of self-hosting options there is no reason to believe that it's not that case that every government on Earth is doing it to some extent.

Perhaps the real problem is too much acceptance of shoddy verification.

gustavus[hidden]
I don't think it is a verification problem. It wasn't like before in the days of controlled media and radio there wasn't a narrative to push, it's just that most people didnt have any awareness of alternatives. Nowadays people are more aware of alternative viewpoints or perspectives. The solution isn't to try and force one approved source of truth, but to just assume everyone is trying to push their version of the truth and examine it critically.

My understanding is that this is the way France operates where you assume all the media is hostile and you view it all critically. In America this is a bit of a shock because for a long time we had trusted arbiters of truth and now that they are less trusted we've been trying to look for alternatives rather realizing it should all be examined critically.

capitainenemo[31 comments hidden]
There's also the other side of it, poisoning the AIs... https://www.newsguardtech.com/special-reports/moscow-based-g...

"Massive amounts of Russian propaganda — 3,600,000 articles in 2024 — are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda."

in 2024.. hopefully filters when training have gotten better, but maybe not..

capitainenemo[4 comments hidden]
(I do find it interesting, btw, that a few seconds after post this did get an immediate downvote to 0 - it didn't last, but it makes me wonder if there are bots monitoring the HN firehose for anything critical of country X)
SoftTalker[hidden]
Possible, certainly. On the one hand the HN audience is not very big in the grand scheme of things, but they probably do skew to more influential and likely to be politically active than the average person. If it's worth doing, it's almost certainly being done.
mdp2021[2 comments hidden]
> if there are

There are easier explanations. A hundred people are checking and occasionally the page, with new posts added every couple of minutes: clearly some will meet a post at "0 minutes", read it as fresh, and possibly react. I have seen it.

capitainenemo[hidden]
sure, that's simplest scenario, just seemed odd that was the first vote. and it's not like writing or running a bot is hard and political stories seem like they'd be a popular target to me.
grafmax[2 comments hidden]
Those Russian propagandists. Lucky the US doesn't engage in propaganda.
stickfigure[hidden]
There's something special about propaganda served up with hot polonium tea.
vkou[hidden]
What about other hostile propaganda? Do you think the completely deranged, self-serving, lying, dangerous shit that regularly comes out of DC is being incorporated in those training materials?

This regime and its enablers have done more quantifiable harm to more people and institutions in America than anything Russia has, and one of its favorite tools is propaganda.

rayiner[15 comments hidden]
I wonder how much Russian propaganda worked its way into western academia over the 20th century? Very similar.
solid_fuel[12 comments hidden]
Somehow I am not surprised to see you repeating the red scare here. This attitude is anti-intellectual and destructive.
rayiner[11 comments hidden]
The New York Times did a multi-article hagiography of communism on the 100th anniversary of the russian revolution, with articles like “how to parent like a bolshevik.” https://www.nytimes.com/column/red-century. Insofar as “scare” implies a reaction to something that isn’t real—no, there was no “red scare.” The communist infiltration was real.
solid_fuel[10 comments hidden]
The New York times publishing an article you don’t like is not “communist infiltration”. In this country, we have a right to free speech, including free press.
rayiner[9 comments hidden]
Wistful coverage of the Soviet Union in 2017 is communist infiltration. Sure, communists have the right to free speech. But I'm likewise entitled to call them what they are.
jrflowers[8 comments hidden]
You think Soviets were writing in the New York Times three decades after the collapse of the USSR?
rayiner[7 comments hidden]
I think people one-shotted by "Russian propaganda" were working at the NYT in 2017, yes.

Remember, the discussion upthread was about Russian propaganda making its way into AI models. Similarly, my point was about Soviet propaganda making its way into the training materials for academics, journalists, etc. I didn't say they were actual Soviets.

jrflowers[6 comments hidden]
Why would Russia be pushing communism in 2017?
rayiner[5 comments hidden]
My point is that Russian propaganda from the 20th century made its way into intellectual water supply just like Russian propaganda from the 21st century. The propaganda is different, but the mechanism has parallels.
jrflowers[4 comments hidden]
So it is communist infiltration of the variety where there were no communists and there was no infiltration, just somebody read a book or whatever and then wrote about it?
rayiner[3 comments hidden]
You keep changing the words. Above you said “Soviets.” I don’t think they’re soviets. But I do think they’re communists. So “communist infiltration” is accurate.
jrflowers[hidden]
Why would I keep asking about Soviets after you clarified that you meant Russian propaganda in your first response?

I’m no communist infiltration expert or anything but do you have something to support your assertion other than suggesting people read “How to Parent Like A Bolshevik”?

lukan[hidden]
Ok, who are "they", the communists now?

(the russians today?)

tbugrara[hidden]
Well considering the USA pumps out the largest amount and widest reach of propaganda on the earth, and has for decades, I'd argue it makes more sense to incorporate all the propaganda - omitting any only strengthens/biases the ones included.

That said, your focus on Russia specifically just smells like red scare paranoia, one of many successful psyops from the USA.

rexpop[hidden]
Yeah, and the Iowa Writers’ Workshop and Jackson Pollock were promoted by CIA as a psyop—but that's a fact. We don't have to speculate and insinuate.

What sort of "infiltration" is that?

sedan_baklazhan[2 comments hidden]
AI companies: stealing data everywhere to feed their models, paying no one for it.

Also AI companies: oh no, You’ve poisoned our models with your data!

mdp2021[hidden]
> stealing

As disturbing as "you would not steal a <whatever>" in the times of pro-copyright brainwashing. What is published is there for people to read, and most societies already decided through libraries: for free, with charges on the State.

> [all caps]

Pls reread the guidelines. Check the page visually, see why. Edit the post pls. while you can.

mdp2021[4 comments hidden]
> poisoning the

Reminder that an intelligent entity does not believe or repeat something it has read even a million times: reminder that now that LLMs exist, we must find the real recipe ASAP.

jddj[3 comments hidden]
I'm concerned there are probably far more counterexamples of this in human history and society than there are examples.
mdp2021[hidden]
I wrote «an intelligent entity», very clearly - not "somebody supposed to be intelligent".

Of course there can be more examples of intellectual failures than of intellectual accomplished ones. And again, this calls for the urgency of finally injecting supplies of intelligence into the systems.

OgsyedIE[hidden]
https://en.wikipedia.org/wiki/Illusory_truth_effect
gregglain[2 comments hidden]
We know the solution to the decline of journalistic integrity but it's hostile to profits.
capitainenemo[hidden]
I'm curious what that would have to do with a state controlled news company generating millions of spam articles.
thih9[hidden]
Finally an article that doesn't say "ChatGPT did something" and puts the spotlight on the operators.

Could we keep that wording for other news? If not, could we then say here that ChatGPT planted fake articles, for consistency?

card_zero[8 comments hidden]
Seems I'm uneducated about how journalism works: you can just write a polished-looking opinion piece and submit it to obscure outlets, who pass it on to news sites and newspapers? Do you get paid? Or if you offer it for free, is that normal, and not suspicious?
baby[hidden]
Yeah, and it’s not just obscure outlets, you can do this for big magazines people here read for the right price
SoftTalker[3 comments hidden]
Been happening forever, PR departments plant news stories all the time. They just need a few willing reporters/writers to get something published to start the ball rolling.

I thought I remembered it being called "submarining" but that seems to be something else. I also thought I remembered Joel Spolsky writing about it but can't find it now.

martey[2 comments hidden]
I think you might be thinking about Paul Graham: https://paulgraham.com/submarine.html
SoftTalker[hidden]
Yes! That's the post I was thinking of. Thanks.
glenngillen[hidden]
It's quite old now, but check out the book "Trust me, I'm lying" by Ryan Holiday. Talks about how he did this as part of a marketing strategy when he was working at (IIRC) American Apparel, among other places. And how easy/often it happens in politics. Keep in mind the book is almost 15 years old, and talking about things that even at that point were what he'd done in a previous life/career.
PeterStuer[hidden]
"Journalism" hasn't worked for decades. It's all nepo-baby emo diaries, subsidy milking and clickbait farming these days.
pessimizer[hidden]
This wasn't in newspapers, this was only in marginal publications that take anything, such as Daily Kos, Middle East Monitor, and apparently the "Port St. Joe Star, a tiny Florida weekly newspaper."

Other than those, it was literally Medium and social media.

What you can do is write a polished opinion piece and tell one of your assets, or pay some politician, to publish it as an OpEd in outlets like the WaPo and NYT. Or you can get nobodies who know nothing to publish similar articles to each other in Wired, the International Business Times, and Forbes (either by "leaking" to them or because they're on your payroll), then get politicians to mention those articles in Congress and to write furious letters to WaPo and NYT accusing them of covering this story up - and then they'll write your propaganda themselves.

delichon[6 comments hidden]
I prefer an LLM that can steelman any argument. On the whole I think it's more effective to engage with evil than to ignore it. For that purpose these planted articles are a feature. I say that as an enemy of the government of Iran.
supjeff[hidden]
I kind of had the same take. If an LLM spits out legitimate arguments and they get published, i mean...
zahlman[hidden]
You seem to presume that the media owners are interested in critiquing and debating with submissions, rather than just selecting whatever fits their narrative.
axus[hidden]
You can fool some of the people, all of the time. Attempting censorship to stop that isn't worth it.
cloverich[hidden]
Need to take into account how belief and human brains work. Once human brains become locked on to something ideologically, similar to religion, its not the quality of the evidence or argument that changes minds. To an extent we're designed this way to keep the world moving, as the natural order of things is relatively indifferent to suffering. You can indeed flood segments of the world with disinformation, and by the time you move to refute it, its roots are set and actions in motion.

I don't really know what the best approach is to it, but I don't think a simple hands off, engage with the argument as such, is a winning strategy. I don't really like the alternatives any better.

watwut[hidden]
That is not steelmanning. Nor a feature.
tastyfreeze[2 comments hidden]
All news has been a tool of spooks for a very long time. The official CIA Operation Mockingbird started in the late 40s. Now that AI is involved its news? None of it can be trusted.
patrickk[hidden]
One can only imagine how much of the news is just dictated by US intelligence. Social media too- remember the Elgin AirForce base being revealed as the top most Reddit addicted city?
sva_[4 comments hidden]
https://archive.is/EFOhL
charlieflowers[3 comments hidden]
Edit: Resolved now it was on my end. > fyi not working for me
thih9[hidden]
N=1 but it's working for me at the moment.
charlieflowers[hidden]
Resolved it was on my end
mahdihabibi[hidden]
Reading this from Tehran, I'm afraid of losing my access to chatGPT.
BeetleB[3 comments hidden]
I'm sure there are a lot of differing (valid) opinions on this.

All I can say is that this highlights the need for open weights models.

> OpenAI said it had banned the accounts behind the campaign

If we knew OpenAI would always ban such activity regardless of the actor, I'd be happy. I don't know this.

cvoss[2 comments hidden]
How does open weights address the issue of nation states injecting covert propaganda into another nation's public discourse?
BeetleB[hidden]
Open weights ensures one company (or country) can't decide which actors are allowed to and not allowed to engage in this behavior.

Propaganda has always been there. The dynamics that allowed for all the fake news articles in 2016 to impact the elections are the same dynamics that allowed the articles in this case (shoddy news editors who aren't even verifying the identity of the authors). We didn't solve the original problem in the decade since - it's not going to be solved and is here to stay. The issue has shifted to who has access to this power.

Some random perspective: For years, I lived in an autocratic country where there was no freedom of speech and the government closely monitored all news agencies/publications in the country (before most people had Internet access). The dialog/debate was the same: "If we give freedom of expression, how will we address all the false propaganda coming in?"

Of course, times changed. They couldn't block the whole Internet, and false propaganda came in. Which version would you prefer - the one where they blocked it (but ran their own propaganda), or the one where they couldn't block it?

gizmodo59[8 comments hidden]
how is this news? Adversarial country using AI to generate fake news goes way before AI.
hammock[4 comments hidden]
Why does it have to be an adversarial country? Why does it have to be a country at all? Couldn’t it be the mayor of some town? Or a PR team at a fast food franchise group? Or one of SBF’s polycule members?

The notion we only see this news when it is calling out a country with which the federal government is in conflict makes me question to veracity and intent of the article itself, more than the content of it.

deepsquirrelnet[2 comments hidden]
And what do you call it when a country acts in adversarial ways to its own citizens? Call it cynical, but it sure seems like there's a lot of that in the world right now.
fakedang[hidden]
You call it a country :P
bilbo0s[hidden]
Well it’s mostly other entities running influence campaigns.

Thats true.

But this article is about the influence campaigns presumably tun by the Iranian government.

bitmasher9[2 comments hidden]
This is classic FUD, Fear Uncertainty and Doubt.

It’s designed to make the reader question which articles they have read recently that are Iranian propaganda, and sow doubt about future articles that seem to benefit Iran.

This amount of FUD always seems to increase around election time…

Telemakhos[hidden]
It's a well-researched article that lays out evidence, not suspicions. It includes OpenAI's own investigation into accounts that were found generating op-eds for Iranian propaganda. It shows the route by which these op-eds entered into smaller publications, namely by a pacifist NGO in Oregon, and neither then NGO nor the publications saw them as propaganda, despite the origins of the op-eds.
JoBrad[hidden]
It’s news because it still needs to be called out. Not everyone recognizes AI-generated content, but even if everyone did, calling it out serves a real social purpose.
nba456_[hidden]
Probably posted to this very site, no doubt.
jillesvangurp[2 comments hidden]
Fake news on the internet is at this point a tool that has been widely used for probably at least twenty years. The history of propaganda is much older.

A big part of the problem is social networks treating very non authoritative sources as if they are authoritative based on all sorts of factors that can be manipulated or based on financial gain (somebody pais to get their content promoted).

Users of these tools include lobby groups, nation states, politicians, criminals, scammers, etc. Anyone interested in promoting their cause, sowing FUD about other people's causes, spreading misinformation, or distracting from inconvenient but accurate information.

We'll always have people like that. But we might want to think about getting better tools to deal with them. I prefer getting my news from reputable sources instead of random clickbait distributors or whatever. Keeping track of reputation for domains, people, etc. is currently hard. I don't think it's necessarily a technical problems. We've had the tools to digitally sign stuff for decades. But we're not using them effectively. People can't be bothered to sign their emails, documents, photos, articles, etc. The tools to add and check those signatures are not there or an afterthought at best.

mkovach[hidden]
Drew Curtis figured this out in 2007 and even wrote a book about it.
gavinray[3 comments hidden]
The content of these articles seems to be "the Iran war is bad for Americans", which is not propaganda though?
Georgelemental[2 comments hidden]
Just because something is true doesn't mean it's not propaganda. Propaganda is not defined by how true or false it is. https://www.merriam-webster.com/dictionary/propaganda
8note[hidden]
propaganda is also value neutral on those definitions
xbar[hidden]
I can think of one media outlet for whom this would be an upgrade to its accuracy.
uraniborg[2 comments hidden]
LLMs are about as safe as the internet in 1993. Tampering with the training data to inject bias is a massive attack surface that is being ignored. It makes sense: trying to address corruption of models this way would slow down accelerationists and their zealot minions. Can’t get in the way of those sweet sweet trillions.
uraniborg[hidden]
Downvoting out of fear.
gustavus[3 comments hidden]
Remember 95% of what you see or hear on the Internet is designed to sell you something either an idea or a product, and it isn't done honestly.
everdrive[hidden]
Much of modern internet content is far worse than cable advertising in the 90s. I remember when I was a kid; I'd aggressively mute the commercials because I hated the idea that someone was attempting to influence me. The modern internet is orders of magnitude worse buy nearly any measure. The nastiest, most outrageous content is plastered on every wall. Every single person has an angle. Every bit of content meant to pull you in one direction or another, and given the addictiveness of novelty and powerful emotions, we keep coming back for more. It's an absolute disaster we don't know how to walk away from.
bilbo0s[hidden]
>95%???

That low?

In all honesty, given the volume of comments, posts and articles that bots can generate in a given time period? I mean it’s gotta be several orders of magnitude higher than what humans can generate.

And then you factor in the fact that even most of the humans are astroturfing..

The percentage of influence campaigns we interact with has just gotta be really high.

VLM[hidden]
They should compare it to the scale of Israel's similar activities. At least Iran isn't interfering in our elections.
WhereIsTheTruth[hidden]
Iran? how convenient

It's like when SEA starts to distance itself from US's grip, we suddenly hear about North Korea doing missile tests

jadar[2 comments hidden]
democracy dies in the darkness of paywalls, I guess.
baggachipz[hidden]
It also dies in the darkness of a takeover by billionaire with an agenda.
sleep_debt[hidden]
...as if the American government hasnt done this.
mmooss[hidden]
Speaking of disinformation, you might notice a pattern recently:

* Opinions opposed to US administration policy on AI (including data centers) is called Chinese propaganda. A month or two ago, when someone tried to roll out a bunch of pro-data center opinions across the media (coincidentally all on the same day making similar arguments), they included this theme.

* Opinions opposed to US administration policy on Iran is called Iranian propaganda.

That's not really new: News reporting and opinions critical of the US administration have lead to the authors being called traitors, 'enemies of the people', and sometimes being prosecuted by the Department of Justice. Some leaders in the defense industry call critics of military spending and pro-warfare policies, 'traitors', etc.

Of course some could be, might be, likely is true. That's the nature of effective propaganda.

(A side effect of nationalism - 'nation uber alles', defining nation as 'what I believe' - is that individuals and freedom are relegated to second place or lower.)

supjeff[2 comments hidden]
Looking at today's headlines on Washington Post, I'm asking myself if it's always been a pro-war, pro-trump, right-wing rag, or if it's only since Bezos bought it?
punchmesan[hidden]
Only since Bezos. I used to actually pay for WaPo pre-bezos. It was fine for a while and then he started clearly messing with editorial decisions and I bailed. It's now a garbage source.
tyjen[6 comments hidden]
Not a bad actor spreading misinformation, but I encountered a similarish weird, dystopian experience with ChatGPT.

Utilized ChatGPT on my PC to search local plant nursery inventories for a specific tree. Output showed several local nurseries with rough probability descriptions regarding whether they have the tree in inventory, except for one nursery. For that nursery, ChatGPT provided a direct link to the specified tree, which was directly on the nursery's domain, showed a picture and description of the tree, and even showed the item in-stock with 6 in inventory at the 5-gallon size.

Excited, I plan my trip and am about to pick up my tree, I then double-check the nursery site on my phone to verify location and stock. As I attempted to navigate the website inventory to find the tree, I noticed that the tree no longer existed. There is no page for that varietal or species of tree on the entire website's domain.

I head back inside and look at the ChatGPT provided link on the PC and there it is, showing the inventory that doesn't show up when you use the nursery's website to directly find it without the provided link. A call to the nursery, further verified that the tree is not in stock, nor have they ever carried it.

ChatGPT made it all up. Did ChatGPT decide to do this, or did the nursery pay for this type of "advertising" to generate in-person customers? Nevertheless, still an incredibly weird and dystopian experience.

garyfirestorm[hidden]
It didn’t generate a link (sources) when it claimed it was in stock
pmontra[hidden]
It seems that it generated a page with a plausible answer for a very long tail subject for which the random part of the generation could become prominent. We used to say that it hallucinated the answer. Another term is confabulation.

If only the nursery had a MCP server /s

Avicebron[hidden]
Not being able to trust your tools is certainly dystopian...
ButlerianJihad[hidden]
Did you consider that the website made it up and fed it to ChatGPT?

It sounds like a bundle of very common SEO tricks combined with the differential between your human user agent’s visit.

Perhaps ChatGPT only showed you what the nursery told it about.

This phenomenon is not unique to LLM. If you telephone a store, chances are that the salesman who answers the phone will tell you only what you want to hear. They don’t even strictly need to lie. The store only wins if you drive there and walk in the door. Why should they discourage you from doing that very thing?

This concept was driven home to me at the dry cleaners. Why should a dry cleaner stop work and answer the phone, only to tell a customer that their batch isn’t ready? They just got taken off-task for a non-revenue interaction. It doesn’t matter, because even if the customer’s batch is ready, they aren’t there to pay and pick it up!

mrguyorama[hidden]
There is no measurable difference between a neural network generating text that happens to be true and a neural network generating text that happens to be false.

It's not "Hallucination" or a conspiracy, it is their defined operating mode.

It told you they had that tree because you wanted that tree, and because it was trained to tell you what you want to hear. By design.