Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Strengthening the fight against online child sexual abuse (https://openai.com)

125 points by rossant 4 days ago | 323 comments | View on ycombinator

maxgee 4 days ago |

Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.

We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.

OpenAI does not care about fighting child sexual abuse, this is completely performative.

EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com

gwd 4 days ago |

So I've been working on having Claude using GPT images to illustrate various fairy tales. At some point there was a Russian one of a girl left in the forest, but in the first cut she looked peaceful and happy. I told Claude I didn't think that was realistic, and it said that it had already asked GPT images to show the girl "scared", and it refused 12/12 times. It also refused to have Frost (who in the fairy tale ends up giving her a coat and a chest of gold) standing over her; he had to be at her level, an arm's length away. (Showing the girl as "brave" and then "nervous" had better results.)

Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.

nolist_policy 4 days ago |

Relatedly, Openrouter now does CSAM scanning and retention on so called "Zero Data Retention" endpoints as well.

https://openrouter.ai/blog/announcements/csam-screening/

RIMR 4 days ago |

This kind of thing is always an excuse for creating surveillance tools that scan your files and report you to the government. It starts with CSAM because they know you don't want to argue against CSAM detection, but you absolutely should.

I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!

It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.

Donald 4 days ago |

It is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it's detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user's local law enforcement jurisdiction, and they're arrested within a short time period, like one or two weeks.

eutropia 4 days ago |

This is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.

When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)

A model capable of shielding investigators from additional trauma is an excellent thing.

itake 4 days ago |

My colleagues in Singapore trained pornography detection model.

Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.

AmazingTurtle 4 days ago |

reminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)

callc 4 days ago |

Curious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.

I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.

How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?

mrandish 3 days ago |

I guess it's sign of the times that upon seeing the headline "Strengthening the fight against online child sexual abuse", my first reflexive thought wasn't positive. Instead it was a jaded assumption this would be another perhaps well-intended effort likely to either go awry, further undermine the privacy and workflows of law-abiding users, or be largely ineffective - and quite possibly all three.

avazhi 4 days ago |

NCMEC is cancer.

Remember when they wanted Apple to scan everybody’s devices locally?

I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.

If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).

nonethewiser 3 days ago |

The consent/harm sexual ethics framework has a problem here.

Most people think AI-generated child porn is wrong. But the consent/harm framework can't explain why AI generated child porn is wrong. Either you reach for indirect harms which can just be stipulated away (its not shared so police aren't confused, local model, etc.), or you have to argue in favor of AI generated child porn. In fact we have seen plenty of that here. And this framework has similar problems with necrophilia, bestiality, and adult incest.

A framework that allows for some natural order does not have this issue. If there is some natural order, and right/wrong are not just a function of preferences, you dont have to judge actions solely by effects. From that view, you can say sex has a proper place and purpose, and children have a proper role -- to be protected, raised, and not sexualized. Sexualizing a child violates that order whether or not a real child is ever involved. So the wrongness doesn't require any specific child to be harmed.

I think thats what most people already sense - that the wrongness was never only about harms to children.

eastbound 4 days ago |

Finally Sam Altman found something good to do in terms of my security, I'm relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).

stickfigure 4 days ago |

How is this different from omni-moderation, which OpenAI already has?

https://developers.openai.com/api/docs/models/omni-moderatio...

kittikitti 4 days ago |

This is just public relations and no reasonable person would believe OpenAI will do anything to fight against online child sexual abuse. In fact, it's much more reasonable to assume they will continue to make the problem much worse.

nrmitchi 4 days ago |

Frankly, and I don't mean this in a bad way about the problem, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. "Scan all your images through our CSAM detector" means "send all images to us, including the 99.99% that are not CSAM!"

Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.

undefined 4 days ago |

undefined

andai 4 days ago |

>It is well established that combatting online child predators requires coordination across many organizations.

It requires all devices integrated into one single consciousness.

bearjaws 4 days ago |

If only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.

Could be a great start for "rogue AI"

Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.

sandboxdevelope 3 days ago |

Don't forget that OpenAI used CSAM to train it's models

undefined 4 days ago |

undefined

rfgplk 4 days ago |

[flagged]

haickernews 4 days ago |

[flagged]