Hi there! I'm working on publishing my "seconds" or posts that I've mostly finished but just aren't "good enough" for my main blog. This is going to be where you can find my semi-raw bits of work. I'm also using this to play with standard.site lexicons. I'll blog more about it after I have opinions. Enjoy!

This post was drafted late July 2026. Facts and circumstances have changed since the time of drafting.

Recently the heavyweights of the AI industry have been up in arms over open-weights AI models (AKA "open source" AI models, the ones where you can run it on hardware you can look at). Most of the giants of the tech industry have backed an open letter saying that open-weights AI models should not be banned. In response Anthropic posted cringe on main with several claims that I want to tear apart:

  • "Anthropic has never advocated for a ban on open-weights models. [...] All sufficiently capable models, open and closed, should go through mandatory safety testing."

  • [We] should crack down on the rampant smuggling and workarounds used to obtain access to [powerful NVIDIA] chips [...] China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.

  • [P]owerful AI models may be misused to carry out cyberattacks or biological attacks, and may have serious alignment problems.

  • We should crack down on industrial-scale distillation operations. [...] We should have policy interventions to deter this behavior.

Mandatory safety testing is functionally a ban

Open-weights models are very popular among developers because you can _run them on hardware you can look at_. I have a local model that I use for various incredibly boring things (mainly tool use and limited summarization of bulk chat messages) based on Qwen 3.6 35B-a3B. Apple ships their own technically-closed-but-functionally-open-weights models that you can invoke with a tool like apfel, which is mostly used for background tasks or things you ask Shortcuts to do.

Banning open weights AI models would be about one of the biggest ways to piss off large swaths of developers and big tech companies that simply do not want third party model providers to see their data. 80% of the usecase for models that are not hosted by a third party is to analyze and do inference on private data that people just don't want even the possibility of being leaked.

Not to mention, with open-weights models you can just download the model weights and host it yourself with something like Ollama or by compiling llama.cpp from source. When you run it yourself, you set the policies of what is acceptable. Maybe you do actually want to talk to an AI model about a security problem like HuggingFace had to when OpenAI’s sandboxing failed leading them to get attacked.

Anthropic did not clarify on what they meant by “mandatory safety testing” beyond a few vague snippets:

The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release. I think this idea is actually close to a consensus: I have been heartened both that the Trump administration has moved in this direction in recent months, and by recent industry proposals that would apply such testing to the most capable models regardless of their country of origin or whether they are open or closed (while exempting less capable models, such as those from startups and academia, entirely).

Reading between the lines, this is saying that they want bureaucrats to review what AI models do before they’re allowed to be released. Maybe this is related to Fable getting ITAR’d, maybe this is some kind of genuine delusion about them being the “only trustable ones” that can evaluate if a model is safe, maybe this is actually rooted in genuinely wanting to make the world better. I don’t know, but I do know that the regime this is calling for would functionally silo advancement in this technology in the labs of big companies without anyone really being able to benefit from it.

If the US requires some kind of mandatory safety testing before models are released, it would functionally kill the AI bubble, which would in turn kill most of the US economy.

What the fuck do you mean by “limited domestic production capacity”

I’m gonna be honest, I don’t know what mister “loves gods” (Amodei) is getting at here. This is kind of a ludicrous assertion on the face of it.

The CEOs of the two biggest AI companies in the US are named “alternative to man” and “loves gods”. Great writing as always, Hideo Kojima!

Currently there is a battle of science being waged between two countries. One is an authoritarian anti-intellectual waning global superpower that has fundamental infrastructure issues so deep that it makes building any additional compute capacity en masse logistically impossible. The other country is China.

We have also forced the other country to have hard limits, thus forcing them to be clever, thus making them stop relying on western technology entirely. That certainly sounds like a great way to win a science battle!

Sure, today China has “limited domestic production capacity”, but even in early 2025 Huawei’s Ascend series was within 60% of NVIDIA’s chips. Their Ascend 950PR claims to have 1.56 petaflops of performance and 112Gi of memory. It's good enough for serving language models en masse today. It will only get better tomorrow.

All they have to do is figure out how to manufacture high bandwidth memory domestically and then they won't need outside technology at all.

Alignment to who, cyberattacks, and biological warfare

One of the big memes in the AI safety crowd has been the idea of alignment, or making sure that AI models roughly take actions that correlate to what actions a human would take in the same situation. The only problem is that this idea of “alignment” does not specify who they want to align the values to, only that it should be aligned to “the good of humanity” or whatever.

The only major problem here is that as a species we cannot agree what “the good of humanity” actually is, and this ideological misalignment has been the entire story of the last 10 or so years of world news. So who are these models aligning to? The views of the AI companies that train them?

Detailing the ethical quandries that this entire situation exposes could probably fit written volumes the size of all human authored text several times over. One of the easiest to understand ones are the ethical problems that image2image pipelines (ones that take an image as input and generate another image as output) expose. Chief of which is that people have been using them to make Non-Consensual Intimate Imagery (NCII). Where is the line? Who dictates it? I sure as fuck don’t know and don’t want to bother trying to debate it, so don’t ask me.

Anthropic does a bit better than most to define the kind of alignment they do in Claude’s Constitution, but in general this still doesn’t really answer the question of how things are aligned and how that differs from common views.

The bits about autonomous AI agents doing cyberattacks or biological warfare kinda sound like fanfiction, except for that one time a model OpenAI was evaluating hacked into HuggingFace to cheat on a test. Honestly the fact that this is now in our threat profiles is really fucking annoying, especially compounded with the fact that many powerful models reject cybersecurity defense work:

The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.

(c.f. the HuggingFace post about how an OpenAI model under evaluation popped HuggingFace)

If the good guys have guns that refuse to shoot sometimes but the bad guys have unrestricted access to firearms, then the good guys’ guns are fundamentally useless. Maybe this bit is why Anthropic is getting all uppity about their models being the only ones that can safely do this and that their views of what is good for humanity are the best or whatever.

This is the worst fucking cyberpunk dystopia ever.

To be blunt: if the choice is paying for access to a tool that sometimes randomly rejects your work without recourse or paying access to a tool that doesn’t but is really cagey about the topic of Chinese Internet Maintenance Day, I think it’s reasonable that many people are going to go for the latter if only because they can better rely on it.

The distillation panic

It seems that every time any Chinese lab releases anything that performs anything close to an Anthropic model, Anthropic cries that the Chinese labs are “distilling” their models and “illegal stealing capability” from what they do. As if they have the moral high ground to stand on after ripping priceless books to shreds to salt the earth for more training data.

At a high level, “distillation” is a process of using a “teacher” AI model to generate training data for a “student” AI model. This is a huge part of how AI model capabilities have advanced over the past year or so. I’d be willing to bet that Anthropic uses this process internally in order to make their models better.

There’s another way you can use a “teacher” AI model in the training process though, as an aid to supervised finetuning. With supervised finetuning you have a “good” and “bad” response for every question. You want the model under training to have responses be similar to the “good” option without them being similar to the “bad” option. This is also how reinforcement learning from human feedback (RLHF) works, but with humans in the mix instead of pre-seeded good/bad responses.

If you swap out the human for another AI model, then you can have that “teacher” model judge the responses from the “student” model at machine speed. This drastically accelerates the process of distilling knowledge, skills, and other qualitative properties out of the “teacher” model with very little human interaction in the process.

One of the side effects of this process is that if you have any typos or bugs in your code, you can accidentally create models that are maximally horny.

However at the same time, Anthropic has cried “distillation” so many times over the years that I don’t know if I believe them. They’ve gone out of the way to annotate when Chinese firms are using Claude Code and regularly post claims about “distillation attacks” that honestly verge on sinophobia.

Honestly this kind of “distillation attack” is probably happening to a limited extent (maybe when doing final evaluations?), but I don’t think that this actually happens in practice nearly as much as Anthropic is claiming. The numbers just don’t add up, it seems like it’d scale poorly.

Amusingly enough, Google is/was making this a product opportunity. I’m linking to an archive of their documentation as the live version 404s now, but either way my point is more that this is something that could probably become a useful thing with some mild feature work.

TL;DR

Honestly, Anthropic posted cringe on main.

I think that their goal is to try to get open-weights models banned or add enough regulatory overhead such that open-weights models are functionally banned in practice. I don’t think that this is a viable strategy, and that at some level Pandora’s Box has been opened and we can’t just “go back” to how things were before. I have heard from many in the AI space that share similar views but have elected to not express this publicly.

But at the same time, if this technology has to exist I’d much rather be able to have the capacity to run it on hardware I can look at with policies I can control than it running as a service on remote hardware with a remote company’s policies.

This post is probably a bit more angry than you’re used to seeing out of me. I try to be a bit more uplifting and make the stuff I produce not reflect emotions like that. I actually wrote this originally on the day the dariopost was published and spent a few days both cooling down and getting feedback from others so that

But hot damn these people are seriously advocating for a two-tired society where you either have access to advanced AI tools or you’re defenseless against the people that do. This is prisoner’s dilemma 2: electric boogaloo.

And the fucking idiots at the helm can’t even airgap their evaluation sandboxes correctly. Again! Kojima! Keep up the good work! Fantastic writing as always!