Now a full post

LMAOU congrats David.

Another ironic point… Lesswronger’s actually do care about ML interpretability (to the extent they care about real ML at all; and as a solution to making their God AI serve their whims not for anything practical). A lack of interpretability is a major problem (like irl problem, not just scifi skynet problem) in ML, you can models with racism or other bias buried in them and not be able to tell except by manually experimenting with your model with data from outside the training set. But Sam Altman has turned it from a problem into a humble brag intended to imply their LLM is so powerful and mysterious and bordering on AGI.

Not gonna lie, I didn’t entirely get it either until someone pointed me at a relevant xkcd that I had missed.

Also I was somewhat disappointed in the QAA team’s credulity towards the AI hype, but their latest episode was an interview with the writer of that “AGI as conspiracy theory” piece from last(?) week and seemed much more grounded.

I mean if you ever toyed around with neural networks or similar ML models you know it’s basically impossible to divine what the hell is going on inside by just looking at the weights, even if you try to plot them or visualise in other ways.

There’s a whole branch of ML about explainable or white-box models because it turns out you need to put extra care and design the system around being explainable in the first place to be able to reason about its internals. There’s no evidence OpenAI put any effort towards this, instead focusing on cool-looking outputs they can shove into a presser.

In other words, “engineers don’t know how it works” can have two meanings - that they’re hitting computers with wrenches hoping for the best with no rhyme or reason; or that they don’t have a good model of what makes the chatbot produce certain outputs, i.e. just by looking at the output it’s not really possible to figure out what specific training data it comes from or how to stop it from producing that output on a fundamental level. The former is demonstrably false and almost a strawman, I don’t know who believes that, a lot of people that work on OpenAI are misguided but otherwise incredibly clever programmers and ML researchers, the sheer fact that this thing hasn’t collapsed under its own weight is a great engineering feat even if externalities it produces are horrifying. The latter is, as far as I’m aware, largely true, or at least I haven’t seen any hints that would falsify that. If OpenAI satisfyingly solved the explainability problem it’d be a major achievement everyone would be talking about.

well, I can’t counter it because I don’t think they do know how it works. the theory is shallow yet the outputs of, say, an LLM are of remarkably high quality in an area (language) that is impossibly baroque. the lack of theory and fundamental understanding presents a huge problem for them because it means “improvements” can only come about by throwing money and conventional engineering at their systems. this is what I’ve heard from people in the field for at least ten years.

to me that also means it isn’t something that needs to be countered. it’s something the context of which needs to be explained. it’s bad for the ai industry that they don’t know what they’re doing

EDIT: also, when i say the outputs are of high quality, what i mean is that they produce coherent and correct prose. im not suggesting anything about the utility of the outputs

Gentoo is firmly against AI contributions as well. NetBSD calls AI code “tainted”, while FreeBSD hasn’t been as direct yet but isn’t accepting anything major.

QEMU, while not an OS, has rejected AI slop too. Curl also famously is against AI gen. So we have some hope in the systems world with these few major pieces of software.

ah, and they’ve got a community feedback forum post, where it isn’t going the way they might have expected: https://connect.mozilla.org/t5/discussions/building-ai-the-firefox-way-shaping-what-s-next-together/td-p/109922

I don’t doubt you could effectively automate script kiddie attacks with Claude code. That’s what the diagram they have seems to show.

The whole bit about “oh no, the user said weird things and bypassed our imaginary guard rails” is another admission that “AI safety” is a complete joke.

We advise security teams to experiment with applying AI for defense in areas like Security Operations Center automation, threat detection, vulnerability assessment, and incident response.

there it is.

Does this article imply that Anthropic is monitoring everyone’s Claude code usage to see if they’re doing naughty things? Other agents and models exist so whatever safety bullshit they have is pure theater.

I doubt every word of this. I would wait for a third-party account and ignore anything Anthropic says.

[…] Curio’s Grok, an anthropomorphic rocket with a removable speaker, is also somewhat opaque about its underlying tech, though its privacy policy mentions sending data to OpenAI and Perplexity. (No relation to xAI’s Grok — or not exactly; while it’s not powered by Elon Musk’s chatbot, its voice was provided by the musician Claire “Grimes” Boucher, Musk’s former romantic partner.)

<applies brain bleach liberally>

someone is definitely going to wind up shagging that child’s toy.

Ah, the site requires me to agree to “Data processing by advertising providers including personalised advertising with profilingConsent” and that this is “required for free use”. A blatant GDPR violation, love-lyy!

Btw “Don’t Die” is a Bryan Johnson adjacent longevity community slogan which the writer is very likely to have seen often around twitter

Or it could be a reference to what often is said before the start of a match of a video game (though they probably left out “kick ass” for marketing purposes).

Edit: actually, considering that, maybe there’s a reveal at the end that they’re in the Basilisk torture sim, so… there might be something there?

Previously, on Awful, I wrote up what I understand to be their core belief structure. It’s too bad that we’re not calling them the Cyclone Emoji cult.

Rationalism described as a cult incubator

I see my idea is spreading. (I doubt im the only one who came up with that, but I have mentioned it a few times, it fits if you know about the silicon valley tech incubator management ideas).

Uzumaki intensifies

I think I’ve heard Rationalism described as a cult incubator

Aside from the fact that rationalism is a cult in and of itself, this is true, no matter how you slice it. You can mean it with absolute glowing praise or total shade and either way it’s still true. Adhering to rationalist principles is pretty much reprogramming yourself to be susceptible to the subset of cults already associated with Rationalism.

Epstein did fund MIRI.

tangent: I’ve seen people using this Bubblegum Slop (BS for short) in their social media stories. My guess is that fb/insta has started suggesting you use their slop instead of using music licensed from spotify, or something.