Wrong answer. Or at least, obvious and not particularly useful.
Truth is, none of those parties are "nefarious" - they're all just not on your side. And "security" is never an unqualified good thing to have (it's not an unqualified bad thing either). It's just a framework of coercion.
The most important questions to answer about any security system is, what is being protected, for who, and from who. People don't ask that much, not even in the industry - it's an implicit assumption that everyone themselves is a "good person" and is on the protected side of security systems. And then they're confused because it turns out end-users are more often seen as threat actors. All the players mention, but perhaps especially Apple, in its own special way, is protecting the computer from the user just as much as they're protecting the user/user's data from third parties.
They kinda do though, in that instances have been observed to send unrequited messages even when the person/people in charge of some account didn't expressly ask the models to do so.
For my own use of LLMs, I do try to avoid anything which I know has a risk the artefacts they produce may end up DoSing or spamming, and I've avoided the OpenClaw-type pattern for a broader range of reasons of which this is simply one tiny part, but I'm not absolutely confident I could avoid this even in the code coming out of the free tier of the web chat interfaces except by checking every single line of output every single time.
At 1000, you can afford better tools and better employees, and replacement parts get cheaper as you order in bulk, and you can explore clever strategies to smooth risk curves.
At 100 000, you can afford a better and continuously improving process, and dedicated facilities, and skilled experts, and parts get even cheaper because you're a volume buyer or perhaps own the supply side, and you get to set your own risk curve.
Lots of things get cheaper at scale. Insurance, too.
No, it's the actual reasonable approach that sane people have to security. In the real world, security is always about costs and benefits, because you can always make something more secure than it is by spending more money, but it also doesn't make sense to spend more than you're getting from it.
Normally, you secure things up to minimize (${cost of security measures} + ${expected damage from attacks that materialized}), writing off actual material damage with insurance wherever possible. You pick security measures based on their effectiveness, which usually translates to "how expensive will it make success for attackers", aiming to push that above the value the attackers can expect to gain.
There are obvious exceptions to that, like risk to life and limb, as well as some other special situations where attackers may have unusual motivations and thus the economic logic of "make stealing treasure cost more than the treasure" stops applying. But those are exceptions. Almost everything you deal with in your life - from your bike shed to the corporation that owns your bank - follows the above logic in terms of security.
--
I spell this out because I've noticed that tech industry circles have this weird, belief in security as some kind of binary, holy good, that you either have and are blessed, or don't and sin. This obsession starts with failing to even recognize, much less ask, the most important questions about security: why do you want to protect it, and who are you protecting it from?
100% agree, and so happy to see somebody call this out. If you go on /r/SelfHosted or any other novice oriented forum, you’ll quickly realize that most users are simply “keeping up with the joneses” when it comes to security & redundancy. That itself is fine I guess, but the zero tolerance they have for anything else is just absurd.
My approach for AI-first code review, or really any kind of AI technical opinion, is that if the claim AI made is both important and not obviously true at a glance, it has to prove it to me, and keep trying until I'm convinced or can spot an obvious mistake in the proof.
With reviews, this is usually the case where AI is making a claim that something in the PR will fail because of some assumptions or behaviors in code outside of the PR - e.g. "this change will fail in scenario X, because foo is null in this case, because the SQL query doesn't populate it when bar == quux, and it gets propagated as null through the JSON deserialization (optional field)...", where all the SQL and JSON parsing was not part of the code under review, and "bar == quux" is some weird domain special case.
Stuff like this is both critical, and there's no way for me to judge it without an expensive context switch. So I learn to ask for a more detailed walk-through once, and if that doesn't make me "see" it, I just ask it to reproduce it with tests, and confirm it's a real problem. Reviewing the reproduction is usually enough for me to either "see it" or accept they're probably right and ask the author to recheck it.
(Why not jump straight to "reproduce it" for every finding? Because it still takes time to have AI do the repro. It's cheaper than a deep context switch, but not free.)
I think it's not really about having a conversation - I mean, that's part of it, but alone it's an illusion that eventually fades quickly. It's more of because of how it demonstrates intelligent behavior in reaction to requests, both in trivial and complex matter, and all across the board. LLM's response may be completely incorrect or confused, but it's nearly always exactly what you expect from a human[0]. This creates a more general feeling you're dealing with a human-like intelligence.
To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.
This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.
--
[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.
> Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?
Difference in size and complexity and nature of calculations being run?
I'd ask the other way - why do you (general you, people who do not have this inkling) have no problem debating consciousness of meat based brains, but it somehow becomes a category error when talking about silicon? Assuming you don't believe in divine magic, and that divine magic is core to consciousness, there's no reason to assume it's impossible a complex enough machine running complex enough software could be intelligent, or conscious - thinking is computation, and computation is made of math - it's independent of substrate that does the computing in the real world.
LLMs are definitely a different beast than regular software - both in their structure and in their generality. They may not be conscious or intelligent, maybe this specific design could never truly be (though I think it could) - but bucketing them with spreadsheets and terminal emulators is a real category error. If you stop fixating on the underlying substrate, then LLMs are already much more similar to biological minds than to any "regular" computer programs.
But that's still somewhat abstract. In immediate practical terms, it's also why I keep saying that anthropomorphising them gives a better high-order intuition: they are, by design, emulating human thinking in full generality, which makes their overall behavior, including their well-known problems like hallucinations or prompt injections (i.e. manipulation/gullibility), match what you'd expect of a people-like component of a system. It's a real, dangerous mistake, to treat LLMs like regular software components when designing systems.
I wouldn’t say they’re anywhere near “full generality” because that would include believing untruths and passionate rages, existential angsts and the crazed obsession of heartbreak.
Your definitions depend on us being computers who think.
Solving some of the intelligence part of our logical thinking doesn’t get us anywhere near consciousness, which is a superset way beyond the linguistic intelligence used for communication.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.
I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.
> that we glean insights into nature of the simulated phenomena
That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.
Great post. And that's exactly where I think we are with language models... we as a civilization are hypnotized and enchanted by the overfitting of models whose parameters are beyond our understanding, but whose mistakes we are more likely to forget than its accuracies, which again is a central human characteristic that explains our attraction to both psychics and slot machines.
Heck, it even explains my own attraction to overfit sports betting algorithms. No one is immune.
What's dangerous is when something like that replaces independent thought and becomes societally pervasive. That's an "oracle" the likes of which ancient civilizations warned that believing would lead to tragedy (or at a minimum, accidentally boning your own mother).
I'm an atheist, but raised Jewish. I read the Torah as a series of specific warnings and prohibitions against every type of shamanism, magic, witchcraft, prognostication, and deification of systems which predict (as well as systems which attempt to turn language into machinery, and worship the machine they've built ... see also, "Sound of Silence" by Paul Simon and "The Future" by Leonard Cohen, which both express this theme well). The framework requiring proof and disavowing illusion or the belief that all is illusory is notably different from a Buddhist perspective, for example.
We as a culture, right now, are not handling well the rise of a golden idol or an oracle in our midst. The right response is to try to trace the output back to ground truth and figure out why your model made a prediction... or else to build a model from ground truth and see how it performs against the oracle. We are doing neither. We're diving headlong into our own confirmation biases.
[Edit] I just wanted to add, because I got off track, that your conclusion about what's going on with human curiosity in cases where prediction is not the issue seems right to me. Barring some edge cases like predicting an eclipse and using it to slaughter your enemies, I think a lot of us do simply want to understand how things work, because figuring them out is enormously gratifying and is the work of lifetimes of incredible people who came before us. Using that knowledge or those techniques to predict things is technology, not science, and while I'm a fan of both, the former is only ever a practical test of the latter. Moreover, the sense of accomplishment of randomly walking your way to a profitable model is ephemeral and in a way earthbound, limited to the plane of one's own brief existence. Even if it were platonically perfect, a model is only saying how something behaves, not how it works. That's nothing compared to the joy of figuring out even the most trivial or axiomatic thing about how a cell or a compound or a physical structure or anything works, about how the universe actually works. And I think our better angels tell us to seek those answers, because our own life is fleeting, and predicting behavior is, like wealth, something you can't take with you. And not something you'll be remembered for anyway.
I think we're talking about different kinds of models. I was referring to things like fluid dynamics equations that explain why gases and liquids move and how they act when changing states, as a basis for building weather models that predict how things will unfold in the future.
I'm also a fan of going the other direction: I've had a sideline working on code to evolve genetic algorithms for the past 20 years, and while the goal of that is to be predictive and profitable, it's often the underlying real-world dynamics my little mutants surface which are the most interesting and applicable in the long run. So I'm not saying there isn't a place for throwing everything at the wall until you see what sticks and then deriving a hypothesis from that (whether your interest is to predict the future, or merely academic, to explain the past). What I am saying is similar to you: We should not treat any model as an oracle. But I'm also saying that models can be built or they can be evolved, and if we only evolve them without understanding how they work, we are missing a crucial ingredient to knowing how well we should rank them. Overfitting and sample bias and data leakage are not problems when you want an equation to calculate airflow over a wing. If you began with an evolved equation which derived the results and didn't start from the base reality, you couldn't trust that equation to be airworthy even if it were right 99.99% of the time against the data it was trained on.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
This is a good model. If you take an old ROM dump from a video game, it's just a pile of bits. You don't know what bits represent code, what represent an image, what represent text, etc. You have to analyze them contextually to actually figure out what is code and what is "data" in context, because without context they are truly one and the same.
Wrong answer. Or at least, obvious and not particularly useful.
Truth is, none of those parties are "nefarious" - they're all just not on your side. And "security" is never an unqualified good thing to have (it's not an unqualified bad thing either). It's just a framework of coercion.
The most important questions to answer about any security system is, what is being protected, for who, and from who. People don't ask that much, not even in the industry - it's an implicit assumption that everyone themselves is a "good person" and is on the protected side of security systems. And then they're confused because it turns out end-users are more often seen as threat actors. All the players mention, but perhaps especially Apple, in its own special way, is protecting the computer from the user just as much as they're protecting the user/user's data from third parties.
reply