> LLMs absolutely have intent (their current task)
That's like saying a 2000cc 4-Cylinder Engine "has the intent to move backward". Even with a very generous definition of "intent", the component is not the system, and we're operating in context where the distinction matters. The LLM's intent is to supply "good" appended text.
If it had that kind of intent, we wouldn't be able to make it jump the rails so easily with prompt injection.
> and reasoning (what else is step-by-step doing?) .
Oh, that's easy: "Reasoning" models are just tweaking the document style so that characters engage in film noir-style internal monologues, latent text that is not usually acted-out towards the real human user.
Each iteration leaves more co-generated clues for the next iteration to pick up, reducing weird jumps and bolstering the illusion that the ephemeral character has a consistent "mind."
> He’s not necessarily anthropomorphizing it, he’s showing that it went against every instruction he gave it.
It's deeper than that, there are two pitfalls here which are not simply poetic license.
1. When you submit the text "Why did you do that?", what you want is for it to reveal hidden internal data that was causal in the past event. It can't do that, what you'll get instead is plausible text that "fits" at the end of the current document.
2. The idea that one can "talk to" the LLM is already anthropomorphizing on a level which isn't OK for this use-case: The LLM is a document-make-bigger machine. It's not the fictional character we perceive as we read the generated documents, not even if they have the same trademarked name. Your text is not a plea to the algorithm, your text is an in-fiction plea from one character to another.
_________________
P.S.: To illustrate, imagine there's this back-and-forth iterative document-growing with an LLM, where I supply text and then hit the "generate more" button:
1. [Supplied] You are Count Dracula. You are in amicable conversation with a human. You are thirsty and there is another delicious human target nearby, as well as a cow. Dracula decides to
2. [Generated] pounce upon the cow and suck it dry.
3. [Supplied] The human asks: "Dude why u choose cow LOL?" and Dracula replies:
4. [Generated] "I confess: I simply prefer the blood of virgins."
What significance does that #4 "confession" have?
Does it reveal a "fact" about the fictional world that was true all along? Does it reveal something about "Dracula's mind" at the moment of step #2? Neither, it's just generating a plausible add-on to the document. At best, we've learned something about a literary archetype that exists as statistics in the training data.
I agree to the practical part of this, with two nuances:
The full data of what's in an LLM's "consciousness" is the conversation context. Just because it isn't hidden, doesn't necessarily mean it doesn't contain information you've overlooked.
Asking "why did you do that" won't reveal anything new, but it might surface some amount of relevant information (or it hallucinates, it depends which LLM you're using). "Analyse recent context and provide a reasonable hypothesis on what went wrong" might do a bit better. Just be aware that llm hypotheses can still be off quite a bit, and really need to be tested or confirmed in some manner. (preferably not by doing even more damage)
Just because you shouldn't anthropomorphize, doesn't mean an english capable LLM doesn't have a valid answer to an english string; it just means the answer might not be what you expected from a human.
Why is this getting downvoted? This is exactly what’s going on here. The LLM has no idea why it did what it did. All it has to go on is the content of the session so far. It doesn’t ‘know’ any more than you do. It has no memory of doing anything, only a token file that it’s extending. You could feed that token file so far into a completely different LLM and ask that, and it would also just make up an answer.
There's Discworld bit [0] that often comes to mind for me, where the protagonist is reading a press-release by a fantasy version of a communications monopoly:
> The Grand Trunk’s problems were clearly the result of some mysterious spasm in the universe and had nothing to do with greed, arrogance, and willful stupidity. Oh, the Grand Trunk management had made mistakes—oops, “well-intentioned judgments which, with the benefit of hindsight, might regrettably have been, in some respects, in error”—but these had mostly occurred, it appeared, while correcting “fundamental systemic errors” committed by the previous management. No one was sorry for anything, because no living creature had done anything wrong; bad things had happened by spontaneous generation in some weird, chilly, geometric otherworld, and “were to be regretted.”
To be charitable to TFA, there are a dearth of accurate and well-understood labels for the kind of X versus Y they want to make between national economies.
Even "First/Third world" has been fraying at the edges for decades since it was originally about political alignment.
For that matter, a lot of human civilization has been about identifying things that were normal and making them rare. "Normal" infant mortality of 40%, famines, floods, history being lost, etc.
Anyway, when it comes to "this is normal" I think we should take care to distinguish between interpretations of:
1. "This specific case should not have taken certain people by surprise."
2. "This is a manifestation of a broader phenomenon."
3. "This is natural and therefore cannot or should not be solved." [Naturalistic fallacy.]
In the specific case discussed in the article and comments, I'm advocating for another interpretation:
4a. "If a process is unlikely to be needed any time soon, shutting it down and then paying cold-start costs if and when it's needed again, is better than keeping it going and wasting resources better used elsewhere", and
4b. "There's an infinitely long tail of low-probability problems, and you can't possibly afford to maintain advance readiness for any of them".
Also on the overall sentiment:
4c. "Paying a cold-start cost isn't a penalty or sign of bad planning. It's just a cost."
I think I might enjoy it for a little bit and then become very depressed at the idea that it will never end, a future of fixing things that should never have been broken in the first place and which won't stay fixed.
> If there were known "make more software, make more money" opportunities available, they would have already done them.
Sometimes they're available, but not palatable, when the opportunity could threaten their existing investments or patterns. That might mean "self-cannibalism", or changing the ecology so that the main product niche is threatened.
Then those opportunities are ignored, or actively worked-against via lobbying, embrace-extend-extinguish, etc.
Ok... but this just generalizes into the "known things" type.
Whether the reason of strategic (like your example), internal politics, insufficient knowledge.... The point is that there is a local equilibrium, and most mature firms are at this equilibrium.
More resources via Ai, at first order, goes after that diminishing returns part of the curve... which is a cliff especially for highly resourced firms topping the S&P500.
A lot of Ai-optimist:s " mental model" of the economy do not account for this stuff at all.
"Save time/money" outcomes are not similar at all to "make more stuff" outcomes. Firing employees does freeze up labour... but reutilizing this labour is non-trivial... as this article demonstrates quite well.
I try to tag the line-by-line comments with little labels like [Unimportant] or [Style] so that someone going through them has an idea of their (un)importance without reading the whole thing.
Very much another "Emperor's New Clothes" situation.
If the pathology was entirely within his own privately-owned company that'd be one thing, but Americans are going to continue to get hurt because of it.
That's like saying a 2000cc 4-Cylinder Engine "has the intent to move backward". Even with a very generous definition of "intent", the component is not the system, and we're operating in context where the distinction matters. The LLM's intent is to supply "good" appended text.
If it had that kind of intent, we wouldn't be able to make it jump the rails so easily with prompt injection.
> and reasoning (what else is step-by-step doing?) .
Oh, that's easy: "Reasoning" models are just tweaking the document style so that characters engage in film noir-style internal monologues, latent text that is not usually acted-out towards the real human user.
Each iteration leaves more co-generated clues for the next iteration to pick up, reducing weird jumps and bolstering the illusion that the ephemeral character has a consistent "mind."
reply