Not anymore than ants or any other species that's spread across the planet. A virus is something co-opts the replication machinery of a host to make more copies. That's not what humans do.
Since this idea may have come from the first Matrix movie, it's ironic that the character who espoused the idea, Agent Smith, became an actual virus in the 2nd and third movies.
Hmm maybe it always feels "different" to every generation going through automation of something previously un-automated, but don't you feel that to some extent this is different.
Humans have been the dominant species because of our ability to think, not our ability to get from A to B in any particular way/cook food. If we abstract ~the bulk of our thinking to LLMs, don't we lose something as a species which makes us more vulnerable... Or is it always the case during an automation?
We're in the midst of figuring out what the tradeoffs are of LLM usage - all the things you list are already mostly known and found worth it. You trade a tiny amount of time and fuel to cook food in exchange for not getting sick. How is that relevant to anything?
I'm making the point that the viral sensationalism in this article can be equally applied to automobiles, computer, telephones, calculators, and toothbrushes.
If the article had merely been what you said "is AI worth the cost" then I would not have taken issue with it.
The core idea being that each time humanity was going to exceed the carrying capacity of its environment, we invented a new technology which increased the headroom. Obviously this ratchet only goes in one direction. The only way to go backwards is collapse.
> What about cooking food? We've become entirely dependent on cooking food.
In evolutionary time, we lost the ability to digest raw meat because we offloaded so much of process of the digestion to the cultural technology of cooking.
Do you like the looks of a future where that happens but for thinking?
Even if AI were superhuman at all thinking tasks, that doesn't mean we'd lose our abilities.
Chess and games can be a lot more demanding than a lot of people's work lives - and AI is already superhuman there.
The heaviest thing I need to lift in my daily life is a laptop and yet I've gotten significantly stronger over the last few years from strength training.
If AI took over all the boring but hard things I'm required to do it'd free up lots of time for me to spend thinking about things, and solve problems I find more interesting.
I’ve encountered several people now who consult LLMs constantly for nearly every decision. Some proportion of people will absolutely cede their thinking to these things, and some already have.
Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines.
What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?
There is a distinction between two important concepts here. There is access consciousness and there is phenomenal consciousness. The actual paper published alongside the j-space blog post discusses the idea of access consciousness if you are interested.
Phenomenal consciousness is what the hard problem of consciousness is about. I personally believe there is a good reason to think the reducibility of this concept to some substrate independent description is equivalent to having a proper definition, and also believe it is not reducible to such a description. Hence the only definition one can give is akin to pointing at the moon and hoping you don’t look at the hand. I.e., phenomenal consciousness is first person experience, the existence of qualia, and that the “lights are on”.
We are biased as educated people to believe all concepts are reducible to substrate independent description but there is no reason to believe this a priori.
The substrate non-independence of consciousness, if correct, could mean we won’t just build one no matter how intelligent and human-like they become.
I agree with you that there is nothing “magical” about our intelligence. However we have as a species effectively believed in magic since Newton decided gravity is a black box that cannot be reduced to the mechanical philosophy, and went on with describing its properties formally. We don’t call it magic, but physics. We call quantum mechanics unintuitive because it disagrees with the Boolean logic that is appear at the every day level. For some reason we keep making this mistake over and over.
(There is one weakness to my view that I aware of. I think there must have at least at some point and maybe still been a role in which the process that results in phenomenal conscious plays in a correlated intelligence, because it seems unlikely to persist through evolution if not. I.e. certain physical processes encode information better than others, and we happen to live in a universe where such process is aligned with consciousness. Substrate independence doesn’t have to contend with this, but it does have worse problems in my view.)
Can we agree your reply is rude given than I have made a fair effort to explain my view to you already? This is not a simple topic and one should not expect the whole thing to come down to a tweet; neither is an accurate understanding of evolution I’ll add.
Which part can I express better? Do you know what I am referring to by phenomenal consciousness?
I apologize if I've caused offense - I knew it would be contentious but I thought it was more important that you understood my perspective from the outset.
I have looked up "phenomenal consciousness" and I understand it to be roughly the qualia of being conscious. I do not understand then what you meant by "reduce to a substrate independent description".
I hope you can understand why I was so dismissive - your large block of text became unintelligible to me only 5% of the way in.
I’m not more offended than by the army of people here informing me that my IQ is low, so no worries. I don’t have the skill to express these ideas more succinctly, and I apologize for this. I’ve tangled a lot, so I can try to express some parts in more organized manner if you’ll bear with me.
(1) An important topic in consciousness studies is Chalmers’ hard problem; the original essay is very readable and I recommend it. He sets it aside from the “easy” problems of explaining how various human functions correspond to neural mechanisms, since he supposes these are clearly achievable by science with time. Many find this name contentious or the framing misguided. The hard problem is to explain how and why phenomenal consciousness exists, i.e., that there is an experience that goes along with some of those neural mechanisms. This is perplexing because if the functional aspects can all be explained mechanically, there is no a priori reason there must be an experience.
(2) Substrate independence is the quality of the essential features of a process to depend only on their information content, not on the underlying medium expressing the information. For instance, the working of an operating system is substrate independent in the sense that it can run on different hardware. It is a commonly held belief that consciousness is like this, and I suspected this was your view. My own belief is that fundamental physical processes are substrate dependent. Each assumption has different explanatory power and issues.
I fear this is not succinct but my urge is to continue.
> I fear this is not succinct but my urge is to continue.
Regardless, I understood all of it, and I thank you for sticking it out with me.
Re (1) - yes, I am familiar. To cut to the chase - I agree that P-Zombies can never be strictly ruled out, but I find this issue to be akin to solipsism.
Re (2) - yes, that does describe me: neurons are no more conscious-making than transistors. But again, I agree this can never be proven strictly.
The most naive versions of substrate independence have a hard time contending with Putnam’s thought experiment showing rocks are conscious (Chalmers has a webpage discussing this also); the idea is that a sequence of machine states can be made to correspond 1-1 to a sequence of rock states. There are stronger versions which start making assumptions about causal and organizational structure, which is probably your view. This suggests at least that consciousness depends not only on information expression, but the manner in which the information expression happens. This doesn’t preclude your position, but it starts to sound like the substrate might matter because certain causal and organizational architectures may be only expressible in our reality in certain ways.
This rocks thing sounds like an argument I heard Searle make one time - that the vibrating molecules in a wall can be thought of as sentient because you can map them to intelligent output (where, of course, the intelligence is actually hidden in the mapping.)
I don't know how to resolve these things - but my suspicion is these issues get resolved like the question of solipsism: we take a practical explanation because it's so useful.
Because, at the end of the day, you're going to be hard-pressed to prove consciousness when it comes from a circular definition.
I look to vitalism for inspiration. I am hard pressed to find inexplicable behavior in evolved chemistry. It remains chemistry.
The explanation I have against solipsism is that I believe that physical things of similar kind have similar behavior, and brains are extremely similar to one another. This is compatible with my belief in substrate non-independence.
> it comes from a circular definition
Right my original post tried to preempt this common view. My view is that there cannot be a definition of consciousness that isn’t effectively like asking the other person to observe it.
But according to solipsism you can't be sure my brain actually exists at all, right?
And for what it's worth - no - I do not believe neurons are so different than transistors. Sure, it could be possible - but really? That feels like a reach for a theory that wants to find a certain conclusion. They're both made of the same protons and electrons, right? They obey the same laws of chemistry and physics. You think consciousness is really in the carbon and not in the silicon? Do you have any justification for that belief? Is it the atomic weight? How could that possibly be?
> My view is that there cannot be a definition of consciousness that isn’t effectively like asking the other person to observe it.
Right. But that's circular, isn't it? Consciousness is what I feel like consciousness is.
> But according to solipsism you can't be sure my brain actually exists at all, right?
Right but my belief is I live in a universe with regular processes depending on regular arrangements. It seems like a relatively weak axiom to hold and precludes the version of solipsism you mention.
> They're both made of the same protons and electrons, right?
They have different arrangements. A MOSFET is just a block of polysilicon sitting on top of another layer of silicon which has been doped to contain some arsenic or boron atoms (+ some other details). I can put some legos together which correspond conceptually to this, but they will have functional differences. The actual substrate is what causes the mosfet to have the function of a transistor. Of course the neurons in the brain are far more complex and have different physical properties.
To me trying to explain how protons and electrons put together results in consciousness feels the same as the pre-Newtonians trying to explain gravity with their mechanical philosophy. The only thing I know is my brain is conscious, and your brain is physically similar to mine. Substrate non-independence is actually the weaker of the two assumptions in a sense because it makes a positive statement about fewer systems.
> That feels like a reach for a theory that wants to find a certain conclusion.
The problem with this kind of thinking is one can impugn the beliefs of either side with underhanded motives. Maybe people self-identify as being moral and ethical and elevating the talking computer gives them status in their circle. Maybe the personhood for AI researcher has a big ego and wants their work to be extremely important today.
I honestly do not find it plausible that we cannot build a consciousness-free intelligent computer system. A computer system can be made out of marbles, or pulleys, or pen and paper. All of these are having an experience if you put the symbols or objects in the right order? It just seems absurd to me.
> Right. But that's circular, isn't it? Consciousness is what I feel like consciousness is.
A priori why must real concepts be reducible in the way you want? Gravity is the thing that makes stuff fall, actually it’s the curvature tensor of the spacetime metric? What is the spacetime metric actually? Maybe something in string theory? Eventually you get to irreducible primitives that simply correspond to something we observe. Why isn’t consciousness just this with fewer steps?
But this is just the "consciousness is based on information, not substrate" position, right? It seems absurd?
I understand that it troubles you that real computation can be done with marbles... but... it can. So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation?
> A priori why must real concepts be reducible in the way you want?
Because everything we do in science is like this? Do you think consciousness is beyond the ability for science to study it?
> Gravity is the thing that makes stuff fall...
I agree that gravity is akin to an axiom handed down from on-high. But it's an extremely simple axiom. And it's possibly explained by yet undiscovered simpler axioms. I see no such axiom for the theory of conciousness, despite our extensive understanding of the physical mechanisms involved. Given its absence, it seems to me like the most reasonable explanation is the one we already have - rather than insisting it still must be undiscovered.
And... I'm sorry but your argument feels exactly like vitalism. Given the repeating pattern of man insisting he has a special place in God's universe and then fighting for it stubbornly but then being proved otherwise... I think it's likely this instance will fall the same way all the others have.
> I understand that it troubles you that real computation can be done with marbles... but... it can.
I don’t understand what you’re saying here. It doesn’t trouble me at all. I’m actually quite confident I could design a simple computer from scratch in a substrate of marbles and an electric motor to bring them back up.
Maybe we should separate these threads otherwise it’ll be too hard to keep branching out.
I must have misunderstood where you were drawing the line of absurdity.
I'm comfortable going slower - although the site won't let me reply as quickly as it let's you.
So I take it that you disbelieve that consciousness/qualia can be created by computation. But you do believe, I suspect, that consciousness/qualia are a physical process. So I'm wondering, is this the only example of non-computable physical process you know of?
I’ll just leave this comment to start a thread to ask you about your position and then figure out how I’d like to address your question later.
Your position seems to have non-trivial difficulties if its own. I take it you don’t believe that a sequence of unique rock states or wall states is conscious, which means consciousness is not merely caused by an abstract mapping to some sequence of information states. I.e., I gather you must believe there is something about the organization or causal structure of the processing necessary for consciousness. Before continuing I’ll share another thought experiment just to suggest why this must be. Put two supposed conscious AIs into conversation for a day. Take a long time writing down the states line by line in a (very!) big notebook. Then cover up each line with other sheets who paper, revealing them line by line in a sequence. From our perspective the same conversation must be getting expressed by the two AIs, but surely we don’t think this results in a new conscious experience for the two AIs modeled by the states.
Okay then my challenge to your position (and mine of course, but this is why I’m led to my beliefs ultimately) is it must explain (1) which kinds of casual and organizational structures count here and (2) how these result in conscious experience.
It seems to me that even though I also cannot meaningfully answer these, believing there is some special physics going on which is yet to be discovered seems necessary. I.e. the how part is that it’s just a feature of that physical process. The which kinds are “the physical ones of the right kind inherently have consciousness”. And I will point out that there has rarely been a time in the last 2000 years when we as humans did not think we were nearly at the end of understanding physics.
Let me apologize for having to discontinue now, or my girlfriend will dump me for being more interested in talking to you than her tonight. Another time!
> So you must believe that consciousness is entirely uncomputable? Do you have any other examples of physical processes that are beyond computation?
I think all physical processes are distinct from their descriptions, and have a real existence. We can describe some things about them but our descriptions are not equal to them.
Oh whoops I replied above apologizing for my delay. I’m starting a new job and they’re threatening to make me do stuff (a great thing!!) But I will likely send my responses tonight.
I'm sorry - I missed this reply - which is why I asked it a second time below.
So - let me know if I have this right - if I created a simulation of you it would act like you and talk like you in all ways - but not actually feel like you?
And if I asked it if it felt like you (being a perfect simulation of you) it would answer as you do "Yes, I feel like me." But it would be what... lying about having feelings?
Do I have that right?
And if so - is it aware that it is lying to me? Does it believe what it's saying? I guess we could never tell, because our only recourse is to ask it, and it would just continue lying. Or, being a p-zombie, it wouldn't be capable of belief at all? Despite its claims?
And if I injured this simulation - it would cry out in pain and beg me to stop? All the while not actually feeling pain?
I'm curious - why does this scenario seem more likely to you than the more obvious explanation?
Why is it more likely that behavior which appears identical to a feeling creature is actually somehow faked?
And where does all this fakery occur?
Presumably my simulation is just simulating the interactions of atoms in the body. Does the body have systems for faking emotions that we are unaware of - that only come online while its being simulated? Or does my simulation have a secret fake emotion system I am not aware of?
I thought about how to respond to this for quite a while tonight (last night now as I fell asleep).
My position is that there must be something wrong with the idea that a behaviorally identical simulation can be made which is not conscious. I think this because I do think phenomenal consciousness is playing a role in behavior, since surely my consciousness is a cause of memories of my consciousness (certain structure of physical matter in my brain has been caused by my consciousness); this is called the self-stultification argument against epiphenomenalism.
So that just reaffirms my position, it does not argue against yours. I believe our physical models of reality are fundamentally limited and even within the limitations far from complete.
A truly behaviorally accurate simulation, by which I mean the simulation has parts corresponding to everything in the original, is a very high bar of course. So to address your point about faking directly, I would not call this faking, since faking seems to require a separate inner process to decide to do the deception, but I believe it is just a highly accurate mimic, and I don’t actually see anything wrong with this.
> A truly behaviorally accurate simulation, by which I mean the simulation has parts corresponding to everything in the original, is a very high bar of course. So to address your point about faking directly, I would not call this faking, since faking seems to require a separate inner process to decide to do the deception, but I believe it is just a highly accurate mimic
Yes, this is the situation which is most interesting, despite the bar.
Ok - we can call it mimicry if you like. That doesn't change my question -
If the simulation isn't using the same mechanism (feelings) as the physical system, then where is this mimicry coming from? Clearly the simulation is just following the laws of chemistry. No mimicry in there. Is it in the original atomic configuration - that seems unlikely.
You seem to be assuming the existence of some mechanism that was not described in the original physical system. Why do you insist that this must be the case, and is not simply the same as the physical system?
I feel like you're on the wrong side of Occam. There is an obvious answer to this question - and yet you're picking the hypothetical/undiscovered/possibly non-existent one.
(Remember, in this situation we are not describing an LLM, but a system generated from the rules of chemistry. I can see how mimicry might apply in the LLM case - but I don't see where it would be hiding in the physical simulation case.)
> Remember, in this situation we are not describing an LLM, but a system generated from the rules of chemistry
However, it is a system with a representation purported to be a correct description of all the relevant rules of chemistry and physics, and we don’t have knowledge of what all the relevant rules are.
And I might add that there is an assumption here that the relevant rules are representable via arbitrary description.
There is a basic idea I hold which we can highlight now. Since you find it implausible that there is such a description as alluded above beyond the currently known theories, you are led to conclude that there must not be anything else in reality. I don’t see why we can assume a priori that every true aspect of reality is arbitrarily representable independently of the substrate. In fact qualia and experience seem to be a first person view of processes which have third person effects. It seems like any precise description of this system could only refer to the third person effects, not the first person effects (since the description is third person).
> I feel like you're on the wrong side of Occam. There is an obvious answer to this question - and yet you're picking the hypothetical/undiscovered/possibly non-existent one.
I don’t think Occam’s razor or questioning my biases etc.. have any explanatory value for the actual question at hand. They might have value for ethics or sociology, but I want to know the actual truth. The substrate independent view which somehow conjures consciousness just seems less plausible than the alternative to me.
You already conceded that the simulation would act exactly as the real item. That is a concession that all the behavior is captured by the laws of chemistry.
The only thing you doubt is that the qualia is captured. And my question is - if the physical object doesn't need qualia to entirely duplicate the behavior of the creature - then where is this qualia coming from then? And why would Mother Nature add a system that is superfluous? And where is it hiding? And why doesn't it follow the laws of chemistry too?
And I also want to point out - if we ask the simulation if he feels conscious, if he has feelings, he will say yes. All of his behavior is captured by the simulation. He will vehemently insist that he does.
So why are you adding something extra? That's where my Occam question originates. Adding something extra is by definition less simple. And I think Occam is entirely relevant here... why would it not be?
May apologies, but I intended to convey the opposite when I said this. I’ll read the rest of this comment later.
> My position is that there must be something wrong with the idea that a behaviorally identical simulation can be made which is not conscious. I think this because I do think phenomenal consciousness is playing a role in behavior, since surely my consciousness is a cause of memories of my consciousness (certain structure of physical matter in my brain has been caused by my consciousness); this is called the self-stultification argument against epiphenomenalism.
OH! Sorry - that's a different position then. Let's address it:
So you think a computer can't simulate a human brain.
But why do you think so? We've never found a physical system we can't simulate with a computer (and believe me, Searle and Penrose have tried).
Given there's no evidence that this is even possible, and for which you have no evidence to support it - you still believe computers can't simulate brains?
Why?
I think this is an even greater stretch than before.
These things act like people. Yes, of course, it is impossible to tell if they actually have feelings or whether they pretend to have feelings.
But that is entirely irrelevant to the core issue - how do you want the thing to behave? And the truth is - we have absolutely no language to express how we want a non-sentient entity to behave without anthropomorphizing.
Or - you tell me - how would you instruct an LLM to behave in this situation without using personification?
And, again, it's irrelevant - except to those people who are terrified of accidentally personifying them. Do they have feelings or are they faking? DOES NOT MATTER. We use them - we need to adjust them - we use the most convenient language to do so. What precisely is so upsetting about that?
The web Claude wouldn't tell me, but Claude Code had no reservations:
EndConversation (deferred tool): use only for sustained user abuse directed
at the assistant, or when the user explicitly asks to see it demonstrated.
Load the full guidance via ToolSearch("select:EndConversation") before using
it.
There's a much more detailed set of instructions that are provided when the agent goes to invoke it:
Given everything we know about them... do you really think they just spout gibberish because it's funny?
There was clearly some method to this madness. You're being willfully dense if you ascribe it to... what... childish vandalism? A long extended coordinated hallucination? What?
EDIT: Oh right - you still think they're Markov chains.
It could be that or it could be nothing. And we have no way to prove one way or the other without access to the agent logs right?
This reminds me of the plot of Hot Fuzz where the officer comes up with a grand narrative of what's was happening but the truth was such a mundane simple thing.
Short of OpenAI, or the agents themselves, telling the truth, we have no way of knowing what's real so let's not get carried away by grand narratives
Agree. It’s a very limited description. Yes, technically LLMs are like Markov chains or n-grams predicting the next token. But it completely ignores the compression and interpolation capabilities achieved with transformers.
> There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
Well - remember that botnets can wield a great deal of computing power.
I'm almost afraid to ask Claude if he could create a distributed LLM.
EDIT: Someone downvoted me - so I went ahead and asked. Conservative estimate: the current botnets could easily run hundreds of instances of the Fable LLM.
> I'm almost afraid to ask Claude if he could create a distributed LLM.
Or you just add a lot of randomness to a bunch of small semi-smart LLMs. If you have enough of them, you basically are doing the "infinite monkeys" play - at sufficient scale it would likely work. Then add smart coordination and you've got something interesting.
Think of how bacteria can do horizontal gene transfer. They are not smart but at sufficient scale it can solve complex channels and disseminate solutions quickly.
"You should have a sub-hammer to adversarially enforce that your primary hammer accurately drives nails into wood"
We wouldn't accept such behavior from any other tool, machine, or computer program. At least most of us would not. To paraphrase the old saying: Some people, when confronted with a problem with an AI model think "I know, I'll use an additional AI model." Now they have two problems.
I find it very intriguing that two lineages of models -- from two different companies -- that are supposed to be painstakingly RL'd to become human-level programmers are actually consistently bad at it out of the box in very similar ways. One would think that at least one of OpenAI and Anthropic would (could) have pushed their model to a direction different from "if you can solve a 100-line problem in 10,000 lines, do it".
Human level programmers generally suck too. Unending security holes have been human behavior ever since we started sharing computers.
Also, programming isn't a destination, it is a process and everyone has a different ruleset on what that process looks like. Writing a computer program of any complexity is not a determinalistic process. Trying to compare it to a detemanilisic tool will just leave you with broken analogies and a poorer understanding of the world.
Or another way to say this. If programming were deteminalistic we'd have solved it without AI.
> We wouldn't accept such behavior from any other tool, machine, or computer program.
Von Neumann, 1952: "Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components." The whole result is that you get an arbitrarily reliable machine out of unreliable parts by adding more unreliable parts and voting. Every ECC memory and RAID array descends from it.
But why stop at rewriting in another language. Get another sub agent to invent a new language, create a database, query language and maybe another few DSLs. Then you've got an ecosystem!
You can now re-position your initial solution and sell the client access to some agents that will implement & configure the ecosystem to suit their initial needs!
And don't forget the agents that you'll need to train the customer to use the whole thing!
Sorry, I wasn't implying your advice doesn't carry weight. Was more just thinking about the things that (used to) happen when you introduce more parties to process of creating software.
I wonder if I would need a non-openai agent to enforce it.. I have tried so far with skills and agents.md and code stills end up over engineered to the moon.
Will ask OpenAI to write me that agent! Hope the agent is not over engineered or else unsure how to solve the bootstrap puzzle :D
There's a misconception that a mistake one instance of a model makes will be repeated by other instances. In my experience most mistakes are just sampling noise, and having the same model (with empty context) check the work usually fixes it.
It's super easy - just ask your agent to set up a code review commit gate.
In omp you can also have the advisor role, which is off by default, you can enable it with /advisor command. It acts as a model that reviews the default agent's work in the background.
I am omp pilled, but as the other comments say, any good harness lets you do this in one or the other way.
unrelated: all my homies use their claude subs with omp, and aside from sometimes having to rety the connections, it works, and nobody got banned (yet)
What about cooking food? We've become entirely dependent on cooking food.
reply