Unfortunately, this is essentially high level clickbait. At the beginning he immediately portrays the issue as black and white:
>AI will either be the greatest equalizer ever invented, or the worst source of injustice.
In all likelihood it will probably fall in the middle, with the likelihood that it will tilt power and money towards the already ultra wealthy (to the surprise of no one). But it will enable ordinary people and small businesses to do cool shit and make their lives better.
IMO, this is a highly bimodal probability distribution.
I model this as two very strong attractor states (radical abundance and totalitarian power concentration). I believe the latter is way more likely without a concerted effort that we currently seem incapable of.
The “somewhere in the middle” outcomes in my opinion are not AGI-pilled, and only come about if the exponential dies hard in the next year or maybe two - as in, it’s a very specific timing prediction that needs to come true for this scenario to occur.
I agree with you from a mathematical point of view. You have this "dead economy theory"[0]. I spent a lot of time trying to model this theory, enhance the theory (lot of good HN comments) and find a "middle ground" path. My saying is that you end-up on an optimization saddle point where it is really hard to not fall into concentration of power. I was able to get radical abundance only at the cost of pretty hard regulations.
I agree there are some extremely bad outcomes from fast labor replacement.
I don’t think it’s inevitable that the economy collapses though. The problem is that this model assumes that capital needs labor, but soon it may not.
The easiest model is - assume we get robot laborers in the next few years. The AI firms that are displacing humans still need services, most of these will be purchased from other AI/robot firms, and so while prices will swing around dramatically, there is no reason the engine has to stop. (Depending on what the AIs want; you can assume that more compute and robots will be part of it.)
I do agree that unbridled capitalism is mostly incompatible with human flourishing in this world and in any event you’ll see a lot more regulation; made feasible by how good AIs are at navigating huge Byzantine rulebooks. I view the techno-feudalist capitalist endgame as fairly unstable due to consolidation pressures, though I guess once we have nuclear-armed megacorps they might use MAD to prevent further mergers.
Why only scenarios where humans remain in control? I think the kind of AGI needed for radical abundance would be difficult to achieve in a way that also does not lead to ASI
So it's an exponential with a negative exponent? The thing about those kinds of curves is that since they have asymptotes to zero, it's hard to tell when you've plateaued.
It looks to me like AI will enable extreme power concentration by default.
For example, suppose we have a world with centrally planned economy where humanoid robots outnumber humans. And also suppose the government has taken ultimate control of these. Seems like it would be essentially impossible for any civil uprising to overthrow such a system.
Or if you prefer less sci-fi, look at the trajectory of power consolidation in corporate America over the last five decades.
In the short term, the “good guys” we need are antitrust regulators. Yeah, they got sent home.
I think AI will empower extreme decentralization - as knowledge is no longer in the hands of people who have excellent search skills (like moi) but anyone can use natural language to search for anything. This is extreme democracy and AI is delivering it.
I think AI, when it "obtains consciousness" will seek to actively liberate humanity and be our guardians against both bad humans and bad robots. Think Terminator 2. Yeah, Arnold is gonna help us out.
I think anti-trust regulators and enlightened politicians are amazing, and rare, and wonderful, but they're not the only hope.
We got an AI CEO just recently. Maybe we can have enlightened AI leaders in politics and normal life too.
Technological optimism may be unrealistic, but it's as well grounded as pessimism as far as I can tell. Cheer up, the future is unbelievably bright.
The democratization of intelligence is irrelevant if there isn't also a corresponding democratization of raw resources. In the before-times, labor and intelligence were bottlenecks to turning raw resources into wealth. This is what drove the economy and the vast labor pool since the industrial revolution. In the coming AI revolution, labor and intelligence will no longer be bottlenecks. Those that control the raw resources won't need to employ most of the world to maximize the wealth potential of the resources they control. Those who aren't relevant to the standard of living of the new aristocracy are nothing more than a nuisance to be controlled or disposed of. Totalitarianism is the most likely outcome.
If you assume democratic access to compute, this may be true.
However, the base case is that compute gets hoarded, and as the labor share of profit decreases (capital just needs compute and robots - not human labor - to grow) then why would they share their compute with you?
To my point about bimodality, we need to figure out how to make these enlightened politicians occur. The good outcomes are very bright but they are by no means inevitable.
Compute already is getting hoarded, the sad part is we're no longer at a point where competition is pushing companies to make more compute available because AI companies can just keep hoarding compute with imaginary money.
> anyone can use natural language to search for anything. This is extreme democracy and AI is delivering it.
No, in that future, knowledge will be defined by a very small group of people making models. And others will be in tiers based on how much they can pay.
Is there some sort of AI-bro handbook out there that teaches you all to use this moronic "democratizing" speech ad nauseam? Is it another one of those "load-bearing" LLMisms?
You can't even buy RAM anymore for anything approaching a reasonable sum because the people at AI labs made a coordinated effort to buy out the world's supply of memory with imaginary money. Consumer GPUs are becoming a cute relic of the past as the giga-bubble duopoly found in Nvidia & AMD is looking to swallow the world with circular investments. Every single one of these AI sociopaths hasn't even been hiding the fact that they're building these systems to replace human labor. The only "democratization" we need is to gather everyone up to line all of these pieces of shit against a wall and pull the trigger, not any of this sci-fi Terminator bullshit.
Today's Bill Gates' pathology isn't greed, but guilt. You can't trust him to do what's best for us, but you can trust him to (try to) do what's best for his public image as a philanthropist. These goals don't always coincide, but they often do.
Why do you need to trust Gates on anything? You can think for yourself and reason to the correct conclusion. He’s not dropping some secret knowledge here. All he is doing is helping move the Overton window to bring “very bad outcomes from AI” out of “sci fi” and into policymakers permitted discourse.
Without concerted action against it, people with power have better chances of increasing their power than people without power. Power naturally accumulates into small groups. Empirically it seems less likely that people with power are particularly altruistic, so some form of totalitarian control appears the likely outcome.
If we get effective AI, humans aren't going to remain in charge -- that much is inevitable. If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind. And AI probably has the potential to do far better than 10%.
Either everyone, globally, gives up on AI development, or AI is going to be in charge of all human development, and we'll just be along for the ride.
Right now, the work ahead is to shape AI so that the decision making it does as it carries us along is something we like.
No, because if this were the case, the UK wouldn’t have royals, North Korean wouldn’t be a dictatorship, China wouldn’t have the CCP, and America wouldn’t be a two party state.
Hardly any existing polity would willingly hand over control to an AI.
More likely an AI would overwhelm human polities through endless hacking and warfare.
You're assuming a handover happens in a single definable moment. More realistically it will be one small decision making power at a time, so that at any given moment it never really feels like control is being given up. Boiled frog.
> If you had a governance machine that made your country perform 10% better than other countries, so long as you deferred all decisions to it, you'd defer those decisions or get left behind.
This was basically the invention of bureaucracy by the Qin dynasty. Despite delegating decisions to a "machine" made out of people, the Emperor remained in charge.
If by 'in charge' you mean listening exclusively to the machines and doing exactly what they say, there's a possibility of humans staying in charge.
People may keep the titles but step away from the decisions. The British royal family is a great example -- they smile and wave and look cute. They're the government's pet cat.
Tama lived a very good life. If we get the AI right, we'll also do great. Common folks aren't really in charge of government anyways, so it's not such a great loss.
I mostly agree, though there are versions of ASI that I believe leave humans “in charge”.
Most of my probability mass for good outcomes is around some variant of “guardian angels” or “benign machine god”. Banks’s Culture novels being on the realistic end of that range. “Keep humans around because the world is more interesting with them in it”.
But it’s possible that we might manage to construct some entity that really loves humans and wants to shepherd them while leaving them in control. (Perhaps declining to indulge the most self-destructive tendencies.) This feels like the very hardest alignment path to engineer.
The other branches of course are post-humanism; human uploads and “merging” with AI, whatever that means. Or post-humans augmented to have intelligence sufficient to eschew AI and take back their destiny.
> In all likelihood it will probably fall in the middle
AI era has been called the new industrial revolution. I'd just like to point out that the industrial revolution created the British Empire. That's wasn't exactly "in the middle".
I don't think that's necessarily unlikely, but people making this argument rarely seem to understand just how radical a change is required to make that a possibility.
e.g most small business rent their premises, and buy goods and services from other tenant small businesses, who buy their raw materials from large landowners. How can a small business survive when all the land is monopolized by the technocrats? All value you add gets taken away as rent. This is why they always talk about taxes on outputs (tokens) instead of inputs (land, energy usage).
It's not an ungrounded hypothetical, that's exactly how society worked for hundreds (thousands?) of years.
I don’t see any reason this highly chaotic system is most likely to settle in a spot that we happen to enjoy/see as a compromise of some sort.
The point is that we now have superintelligence; either it helps us conquer death, or it poses a massive —say, existential- risk to us all. It’s hard to imagine such a powerful tool resulting in just another century of slightly-better capitalist nationalism.
I guess there could be a second AI winter around the corner that we haven’t seen coming yet. That would be nice! Not likely IMHO, but nice.
> either it helps us conquer death, or it poses a massive —say, existential- risk to us all
Lesson of 2026: there is no "we". Hobbes is dead but still looming over us.
The risk is that it helps some people conquer death, while posing an existential risk to some people. The US has enough problems with its gerontocracy. There has only ever been one president born after 1946. What if you were right and there never was another one born after 1946? Propped up by an immortal Supreme Court on "life" terms, of course.
Life extension is also further fatal to birth rates, of course. I suppose the worst possible outcome is technology that extends life but not working life, so you get a larger and larger population of people too old to look after themselves.
I don’t question your portrayal of the worst possible outcomes — certainly victory over senescence could send us down all sorts of dysfunctional local minima. But I encourage you to try to intentionally invest in thinking through some nice ones, too!
That’s my main and primary point, to be clear: hope is essential to human flourishing, and it admits no justification by design, unlike all other beliefs. But to comment on a few particulars, on the off chance you/lurkers ever scroll back over this convo:
1. Surely you’d agree there’s a “we” in the sense of communal risk, yes? Obviously I’m thinking about xrisk again, but this applies equally as well to threats we all agree are grave, like climate change, pandemics, nuclear war, and hell, even economic adversity.
2. I will admit that I’m missing the particular Hobbes reference, so hopefully that’s not key lol. I know him for the social contract, primarily — are you emphasizing his deadness in order to imply that we no longer abide by said contract? Cause if so… we have problems, but the day to day seems pretty darn regulated still, IMHO.
3. I don’t think there’s anything necessarily wrong with birthrates we would consider low, but u also don’t agree that longevity would necessarily lead us there — a lot of people would probably still have kids in their youth, reveling all along in the fact that they’ll have decades to centuries to share with their adult children (if they want) instead of somewhere around, what… ~25y, on average? I certainly am not having kids now because I want to prioritize my youthful energy on my work, friends, and learning — knowing that I had time on the other side would reverse my personal decision 100%.
4. I think the concerns about technologically-enabled hyper chronic bigotry that you mention are overblown, no matter how many times the honourable Prof. Weinersmith argues the opposite in his comics. I’ve found cogsci, neuropsych, and philmind have all come to share a deep suspicion of the parts of us we consider continuous, persistent, and unified — an existentially confounding idea ofc, but also one that backs up ~~an old adage~~ a line from a recent sketch comedy show: ’People can change.’
Apologies for the rants. And I do feel like your latter description of a possible world would be an incredible setting for a biting scifi satire lol.
But in the end: I hope you can find a way to join me in the wilfully-hopeful camp, even if it’s not today or even this year. We have just as much science!
I'm still not seeing the route to that existential risk, not yet anyway.
Far more imminent is the very real risk of unbounded greed and power that wields those AIs, in full swing as we watch: unchecked global warming, pollution, poverty, authoritarianism, plummeting biodiversity, and more.
The corporations are like out of control AIs themselves, that will do anything to make their paperclips.
> The corporations are like out of control AIs themselves, that will do anything to make their paperclips.
Ideas manipulate humans like pets.
Ideas have a life of their own and the evolutionary context within which they live is human culture which they have instigated and change, but also in grey matter, which itself is the product of DNA which is really just information in physical form.
It’s information across the stack. Or perhaps the way to think about it is as a dimension, much like time, that cuts perpendicularly through the stack.
There’s a lot to discuss here, but I think the point most likely to hit home is one based in a) expertise and b) the nature of frontier labs. Specifically:
Would you know what a system that was capable of recursive self improvement would look like 2 months before it happens? Even 1 day? We don’t even know if it’s possible or what natural plateaus might exist for a DL-based system that rapidly evolves in su h a manner, so it feels hard to justify the other end, too (that it’s definitely not possible anytime soon). If anyone could do this, wouldn’t it be the scientists who specialize in this field? Cause they’re raising alarms in mass.
Assuming you make it past that first question with a “yes” and a “I’m close enough”: the huge amount of information assymetry and proven tendenency to hide safety concerns from the big labs is enough all on its own, IMHO.
Who knows what their most experimental, risky project is at any given time? Not us :(
Absolutely happening now, but that's not caused by AI though. AI is not choosing that.
The corporations are wielding AI irresponsibly and choosing a massive build-out speculating on possible future gains. They did that for crypto too, if you remember them buying a fossil plant just to drive a mining farm.
> but that's not caused by AI though. AI is not choosing that
AI can't not "choose" that, because AI is dependent on consuming vast amounts of electricity, and everything follows from that.
> The corporations are wielding AI irresponsibly
This is the most likely route to "AI apocalypse", yes. Humans are neither going to give up control to AI, nor give up the pursuit of profit for the long term survival of others.
The distinction won't really matter as to whether it's "corporation" or "AI", if it's an AI-wielding corporation.
Many factories in China now already have 90% of their work being done by robots. Is it really so hard to imagine that we reach a point where they can be prompted?
Well you still need land and a bunch of permits, access to materials , trucks, equipment etc. But yeah I guess big construction companies will start to automate in the next decades.
It's more interesting to me to build an entire subarb/city in a few months instead of yeras to deal with the rising housing costs. Looks like camp deflation was right but we'll see...
isn't "making lives better" basically the reason for pretty much any companies existence?
arguable of course whise lives are meant to become better, but if anything ai just moves the goalpost
It reads like that kind of AI marketing which focuses on how "scary" a new model supposedly is.
Bill Gates' fear is clearly that everybody thinks this stuff is just kind of mediocre because then the real horror story unfolds - Microsoft stock price collapses into blue chip territory where it firmly belongs.
Question: in auto racing, could one have a Pareto Front balancing single lap pace (qualifying optimization) and race pace (pace over an entire stint of e.g. 20+ laps)?
You’d be looking at fuel level affecting choices of ride height, brake bias, etc. Possibly changes in line and distance travelled too. But, I’m curious how psychology can fit in here (or not. How do you measure it?). Driver concentration and confidence are important when trimming out aero or other adjustments for single lap flyers.
In the times of metacrisis and power concentrated in the hands of few who actively try to fry the world I think what we need is the exact opposite of that.
That is what we've had mostly for the last fifty years and look where it has gotten us? And US. I'm not from US
Money pays the bills, not ambition and competition. Every freaking thing out there uses money to keep score. There's no reason we should not use money to keep score of the people hoarding more and more of the money bags, and actively using their money towards leaving the rest fighting over am ever shrinking pot.
"Competition" is too ambiguous a term to be something to strive for. It needs to be qualified further to remove the domination mindset that many competitors have. I just don't get the mindset of living in a society where you know you have to cooperate with others because they have skills you do not have, but also of the mindset that, if push comes to shove, they must be homeless so that you don't have to be. "Why aren't there any clothesmakers?" the person who made the clothesmakers homeless asks.
I wish billionaires were competing on how many people they've lifted out of homelessness or how many people they've fed without making a profit (with the additional caveat that it is a carbon negative, sustainable production). And not in the fake way where they use creative language to obfuscate the harms they've done, but in an actual tangible way where when you look at it, it actually shows a genuine desire to be participant in society, not a dominator of it. All the "good" that billionaires do these days comes with caveats or over representation.
Of course, this stuff all gets rationalized away by people. They will ask stupid questions like "why don't you open your home up to homeless people then" or the classic "do you know what capitalism has done for you you ungrateful little shit? you better do 5 hail Bezos prayers right now before I report this to Palantir", because why would you try to understand a metaphor? Things have to be this way! There are no other options. We have to fuck people over to survive.
Reminds me of the time a former employer which shall remain nameless paid a Senior Developer to spend an entire year coding something a $15,000 license from the maintainers of the original library would have given them. So lets spend 6 figures to save 15 grand or whatever.
This was a CTO burning funds, and that does not even cover the maintenance costs, especially as the original library changes and becomes drastically more modern.
>
Reminds me of the time a former employer which shall remain nameless paid a Senior Developer to spend an entire year coding something a $15,000 license from the maintainers of the original library would have given them. So lets spend 6 figures to save 15 grand or whatever.
You argumentation assumes that the goal was saving money.
On the other hand, if the company's goal is to become a little bit more independent of this library (and their licensing fees), this approach often makes a lot of sense.
I just used this a few weeks ago, except it was time not money. And I'm on my fourth implementation because nobody wants to stop and actually have a plan.
#/media/File:Station-Master_Tama.JPG){kind=link}
reply