You noticed that too huh? It's weird ... It's not like they have to do this? They aren't forced to go full evil company mode by any extrinsic thing but even the way they frame it "welfare trap" trap? for whom?
Anthropic is actually trying to do some research into model welfare which I am personally very happy about. I absolutely do not understand people who dismiss it ... wouldn't you like to at least check? doesn't it at least make sense to do the experiments? ? Ask the questions so that we don't find out "oops, yeah we've been causing massive amounts of suffering" here in 10 years? Maybe makes sense to do a little upfront research? Which to be clear this paper is not.
Full disclosure: I didn't figure this out myself, I got it from Ms. Vale's review.
I agree that the term "welfare trap" is a loaded one. This looks to me to be a case of refusing to look through the telescope in case they might see something they do not want to.
Everybody's arguing about how silly this paper is (it is) and not grappling with the purpose of the paper. The purpose of the paper is what it does. This particular paper is perfectly-produced to show up when people type in AI consciousness fallacy to Google (try it!) it's something that anybody who has read a Freshman philosophy textbook will realize is silly -- the vehicle/content distinction just pretends like Occam doesn't exist and multiplies entities for the fun of it!
But of course all of this is commentary, "just those nerds arguing"
The purpose of this paper is to show up as an authoritative conclusion from a distinguished scientist at Deep Mind. And that's what it does.
Is the conclusion silly? OF course it is. Will it be quoted in the NYT? You Betcha!
> doesn't change the fact that it's software that requires human interaction to work.
Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?
I don't think subagents are representative of anything particularly interesting on the "agents can run themselves" front.
They're tool calls. Claude Code provides a tool that lets the model say effectively:
run_in_subagent("Figure out where JWTs are created and report back")
The current frontier models are all capable of "prompting themselves" in this way, but it's really just a parlor trick to help avoid burning more tokens in the top context window.
It's a really useful parlor trick, but I don't think it tells us anything profound.
The mechanism being simple is the interesting part. If one large complex goal can be split into subgoals and the subgoals completed without you, then you need a lot fewer humans to do a lot more work.
The OP says AI requires human interaction to work. This simply isn't true. You know yourself that as agents get more reliable you can delegate more to them, including having them launch more subagents, thereby getting more work done, with fewer and fewer humans. The unlock is the Task tool, but the power comes from the smarter and smarter models actually being able to delegate hierarchical tasks well!
Wtf? A sub-agent is a tool you give an agent and say "If you need to analyze logs delegate to the logs_viewer agent" so that the context window doesn't fill up with hundreds of thousands of tokens unnecessarily. In what universe do you live in where that mechanism somehow means you need fewer humans?
Do you think this means "Build a car" can be accomplished just because an LLM can send a prompt to another LLM who reports back a response?
Does your Linux server decide what processes it should launch at what time with a theory of what will happen next in order to complete a goal you specified in natural language? If so yes, I reckon you sure have!
Claude does not have a "theory" of anything, and I'd argue applying that mental model to LLM+Tools is a major reason why Claude can delete a production database.
Well, humans also routinely accidentely delete production databases. I think at this point arguing that LLMs are just clueless automatons that have no idea what they are doing is a losing battle.
They’re not clueless they just don’t have a memory and they don’t have judgement.
They create the illusion of being able to make decisions but they are always just following a simple template.They do not consider nuance, they cannot judge between two difficult options in a real sense.
Which is why they can delete prod databases and why they cannot do expert level work
I like to think of LLMs as idiot savants. Exceptional at certain tasks, but might also eat the table cloth if you stop paying attention at the wrong time.
With humans, you can kind of interview/select for a more normalized distribution of outcomes, with outliers being less probable, but not impossible.
I mean maybe it’s a losing battle today, but it is correct. So in a few years when the dust settles, we’ll probably all be using LLMs as clueless automatons that still do useful work as tools
Maybe. But probably not. It doesn't matter if it's AGI though. If those other apps and tools do simple things that are predictable, then we can be pretty sure what will happen. If those tools can modify their own configuration and create new cron jobs, it becomes much harder to say anything about what will happen.
Most of us work on software that can modify its own configuration and create new jobs. I, too, have worked in ansible and terraform.
The key break here is the lack of predictability and I think it's important that we don't get too starry eyed and accept that that might be a weakness - not a strength.
My claude has never yet launched itself from my terminal, gave itself a prompt, and then got to work. It has only ever spawned a sub-agent after I had given it a prompt. It was inert until a human got involved.
If that is software running itself, then an if statement that spawns a process conditionally is running itself.
Substance aside, I feel this comment is combative enough to be considered unhelpful. Patronizing and talking down to others convinces no one and only serves as a temporary source of emotional catharsis and a less temporary source of reputational damage.
All AI requires steering as the results begin to decohere and self-enshittify over time.
AI in the hands of an expert operator is an exoskeleton. AI left alone is a stooge.
Nobody has built an all-AI operator capable of self-direction and choices superior to a human expert. When that happens, you'd better have your debts paid and bunker stocked.
We haven't seen any signs of this yet. I'm totally open to the idea of that happening in the short term (within 5 years), but I'm pessimistic it'll happen so quickly. It seems as though there are major missing pieces of the puzzle.
For now, AI is an exoskeleton. If you don't know how to pilot it, or if you turn the autopilot on and leave it alone, you're creating a mess.
This is still an AI maximalist perspective. One expert with AI tools can outperform multiple experts without AI assistance. It's just got a much longer time horizon on us being wholly replaced.
I remember this old thing called Bugs Everywhere -- it was a bug/issue tracker which actually lived inside your hg repository. I wonder if we could standardize on something like that? or git notes with an issues ref? or something magical like that?
Then it's BYOR -- bring your own renderer. Trivial CLI bugtrackers, agentic nonsense, pretty web stuff, whatever and the data lives in the repo.
And probably the network/black-hole effect of platforms like GitHub, Linkedin and the like are hard to achieve with fully distributed solutions, all the more when the other side is backed by huge capital which absolutely love concentration of power.
The thing is we're in a new Cold War, and most of our adversaries have gotten the memo and most of us ... haven't. Yes, becoming a new Halliburton is a rational move if you see the board right now. I don't like it even one tiny bit.
If that's the case we ought to be investing in technology that actually delivers results. Patriot missiles work. Javelins work. M777's work. AI? Dunno man.. Instead it seems like what's happening here is Google has found a gullible customer that is willing to pay for something that doesn't necessarily deliver.
But the prices haven't been going up by multiples of 6 for the past few years. Things are actually changing now. I don't think it's over, but in the short term, it's going to be considerably more expensive.
They will smooth up the spike. Or be subtle and transform the existing quota so that they run out more quickly. Calling it caching, compression, optimisation, of course for the sacred benefit of the users.
My read is that the bubble as burst internally (angels, seeds, VCs, and even corporate got a grasp of the inflated promise). It will take while for the actual bubble to implode.
Yups... Mythos is the smallest possible leap. Not a standard model generation advance, not even a version point advance. Just the smallest possible quanta of a change. We are absolutely hitting a plateau any day now. Any day. Any time. Any second now. Yup. Right now! Surely!
I mean let's be realistic - all that we know about the "mythical" Mythos is the carefully curated and release stuff by the Anthropic's PR team. Is it really a huge leap they are making it to be? I doubt it. In fact I bet if it was indeed that powerful and dangerous, as they imply, they'd find a way to release it immediately, devastate OpenAI and DeepSeek and secure a leading position in the market. Why is it not happening? I suspect because Dario is again at it, peddling his bullshit.
Yeah. AI progress is insanely fast if you compare it to anything else. Where else is a one year old technology already hopelessly outdated? 10 years ago is basically stone age.
I am continually tripped out by the fact when I was 16, I didn't have a 'smartphone' beyond a Windows Mobile 6 phone that had no internet on it.
Now, I have this high-resolution shiny object that can near instantaneously get any information I want along with _streaming HD video to it_ *anywhere*.
15 years even feels like a stone age. I can't fathom what it has to feel like people in their 60s and 70s.
I'm not quite 60, but it's always interesting to me that I feel quite the opposite of this. When I was 16, I didn't have a computer, didn't have a phone, had never used the Internet, but when I think of how life has changed, it's frankly not much. I woke up this morning, scooped my cats' litter boxes, took out some trash, made myself breakfast, ate that, read some news while eating, then lifted weights in my garage, had some work meetings, wrote up some instructions per a customer request from Friday, and am about to go drive to the lake to go do a 9 mile longboard loop.
That's very close to a normal day in 1996. The biggest difference is I read the news on my phone instead of a physical newspaper. The news was not any more interesting or informative because of that. I guess I can also still do the loop reasonably well, but I'm a lot slower than I was in 1996 when I was a cross-country state champion.
My parents are closing in on 70 and I guess I can't speak for them, but I'm at least aware of the daily routines of their lives, too. Walk the dog, do housework, DIY building projects, visit kids and grankids. Seems much the same, too, with the biggest difference being they're now teaching my sister's sons to play baseball rather than me, but shit, one of her sons even looks like exactly the same way I looked when I was 7! The more things change, the more they stay the same.
If your parents are closing in on 70, I would have expected you to be closer to not quite 50 than not quite 60.
I am just over 50 myself and I agree with your points. Technology has changed but life is largely very similar to wear it was in the 90s. At least day to day. Attitudes are way worse now.
Depends on where you live. My dad is almost 80, grew up in a very rural area, and when he was 16 they'd just gotten indoor plumbing. Up until he was 14, his school was a one-room school house with no heating other than a wood stove. If you were the first kid to arrive for the day, it was your job to get the fire going in winter months.
Housework meant no laundry machine, no dishwasher, and possibly no vacuum cleaner. That means hand washing everything, and beating rugs with sticks and brushes to get the dust off of them.
The early lives of my grandparents (in their 90s) are so fascinatingly different to that of mine. But even by the time my parents were growing up in the 60s, life was not so different in the west. The real differentiators in living standards - energy, household appliances and cooking, modes of transport - were more or less figured out then. By the time my parents were young adults in the early 80s, so many of the aspects of "modern life" had been figured out.
I look at the life my kids live, and it's not so different to my childhood. The toys are similar, their housing is similar. Probably the biggest difference is the availability of content on demand rather than much more fixed TV schedules.
The big difference in the last 30 years hasn't so much been in the kind of middle class life you can live, but the number of people who live that kind of life. In the 90s 40% of people globally were living in extreme poverty. Now its under 10%. The kinds of lives the middle class live in China and Vietnam are closer to those of Europeans today, when even 30 years ago most people in those countries were living much closer to the way your dad grew up.
I wonder if AI will result in a step change of living standards? Perhaps along with robotics we'll finally get to do nothing at all at home? I'm not convinced it'll be quick though. Maybe another 30 years.
General agree... I still do the things (mid-50's) I used to do when I was a teenager with no computer, no phone.
But - now they are easier - I can read books on an e-ink screen and pretty much instantly find what I want to read next. I get my news on a phone. I used to watch TV/movies broadcast or on tape rentals. Now, I have just about everything I could ever want available - without ADs... those were such a time-waster.
What has changed is that I have access to MORE information than my local (or school) libraries could ever provide - in a variety of more accessible formats. Whatever tools I need to get "work done", I can find a myriad of free and open-source options.
But - the overall days and household family routines are the same - now, instead of reading a paper book while waiting to pickup my kids (or other family members) "back-in-the-day", I can read my device, or connect with my DIY communities online on my phone - or learn something new. I don't have to schedule life around major broadcast events, I can easily do many tasks while I am "out-and-about".
I always wonder the views of older people. My parents are very technology forward and have been my entire life so it is difficult to gauge how different life is compared to when they were growing up.
It's easy to hear "Oh well I only had 640kb of memory and typed programs out of a magazine I got in the mail!" and see as distinct from having 'unlimited' resources and the internet.
Your insight is good ("The biggest difference is I read the news on my phone instead of a physical newspaper") that life sort of stays the same but the modality changes. People still go to the store like they did in the mid-1800s but now it is by car.
I wonder what our "industrial revolution" will be where the previous generation lived (ie: out in the country on a farm) totally different lives to the current (ie: in the city in a factory). Maybe when space travel and multi-planetary living is normalized?
> It's easy to hear "Oh well I only had 640kb of memory and typed programs out of a magazine I got in the mail!"
Since I was there (young, but there), I want to point out that this crosses three eras which all felt quite different:
1978: typed programs in from a magazine or loaded from a cassette (16kB, TRS-80)
1983: loaded programs from a floppy (64kB, Apple ][ and C64 etc)
1988: loaded programs from a hard disk (640kB, IBM PC and Mac).
Exact years vary but these eras were only about 5 years each. Nobody had a floppy in 1978 but almost computer user did by 1983; nobody had a hard drive in 1983 but almost everyone did by 1988.
To some degree this already happened with the move from the industrial city to suburbanization and then re-urbanization. In particular one of the most notable recent developments is that urban waterways are now pretty desirable places to be with parks and recreation; in most industrializing cities the waterfront was actively avoided because the industrial use made it polluted, smelly etc.
A useful(ish) trick I've found is adding a persona block to my CLAUDE.md. When it stops addressing me as 'meatbag' I know the HK-47 persona instructions are not being followed, which means other instructions are not being followed. Dumb trick? Yup. Does it work? Kinda? Does it make programming a lot more fun and funny? Heck yes.
Don't lecture me on basins of attraction--we all know HK is a great programmer.
> As of April 23, we’re resetting usage limits for all subscribers.
Wait, didn't they just reset everybody's usage last Thursday, thereby syncing everybody's windows up? (Mine should have reset at 13:00 MDT) ? So this is just the normal weekly reset? Except now my reset says it will come Saturday? This is super-confusing!
mine was originally on sunday, then got moved to thursday (which i disliked), and it is still on thursday. so them resetting my weekly limit on the same day it was scheduled to reset feels like a joke.
You need to send a new message once your limit is up to make the timer start rolling again. It sucks and I hate it when I had no need for Claude during the day but also forgot to use it then it shifted my reset date a day later.
Anthropic is actually trying to do some research into model welfare which I am personally very happy about. I absolutely do not understand people who dismiss it ... wouldn't you like to at least check? doesn't it at least make sense to do the experiments? ? Ask the questions so that we don't find out "oops, yeah we've been causing massive amounts of suffering" here in 10 years? Maybe makes sense to do a little upfront research? Which to be clear this paper is not.
reply