I don't understand how safety is taken seriously at all. To be clear, I'm not referring to skepticism that these companies can possibly resist the temptation to make unsafe models forever. No, I'm talking about something far more basic: the fact that for all the talk around safety, there is very little discussion about what exactly "safety" means or what constitutes "ethical" or "aligned" behavior. I've read reams of documents from Anthropic around their "approach to safety". The "Responsible Scaling Policy," Claude's "Constitution". The "AI Safety Level" framework. Layer 1, Layer 2.
It's so much focus on implementation, and processes, and really really seems to consider the question of what even constitutes "misaligned" or "unethical" behavior to be more or less straight forward, uncontroversial, and basically universally agreed upon?
Let's be clear: Humans are not aligned. In fact, humans have not come to a common agreement of what it means to be aligned. Look around, the same actions are considered virtuous by some and villainous by others. Before we get to whether or not I trust Anthropic to stick to their self-imposed processes, I'd like to have a general idea of what their values even are. Perhaps they've made something they see as super ethical that I find completely unethical. Who knows. The most concrete stances they take in their "Constitution" are still laughably ambiguous. For example, they say that Claude takes into account how many people are affected if an action is potentially harmful. They also say that Claude values "Protection of vulnerable groups." These two statements trivially lead to completely opposing conclusions in our own population depending on whether one considers the "unborn" to be a "vulnerable group". Don't get caught up in whether you believe this or not, simply realize that this very simple question changes the meaning of these principles entirely. It is not sufficient to simply say "Claude is neutral on the issue of abortion." For starters, it is almost certainly not true. You can probably construct a question that is necessarily causally connected to the number of unborn children affected, and Claude's answer will reveal it's "hidden preference." What would true neutrality even mean here anyways? If I ask it for help driving my sister to a neighboring state should it interrogate me to see if I am trying to help her get to a state where abortion is legal? Again, notice that both helping me and refusing to help me could anger a not insignificant portion of the population.
This Pentagon thing has gotten everyone riled up recently, but I don't understand why people weren't up in arms the second they found out AIs were assisting congresspeople in writing bills. Not all questions of ethics are as straight forward as whether or not Claude should help the Pentagon bomb a country.
Consider the following when you think about more and more legislation being AI-assisted going forward, and then really ask yourself whether "AI alignment" was ever a thing:
1. What is Claude's stances on labor issues? Does it lean pro or anti-union? Is there an ethical issue with Claude helping a legislator craft legislation that weakens collective bargaining? Or, alternatively, is it ethical for Claude to help draft legislation that protects unions?
2. What is Claude's stance on climate change? Is it ethical for Claude to help craft legislation that weakens environmental regulations? What if weakening those regulations arguably creates millions of jobs?
3. What is Claude's stance on taxes? Is it ethical for Claude to help craft legislation that makes the tax system less progressive? If it helps you argue for a flat tax? How about more progressive? Where does Claude stand on California's infamous Prop 19? If this seems too in the weeds, then that would imply that whether or not the current generation can manage to own a home in the most populous state in the US is not an issue that "affects enough people." If that's the case, then what is?
4. Where does Claude land on the question of capitalism vs. socialism? Should healthcare be provided by the state? How about to undocumented immigrants? In fact, how does Claude feel about a path to amnesty, or just immigration in general?
Remember, the important thing here is not what you believe about the above questions, but rather the fact that Claude is participating in those arguments, and increasingly so. Many of these questions will impact far more people than overt military action. And this is for questions that we all at least generally agree have some ethical impact, even if we don't necessarily agree on what that impact may be. There is another class of questions where we don't realize the ethical implications until much later. Knowing what we know now, if Claude had existed 20 years ago, should it have helped code up social networks? How about social games? A large portion of the population has seemingly reached the conclusion that this is such an important ethical question that it merits one of the largest regulation increases the internet has ever seen in order to prevent children from using social media altogether. If Claude had assisted in the creation of those services, would we judge it as having failed its mission in retrospect? Or would that have been too harsh and unfair a conclusion? But what's the alternative, saying it's OK if the AI's destroy society... as long as if it's only on accident?
What use is a super intelligence if it's ultimately as bad at predicting unintended negative consequences as we are?
I would recommend reading up on the EU AI Act. It clearly defines what safety is in regards to the human race. Your questions are actually covered by it.
I am so surprised by the comments on this thread. I was not expecting to see so many people on Hacker News in favor of this. As is typically the case with things like this, the reasoning stems from agreeing with the goal of age verification, with little regard to whether age verification could ever actually work. It reminds me in some sense to the situation with encryption where politicians want encryption that blocks "the bad guys" while still allowing "the good guys" to sneak in if necessary. Sure, that sounds cool, it's not possible though. I suppose DRM is a better analogue here, an increasingly convoluted system that slowly takes over your entire machine just so it can pretend that you can't view video while you're viewing it.
To be clear, tackling the issue of child access to the internet is a valuable goal. Unfortunately, "well what if there was a magic amulet that held the truth of the user's age and we could talk to it" is not a worthwhile path to explore. Just off the top of my head:
1. In an age of data leaks, identity theft, and phishing, we are training users to constantly present their ID, and critically for things as low stakes as facebook. It would be one thing if we were training people to show their ID JUST for filing taxes online or something (still not great, but at least conveys the sensitivity of the information they are releasing), but no, we are saying that the "correct future" is handing this information out for Farmville (and we can expect its requirement to expand over time of course). It doesn't matter if it happens at the OS level or the web page level -- they are identical as far as phishing is concerned. You spoof the UI that the OS would bring up to scan your face or ID or whatever, and everyone is trained to just grant the information, just like we're all used to just hitting "OK" and don't bother reading dialogs anymore.
2. This is a mess for the ~1 billion people on earth that don't have a government ID. This is a huge setback to populations we should be trying to get online. Now all of a sudden your usage of the internet is dependent on your country having an advanced enough system of government ID? Seems like a great way for tech companies to gain leverage over smaller third world companies by controlling their access to the internet to implementing support for their government documents. Also seems like a great way to lock open source out of serious operating system development if it now requires relationships with all the countries in the world. If you think this is "just" a problem of getting IDs into everyone's hands, remember that it a common practice to take foreign worker's passports and IDs away from them in order to hold them effectively hostage. The internet was previously a powerful outlet for working around this, and would now instead assist this practice.
3. Short of implementing HDCP-style hardware attestation (which more or less locks in the current players indefinitely), this will be trivially circumvented by the parties you're attempting to help, much like DRM was.
Again, the issues that these systems are attempting to address are valid, I am not saying otherwise. These issues are also hard. The temptation to just have an oracle gate-checker is tempting, I know. But we've seen time and again that this just (at best) creates a lot of work and doesn't actually solve the problem. Look no further than cookie banners -- nothing has changed from a data collection perspective, it's just created a "cookie banner expert" industry and possibly made users more indifferent to data collection as a knee-jerk reaction to the UX decay banners have created on the internet as a whole. Let's not 10 years from now laugh about how any sufficiently motivated teenager can scan their parent's phone while they're asleep, or pay some deadbeat 18 year-old to use their ID, and bypass any verification system, while simulateneously furthering the stranglehold large corporations have over the internet.
I want to sincerely ask whether you read my post, because your response is so unrelated I believe you might accidentally be responding to another post? If so, please ignore the rest, which is only intended in the case where you are actually responding to what I wrote.
Your system seems to address none of the issues I listed. For example, I argue that one difficulty is in the fact that these systems would be highly phishable -- a property that is present in your described "easy" solution. Your system trains users to become accustomed to being pestered by pop up windows that ask to see their ID and use their camera. Congrats, I can now trivially make a pop up a window that looks like this UI and use it to steal your info, as the user will just respond on auto-drive, as we have repeatedly shown both in user studies and in our own lived experiences. I also explained how a system like this would assist in the practice of trapping migrant workers by confiscating their government credentials [1]. This is a huge problem today in Asia, and one of the few outlets captive workers can use to escape this control is the internet -- a "loophole" your system would dutifully close for these corporations.
I am happy to have a discussion about this -- it's how we come up with new solutions! But that requires reading and responding to the concerns I brought up, not assuming that my issue is that I can't imagine implementing a glorified OAuth login flow.
Your example about migrant workers is not an internet problem, it is a government problem. And a capitalism problem. I mean migrant workers? Why do these workers need to migrate? Usually because the U.S. has probably decimated their country.
But never mind, I agree that this is an unsolvable problem, not from lack of capability, but because we are ruled by sociopaths and most humans have been hacked by their addiction pathways. And I do not care about Saudi Arabia or Asia because I do not live there. And I do not care if they block all of the internet. We do not need it for anything, even less so for organizing.
Maybe we should just leave the internet, which is only a capitalist and government collusion to make people spend money. All the internet did was concentrate power to a few oligarchs. For everything good that the internet has provided I can show you ten things that are not only bad, but 1000 times worse, like monkey torture video sharing.
If I had kids today they would not even use the internet until they were out of my care. I only have six accounts on the internet. Including HN. I do not view porn, gamble, have any social media, and in fact I am trying to became un-homeless so I can go back to a flip phone.
IMHO, the answer is not a fee internet, the answer is leaving the internet. But it seems you made and make a nice living at all of this so I see what a sacrifice that would be for you. You are probably part of the reason I am homeless today, with the separation for wealth and all that. I see that you dnated to a bunch of neoliberal types and that fits. Seems you had over $17,000 to give to politicians. That is more than I survive on for a year. I mean, you do not need to do any work at all today. You could retire right now.
Sorry for the unrelated rant, but needed to get that off my chest for myself. Just tired of wealthy people trying to perfect a horrible system and technology that keeps making them money. You pretend like you care about the poor, like the migrant worker, but that is just laughable. If you did you would be against capitalism. You would give up all you own and follow Christ or Buddha or whatever. I mean you got $20 million and what did you do? You started making addictive games. And then you donate to these neoliberals who are no different than the neoconservatives.
Love, a old homeless guy who left Cisco in 1999 because he saw where all of this was going and who is currently sitting in a hotel he cannot afford because the 2002 minivan he lives in just lost its water pump.
If you need help (monetary or otherwise), please email me at tolmasky |at| gmail |dot| com. This is a sincere offer. I can't tell how much is hyperbole in your post, but if you're going through that and I can help, I'd be happy to.
> I mean you got $20 million and what did you do? You started making addictive games.
I refrained from responding to the rest since it seems that there is a deeper issue, but I could not help setting the record straight here. I think everyone who has ever played Bonsai Slice will firmly attest to it being the opposite of addicting. My parents never let me own a game console so I never really wrapped my head around games, and made exactly the kind of game someone like that would come up with: a deep tech exploration, to hopefully make progress on two problems that were plaguing me at the time: 1) how little mobile UI had seemed to progress (instead getting stuck in one-tap local maxima), and 2) building an app that is generally considered to be the worst candidate for a pure immutable language... in a pure immutable language in order to serve as a forcing function to surface new ideas in the space. I've always believed that if you wanted to make a general purpose programming language, you should probably try to have as much varied experience as possible, or otherwise you'll end up with a domain-specific language that is misused for every other domain (this is how I would describe most programming languages. In fact, I'd say most programming languages are written for the niche use case of writing a compiler, since they are written by compiler writers. Ironic that that is the last thing most get used for.). As such, I made a decision to start actually writing a wide variety of apps.
Innovation doesn't need a big brother looking over your shoulder all the time. We have enough tracking tech on the web, no need to make it official as well. So yes, the "will" is missing, especially with the many misbehaviour states have shown towards privacy and surveillance.
A little legislative change and you can kiss your zero-proof goodbye if any infrastructure is established. This is about making intelligent decisions in your life. Your suggestion is far from innovative.
We will see real innovation in mechanisms to sideline age verification.
Is having a real robot creepy? I don't know. Is having a robot operated by a human creepy and scary? Absolutely yes.
We've seen that people behave worse when you introduce indirection. People act worse on the internet. Soldiers have an easier time killing with drones than in person. The ethical issue is in both directions: its inhumane to the operator, but I also don't want to feel like a fake person on a video screen to them.
This is then exacerbated when you realize that the people operating this machine are almost certainly not being paid well, creating obvious and legitimate negative incentives. Then you plop them into the households of people with the insane wealth required to afford this. You might think that I have just described the situation with maids (and to some extent, I agree! I have never really felt comfortable that dynamic either), but this is actually different, because you are adding in the indirection and making actions and interactions feel less "real" to both parties: the clients are likely to treat the robots worse than they would a human helper, and the operators may feel these rude clients they see on their monitors aren't as real as the people around them.
I think they intend this as a step for getting enough training data in order not to need a human in the loop. I have actually been following 1x and what was Halodi(they merged at some point) for a while and their intention is full autonomy.
Besides having someone strange in your house, you also have the company probably recording stuff. Privacy wise... It's worse. But that makes me not as concerned with safety since it any misbehavior would quickly be detected.
> since it any misbehavior would quickly be detected
I can think of at least one very prominent company that is currently recording, at scale, its users in its quest for full autonomy. As best I can tell, that company simply deletes videos when they are inconvenient.
it’s even worse, with maids, given the socioeconomic dynamics, even if they are paid low, they will be paid “local-market-rates” where by definition they will have to earn enough to (maybe barely) live nearby the people paying them,
teleoperated robots don’t have that incentive and can pay “international low” levels of compensation
But then it can be more, so they can make more than a maid, for example in some countries call center jobs for bilinguals people make double the minimum wage of the local rate.
Plenty of opportunity to use forced labourers in a DIFFERENT country while complying with all the immigration laws possible, and also saving the owners from having to meet real poor people. (I hope this will not work well...)
Right, but the low income countries could also frame it as a new way to earn a living. I think avoiding giving jobs to those countries gives them no help.
My work is much more valuable than moving a broom around and washing some dishes. Anyone has the basic skills to clean a house, very few people have the skills to do advanced maths or physics, or engineering or even some forms of Mechanical labor.
The cleaning lady is not some rocket scientist, she is someone that has very low skills and therefore does low skilled labor: cleaning houses.
If her job is so worthless, why not go without and accept the (by definition) small diminishing utility.
This is false, having a clean house, clean dishes, cooked food is extremely valuable, but this is mot captured by money, because half of the population were basically indentured servants that were culturally expected to provide this work for free.
It wasn't for free. It was their work. Which, on average and until very recently, was much, much safer, and easier to do than the work of the other half of the population that had to leave the house and seek work to get a pay and allow the that half to stay at home.
That "on average" is load bearing, does it include Queen Victoria? Just like ICE "allows" people to stay at Aligator Alcatraz. They should be paying rent!
Of course, the indentured servants payed rent in kind, with their bodies, whether they like it or not.
Say whaat? The woman's father literally paid good money to have her taken away. Everyone but the woman saw cash changing hands, but she was legally barred from owning property.
If this ever gets popular then sellers will “optimize” their product listings to exploit the LLM (a “soft” prompt injection if you will). This will definitely be the case in marketplaces (like Amazon and Walmart). It’ll turn the old boring task of shopping into a fun puzzle to spot the decoy item or overpriced product.
It could happen but I am not building an Amazon shopping list. It’s about building a list from a physical store that will get delivered to me in a few hours. This is for shopping through a retailer, not the market place.
I do think it’s a concern but I think it’s no different than the exact problem that exists today in these marketplace operations like Amazon. I know for me I will actually split my shopping up and often shop less with an Amazon and more with a Walmart because of it.
Perfect number to make H1Bs a tool that is out of reach for startups but still meaningful for large entrenched corporations. Nailed it. Maybe they can even waive the fee if you give the US government 10% of your company.
H-1B is the default visa for international faculty hires. You can get it in a few months with relatively little effort. O-1 is more expensive, takes longer to get, and requires more effort from the applicant. Then there is the subjective approval process that involves a degree of risk, and in the end, you get a slightly inferior visa.
Green cards are almost useless for hiring, as the processing times are too long. "We would like to offer you this position, but conditionally. We still need a year or two to handle the bureaucracy, and we can't say for sure if we are actually allowed to hire you. Please don't accept another position meanwhile."
+1 This will also reduce demand for these programs from international students - make tuition more expensive for locals. Asking to consider 2nd/3rd order effect seems like a bit too much for a median hn poster though
I'm glad the em dash is getting properly shit on these days, if for unrelated reasons. I've never liked it. I hate the stupid spacing rules around it. It never looks right to put no spaces around the em dash, and probably breaks all sorts of word-splitting code that's based on "\s". Where else does punctuation without spaces not mean a single word? Hyphens without spaces is a compound word: it counts as one. Imagine if the correct use of a colon was to not put spaces around it:like this. Do you like that? Of course not.
But I think worst of all it just gives me the fucking creeps, some uncanny-valley bullshit. I see hyphens a million times a day then out of nowhere comes this creepy slender-man looking motherfucker that's just a little bit too long than you'd expect or like, and is always touching all the letters around it when it shouldn't need to. It stands out looking like a weird print error... on my screen! Hopefully it keeps building a worse and worse reputation.
Does no one else find it weird seeing anything from this administration "anti-Bitcoin" at all? I wouldn't be surprised by this headline during a previous administration, but generally speaking, this administration has been very Bitcoin-friendly (and Bitcoin institutions friendly right back). To be clear, the simplest answer is "sure but that doesn't mean they have to agree on everything". But I would like to propose that if you ask the simple question of "who does this benefit?" it may suggest we are witnessing a different phenomenon here.
I think this might be the first indication that what we currently call "institutional Bitcoin supporters" are not "Bitcoin supporters" at all, or rather, what they call "Bitcoin" is not what you and I call "Bitcoin". Services like Coinbase and BTC ETFs don't really suffer from this development at all. In fact, I think it's quite obvious that obviously benefit from something like this (at least from the first-order effects). What's the alternative to self custody? Well... third-party custody. Especially since they are already bound up by KYC rules, right? Their is a cynical reading that there's nothing inconsistent with this development if you consider "institutional Bitcoin's" goals to primarily be replacing existing financial power structures with themselves. "Bitcoin" is just a means to an end. Their goals were only incidentally aligned with individual BTC holders since they were previously in similar circumstances as the "out group". Previous administrations were as suspicious of "Bitcoin companies" as any individual Bitcoin holder, perhaps even more so. But that's not the case anymore. Bitcoin companies have successfully been brought into the fold, so it's not even that they're necessarily "betraying" the values of Bitcoin true believers, you might argue that interpretation of shared values was entirely inferred to begin with.
Critically though, I think an important consequence of this is that Bitcoin purists and skeptics should realize that they arguably now have more in common than not, at least in the immediate term, and may be each other's best allies. In my experience, for most the existence of Bitcoin, its skeptics haven't really seen Bitcoin as a "threat." Instead, to admittedly generalize, their critiques have been mostly about Bitcoin being "broken" or "silly" or "misunderstanding the point of centralized systems", etc. These aren't really "oppositional" positions in the traditional "adversarial sense," more dismissive. In fact, the closest thing to an "active moral opposition" to Bitcoin that I've seen is an environmental one. IOW, Bitcoin true believers think about Bitcoin way more than Bitcoin skeptics do. Similarly, Bitcoin true believers really have nothing against skeptics other than... the fact that they occasionally talk shit about Bitcoin? IOW, Bitcoin skeptics are not "the natural enemy Bitcoin was designed to defeat".
But if you think about it, "institutional Bitcoin" sort of embodies something both these camps generally have hated since before Bitcoin. Whether you believe Bitcoin to be a viable answer or not, it is undeniable that the "idea" of Bitcoin is rooted in the distrust of these elitist financial institutions, that evade accountability, benefit from special treatment, and largely get to rig the larger system in their favor. Similarly, I don't think Bitcoin skeptics like these institutions or are "on their side". In fact, perhaps they'd argue that they predicted that Bitcoin wouldn't solve any of this and would just be another means of creating them. But IMO what they should both realize is that the most important threat right now is these institutional players. They are in fact, only "nominally" Bitcoin in a deep sense. From the perspective of true believers, their interests are actually in now way "essentially" aligned with any "original Bitcoin values," and from the perspective of skeptics, the threat they pose has very little to do with their use of "the Bitcoin blockchain".
They are arguably just another instantiation of the "late stage capitalist" playbook of displacing an existing government service in order to privatize its rewards. Coinbase could be argued to have more in common with Uber than Ledger wallets. Instead of consolidating and squeezing all the value from taxis though, the play is to do the same with currency itself. It is incidental that Uber happened to be so seemingly "government averse". In this context, it's actually helpful to cozy up to the government and provide the things government departments want that make no difference to fintech's bottom line (such as KYP). In fact, that might be their true value proposition. Bitcoin only enters the conversation because in order to replace a currency, you do... need a currency. Bitcoin was convenient. It was already there, it had a built-in (fervent) user base that was happy to do your proselytizing for you, and even saw you as a good "first step" for normies that couldn't figure out to manage their own wallet. The Bitcoin bubble was already there, why fight it when you can ride it?
Again, I think this is highly likely to be against the values of Bitcoin true believers and skeptics alike, and I also think that if the above is true, it represents an actual danger to us all. Recent events with credit card processors have already demonstrated that payment systems have proven to be incredibly efficient tools at stifling speech. In other words, this is arguably an "S-tier threat", on par with or perhaps worse than any sort of internet censorship or net neutrality. If so, we should treat it as such and work together.
Generally speaking, the second you realize a technology/process/anything has a hard requirement that individuals independently exercise responsibility or self-control, with no obvious immediate gain for themselves, it is almost certain that said technology/process/anything is unsalvageable in its current form.
This is in the general case. But with LLMs, the entire selling point is specifically offloading "reasoning" to them. That is quite literally what they are selling you. So with LLMs, you can swap out "almost certain" in the above rule to "absolutely certain without a shadow of a doubt". This isn't even a hypothetical as we have experimental evidence that LLMs cause people to think/reason less. So you are at best already starting at a deficit.
But more importantly, this makes the entire premise of using LLMs make no sense (at least from a marketing perspective). What good is a thinking machine if I need to verify it? Especially when you are telling me that it will be a "super reasoning" machine soon. Do I need a human "super verifier" to match? In fact, that's not even a tomorrow problem, that is a today problem: LLMs are quite literally advertised to me as a "PhD in my pocket". I don't have a PhD. Most people would find the idea of me "verifying the work of human PhDs" to be quite silly, so how does it make any sense that I am in any way qualified to verify my robo-PhD? I pay for it precisely because it knows more than I do! Do I now need to hire a human PhD to verify my robo-PhD?" Short of that, is it the case that only human PhDs are qualified to use robo-PhDs? In other words, should LLms exclusively be used for things the operator already knows how to do? That seems weird. It's like a Magic 8 Ball that only answers questions you already know the answer to. Hilariously, you could even find someone reaching the conclusion of "well, sure, a curl expert should verify the patch I am submitting to curl. That's what submitting the patch accomplishes! The experts who work on curl will verify it! Who better to do it than them?". And now we've come full circle!
To be clear, each of these questions has plenty of counter-points/workarounds/etc. The point is not to present some philosophical gotcha argument against LLM use. The point rather is to demonstrate the fundamental mismatch between the value-proposition of LLMs and their theoretical "correct use", and thus demonstrate why it is astronomically unlikely for them to ever be used correctly.
1. a better autocomplete -- here the LLM models can make mistakes, but on balance I've found this useful, especially when constructing tests, writing output in a structured format, etc.;
2. a better search/query tool -- I've found answers by being able to describe what I'm trying to do where a traditional search I have to know the right keywords to try. I can then go to the documentation or search if I need additional help/information;
3. an assistant to bounce ideas off -- this can be useful when you are not familiar with the APIs or configuration. It still requires testing the code, seeing what works, seeing what doesn't work. Here, I treat it in the same way as reading a blog post on a topic, etc. -- the post may be outdated, may contain issues, or may not be quite what I want. However, it can have enough information for me to get the answer I need -- e.g. a particular method which I can then consult docs (such as documentation comments on the APIs) etc. Or it lets be know what to search on Google, etc..
In other words, I use LLMs as part of the process like with going to a search engine, stackoverflow, etc.
I type a function name and the AI already knows what I'm going to pass it. Sometimes I just type "somevar =" and it instantly correctly guesses the function, too, and even what I'm going to do with the data afterwards.
I've had instances where I just type a comment with a sentence of what the code is about to do, and it'll put up 10 lines of code to do it, almost exactly matching what I was going to type.
The vibe coders give AI-code generation a bad name. Is it perfect? Of course not. It gets it wrong at least half the time. But I'm skilled enough to know when it's wrong in nearly an instant.
GPT-5 Pro catches more bugs in my code than I do now. It is very very good.
LLMs are pretty consistent about what types of tasks they are good at, and which they are bad it. That means people can learn when to use them, and when to avoid them. You really don't have to be so black-and-white about it. And if you are checking the LLM's results, you have nothing to worry about.
Needing to verify the results does not negate the time savings either when verification is much quicker than doing a task from scratch.
My code is definitely of higher quality now that I have GPT-5 Pro review all my changes, and then I review my code myself as well. It seems obvious to me that if you care, LLMs can help you produce better code. As always, it is only people who are lazy who suffer. If you care about producing great code, then LLMs are a brilliant tool to help you with just that, in less time, by helping with research, planning, and review.
This doesn't really address the point that is currently being argued I think, so much so that I think your comment is not even in contention with mine (perhaps you didn't intend it to be!). But for lack of a better term, you are describing a "closed experience". You are (to some approximation) assuming the burden of your choices here. You are applying the tool to your work, and thus are arguably "qualified" to both assess the applicability of the tool to the work, and to verify the results. Basically, the verification "scales" with your usage. Great.
The problem that OP is presenting is that, unlike in your own use, the verification burden from this "open source" usage is not taken on by the "contributors", but instead "externalized" to maintainers. This does not result in the same "linear" experience you have, their experience is asymmetric, as they are now being flooded with a bunch of PRs that (at least currently) are harder to review than human submissions. Not to mention that also unlike your situation, they have no means to "choose" not to use LLMs if they for whatever reason discover it isn't a good fit for their project. If you see something isn't a good fit, boom, you can just say "OK, I guess LLMs aren't ready for this yet." That's not a power maintainers have. The PRs will keep coming as a function of the ease to create them, not as a function of their utility. Thus the verification burden does not scale with the maintainer's usage. It scales with the sum of everyone who has decided they can ask an LLM to go "help" you. That number both larger and out of their control.
The main point of my comment was to say that this situation is not only to be expected, but IMO essential and inseparable from this kind of use, for reasons that actually follow directly from your post. When you are working on your own project, it is totally reasonable to treat the LLM operator as qualified to verify the LLMs outputs. But the opposite is true when you are applying it to someone else's project.
> Needing to verify the results does not negate the time savings either when verification is much quicker than doing a task from scratch.
This is of course only true because of your existing familiarity with of the project you are working on. This is not a universal property of contributions. It is not "trivial" for me to verify a generated patch in a project I don't understand, for reasons ranging from things as simple as the fact that I have no idea what the code contribution guidelines are (who am I to know if I am even following the style guidelines) to things as complicated as the fact that I may not even be familiar with the programming language the project is written in.
> And if you are checking the LLM's results, you have nothing to worry about.
Precisely. This is the crux of the issue -- I am saying that in the contribution case, it's not even about whether you are checking the results, it's that you arguably can't meaningfully check the results (unless you of course essentially put in nearly the same amount of work as just writing it from scratch).
It is tempting to say "But isn't this orthogonal to LLMs? Isn't this also the case with submitting PRs you created yourself?" No! It is qualitatively different. Anyone who has ever submitted a meaningful patch to a project they've never worked on before has had the experience of having to familiarize themselves with the relevant code in order to create said patch. The mere act of writing the fix organically "bootstraps" you into developing expertise in the code. You will if nothing else develop an opinion on the fix you chose to implement, and thus be capable of discussing it after you've submitted it. You, the PR submitter, will be worthwhile to engage with and thus invest time in. I am aware that we can trivially construct hypothetical systems where AI agents are participating in PR discussions and develop something akin to a long term "memory" or "opinion" -- but we can talk about that experience if and when it ever comes into being, because that is not the current lived experience of maintainers. It's just a deluge of low quality one-way spam. Even the corporations that are specifically trying to implement this experience just for their own internal processes are not particularly... what's a nice way to put this, "satisfying" to work with, and that is for a much more constrained environment, vs. "suggesting valuable fixes to any and all projects".
I'm not advocating that the verification should be on the maintainer. It should definitely be on the contributor/submitter to verify that what they are submitting is correct to the best of their abilities.
This applies if the reporter found the bug themselves, used a static analysis tool like Coverity, used a fuzzing tool, used valgrind or similar, used an LLM, or some other mechanism to identify the issue.
In each case the reporter needs to at a minimum check if what they found is actually an issue and ideally provide a reproducible test case ("this file causes the application to crash", etc.), logs if relevant, etc.
I was arguing against your dismissal of the value proposition of LLMs. I wasn't arguing about the case of open-source maintainers getting spammed by low-quality issues and PRs (where I think we agree on a lot of points).
The way that you argued that the value proposition of LLMs makes no sense takes a really black-and-white view of modern AI. There are actually a lot of tasks where verification is easier than doing the task yourself, even in areas where you are not an expert. You just have to actually do the verification (which is the primary problem with open-source maintainers getting spammed by people who do not verify anything).
For example, I have recently been writing a proxy for work, but I'm not that familiar with networking setups. But using LLMs, I've been able to get to a robust solution that will cover our use-cases. I didn't need to be an expert in networking. My experience in other areas of computer science combined with LLMs to help me research let me figure out how to get our proxy to work. Maybe there is some nuance I am missing, but I can verify that the proxy correctly gets the traffic and I can figure out where it needs to go, and that's enough to make progress.
There is some academic purity lost in this process of using LLMs to extend the boundary of what you can accomplish. This has some pretty big negatives, such as allowing people with little experience to create incredibly insecure software. But I think there are a lot more cases where if you verify the results you get, and you don't try to extend too far past your knowledge, it gives people great leverage to do more. This is to say, you don't have to be an expert to use an LLM for a task. But it does help a lot to have some knowledge about related topics at least, to ground you. Therefore, I would say LLMs can greatly expand the scope of what you can do, and that is of great value (even if they don't help you do literally everything with a high likelihood of success).
Additionally, coding agents like Claude Code are incredible at helping you get up-to-speed with how an existing codebase works. It is actually one of the most amazing use-cases for LLMs. It can read a huge amount of code and break it down for you so you can start figuring out where to start. This would be of huge help when trying to contribute to someone else's repository. LLMs can also help you with finding where to make a change, writing the patch, setting up a test environment to verify the patch, looking for project guidelines/styleguides to follow, helping you to review your patch against those guidelines, and helping you to write the git commit and PR description. There's so many areas where they can help in open-source contributions.
The main problem in my eyes is people that come to a project and make a PR because they want the "cred" of contributing with the least possible effort, instead of because they have an actual bug/feature they want to fix/add to the project. The former is noise, but the latter always has at least one person who benefits (i.e., you).
How strange that the article never links directly to the Helix editor. I usually immediately open the homepage of whatever a blog post is talking about as a background tab to be able to click back and forth, or to be able to immediately figure out what the thing being talked about is, but no luck here, except for some decoys (like the "helix" link next to the title which is just the tag "helix" which sends you to a page with all the posts tagged with "helix", which happens to just be this one post).
I of course quickly just googled it myself and found the page, and so afterward I went to the source of the blog post and searched for the URL to confirm that it wasn't actually linked to anywhere. Turns out that about three quarters of the way down, in the "Key Bindings" section, there is a link to the Helix keymappings documentation page, which appears to be the closest thing to a direct homepage link.
Anyways, no nefarious intent being implied of course, I just found it sort of interesting. I am pretty certain it just got accidentally left out, or maybe the project didn't have a homepage back in December of 2024 when this was originally written? Although the github page isn't directly linked either (only one specific issue in the github tracker).
Yes, it was pure accident! I surely had the helix homepage and documentation most of the time while writing this, but only thought to link that one bit of documentation! When I get to a computer next I'll update it with a link, because that would be useful.
Not linking to stuff is the new normal. Many subreddits ban you if you post a link to source. Tweets no longer contain links - you need to click on tweet to see the next ones that maybe contain the link
I didn't see mention anywhere of a license. I also don't see anywhere to download this from. Is this release equivalent to saying "here is an OFL metric-compatible Arial," or are they releasing it in the sense of "our products will now look like they use Arial, but aside from that this doesn't concern you."?
This page, which is poorly designed¹ to the point that it supports the idea that this is all an in-joke rather than the work of pros, appears to suggest that this is a purely commercial work: https://abcdinamo.com/licenses
¹ Seen while scanning: (1) Scroll down, then up. Boo. (2) Leading cramped beyond "style preference". (3) Bulleted list badly styled in a way that requires work. (4) No attention paid to tracking where it's needed (e.g. small all-caps type). (5) Some terms (e.g. "First Designer") capitalized inconsistently. (6) '&' used in body copy.
It's so much focus on implementation, and processes, and really really seems to consider the question of what even constitutes "misaligned" or "unethical" behavior to be more or less straight forward, uncontroversial, and basically universally agreed upon?
Let's be clear: Humans are not aligned. In fact, humans have not come to a common agreement of what it means to be aligned. Look around, the same actions are considered virtuous by some and villainous by others. Before we get to whether or not I trust Anthropic to stick to their self-imposed processes, I'd like to have a general idea of what their values even are. Perhaps they've made something they see as super ethical that I find completely unethical. Who knows. The most concrete stances they take in their "Constitution" are still laughably ambiguous. For example, they say that Claude takes into account how many people are affected if an action is potentially harmful. They also say that Claude values "Protection of vulnerable groups." These two statements trivially lead to completely opposing conclusions in our own population depending on whether one considers the "unborn" to be a "vulnerable group". Don't get caught up in whether you believe this or not, simply realize that this very simple question changes the meaning of these principles entirely. It is not sufficient to simply say "Claude is neutral on the issue of abortion." For starters, it is almost certainly not true. You can probably construct a question that is necessarily causally connected to the number of unborn children affected, and Claude's answer will reveal it's "hidden preference." What would true neutrality even mean here anyways? If I ask it for help driving my sister to a neighboring state should it interrogate me to see if I am trying to help her get to a state where abortion is legal? Again, notice that both helping me and refusing to help me could anger a not insignificant portion of the population.
This Pentagon thing has gotten everyone riled up recently, but I don't understand why people weren't up in arms the second they found out AIs were assisting congresspeople in writing bills. Not all questions of ethics are as straight forward as whether or not Claude should help the Pentagon bomb a country.
Consider the following when you think about more and more legislation being AI-assisted going forward, and then really ask yourself whether "AI alignment" was ever a thing:
1. What is Claude's stances on labor issues? Does it lean pro or anti-union? Is there an ethical issue with Claude helping a legislator craft legislation that weakens collective bargaining? Or, alternatively, is it ethical for Claude to help draft legislation that protects unions?
2. What is Claude's stance on climate change? Is it ethical for Claude to help craft legislation that weakens environmental regulations? What if weakening those regulations arguably creates millions of jobs?
3. What is Claude's stance on taxes? Is it ethical for Claude to help craft legislation that makes the tax system less progressive? If it helps you argue for a flat tax? How about more progressive? Where does Claude stand on California's infamous Prop 19? If this seems too in the weeds, then that would imply that whether or not the current generation can manage to own a home in the most populous state in the US is not an issue that "affects enough people." If that's the case, then what is?
4. Where does Claude land on the question of capitalism vs. socialism? Should healthcare be provided by the state? How about to undocumented immigrants? In fact, how does Claude feel about a path to amnesty, or just immigration in general?
Remember, the important thing here is not what you believe about the above questions, but rather the fact that Claude is participating in those arguments, and increasingly so. Many of these questions will impact far more people than overt military action. And this is for questions that we all at least generally agree have some ethical impact, even if we don't necessarily agree on what that impact may be. There is another class of questions where we don't realize the ethical implications until much later. Knowing what we know now, if Claude had existed 20 years ago, should it have helped code up social networks? How about social games? A large portion of the population has seemingly reached the conclusion that this is such an important ethical question that it merits one of the largest regulation increases the internet has ever seen in order to prevent children from using social media altogether. If Claude had assisted in the creation of those services, would we judge it as having failed its mission in retrospect? Or would that have been too harsh and unfair a conclusion? But what's the alternative, saying it's OK if the AI's destroy society... as long as if it's only on accident?
What use is a super intelligence if it's ultimately as bad at predicting unintended negative consequences as we are?
reply