The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to focus its review on one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude).
IMO the big "innovation" being shown by Mythos is the effectiveness with prompting LLMs to look for security vulnerabilities by focusing on specific files one at a time and automating this prompting with a simple script.
Prompting Mythos to focus on a single file per session is why I suspect it cost Anthropic $20k to find some of the bugs in these codebases. I know this same technique is effective with Opus 4.6 and GPT 5.4 because I've been using it on my own code. If you just ask the agent to review your pr with a low effort prompt they are not exhaustive, they will not actually read each changed file and look at how it interacts with the system as a whole. If the entire session is to review the changes for a single file, the llm will do much more work reviewing it.
Edit: I changed my phrasing, it's not about restricting its entire context to one file but focusing it on one file but still allowing it to look at how other files interact with it.
Instead of asking the model: "Here's this codebase, report any vulnerability." you ask. "Here's this codebase, report any vulnerability in module\main.c".
The model can still explore references and other files inside the codebase, but you start over a new context/session for each file in the codebase.
Honestly, that's the only way I've ever been able to trust the output. Once you go beyond the scope of one file it really degrades. But within a single file I've seen amazing results.
Are you not supposed to include as many _preconditions_ (in the form of test cases or function constraints like "assert" macro in C) as you can into your prompt describing an input for a particular program file before asking AI to analyze the file?
Please, read my reply to one of the authors of Angr, a binary analysis tool. Here is an excerpt:
> A "brute-force" algorithm (an exhaustive search, in other words) is the easiest way to find an answer to almost any engineering problem. But it often must be optimized before being computed. The optimization may be done by an AI agent based on neural nets, or a learning Mealy machine.
> Isn't it interesting what is more efficient: neural nets or a learning Mealy machine?
...Then I describe what is a learning Mealy machine. And then:
> Some interesting engineering (and scientific) problems are: - finding an input for a program that hacks it; - finding a machine code for a controller of a bipedal robot, which makes it able to work in factories;
You can go on OVH and get a dedicated server with 384 threads and a Turin cpu for $1147 a month. You have to pay $1147 for installation and the default has low ram and network speeds but even after upgrading those it's going to be 1/5 of what it would cost on public clouds.
If I hand my shopping list to AI, why wouldn't I tell it to price match everything? People will start doing this sooner than you think. I still remember when people were scared to buy things on the internet, this will be faster.
Are you going to choose to buy your protein bar online from mysteryBargainBar[.]com for a $1 savings, or just pick it up as part of your local grocery trip?
> I still remember when people were scared to buy things on the internet
People still /are/ scared to buy things from Amazon for things that go on or in their body.
> Are you going to choose to buy your protein bar online from mysteryBargainBar[.]com for a $1 savings, or just pick it up as part of your local grocery trip?
ChatAI - show the top 50 online retailers by revenue in the US and note any that have credible new stories about quality control issues. Save all of them except StoreX and StoreY in your list you use for comparison shopping.
Or maybe another one, scan all my credit card purchases for all time that you have history and record all the stores.
Done. And plenty of third party sites (consumer reports, wirecutter, etc...) will do this kind of thing too. And you could perhaps transitively trust them - either view direct lists or just scraping the places they recommend.
And the average person doesn't need to figure this out ... skills encoding this will propagate.
In other words, switching costs go to 0, margins collapse. Middle men and people with products that aren't differentiated get hit hardest.
A human can't search 10 apps for the best rates / lowest fees but an agent can.
Thinking ahead 100 years from now, companies like doordash and uber eats don't exist and are instead protocols agents use to bid for items their user asks for and price discovery happens in real time.
Go to a supermarket, witness that dozens of brands sell the same things at wildly different prices, they still all make a profit, same for most services, you have comparator for subscriptions, mortgage rates, &c.
And a human can 100% search 10 apps and use his brain to do basic maths, that's what we've been doing until now. Sometimes I wonder if ai shills live in a parallel universe because it truly feels like they're living a completely different life than the vast majority of people...
> a human can 100% search 10 apps and use his brain to do basic maths
A human _can_ do all of that, but it takes time. If I have to search 10 apps for each item I want to buy (clothes, daily food, movie tickets, laptops, etc.), I will spend the rest of my life just searching for better deals. I'd rather have a bot do all of these searches for me.
I don't see what the role of AI is in this. You don't need an AI to aggregate data from a bunch of sources. You'd be better off having the AI write a scraper for you than burning GPU time on an agent doing the same thing every time.
> A human can't search 10 apps for the best rates / lowest fees but an agent can.
Why would those apps permit access by agents?
It's always been the case that “agents” could watch content with ads, so that the users can watch the same content later, but without ads. The technology never went mainstream, though. I expect agents posing as humans would have a similar whiff of illegality, preventing wide adoption.
Local agents running open weights models won't really work because everybody will train their services against the most popular ones anyway.
As a U.S. consumer, can you buy a DVR that can record HDCP streams (without importing it yourself from a different country)? Even one that does not automatically edit out ads?
If I search "HDCP remover" on Amazon I see tons of results for $15-$30, sure. Reviews say they work as advertised. That typically exists in a different space from DVRs since it's not relevant for broadcast TV as far as I know (AFAIK there's nothing for DVRs to remove in the first place), but it'd be easy enough to chain it if you needed to.
Right, but why the heck would you guess 100 years when we could build and adopt that in less than two weeks? There are already many people working on this type thing. Some of them have been working on it for years and a few probably already have solutions ready to go or even in use.
https://youtu.be/1sd26pWhfmg?t=204
https://youtu.be/1sd26pWhfmg?t=273
IMO the big "innovation" being shown by Mythos is the effectiveness with prompting LLMs to look for security vulnerabilities by focusing on specific files one at a time and automating this prompting with a simple script.
Prompting Mythos to focus on a single file per session is why I suspect it cost Anthropic $20k to find some of the bugs in these codebases. I know this same technique is effective with Opus 4.6 and GPT 5.4 because I've been using it on my own code. If you just ask the agent to review your pr with a low effort prompt they are not exhaustive, they will not actually read each changed file and look at how it interacts with the system as a whole. If the entire session is to review the changes for a single file, the llm will do much more work reviewing it.
Edit: I changed my phrasing, it's not about restricting its entire context to one file but focusing it on one file but still allowing it to look at how other files interact with it.
reply