I formed a PBC and worked at a well-known PBC. Personally, I opted for a PBC because I liked that I could balance a specific cause with shareholder benefit.
In most cases, it doesn't really matter. The board + management is still in charge, and they have significant legal leeway regardless of the structure. But there's little additional cost to opt for a PBC, and it does give you more legal defensibility to be truly mission driven. Standard C Corps weren't really intended for mission driven companies (see the shareholder primacy norm).
I think it's popular for AI startups, because many great researchers understand the risks involved, and they don't want what they build to be controlled solely for shareholder benefit.
While non-profits are also an option for a mission driven org, it's harder to raise the large amounts of cash that some AI startups need, and laws around deferred compensation and private inurement (e.g. options-like structures) make employee compensation harder.
> Kimi K3 performs significantly below the most recent frontier cyber-capable models
UK AISI cyber evals seem to under-elicit capabilities from quirky models [1]. Kimi K3 is a token-hungry model, and I suspect it hit the eval's 100M token limit well before saturating scores [2].
This gap was true for GLM 5.2 as well; they ranked it at Opus 4.5 level [3]. Both anecdotally and with a held-out eval, I've found GLM 5.2 to be better at security research than Opus 4.6 [4]. But it's a quirky model that degrades quickly at long context lengths.
Personally, I'd rank Kimi K3 above Opus 4.8 and lower than GPT 5.6 Sol in its ability to find vulnerabilities and exploit them. But it's not far from the frontier.
[1]: From the UK AISI: "Our setup likely slightly underestimates open weight models’ maximum capability: we didn’t pursue specific elicitation or optimisations which could have improved performance" (https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...).
[2]: Their eval also counts cache hits towards the token budget; the 100M token budget is comparable to a ~5M token budget in other evals.
The gap seems to be so small I'm not sure how much we should care. If we assume that the trend in the graph does continue as a rough linear improvement, the open models will have reached the same level as the present closed models in 6 months and likely saturated the benchmark by mid next year. That seems more important than where we are now and the exact level of measurement accuracy in July 2026. There is a difference between US and Chinese models but it doesn't look like it is going to be strategically significant.
Counting cache hits towards the token budget is exactly how it should be done for these kind of evals, and at any rate, for cyber evals all frontier models benefit from more tokens not just Kimi K3, so the comparison is still apt.
> all frontier models benefit from more tokens not just Kimi K3
Past a point, that doesn't hold and the score plateaus.
Token hungry models tend to plateau at a much higher token count. Because Kimi K3 is a token hungry model – and 100M tokens (including cache hits) seems at the edge of the plateau for these evals – it could disproportionately benefit from a higher token budget.
For Kimi K3 specificially, policymakers are interested in whether it can find and exploit the same scope of vulnerabilities as models like Mythos. In that context, an answer of "yes, but with quintuple the token budget" is materially different from "no, it performs significantly below the most recent frontier cyber-capable models".
(As an aside, I like the UK AISI and think they're the best example of that kind of group!)
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1].
When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree.
I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings.
I built one of the connected tools included in this launch (the Biomni HPC [1]), and I have spent an inordinate amount of my life working on this problem. (I also worked at Anthropic, but not on this product.)
As other comments have pointed out, this is for data science – but it's capable of more than making plots and writing papers [2]. It has integrations with many databases and computational tools, including a researcher's institutional cluster.
That alone is valuable. I founded a startup after struggling with this problem at a bio startup; integrating these tools and databases is hard and time consuming. If the only outcome of this product is that great APIs are built for LLMs, it will be a massive positive impact. Many databases used in computational genomics are still only accessible through FTP!
LLMs are particularly good at navigating these tools and databases. It's often very specialized, but straightforward, work that benefits from in-context skills. Seeing an early glimpse of my former customers – bioinformaticians – using LLMs to solve this problem is what led me to join Anthropic in 2024.
Also, this pattern isn't fundamentally constrained to data science: you can also integrate with a wet lab or a CRO for some kinds of science. This is what I'm spending my time on now.
This type of science doesn't solve everything, but it's useful in some niches. For example, progress on many rare diseases is bottlenecked by researcher attention rather than a fundamental breakthrough.
Connecting AI directly to the data sources (instead of just asking it to provide code that I run locally for myself) can get quite complicated in terms of meeting institutional policy, applicable law, data access-storage requirements (e.g. NIH data repositories), and can require legal agreements between institutions and the AI provider.
If you put your data in Snowflake then there is a built in AI (ok it’s Claude) that can access the databases. This sidesteps a lot of the issues in that the data is clearly already with Snowflake.
Can you speak to what makes this different from simply including or configuring various agent skills? Or is it simply the combination of lots of helpful defaults that makes this product useful?
I can't speak for Claude Science, but I prefer using Biomni as an agent for bio over Claude Code with a custom setup because a) Biomni stays on the frontier for bio, b) it has a config that just works and skills I trust are correct, and c) it has better built-in abstractions for long-running sessions.
As a concrete example, computational biology jobs sometimes run for hours on the Biomni HPC. When they're done, the session needs to reawaken, process the results, iterate, etc. You can implement something like this with agent callbacks, but it's not as straightforward.
This repeats many times for many integrations, so it's just simpler for me to use an agent that's built for exploratory bio and already has all of this. Claude Science has some of these features, so I imagine they're aiming for something similar.
When you set up your Claude Science instance you can see that they're connecting to crossref, semantic scholar, pubmed, ArXiv, FDA. They instruct the LLM to validate citations.
My testing with this technique indicates that method they seem to be using (rag with an instruction to check sources) will reduce the confabulation rate for citations from the base rate 50-60% for regular models (e.g. regular Claude) to 5-15% (depending on how they implemented it).
On the one hand this is way better. On the other hand it's just good enough that your spot check will look good and your work will still contain hallucinations (which is probably worse than obviously bad).
Getting to zero confabulation would require a different process. (stand-alone validation engine running in parallel in real-time which is hard but not impossible.)
I assume they do hallucinate, just like with coding or finding vulnerabilities.
You can try to minimize it (e.g. with a reviewer agent, which Claude Science and Biomni have), but nothing is perfect, so I limit autonomous work to verifiable problems and review it.
I'd really like to see much better visualization from Claude Science at some point. Educational-esque, with full threejs + shaders scenes over just these plots and protein/chemical structures. This for a lot of papers in the literature review would be awesome.
Sounds like the perfect use case for some kind of framework where you have a local LLM (that can run on lower spec hardware) collaborating with the main LLM to optimise latency and all the other niche and legacy use cases ?
I have found that some models consistently find or miss specific bugs, and which bugs are hard don't completely line up across all models, so I believe that. I just refactored the security bug-finding harness I've been working on completely (not checked in yet, testing it currently) to strongly encourage "multi-model, multi-pass" scans and make them easy to orchestrate with de-dupe and weeding false positives with a strong model, rather than one model or doing just one pass over each file. Giving a model a second attempt increases their findings by 20-30%, and giving them a third, adds another 10-15%.
I'm inclined to use DeepSeek V4 Pro the most, because it is consistently extremely strong, it's very fast, it's very cheap and has excellent caching and cheap-as-free cached input tokens (something like 80% of token usage is cached when I'm using it for security scanning). So, my probably "pair" of frontline security researchers will probably be DeepSeek V4 Pro and Gemma 4 31B self-hosted (another shockingly strong contender, competitive with the best models once you let it loop on the same file a couple/few times). But, I won't be surprised if GLM 5.2 turns out better than DeepSeek V4 Pro...it costs quite a bit more.
The loop is "look at this file in this repo, find bugs" iterated over every file in a project, with the ability to look at the rest of the repo for cross-file bugs related to the file they're instructed to look specifically at, but yes. The Anthropic folks have basically said that's how they're doing security audits (Nicholas Carlini is an Anthropic employee and he's done talks about it), so I assume that's how Mythos found its bugs.
I've benchmarked it, and the "here's a repo, find bugs" approach finds far fewer bugs. Like, dramatically fewer. Models are good and contexts have expanded, but focus still wins with hard problems. You could probably tell the good models to make a plan to audit the repo, and it would end up making its own "loop" in the form of a checklist of files to look at over several sessions or via subagents, I assume.
Not sure if helpful but in my experience when something a bit more complex needs to be done, manually making it read the context I know the model will need for it to solve it well (like making it consume all the project docs first) helps with getting a more satisfactory result instead of only giving it the task and let it look around and consume the context it thinks it needs.
Will test your bug finding method in a current project of mine both with my "manual" context preloading and without.
In most cases, it doesn't really matter. The board + management is still in charge, and they have significant legal leeway regardless of the structure. But there's little additional cost to opt for a PBC, and it does give you more legal defensibility to be truly mission driven. Standard C Corps weren't really intended for mission driven companies (see the shareholder primacy norm).
I think it's popular for AI startups, because many great researchers understand the risks involved, and they don't want what they build to be controlled solely for shareholder benefit.
While non-profits are also an option for a mission driven org, it's harder to raise the large amounts of cash that some AI startups need, and laws around deferred compensation and private inurement (e.g. options-like structures) make employee compensation harder.