Honestly, I bought it more for collection than actually using. If I ever do anything with it, it will be from Plan 9. However, searching around there is plenty of documentation on the architecture in addition to a github repo: https://github.com/veos-sxarr-NEC
> In practice, tps is a reflection of vram memory bandwidth during inference.
> Comparing tps ratios- by saying a model is roughly 2x faster or slower than another model- can tell you a lot about the active param count.
You sure about that? I thought you could shard between GPUs along layer boundaries during inference (but not training obviously). You just end up with an increasingly deep pipeline. So time to first token increases but aggregate tps also increases as you add additional hardware.
Hint: what's in the kv cache when you start processing the 2nd token?
And that's called layer parallelism (as opposed to tensor parallelism). It allows you to run larger models (pooling vram across gpus) but does not allow you to run models faster.
Tensor parallelism DOES allow you to run models faster across multiple GPUs, but you're limited to how fast you can synchronize the all-reduce. And in general, models would have the same boost on the same hardware- so the chinese models would have the same perf multiplier as Opus.
Note that providers generally use tensor parallelism as much as they can, for all models. That usually means 8x or so.
In reality, tps ends up being a pretty good proxy for active param size when comparing different models at the same inference provider.
Continuously dissipating 1 gigawatt of energy by boiling room temperature water would require approximately 1.38 million liters of water per hour.
Seems like the environmentally responsible thing to do be to build the datacenter near the coast and use the waste heat to desalinate water. Or at least dissipate the heat into the ocean rather than boiling off an inland freshwater supply.
Setting aside a small patch of ocean for the task seems like a much better plan than the current practice. Provided you dump it in a place with a decent current any adversely affected area should be exceedingly small.
Keep in mind that the sun is constantly dumping energy on us. Absorption averaged across the entire earth is ~200 W/m^2. Assuming I didn't misplace some zeros somewhere then a gigawatt corresponds to ~5 km^2 of ocean surface. That's the daily flux. Penetration falls off exponentially so 75% of that only ever makes it ~10 m down.
I think the takeaway here is the utterly incomprehensible scale of the ocean.
This idea is probably more worth it in middle eastern countries given that 90% of their water comes from Desalination Plants. But given the recent war within region, I don't really expect Datacenters to be built within the region for quite a long time.
I guess when you're dissipating upwards of a gigawatt of power at a single site boiling water starts to look attractive. It's a pretty impressive curveball; I definitely would never have predicted "an evil corporation boils off all the local drinking water" to be a legitimate concern. I'm pretty sure that's too absurd a plot point for even a children's movie.
And yet the term luddite seems to fit the anti-ai crowd perfectly. They are largely concerned about employment (and more generally economic stability) and to that end seek measures intended to protect workers.
There's also some environmentalist concerns which the term luddite again fits perfectly. You just have to generalize, transferring laterally from economic wellbeing to environmental wellbeing.
So I don't think GP qualified as an ad hominem dismissal but rather an accurate description of the situation. Take what's being discussed (restrictions on specifications and interoperability), project it backwards in history, and imagine what an alternate present day would look like. I think it would be pretty bad.
>They are largely concerned about employment (and more generally economic stability) and to that end seek measures intended to protect workers.
Pffft no. Most of us think that AI is being used as a political trick - like firing unionized workers "to replace them with AI" and then hiring new un-unionized workers to replace them, 2 weeks later. Replace the AI with an empty cardboard box labeled "AI" in black marker, and nothing changes.
See also: using AI to launder pirated material, for big businesses.
>a political trick - like firing unionized workers
1. Since when have companies needed trillions of dollars of AI to do that? In the US they've been able to get away with getting rid of unions for decades now.
2. Since when has HN given a shit about unions. Posting about unions, at least till recently has been a great way of getting your comment downvoted to [dead] in one easy step. For longer than LLMs have existed the HN answer to unions was "They are just there to keep me as an SWE from making as much money as I can". Only now do we see a little bit of pushback now that their heads may be next on the chopping block.
Reset it back to 20 years and make that a hard limit for both patents and copyright. No renewals. Zero exceptions. Let the market sort the rest out.
There's always going to be downsides and edgecases when granting any party a monopoly over anything. At least if it's limited to 2 decades any unintended consequences, philosophical objections, and etc are hopefully kept within reason.
That would be insane for aerospace software, where you might spend most of that time getting the code certified (required to break the $0 revenue threshold), let alone paying back your costs and then making an actual profit.
Meanwhile, there are cases where copyright of more than 2 years is overkill.
I don't know what, but it seems like we need some sort of mechanism for variable-length IP duration is needed.
Is copyright meaningful for aerospace software? I'm largely unfamiliar with that domain but I have trouble imagining that (for example) Boeing cares much about people redistributing or hacking on the control software for a 777. How would that impact their bottom line?
I could understand for medical devices maybe but even then it seems like the software is a tiny part of the overall cost of a given design. A competitor could already do a clean room reimplementation in that case.
But I guess it wouldn't be all that bad if there were a carefully crafted extension for government certified software that was explicitly tied to the length of the certification process.
The only problem with this certified software exception is I foresee they'll write the law as "expiration timer starts when software has finished certification" then some lobby group will get the regulatory departments to adopt a new process of partial certification where said software is usable in devices but the 'finished certification' never gets reached so the copyright gets dragged out forever.
Nope, it falls more under trade secrets than copyright.
If you do something that requires stealing the code (publishing it, selling it, etc) the company can legally fuck you up.
Now, once it's in tbe wind, it becomes almost impossible to pursue from a practical point of view, as any implementer can claim trade secrets to avoid showing you the code.
If certification is the actual cost, you don't need copyright, at all. SQLite is in the public domain. Your moat is the certification itself, not the code.
Wait for the great new times when an AI will certify aerospace, automotive and medical SW. Waiting for that. It will be 1000x better and faster than the existing processes
Have you seen the quality of regular software though? And the failure rate of regular physical items? The only reason I trust aircraft is because of the process.
Consider if you will that if some guy were to fly a drone the size of a car that he knocked together in his garage over a residential area people would not accept that. Yet private pilots in cessnas fly over neighborhoods constantly.
9% is an absurd failure rate for solid state electronics. Particularly considering the profit margins. I assume it's related to the power densities involved. Would you happen to recall the source?
Would it even require a particularly high level of resourcefulness? Purchase the GPU along with the mobo that slots it. It's not as though companies typically swap out CPU and GPU while keeping the rest of the box.
They should max out a bit below 6 kW? The H100 SXM5 is 700 w which would place the system at 5.6 kW plus change. Too much for a standard circuit but well within the bounds of a residential appliance.
It's a monolithic 8U rackmount appliance so perhaps a dishwasher would make for a decent size comparison?
Definitely no good if you rent but homeowners should have little to no difficulty. The sort of people interested in such gear usually have multi kW racks already.
reply