Serving a model efficiently at 1M context is difficult and could be much more expensive/numerically tricky. I'm guessing they were working on serving it properly, since its the same "model" in scores and such.

Thanks - still not clear what they did really. Some inference time hacks?

That would imply the model always had a 1m token context but they limited it in the api and app? That's strange because they can just charge more for every token past 250k (like google does, I believe).

But if not shouldn't it have to be completely retrained model? it's clearly not that - good question!

Most likely still 32k tokens under the hood, but with some context slicing/averaging hacks to make inference not error out on infinite input.

(That's what I do locally with llama.cpp)

They already had 0.5M context window on the enteprise version.