How would a machine know that it doesn't know?

Experience of encountering things that were previously unknown unknowns would teach it of the general existence of such things.

Probably by comparing what it experiences to what it can explain.

Sure, but I'm still not sure it would realistically function. All data in this scenario is obviously synthetic data. It could certainly identify gaps in its "experience" between prediction and outcome. But what it predicts would be limited by what it already represents. So anything novel in its environment would likely confound it.

It's a cool sci-fi story. But I don't think it works as a plausible scenario, which I feel it may be going for.