Is it though? I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..)
I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.
You can click through the Lichess opening database (click the book icon, and then the Lichess tab) to get an idea: https://lichess.org/analysis
But the answer is insanely unlikely, past a certain number of moves. The combinatorial explosion is inescapable. Even grandmaster games are often novelties in <10 moves.
So, it has a to have some kind of internal representation of board state and what makes a reasonable move and such that enables it to generalize (choosing random legal moves is almost unbelievably bad, so itβs not doing that).
I also doubt that it has been trained on the full (massive) database of Lichess games, but that would be an interesting experiment: https://database.lichess.org/
>I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..)
Very low. On lichess when you analyse your games you can see which positions have been reached before, and you almost always diverge in the opening.
The lichess db has orders of magnitude more games of chess than the chatGPT training data does, so there is absolutely no way that chatGPT could reach 1400 purely based off positions in its training data.
You could certainly test this by making completely random moves and seeing whether it's more likely to make illegal moves in those positions.
Though I think you're overestimating how many positions have occured. Frequently, by move 20-25 you have a unique position that's never been played before (unless you're playing a well known main line or something)
I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.