I think part of the problem is that hiring bars are unrealistically high. Pulling numbers out of my ass for illustration, it's like companies that control 10% of the software engineering jobs all only want to hire the top 5% of software engineers. They can't all get their way, so the top 5%[1] bounces between these companies whilst the companies whine about "talent shortages".
[1] Should be noted that it isn't the entire group, since some will choose to work for organizations outside this group because the work is more interesting or fulfilling.
I don't know that there is a "right" level for the bar. There might be a game theoretic way of looking at the problem which explains how we wind up with the levels that we do.
In relative terms, the bar for, say, the NBA is much higher than for top universities and tech companies. They set the bar high enough so that, for all practical purposes, minor league teams cannot come close to matching their combined level of play. Also, if you don't compete regularly against NBA caliber players, with NBA quality coaches and trainers, it would be hard to reach the same level of skill even with comparable native ability. I don't think the league is that concerned with missing out on marginal players, as long as on balance it gets those correct. The disaster would be to miss out on the Michael Jordans or Lebrons, Bird and Magic. So they scout and pay through the nose even for the slim chance of that kind of greatness, with the tolerable outcome being a good role player.
At some level, we're all looking for greatness. Nobody wants to see a league entirely made up of role players doing somewhat above average things.
The analogy breaks down in that it's a lot easier to accurately measure (though perhaps not predict) athletic ability. I do think that investors in tech tend to heavily favor obviously stellar performers over "good but not great" ones.
[1] Should be noted that it isn't the entire group, since some will choose to work for organizations outside this group because the work is more interesting or fulfilling.