It's partly technical (BSD Sockets being bad API that hard does low level proto ok details in applications) and partially business - vendors didn't want to do the work to upgrade software and hardware - especially with advent of CEF and similar hardware routing options. And by 1990s the government-led standardisation efforts that gave us widespread ethernet and IPv4 got axed, and efforts to make vendors update if only for federal contracts died in waiver hell.

The others kinds of problems are from there over time.