If not following the guidelines makes your app hard to use or glitchy on some of Apple's lineup, I don't see why you wouldn't expect Apple to reject the app. This reflect poorly on Apple because it makes it look that some of their devices are 2nd class citizens within their ecosystem.

They should follow the guidelines when not doing so causes usability issues. If your games is usable on all devices then obviously there's no problem.