It can be difficult to pass some PCI compliance tests if your ssh port is available to the world. OpenSSH also leaks some information about your server unless you recompile it with those options removed.