Separation of duties. You want the hsm to be unchanged in many years with minimal maintainance. While your phone is already being used for too many things which means it will need an upgrade frequently and you will have a harder time securing it.

Think of the hsm as just another key on your keychain. It has no ui other than a button,does not need charging and will remain in your pocket until needed. You don't want someone grabbing it off your hand while texting or making a call,that would mean they can access all your accounts.