SSN is a (mostly) permanent number you use for many many things. A voter ID would be unique with every upcoming vote. Election specific public key would probably be a more correct term.

Abuse of it would definitely be bad, but it would be more difficult as you'd have to repeat the identity theft every election, rather than knowing someone's lifetime secret number.