There does appear to be ARM64 assembly in Go's crypto code, but perhaps it's suboptimal for M1. Or perhaps there's a build config issue, and the assembly isn't being used. Just my speculation, though. I haven't looked into it.

https://github.com/golang/go/blob/master/src/crypto/aes/asm_...