Domains make IPs human-readable and human-rememberable, and allow multiple servers to identify as the same site (ie, for human and CA purposes), and allow them to move. Somewhere you need a translation and connection system.
You could most definitely change your links to go to IP addresses. But they're more fragile.
Server IP addresses change every time you switch to a different provider.
Ironically the DNS actually provides a layer of resilience against being forced offline by targeting your provider, but that moves the vulnerability up one level, now you rely on the DNS to do its thing, and it too is under the control of a single institution.
The bar is a bit higher than a C&D to your provider but apparently not nearly high enough.
You could most definitely change your links to go to IP addresses. But they're more fragile.