This looks really cool, congrats!

Just to share a couple of similar/related projects in case useful for reference:

http://strobe.audio multi-room audio in Elixir

https://www.panaudia.com multi-user spatial audio mixing in Rust