That only applies between power boats. Between sailboats the direction of wind is what determines the right of way.

And of course power boats must give way to sailing vessels.

The rule of thumb I go for with boats is that the vessel with worse steering has right of way. For instance a freighter has right of way over a sailing boat regardless of direction or wind because the freighter isn't going to be able to change course in any appreciable amount of time.

Freighters have right of way in constrained waters, but offshore they still need to make way for sailboats.

(Yes, I know colregs is a bit more nuanced than that, but I think this is a decent approximation)

Plastic boats give way to wooden ship, wooden ships give way to metal ships, and nothing gives way anything much smaller than itself - Not the colreg rules, but a good approximation for unskilled crew members. Especially on a wooden ship.