That's fair, Crystal seems to also take a lot of time with type inference. However the LLVM codegen is almost constantly slower than the alternatives. For example, Haskell has a non-LLVM backend and a LLVM backend, and the LLVM backend is slower. Zig uses LLVM, and they are working on a non-LLVM backend to speed up development builds. A sibling comment mentioned that the situation is the same for D too. For C, I'm almost certain that TCC is faster than Clang with LLVM.
There seems to be a pattern here. I'm not saying that LLVM is bad, in fact if it's used by so many projects it's for a reason. But it does have a cost.
In Swift, type inference is one thing that can be extremely slow.