No, F-Droid builds almost all apps from source. Even some open source apps don't make it to F-Droid if the F-Droid maintainer doesn't manage to build it themselves on their build server.
The reason F-Droid does this is reproducible builds. Which is a big advantage because the code you see on GitHub is the binary you get in your device. It also means it's quite obvious when code is being added because you can reproduce it.
Of course the build platform being compromised is possible but that can happen even with binary distribution.
I thought they only allow you to guild/sign your own apps if it is a reproducible build, and they verify that the version they build is identical to the one you supply.