You can think of a compressed file as a program (a set of instructions) executed by an interpreter (the decompressor) that builds the original file.

A smaller compressed file is usually going to contain fewer instructions for the decompressor to execute, so there's less work to do to reconstruct the original file.