As another data point, I'm a pretty small guy (5'8" and about 150 lbs) and I can't stand wearing L shirts - I feel like I'm wearing a dress. Given the choice, I'd much rather a S than a L.
A short friend of mine complains that even the small American Apparel shirts end up looking like dresses on him. "It feels like I'm wearing a mini-skirt" he says.