I tried 4.0 in a VM upon release, and that scared me away from the 4.x series until 4.2. Even in 4.2, I had a lot of issues because of my heavily customized setup. I'd say that if you're happy with 3.5.x, stick with it as long as there are no major security holes.

I got tired of basic functionality being broken even in 4.3, and left for greener pastures (Openbox). I only briefly experimented with 4.4, so I can't really comment on it (though from what I did see, I was not enamored at all).