That is: you're presupposing that a JavaScript framework is actually necessary in the first place. I'd argue that the whole concept of a JS framework is the epitome of chasing after shiny and complicated things. If instead you start with just plain ol' HTML and incrementally add CSS and JS as actually needed, you're much more likely to end up with a much simpler and more maintainable end result (or, at the very least, you will have figured out the actual and specific reasons why you might need the additional complexity of, say, Sass or this week's JS framework).
The downsides might be (I'm guessing) performance for very complex apps and lack of pre-built third-party components.
There is definitely a spectrum of simplicity among the frameworks. Angular seems to sit on the opposite side of the spectrum. It has many benefits, I'm sure, but in every aspect they've chosen the more complicated solution.
But seriously while Svelte is the new kid on the block, they do claim to have minimal API design and no virtual DOM. So I guess the real answer is Svelte.
It does have a compiler, though. So compared to the old-fashioned JavaScript workflow it adds a mandatory build step. And I don't think it would work with languages that compile to JS – or does it?
