So how do I know you (aes.io) changed the JS to leak my password to you? I can never be sure without comparing the JS you send me to some JS I know to be secure.
disclaimer: I'm working on a client-side JS crypto web-app myself :P
I'll tell you how it's different: because you, the server operator, can at any time compromise the user. But you could not have compromised the integrity of their browser.
Since the whole point of clientside encryption is to prevent the server from compromising data, the fact that the server retains the ability to do this almost defeats the purpose of encrypting.
One is intentional leakage, or trusting the vendor to be "not evil". Here you are right - as you mentioned in another comment, there is no easy way for users to "police" the vendor of JS crypto. I do not see, however, how this is different from users trusting other vendors of auto-updating software.
You can probably argue that OSS pieces such as Linux and FF have this "policing" enabled through source code availability. But there are many closed-source vendors (Microsoft, Google, anti-virus vendors) that many users trust. If you don't trust the companies mentioned above, then yes, you should not trust aes.io and other JS-crypto based vendors. But if a user has Microsoft/Apple updates enabled, and browsers (other than FF) auto-updating on their systems, etc., then this becomes a question of vendor being "not evil", not weaknesses of JS crypto, and as such somewhat out of scope of scrictly "JS crypto - yes or no" discussion.
The second issue here are unintentional leaks and breaches. Read-only breaches, for example, become more or less non-issue with JS crypto, because stealing properly encrypted data is not that useful. And there are probably orders of magnitude more read-only breaches than read-write, simply because inside companies more people have read-only access to production data than read-write access. Thus JS crypto seems to help substantially reduce the risk of unintentional compromises.
The second issue needs more attention and discussion from the technical point of view, I agree, and I am open to it. The first issue, however, becomes "I trust this vendor but I don't trust that vendor", and this is not a discussion that I believe can be productive here and now.
You ask people to use your service because it is more trustworthy than others because it uses (presumably) AES encryption.
Then, when it's pointed out that your service is no more secure than any other service, in that all these services rely entirely on a promise by their operators not to breach privacy, you say "well that's no worse than any other service".
Now, I think there are a bunch of ways in which services like this are worse than other services. But I don't think I need to make that case, because, more importantly, they're not any better.
disclaimer: I'm working on a client-side JS crypto web-app myself :P