We, an online store, have found deliverability to yahoo.com and btinternet.com addresses by far the worst, I don’t have statistics to hand but it’s up to a 10% failure rate. Much like the OP it seems to be silent too. We have literally no issues with other providers.
The unfortunate thing is that we also find the demographic of customers here in the UK still using btinternet.com email closely aligns with customers who tend to be more problematic…
For reference Yahoo are absolute shits when it comes to mail delivery. They literally will black hole addresses, domains and IP addresses with no recourse ever. The sending server will get a response code back of roughly "fuck you until you've manually filled in this form at this URL". The form is submitted by any attending mail admin if they even notice the response code, then silently with no confirmation they will either ignore you until the universe dies or suddenly mail will start working. If you fuck the form up, you have to wait 6 months to try again.
They actually stopped me hosting my own email because I couldn't be bothered to deal with this shit any more.
They also have recently become absolute shit at IMAP support, so has AOL. Both of them have set up a COPY Rate limit, basically if you move X amount of messages, (10 on aol, around 50 on yahoo) the move will fail with this message:
> IMAP< 72 NO [LIMIT] UID COPY Rate limit hit.
> \* IMAP error on imap.aol.com: UID COPY error
The mail clients themselves could implement a bandaid fix for these types of servers but still, I see a future where you can only use webmail/apps coming, either pay or don't use email.
I got a call from a recruiter some years ago to take over yahoo mail as a manager. In theory this would have been a huge step in career. I noped out of that hard. Did not want to oversee a declining product.
If I recall correctly didn't Yahoo used to provide the BTinternet.com email service? I seem to remember some announcements about BT eventually moving off it but not sure if the backend MTAs changed.
They did indeed! I'm not sure on the infrastructure, but I'd love to know if it was run within Yahoo's data centres and white-labelled, or run in BT data centres using Yahoo Mail's application.
> The unfortunate thing is that we also find the demographic of customers here in the UK still using btinternet.com email closely aligns with customers who tend to be more problematic…
Maybe this is just wishful thinking, but if the more problematic customers are losing you money, then isn't it a valid business case to stop doing business with them given that you can identify them in advance? Wishful thinking because it'd be really nice to create a feedback loop such that users of email providers start to understand their reputation, which would make provider reputation for deliverability matter, which might improve their behaviour. But most businesses probably aren't in a position to turn away business like that.
There is “losing money to deal with this issue” and there is “losing more money than you are making from these users”, I imagine that this is more the former.
It doesn't help with BT that you often have to figure out how to log into the webmail interface (often outlook/Hotmail) and only then do you seen the spam folder. If you don't do this and currate the spam folder it all get steadily worse... And you never see why if you are using an email program set up years ago on your PC.
Trying to get detailed logs from Outlook365 hosted accounts is near impossible.
You'd think you'd be able to log in to your Exchange dashboard and just look at them. They hide it from you and only show you abbreviated logs. You can go through support, and if you pay enough money you'll get an cryptic answer within a week (if you're really lucky the support person might show you the real logs).
Pain in the ass when you're trying to figure out why your users aren't getting email from someone.
You can pry the mail logs from my cold dead hands.
I really don’t care anymore if my email reaches Outlook or Hotmail addresses. It’s impossible to make sure. And to some extent it’s also the users problem.
With Office 365 I have far less issues. So Microsoft can do it, if they want to.
Even Microsoft understands what it means for their business if a paying customer with access to legal representation such as a lawyer or a medical professional randomly started losing correspondence.
If it was proven in court that they were silently discarding correspondence of paying customers because they were over confident in their spam filtering, everyone would jump ship in a heartbeat.
Mail is a very resilient protocol, this isn't UDP, this is the result of a deliberate decision being taken.
Exactly, this is not "randomly losing correspondence". This is simply spam filtering that is done after accepting the e-mail for delivery. Whether that should be an acceptable behavior or not is another discussion.
I'd be interested in how anyone could defend that silently discarding email after "after accepting the e-mail for delivery" based on a defective filter is acceptable behaviour.
Sure, and if it fails move it to the spam folder. In the article Microsoft are discarding emails because the sender hasn't got much of a reputation yet. That's a very weak reason.
If storing vaguely suspected spam is intolerably expensive then maybe just keep a list of discarded messages (sender, title and date). At least users can see what's happening.
They could still at the very least send a notification email back when the deferred analysis rejects the message. Email is multi-hop anyway in the general case.
After accepting an email for delivery, the only choices the mail server should have are "inbox" or "spam folder", not "deleted forever in a black hole".
A lot of mail servers are configured like that. Messages with very high SPAM rating are rejected, but still delivered into the SPAM folder.
The server rejects the message, to get rid of spammers. After a few rejected messages, they usually stop delivering SPAM for quite some time (it's unnecessary work to do, and a higher risk to land on blacklists).
Are you sure that the infrastructure behind Outlook and Office 365's Outlook is sufficiently different? I was under the impression that it's similar if not the same. Compare Outlook:
You're not wrong, but Microsoft's Exchange 365 does have a lot of knobs to control (https://docs.microsoft.com/en-us/microsoft-365/security/offi...) and the defaults are more relaxed, probably as concession to those migrating from on-premise solutions. I'm guessing that they turn the knobs into the most aggressive position for their personal services.
This is fairly normal in today’s world of email providers, and one of the reasons it can be painful to manage your own SMTP server, right?
Mail delivery is an anarchy and I’m impressed the problem of spam has been solved to the degree we currently see. Yes it’s much more centralized than before, and no that isn’t a good thing, but from a customer’s experience point of view, I couldn’t go back to the old days of manually training your email client’s spam filter and whatnot.
> This is fairly normal in today’s world of email providers, and one of the reasons it can be painful to manage your own SMTP server, right?
Yes. Silently discarded e-mail is the worst thing that can happen because you can't detected it as a sender. Your message reaching the spam folder is bad, but not as bad as getting silently discarded.
The best among the bad options is probably getting rejected with the SMTP return code 550 along with instructions how to delist in the return message.
If you are running a business and have customers having emails that are silently being dropped, adding a "verify your email address" page in your process is probably a good step to have anyway.
If they're consistently dropping mail, the user stays unverified. But a verification page is practically mandatory anyway for all sorts of reasons, not least checking the user hasn't mistyped their email.
It's increasing becoming the case that "terrible reputation" now includes the low reputation self hosters who don't have enough volume to even use the tools Google and Microsoft offer.
Google's Postmaster tools all show "No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain conditions before data is visible for this chart."
Microsoft's Safe Network Data Service shows *"No data for specified IPs on this date" for every day.
Don't worry they do that as well. First VPS I got had been completely blacklisted at the provider level and they were refusing everything with 550. Support didn't reply either.
> This is fairly normal in today’s world of email providers
Certainly not normal, but done sometimes, yes.
> and one of the reasons it can be painful to manage your own SMTP server, right?
It's part of the job, but it's more likely that the big providers are the least of your problems. They can be reached and you're usually not alone with your problems.
It's the small providers who have terrible filtering, who use low-quality shit-lists like UCEProtect/Backscatter or their own "heuristics", that are the most painful.
Well I've gotten blocked from being delivered Outlook-hosted mail because my primary MX is on a Linode block that at times get listed with UCEProtect (level 3 even!!!)
Why can't the big ones just check SPF and DKIM etc and let us responsible self-hosters do our thing
> Well I've gotten blocked from being delivered Outlook-hosted mail because my primary MX is on a Linode block that at times get listed with UCEProtect (level 3 even!!!)
I really don't think Microsoft would ever touch that pile of shit, most likely your spammy neighbour got your neighbourhood blacklisted at both places simultaneously.
> Why can't the big ones just check SPF and DKIM etc and let us responsible self-hosters do our thing
Most of the spammers have valid SPF and DKIM. Quite a few have entire subnets of IP addresses to spew spam from. That's why a random IP and just those two aren't sufficient to just start blasting.
What porbelm means is that when SPF and DKIM are valid, reputation check should be based on the sending domain, not on the sending IP.
This way spammers still get thwarted (because of their domains with zero or low reputation) and legit senders can send from any IP (since their domain CAN build good reputation)
Besides, it's also easier to track reputation of ~360M domains than billion of IPv6 and IPv4 ranges.
Unfortunately many legitimate sending domains are newly registered, so have no reputation. It's not possible to rely on age either: if requiring domains to be aged becomes commonplace, they're cheap enough that spammers would just register them and wait for a bit, maintaining a pipeline of them.
So it becomes necessary to consider the reputation of whoever is enabling the sending of the email. The only way to do that is by IP.
If a domain uses DKIM, has a good reputation but comes from a bad IP, then sure, you might trust it. I don't think that'll help much in practice though.
Everything you said applies to IP-reputation as well. There is nothing that makes domain-reputation inherently harder than IP-reputation.
Consider this: every legit IP address once started with the problem of having zero reputation.
How you build reputation from zero is simple: you start by being allowed to send only small amount of daily emails (rest goes to spam or is outright rejected by the recipient SMTP server). Then adjust reputation based on the usual factors (observe human recipients flagging emails as spam/not spam, watch for keywords like Viagra, etc, all the exact same stuff email providers already do). You gain or loose reputation over time, which affects how much you can send: a lot, or nothing.
Some hosting providers have a very negative IP reputation. They seem to be unable to stop their users from sending out spam. They don't seem to react to abuse reports, let alone take action to stop abuse from their ASN generally. So I block their entire netblocks, IP-wise. I'm not aware of any legitimate emails being caught up in this. If somebody is unfortunately caught up, my response is that they should use a more reputable provider.
According to your previous post, I shouldn't do that for DKIM/SPF -valid emails by IP, but I do. I hope this explains I do, and why purely domain-based reputation doesn't work.
Big spammers own multiple ranges of IPs just as easily as they own multiple domains. When you block only 1 of their range, they can re-use their same domains to spam you from other ranges. If you blocked by domain, this would be effective across all their ranges of IPs.
So, in your example, blocking by IP isn't necessarily superior to blocking by domain (and vice versa.)
Once again: it's possible to have the tools and infrastructure manage domain reputation and do a job as good as (and arguably better than) IP reputation. Also to consider: IPs can be acquired by spammers for nearly $0 (eg: destroy+recreate VPS to get a new IP for free), whereas domain names have a fixed cost of ~$10 USD per year per domain. So if you have a system that manages domain reputation and has 1 million domain in the blacklist, you know for a fact that you are burning ~$10M/year of spammers' money.
I'm fairly sure all what you're desribing is and has been done. It's just not a simple problem and spammers do all that you've described as well, plus more.
Recent trend I'm seeing is creating malicious Google and Microsoft account or hacking websites and using those to spew spam.
> Why can't the big ones just check SPF and DKIM etc and let us responsible self-hosters do our thing
I self-host and frequently block entire ISPs. It's very common to get the same mail for days on end from a different IP, and often a different domain each time. Some months DO accounts for over 50% of my spam, for example.
I'm sure there's more graceful ways to handle it if I was willing to put far too much of my own time into fighting a losing battle. but adding a REJECT line for a whole netblock takes seconds.
I don't begrudge the big hosts for treating whole ISPs as cesspits as I do the very same thing.
In the end it is Microsoft's problem, since they are the ones that are losing valid SMTP traffic. If a sufficient amount of users are experiencing a loss of e-mail as a result, then they will be forced to fix it.
There is almost never any valid reason to block an entire ip address. Single user accounts (e-mail addresses), maybe, if compromised.
E-mail, as a standard, is still pretty straight forward. Ideally you should be able to send e-mail directly from your laptop IP, if approved to send e-mail on behalf of your e-mail account / host name. Decentralization is important.
>In the end it is Microsoft's problem, since they are the ones that are losing valid SMTP traffic. If a sufficient amount of users are experiencing a loss of e-mail as a result, then they will be forced to fix it.
One might think that would be the case eventually, but at least in the case of universities insisting on using Office365, it really doesn't seem to be. In academia, where emails to and from university domains are the norm, Office365 seems to frequently treat with extreme skepticism odd domains, like those with .edu or .org TLDs, or prominent universities on ccTLDs in Europe, assuming that most legitimate mail will come from inside an organization. I've seen it send directly to spam emails about papers, peer review, talks, emails from easychair.org, emails from major journals. I know of a case where it has sent emails from Nature's submission system to spam. I've seen it send emails from frequent collaborators to spam. Classifying emails as not spam seems to do little to nothing. Meanwhile, emails from major providers' email services, like gmail and outlook, often seem to get through, despite mostly being spam, and in some cases, rather obvious spam.
It seems likely that people have lost publications and talks as a result: the emails being rejected and spam-filtered are important emails. Yet as much as everyone seems to complain—and these are complaints I have heard from many people, at many universities—Microsoft seems to have continued success in selling Office365 to university IT services, and pushing everyone to use it.
I don't think that's how it plays out. Microsoft has a lot of users on their mail services. If you -- as a business -- cannot reach these users, most of them aren't going to blame Microsoft. They're going to blame you and probably choose not to use your website/shop/... unless there is a previously established relation. Thus your deliverability troubles are now a "you" problem. Because you're going to lose customers over it.
The issue is that domains are cheap and plentiful, and a spammer can easily claim to be sending mailflow on behalf of let's say, 1000 users, each sending 5-10 emails a day, and thus, a single IP "forwarding" (sending) mailflow "on behalf" of 10k domains, with 1k users each, with each user sending just 10 emails a day - would result in a pretty nice batch of spam making it through.
As things currently stand, the only "expensive" resource that is hard to acquire or fake somehow is source IPs with good reputation. Fair? I'm not saying it is. It's just effective.
The cost of a domain looks conveniently close to the cost of keeping track of some entity's reputation to me.
A domain is relatively cheap. Accepting some 20 messages before you are sure it's spammy is also cheap. You will have to filter out the malicious domain registers, but that's a quite immutable set, as becoming a register is a bureaucratic process with a non-trivial cost.
The statement is just really quite absolute. The most trivial example would be an IP address being used by an abuser behind bulletproof hosting. Someone trying to deliver 200 000 spam letters in a minute and you rejecting each one is a significant amount of load - best case it still reduces your logs' SNR.
>If a sufficient amount of users are experiencing a loss of e-mail as a result, then they will be forced to fix it.
The trouble is, how many users are sufficient? Patch Tuesdays have their reputation for a reason, and if your issue isn't one shared by millions of people, it's unlikely a fix will come from Microsoft. Meanwhile email continues to get lost, and Microsoft ignores the issue.
I stopped using my own domain (managed by an association) for sending to outlook.com and yahoo because I know it's a fight that will be a net loss for me. Senders will change provider (because it's critical) before these major actor starts planning a fix.
And in an organisation, if a business person asks the IT to fix the issue, the only way to do it in a timely manner is to changer provider. The requester will not care if it's Microsoft fault, sending mail works at home and in other companies, not here, do something, business comes first, bla bla bla.
Barracuda ESS appears to block everything from Amazon SES even with valid DKIM and SPF. Of course they can’t be bothered with DMARC reports. This really sucks for users trying to receive Cognito password reset emails.
I block SES too. I couldn't find a reasonable way to report abuse, and the system does nothing to enforce opt-outs/unsubscribe. It's an absolutely horrible service to be on the receiving end of.
It handles standard complaints or bounces while adding to global or account supression lists. If an account exceeds a complaint ratio, it is suspended.
What are you aiming to accomplish by blocking a cloud services provider and not a particular sender domain?
So I should point out unfair sample set - self-host me, myself and I, and no-one else.
When I gave up on SES, I was getting annoyed by a commercial list that had no unsubscribe (it did, but went to a domain that didn't exist). So I had a grep through to see what I'd be missing, and it was 100% worth missing. I don't think I've seen any other provider that'd be 100% - usually when they get that bad it's just here-today-gone-tomorrow hosts.
The impression I get is that providers like mailchimp where they're actually packaging it as a service, so they handle unsubscribe, campaigns, they have an abuse@ contact that actually works .. these services deliver content. Dumb pipes that only exist because the cloud provider's regular IP space has a trash reputation, tends to carry everything that gave the regular IP space that reputation in the first place. They're just weaponising the idea that no-one would be dumb enough to block amazon. I'm dumb enough.
Actually their lack of proper enforcement of unsub etc is indeed dumb, I was really just curious because out of everyone I seem to get less junk via SES (pretty much everyone else ends up spewing junk from stolen accounts or trial accounts etc) and I've reported stuff to the usual clowns like mailgun in the past and got zero response so they're out, along with sendgrid.
Also self hosted, oldest domain probably about 15 years - dropping the above hasn't actually affected my as negatively as people getting gsuite wrong and failing dkim etc
I had same problem with deliverability of emails to Hotmail accounts a few years ago, they blocked AWS SNS IP email servers so sometimes were received but most of them directly sent to SPAM or never received. We complained Microsoft and their solution was to use one of their cloud email service or partner because the "black magic" of their AI filter. This is a shame
My favourite response from their so called "support" was to claim nothing was being blocked. They weren't going to let the full headers and error message from their outlook.com server convince them otherwise. Thankfully two days later it magically unblocked again. All the time enrolled in their "Smart Network Data Service" which did nothing. I probably just got an auto reply from the same bot Google uses to handle ban appeals.
I worked at MS during the launch of outlook.com, got some good email addresses without numbers on it, and used them for years. This and other issues caused me to give up and change to a different provided within the last few months.
The Outlook teams are weird. They all have their own special feedback mechanisms that are different from the rest of Office, and they for the most part ignore them.
Earlier this year, for some reason the SMTP servers were changed from smtp.live.com to smtp.office365.com, breaking a number of my workflows and integration with other tools. This is meaningless pain that served no point - you could just point the DNS records for smtp.live.com to the exact same servers smtp.office365.com points to. Combined with other things Microsoft has done (breaking decades of links to MSDN blogs and support pages without providing redirects) I have no faith in the stability of the product in the future.
Their web view generally sucks. It doesn't play nicely with the back button - selections are lost, search results are bypassed. It gets randomly stuck where it won't load or will load the wrong CSS for hours at a time on multiple of my machines.
The Android and iOS apps have largely not changed since they used to be Accompli. I can't name a new feature in the last decade.
Why, in the left nav bar, is there an icon with an envelope and a plus button that is "connect a GMail account"? That icon suggests "write a new mail", which is a function I would do quite often. I never want to connect a GMail account - much less have it take up a dedicated button that's always present in the UI.
Outlook on Android has its section buttons (mail, calendar, etc.) on the bottom. Mail for Windows has them on the bottom. Outlook 2016 and 2019 have them in the lower left. Outlook.com for many years had them in the lower left (if vertical). At some point they've moved to the upper-left, which is inconsistent with every other Microsoft-provided way that I check my mail and a regular source of frustration when I throw my mouse and eyes to the lower left corner and find nothing there.
I could go on and on (my favorite bug is that they removed the Send Feedback button that their docs refer to so I can't tell them any of this). I was an Outlook fan for several years, but I could have written most of this feedback in 2015 and nothing's changed; at this point I would encourage anyone still using it to just forward their mail to some other provider and be done with it.
The document attachments which come along with incoming email are still listed in "Documents" tab in Outlook even after they have been purged from secondary trash. The documents themselves aren't available (cannot be opened) but a list of all documents which have ever arrived (including JPGs/PNGs in signaturss) are still visible in that tab. I flagged this issue once but no resolution yet
I think at least part of the solution should be replying to all such instances with "Hello your client is broken. Would you mind using a different client."
It's a ridiculous and easily dismissible statement at first, but not so ridiculous if it becomes commonplace/familiar/canon (it worked for internet explorer!)
Another one to watch out for is Gmail silently truncated emails (if the unsubscribe button is missing, often it is in the truncated part). You can't get to the missing parts easy on mobile I think.
The unfortunate thing is that we also find the demographic of customers here in the UK still using btinternet.com email closely aligns with customers who tend to be more problematic…
(My parents have a btinternet.com email address)