Last modified: 2014-08-11 20:58:00 UTC
Noted on Checkuser-L mailing list. It appears that emails being sent out to subscribers with Hotmail/MSN email addresses are being bounced, and may not be sending out properly. The Checkuser-L list admins have received a bounce notice for 19 users, which reads: SMTP error from remote mail server after MAIL FROM:<checkuser-l-bounces@lists.wikimedia.org> SIZE=17772: host mx2.hotmail.com [65.54.188.72]: 550 SC-004 (BAY0-MC1-F13) Unfortunately, messages from 208.80.154.4 weren't sent. We recommend that you contact your Internet service provider. The problem is that too many unwanted messages have been sent from the following IP address above. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. Affected users have been sent an email informing of this, and asking them to contact their service provider. Nonetheless, want to ensure you are aware of this in case there is something that can be done from the WMF side.
It seems that the WMF IP adddress has been added to the MSN/Hotmail Junk mail block. This may require negotiation with Hotmail from the WMF end. "550 SC-004: Mail rejected by Outlook for policy reasons. A block has been placed against your IP address because we have received complaints concerning mail coming from that IP address. We recommend enrolling in our Junk Email Reporting Program (JMRP), a free program intended to help senders remove unwanted recipients from their email list. If you are not an email/network admin please contact your Email/Internet Service Provider for help."
Alexandros: Is this something you could take a look at? Or any idea who could? Confirming also for other mailing lists: SMTP error from remote mail server after MAIL FROM: <wikitech-l-bounces@lists.wikimedia.org> host mx2.hotmail.com [65.54.188.72]: 550 Unfortunately, messages from 208.80.154.4 weren't sent. We recommend that you contact your Internet service provider. The problem is that too many unwanted messages have been sent from the following IP address above. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors
There is an rt ticket with some more info on this https://rt.wikimedia.org/Ticket/Display.html?id=7051 as well as another comment on https://bugzilla.wikimedia.org/show_bug.cgi?id=59731#c30 that seems related.
Is it possible for someone with a hotmail account to try adding the mailing list to their address book/contacts? That 'should' get around a blacklist issue if that's the problem and so might help troubleshoot. (if it is a blacklist issue it appears to only be for lists.wikimedia.org since I've received emails to my @wikimedia.org address.
This is the counterpart of bug 56414; we could merge them and avoid mentioning specific ISP in the summary, but the dependencies for mailman and MediaWiki mailing problems are different: a different IP is being blocked so this should be unrelated to MediaWiki issues, unless Microsoft extends blocks to the IP range (do they?). Is there any setting that we have forgotten or something else that mailman should do better? Can it be some attack, e.g. people sending masses of emails with fake From: or Reply-To and hence causing masses of bounce messages to nonexisting addresses? Most likely, however, this is just about bug 59731 causing big backlogs of messages to be cleared at once, causing spikes of deliveries that ISP interpret as malicious, no?
(In reply to Nemo from comment #5) > This is the counterpart of bug 56414; we could merge them and avoid > mentioning specific ISP in the summary, but the dependencies for mailman and > MediaWiki mailing problems are different: a different IP is being blocked so > this should be unrelated to MediaWiki issues, unless Microsoft extends > blocks to the IP range (do they?). I don't think this is the same thing. The email service provider has specifically blocked emails from one of the WMF IP addresses. It is affecting all users, whether or not they have the mailing list in their "contacts". There is no delay involved, they're never getting through at all. > Is there any setting that we have forgotten or something else that mailman > should do better? Can it be some attack, e.g. people sending masses of > emails with fake From: or Reply-To and hence causing masses of bounce > messages to nonexisting addresses? Most likely, however, this is just about > bug 59731 causing big backlogs of messages to be cleared at once, causing > spikes of deliveries that ISP interpret as malicious, no? As noted, there are no "spikes" at all, the emails are not getting through at all.
Logs indicated messages are being rejected by hotmail with reason 550 SC-004. Still unclear what was the cause of this but this page https://mail.live.com/mail/troubleshooting.aspx clearly states complaints have been received. We will try to contact hotmail/live and solve this.
I've submitted Live's (incredibly verbose) form: https://support.live.com/eform.aspx?productKey=edfsmsbl3&ct=eformts&scrx=1 and I'm waiting to hear back. I'll keep you updated.
I just got back this from Microsoft/Hotmail/Live.com/Outlook.com: "We have completed reviewing the IP(s) you submitted. The following table contains the results of our investigation. Mitigated 208.80.154.4 Our investigation has determined that your IP(s) were blocked because too many people have reported your mail as unwanted. However, we have concluded these IP(s) qualify for mitigation. Please note that mitigating this issue does not guarantee that your email will be delivered to a user’s inbox. Mitigation may take 24 - 48 hours to replicate completely throughout our system. If you feel your issue is not yet resolved, please reply to this email and one of our support team members will contact you for further investigation." Let's keep this open and monitored for the next 48 hours.
(In reply to Faidon Liambotis from comment #9) > I just got back this from Microsoft/Hotmail/Live.com/Outlook.com: > > "We have completed reviewing the IP(s) you submitted. The following > table contains the results of our investigation. > > Mitigated > 208.80.154.4 > Our investigation has determined that your IP(s) were blocked because > too many people have reported your mail as unwanted. However, we have > concluded these IP(s) qualify for mitigation. Please note that > mitigating this issue does not guarantee that your email will be > delivered to a user’s inbox. > > Mitigation may take 24 - 48 hours to replicate completely throughout > our system. If you feel your issue is not yet resolved, please reply to > this email and one of our support team members will contact you for > further investigation." > > Let's keep this open and monitored for the next 48 hours. Thanks Faidon, Do they have a definition of mitigation anywhere?
(In reply to Faidon Liambotis from comment #9) > I just got back this from Microsoft/Hotmail/Live.com/Outlook.com: > > "We have completed reviewing the IP(s) you submitted. The following > table contains the results of our investigation. > > Mitigated > 208.80.154.4 > Our investigation has determined that your IP(s) were blocked because > too many people have reported your mail as unwanted. However, we have > concluded these IP(s) qualify for mitigation. Please note that > mitigating this issue does not guarantee that your email will be > delivered to a user’s inbox. > > Mitigation may take 24 - 48 hours to replicate completely throughout > our system. If you feel your issue is not yet resolved, please reply to > this email and one of our support team members will contact you for > further investigation." > > Let's keep this open and monitored for the next 48 hours. Thank you very much for the prompt attention to this. We have now received confirmation from one of the affected users that they are now receiving emails per normal.
I've no idea what's done by whom, but I think it's solved since I receive the emails from the lists again. So thanks everyone for helping! /me is happy
(In reply to James Alexander from comment #10) > Do they have a definition of mitigation anywhere? My understanding is that it means removal from the blacklist and we have evidence of this being the case already. I believe the "no gurantees" statement they make at the end refers to spam classification that happens after that and decides whether the email would reach the user's inbox or their spam folder.
Champion! From my days at RootsWeb as listmaster, it was the case of subscribers not being able to work out how to unsubscribe, and then just calling it spam to deal with the issue. My understanding of the message from hotmail-msn is that we are not globally blacklisted, however, we won't be getting through each user's personal blacklist. Woopee. =Solutions= It would be worthwhile reviewing our landing page for mailing lists, to make sure thta we have SIMPLE information for people to follow to understand what they need to do get off a mailing list. Might also be worth poking Signpost, and seeing if they can do a little article that also gives some background to the problem, and that too has a link to simple instructions. Long-term. As we have email addresses in SUL db, having an ability to have the subscriptions to mailing lists visible in or from for each SUL login (paired on email address) would be a useful development. As we increase the numbers of mailing lists, having the one facility for a user to see their subscriptions in one place becomes increasingly important, especially for a integrated site.
(In reply to billinghurst from comment #14) > It would be worthwhile reviewing our landing page for mailing lists, to make > sure thta we have SIMPLE information for people to follow to understand what > they need to do get off a mailing list. Users who consider mailing lists they subscribed to spam don't reach the landing page. Additionally, hotmail users are notoriously even less tech savvy than average, on average (obviously none of those who write or read this bug is ;) ). And the average knowledge of email functioning is very low; surveys show that usage of email by teens is going down and is now around 40 % (being replaced by walled gardens). I think everyone should just do their duty properly. Gmail has a feature which tells you "hey, this email sender even tells you where you can unsubscribe [with List-Unsubscribe], are you sure they're spammers?" when you click the spam button; maybe Hotmail needs something like that. And our mailing delivery system is full of actual bugs, so we should be working on those (some are found in "see also")... and we are, in part. > Might also be worth poking Signpost, and seeing if they can do a little > article that also gives some background to the problem, and that too has a > link to simple instructions. Nonsense. Such people don't even read the footers of emails they consider spam.
I am marking this as resolved now since this specific problem has been solved.
And I'm experiencing the same problems again. Only a few emails came through the last few days, but I also missed a complete discussion from stewards-l. Perhaps people can contact Hotmail again? (I expect that's the cause again...) Hopefully then the problems are solved for good.... because this is pretty annoying.
(In reply to Trijnstel from comment #17) > And I'm experiencing the same problems again. This may be because of https://wikitech.wikimedia.org/wiki/Incident_documentation/2014-07-14-Lists ; we sent in few minutes the emails of 16 hours and probably their system wasn't amused? Need to ask whitelisting again...
RootsWeb/Ancestry went through a similar issue a week or two ago, and RootsWeb have made significant changes to mailing list "reply-to" addresses to manage this. It is total BS, that well-structured and email RFC complaint mailing lists have to put up with this rubbish from lazy postmasters, and may be at the behest of lazy mail subscribers who decide that something is now spam. :-/
(In reply to billinghurst from comment #19) > RootsWeb/Ancestry went through a similar issue a week or two ago, and > RootsWeb have made significant changes to mailing list "reply-to" addresses > to manage this. It is total BS, that well-structured and email RFC > complaint mailing lists have to put up with this rubbish from lazy > postmasters, and may be at the behest of lazy mail subscribers who decide > that something is now spam. :-/ Hello? Can someone fix this please asap? It's pretty annoying to miss everything without knowing when it will be fixed, and I'm sure I'm not the only one who uses Hotmail...
> Hello? Can someone fix this please asap? It's pretty annoying to miss > everything without knowing when it will be fixed, and I'm sure I'm not the > only one who uses Hotmail... In the long run, this will get fixed, once our VERP-BounceHandler extension goes live in WMF. The extension is completed (AFAIK) , and now requires little testing before we deploy in the WMF cluster. Hope things go well. https://www.mediawiki.org/wiki/VERP https://github.com/wikimedia/mediawiki-extensions-BounceHandler
The above handling is only for Mediawiki, and not the Mailman. Thanks to Richard for notifying me.
(In reply to billinghurst from comment #19) > RootsWeb/Ancestry went through a similar issue a week or two ago, and > RootsWeb have made significant changes to mailing list "reply-to" addresses > to manage this. Any link to more information handy? (In reply to Trijnstel from comment #20) > Can someone fix this please asap? How?
I don't experience any problems anymore - it was just a few days. So I'll close this bug again and let's hope this won't happen anymore. :)
