Messages in this thread | | | Date | Wed, 27 Oct 2021 21:05:25 -0700 | Subject | Re: Unsubscription Incident | From | John 'Warthog9' Hawley <> |
| |
On 10/27/21 17:27, Slade Watkins wrote: > On Wed, Oct 27, 2021 at 4:42 PM John 'Warthog9' Hawley > <warthog9@eaglescrag.net> wrote: >> >> On 10/27/21 12:34 PM, Jakub Kicinski wrote: >>> On Mon, 25 Oct 2021 11:34:28 -0700 Shannon Nelson wrote: >>>>>> It happened to a bunch of people on gmail: >>>>>> https://lore.kernel.org/netdev/1fd8d0ac-ba8a-4836-59ab-0ed3b0321775@mojatatu.com/t/#u >>>>> I can at least confirm that this didn't happen to me on my hosted >>>>> Gmail through Google Workspace. Could be wrong, but it seems isolated >>>>> to normal @gmail.com accounts. >>>>> >>>>> Best, >>>>> >>>> >>>> Alternatively, I can confirm that my pensando.io address through gmail >>>> was affected until I re-subscribed. >>> >>> Did it just work after re-subscribing again? Without cleaning the inbox? >>> John indicated off list that Gmail started returning errors related to >>> quota, no idea what that translates to in reality maybe they added some >>> heuristic on too many emails from one source? >> >> At least for the users I've had anyone mention to me (which for the >> record apparently this happened on the 11th, and people are only >> reaching out now about), the reasons for the unsubscribe was that the >> upstream servers were reporting that the users in question were over >> quota permanently. We take that hinting at face value, and since the >> server is telling us (basically) that the user isn't going to be >> accepting mail anytime soon, we go ahead and unsubscribe them and clear >> the queue so that the users don't cause unnecessary back log. Noting, >> this is an automated process that runs and deals with this that runs >> periodically. >> >> Also noting, that there's not a good way to notify individuals when this >> happens because, unsurprisingly, their email providers aren't accepting >> mail from us. >> >> If folks reach out to postmaster@ I'm more than happy to take a look at >> the 'why' something happened, and I'm happy to re-subscribe folks in the >> backend saving them the back and forth with majorodomo's command system. >> > > John, > That's great. > >> >> If I had to speculate, something glitched at gmail, a subset of users >> got an odd error code returned (which likely wasn't the correct error >> code for the situation, and noting the number of affected users is >> fairly small given the number of users from gmail that are subscribed). >> Likely similar to when gmail had that big outage and it reported >> something way off base and as a result every gmail user got unsubscribed >> (and subsequently resubscribed in the backend by me when the outage was >> over). >> > > Is there any way to detect if something like that affects Google > Workspace hosted inboxes too? Sounds like those in that group who were > affected were very few though but I thought I'd ask anyway.
Looking back around then it looks like a few individual users might have been, but it's sporadic on the domains that aren't gmail.com, so it doesn't look like (to what I can see anyway) it was a full worksapce for instance.
The real problem is the error that was getting kicked back was a generic enough error, that's it's hard to tell what users may have actually been over quota vs. something else.
- John 'Warthog9' Hawley
| |