×

196
197

Office365 email delays? We seem to be getting a few our end... (self.sysadmin)

submitted by tomhudsonnSysadmin

Reason: [{LED=422 4.4.1 Unable to identify upstream MTA for 'ourdomain.com' (db224dde-d05b-44c5-b29b-411bc9bddedc,f5ba5e3f-a116-4517-ae78-02e651b46c63)};{MSG=};{FQDN=uk1.smtp.exclaimer.net};{IP=51.140.37.132};{LRT=7/6/2022 9:04:44 AM}]. OutboundProxyTargetIP: 51.140.37.132. OutboundProxyTargetHostName:

​

Edit - Seems to be resolved now...

all 53 comments
sorted by:
best
topnewcontroversialoldq&a

[–]Zedilt 83 points84 points  (15 children)

EX397744

Title: Some users may experience delays when sending or receiving email messages in Exchange Online

User Impact: Users may experience delays when sending or receiving email messages in Exchange Online.

More info: Users may see the message status being stuck on 'Pending' or 'Getting Status'.

Current status: We're analyzing service telemetry information to determine the next troubleshooting steps.

Scope of impact: Impact is specific to some users who are served through the affected infrastructure in Europe, Middle East, and Africa.

[–]CreshalEmbedded DevSecOps 2.0 Techsupport Sysadmin Consultant [Austria] 42 points43 points  (11 children)

Final status: We've identified that a section of infrastructure, responsible for email attribution and routing, unexpectedly became unresponsive and resulted in DNS lookup failing to resolve hostnames. Additionally, this DNS issue prevented user traffic from being directed as expected and resulted in mail flow delays. We've rerouted user traffic to alternate infrastructure which restored the mail flow delays and remediated impact.

Why am I not surprised that it's DNS.

[–]fahque 22 points23 points  (2 children)

It's not dns. A network section went down. You can call it an infrastructure problem or a routing problem.

[–]supratachophobia 0 points1 point  (1 child)

Kinda coincidental that Network Solutions has had their DNS down twice in 24hrs though, right?

[–]painted-biird 1 point2 points  (0 children)

Yup- was on the phone with them trying to get a client’s DNS records restored yesterday and they had no idea how or why they were changed.

[–]billy_teats[🍰] 22 points23 points  (7 children)

The literally said “infrastructure became unresponsive” right before they said “then dns failed”

It’s a wonderful haiku but dns was not the problem here.

Recently my house blew a fuse. That fuse was connected to my pool pump. After a week of not pumping water my pool started growing algae. When I discovered this, I didn’t yell at my pool pump. Ya it was his job to move water around but he didn’t fail, the underlying infrastructure did. Stop shooting the messenger

[–]corsicanguppyDevOps Zealot 20 points21 points  (2 children)

I'm understanding you need to spend more time in your pool.

[–]billy_teats[🍰] 4 points5 points  (1 child)

Lol

Instead I ended up with 2 raspberry pi’s and a host of environmental sensors that will feed in to a central kubernetes cluster to monitor and alert me when things go wrong. So if my pool goes up or down too far in pH, if the water temp gets too low, if the current stops flowing, I get an alert. Alternatively, if any of the fuses should trip, I will also get an alert.

If all the power goes out to my house then I get no alerts

[–]Frothyleet 3 points4 points  (0 children)

Don't give up before the monitoring automation finish line! Does your power company have an outage website you can scrape?

[–]xixi2 5 points6 points  (3 children)

wow you're really defensive about DNS

[–]billy_teats[🍰] 11 points12 points  (0 children)

I would like to think I am more particular when I assign blame to Microsoft. I have had to prove their services were offline too many times. If you come to them and say you had dns problems and you want a refund, they say they didn’t have any dns problems and you should pound sand. If you say that azures infrastructure went down, they take a few bucks off your 7figure monthly bill

[–]corsicanguppyDevOps Zealot 0 points1 point  (0 children)

"Render unto caesar..."

[–]OhioAspy 0 points1 point  (0 children)

He is using proper terminology. It wasn't DNS, but device failures. If a device can't respond, DNS will never occur. Duh!

[–]aenae 7 points8 points  (2 children)

Brilliant move to put it behind a login so the general public can't access it. /s

I also noticed lots of dns errors starting around 4 hours ago, but their public status page said everything was al right and there were no Americans in Baghdad.

[–]xixi2 3 points4 points  (0 children)

Yeah I am just a normal "user" as far as O365 goes now at my new job as a SQL developer. But I still read here to know what's going on, but I can't get to the status :(

[–]xSnakeDoctor 0 points1 point  (0 children)

I think this is public facing - https://portal.office.com/servicestatus

[–]quaverIT Manager 19 points20 points  (7 children)

I'm getting the same (UK M365 BP tenant). We're also an Exclaimer user.

[–]Pie-Otherwise 8 points9 points  (5 children)

Meanwhile I got up early here in the US, sitting here at 5:37 in the morning hoping this gets fixed in the next couple of hours.

[–]m9832Sr. Sysadmin 2 points3 points  (3 children)

why? what are you going to do, stare it into fixing itself?

[–]IdiosyncraticBond 5 points6 points  (0 children)

That usually works for me. If I use the keyboard it gets worse

[–]VictoryNapping 0 points1 point  (0 children)

Sometimes a threatening hand gesture can help speed it up a bit too.

[–]Pie-Otherwise 0 points1 point  (0 children)

No, I share stuff I find here at work and they all think I'm some kind of uber nerd who has his fingers on the beating pulse of the industry, even at 5:30 in the morning.

In reality I was up at 4 because the geriatric dog was loudly licking her ass and my airpods died so I couldn't put an audiobook on and go back to sleep.

[–]quaverIT Manager 4 points5 points  (0 children)

Best of luck! Not much to do in these situations except grab a coffee and keep an eye on the status pages (and here, naturally), it seems!

[–]PipboyOGSysadmin 1 point2 points  (0 children)

Same here (Belgium). Our clients experience the same issue (with and without exclaimer)

[–]zewo_IT Manager 5 points6 points  (0 children)

Having problems sending emails too : "Microsoft is aware of this problem" on quarantine page

[–]_den_den 3 points4 points  (0 children)

Same AU O365 + excliamer. Excliamer have sent outage notification saying it's an issue on O365 end.

[–]Glanza 2 points3 points  (0 children)

Same here UK based, affecting us and customers who do and don't use Exclaimer.

[–]stef147 2 points3 points  (0 children)

There is an incident: Some users may experience delays when sending or receiving email messages in Exchange Online EX397744

[–]True-Shower9927 2 points3 points  (0 children)

It’s probably another power issue in one of their data centers 😩

[–]Zestyclose_Wait9892 2 points3 points  (0 children)

July 6, 2022 11:03 AM

Title: Some users may experience delays when sending or receiving email messages in Exchange Online

User Impact: Users may experience delays when sending or receiving email messages in Exchange Online.

More info: Users may see the message status being stuck on 'Pending' or 'Getting Status'.

Current status: We're analyzing service telemetry information to determine the next troubleshooting steps.

Scope of impact: Impact is specific to some users who are served through the affected infrastructure in Europe, Middle East, and Africa.

Next update by: Wednesday, July 6, 2022, 1:30 PM (11:30 AM UTC)

[–]vlanche 2 points3 points  (1 child)

Which symptoms are you seeing? In my case, outgoing e-mails seem to be working fine, but incoming e-mails from on-premise and external domains do not arrive at all.

[–]HamiltonFAIVMware Admin 0 points1 point  (0 children)

We've been getting customer complaints about our emails not going out, but it looks like it might be unable to receive on their end. This makes so much more sense since I havnt been able to find anything wrong on our side.

[–]iguru129 2 points3 points  (0 children)

With a SaaS solution, you get what you get and dont pitch a fit.

[–]BrandhorJack of All Trades 3 points4 points  (6 children)

looks like a dns issue, I can't resolve *.mail.protection.outlook.com

[–]tomhudsonnSysadmin[S] 1 point2 points  (5 children)

It's ALWAYS a DNS issue..

[–]Cuntable 1 point2 points  (0 children)

It can't be DNS.

[–]AlexIsPlaying -3 points-2 points  (3 children)

somebody junior fuuuk it up.

[–]newaccountzuerichLinux & Infosec S.Admin 4 points5 points  (2 children)

This is more a senior-level item for allowing anyone to fuck this type of thing up.

Never appropriate to blame the junior. Blame the seniors that enable the process that allows a junior to do it.

[–]tomhudsonnSysadmin[S] 2 points3 points  (1 child)

Agree with this, if a task that can go wrong can cause this damage, then the JR should be no where near it, and the SNR should be the one to point this out

[–]newaccountzuerichLinux & Infosec S.Admin 1 point2 points  (0 children)

Either the approval process for production changes is flawed, or the reviewers didn't do their jobs right (also part of a flawed change process really).

Hopefully the RCA (root cause analysis) is published, though it's likely it will not as it may expose internal processes that could be games when known by red teams.

[–]rainer_d 1 point2 points  (0 children)

Does Microsoft have a status page for their DNS servers?

[–]vlanche 1 point2 points  (0 children)

Seems to be resolved for us, mail flow looks good now.

[–]VinzentValentyn 1 point2 points  (0 children)

We have delays and some customers have delays. All using Exclaimer in the UK

[–]JohnMSP 1 point2 points  (0 children)

We are also seeing some strange NDRs for internal emails - bounce back being generated by a completely different tenant’s onmicrosoft address with unauthorised relay messages.

[–]greenstarthree 1 point2 points  (0 children)

Was an MS side issue for EMEA customers. Resolved as of 12:30 GMT as far as we can see

[–]AmiDeplorabilis 1 point2 points  (0 children)

So, if your car runs out of gas and won't start, do you have engine problems?

If the physical infrastructure fails, do you have DNS problems?

No to both...

[–]_rickjames2nd Line Misery 0 points1 point  (0 children)

We can't download Message Traces despite being Global Admins. Very bizarre...

[–]Kurgan_ITLinux Admin 0 points1 point  (0 children)

Problem is still present right now (14:20 GMT), lots of customers impacted. We don't use Office365 but everyone else does, so...

[–]According_Caramel926 0 points1 point  (0 children)

I've seen this only with exclaimer. Does this really affect M365 without exclaimer?

[–]bwave1 0 points1 point  (0 children)

Every time I see these outages and issues and more and more thankful that we maintain on-premise Exchange and AD, I hope to retire before being forced to go Azure/M365!

[–]Mr_ToDo 0 points1 point  (0 children)

They also chose this day to turn off the basic authentication to one of our clients that are still pushing off their migration.

Sure made troubleshooting interesting when other services are wacky too. Why it doesn't show up as a failed login attempt I'm not quite sure, but whatever.

[–]XXfriX 0 points1 point  (0 children)

Their support tried to convince me that there is OUR DNS problem because THEIR *.outlook.com records were unresolvabe. No comment. Errors were present last three days.

[–]spin_kick 0 points1 point  (0 children)

take a look at network solutions cocking up dns propagation