Rackspace hosted Exchange suffered a disastrous interruption beginning December 2, 2022 and is still ongoing since 12:37 AM December fourth. Initially described as connectivity and login concerns, the guidance was eventually upgraded to announce that they were dealing with a security event.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be solved.
Customers on Buy Twitter Verified reported that Rackspace was not reacting to support e-mails.
This has been rather the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours approximately. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Uncertain how many companies that is, but it’s substantial.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t knowledgeable about the bounce for a number of hours.”
The official Rackspace status page used a running upgrade of the interruption however the preliminary posts had no info other than there was an outage and it was being examined.
The very first official upgrade was on December second at 2:49 AM:
“We are investigating a problem that is affecting our Hosted Exchange environments. More details will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connection concern.”
“We are investigating reports of connectivity issues to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connection and login issues” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the failure, still trying to determine what went wrong.
And they were still calling it “connection and login concerns” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later Rackspace referred to the circumstance as a “significant failure”and began using their customers complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.
The main assistance specified:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any additional concerns while we continue work to restore service. As we continue to resolve the root cause of the concern, we have an alternate option that will re-activate your capability to send out and get e-mails.
At no charge to you, we will be supplying you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until additional notification.”
Rackspace Hosted Exchange Security Occurrence
It was not till nearly 24 hr later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was suffering from a security incident.
The statement even more exposed that the Rackspace professionals had actually powered down and disconnected the Exchange environment.
“After further analysis, we have identified that this is a security event.
The known effect is separated to a part of our Hosted Exchange platform. We are taking necessary actions to assess and protect our environments.”
Twelve hours later on that afternoon they upgraded the status page with more info that their security team and outside specialists were still working on solving the outage.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not released details of the security occasion.
A security event generally includes a vulnerability and there are two serious vulnerabilities presently in the wile that were covered in November 2022.
These are the 2 most current vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack allows a hacker to check out and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy has the ability to run malicious code on a server.
An advisory released in October 2022 described the impact of the vulnerabilities:
“A validated remote attacker can perform SSRF attacks to escalate privileges and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mail box server, the attacker can possibly get to other resources through lateral motion into Exchange and Active Directory site environments.”
The Rackspace interruption updates have actually not shown what the specific issue was, only that it was a security occurrence.
The most present status upgrade as of December 4th stated that the service is still down and customers are encouraged to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in attending to the event. The schedule of your service and security of your data is of high value.
We have committed comprehensive internal resources and engaged world-class external know-how in our efforts to minimize unfavorable effects to consumers.”
It’s possible that the above kept in mind vulnerabilities belong to the security incident affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether client details has been compromised. This occasion is still continuous.
Included image by Best SMM Panel/Orn Rin