24/7/365 Sales & Support 1.866.257.8455 Login

Major Cloud Providers – Monthly Outage Recap October

MS Azure

10/30/19 RCA – Storage – West EuropeBetween 10:17 and 13:52 UTC, a subset of customers using Storage, or Azure services with Storage dependencies, in West Europe may have experienced difficulties connecting to resources hosted in this region.

10/25/19

Azure Databricks and Data Factory v2 – Workspace and Data Flow ErrorsBetween approximately 11:00 and 14:40 UTC, a subset of customers using Azure Databricks may have received ‘No Web App available’ error notifications when logging into a Databricks workspace. Related API calls may have also not returned a response. Additionally, a very limited subset of customers using Data Factory v2 may have received failure notifications for Data Flow jobs.

-DownDetector Problems at Microsoft AzureMicrosoft Azure is having issues since 10:29 AM EDT.Most reported problems

 42% Cloud services

 40% Website hosting

 17% Virtual machines

10/22/19

RCA – Service Availability – West EuropeBetween 21:15 UTC on 22 Oct 2019 and 00:23 UTC on 23 Oct 2019, a subset of customers using Storage in West Europe may have experienced service availability issues. In addition, resources with dependencies on the impacted storage unit may have experienced downstream impact in the form of availability issues or high latency.

Azure Portal Sign In Failure – West EuropeBetween 06:40 UTC and 09:54 UTC a subset of customers were identified as having experienced issues signing in to the Azure Portal in West Europe.

10/21/19

RCA – Storage – West EuropeBetween 23:20 UTC on 21 Oct 2019 and 04:32 UTC on 22 Oct 2019, a subset of customers using Storage in West Europe may have experienced service availability issues. In addition, resources with dependencies on the impacted storage unit may have experienced downstream impact in the form of availability issues or high latency.

RCA – Service Management Errors – East US 2Between 10:08 and 23:37 UTC, a subset of customers in East US 2 may have received error notifications or experienced high latency when performing service management operations – such as create, update, or delete – for resources hosted in this region. Additionally, customers using Azure Databricks and/or Data Factory v2 may have encountered service management errors in multiple regions. A very limited subset of customers using Virtual Machines with SQL Server images, or other SQL IaaS offerings, may have also encountered errors performing service management operations on resources hosted in East US 2.


10/18/19
RCA – Authentication issues with Azure MFA in North AmericaBetween 13:30 UTC and 15:57 UTC, customers in North America experienced issues receiving multi-factor authentication (MFA) challenges. Users who had valid MFA claims during the incident were not impacted. However, users who were required to perform an MFA challenge during this incident were unable to complete the challenge. This represented 0.51% of users in North American tenants using the service during the incident.

DownDetector

10/18/19 Problems at Microsoft AzureMicrosoft Azure is having issues since 10:05 AM EDT. Most reported problems

 60% Cloud services

 35% Website hosting

3% Virtual machines10/16/19 RCA – Azure PortalBetween 13:45 and 14:59 UTC (approx.), a subset of customers may have experienced latency issues with the Azure Portal, Command Line, and Azure PowerShell.

AWS

10/24/19 VMware Cloud on AWS: Host Remediation Degradation VMware Cloud on AWS (VMC) experienced an operational issue that caused us to inadvertently remediate hosts that had not actually failed. Customers may have noticed an unusual amount of host replacement activity due to this error. This incident affected: VMware Cloud on AWS

Start Time (UTC): October 24, 2019, 22:20 hours

End Time (UTC): October 25, 2019, 22:25 hours

Duration: 24hrs 5mins

DownDetector

10/23/19 Problems at Amazon Web Services Amazon Web Services is having issues since 8:48 AM EDT.Most reported problems

 63% S3

 29% AWS Console

 6% Route53

DownDetector

10/22/19 Problems at Amazon Web Services Amazon Web Services is having issues since 2:56 PM EDT. Most reported problems

 73% S3

 18% AWS Console

 8% Route53

10/15/19 VMware Cloud on AWS: SDDC Capacity issue in Asia Pacific Southeast Region New SDDC provisioning may fail for Asia Pacific South East (Singapore). This incident affected: VMware Cloud on AWS SDDC (Asia Pacific (Singapore)).

Start Time: October 15, 2019, 08:32 UTC
End Time: October 18,2019, 07:51 UTC

10/14/19 VMware Cloud on AWS: SDDC provisioning failures in US West (Oregon) Region New SDDC provisioning may fail in US West (Oregon). This incident affected: VMware Cloud on AWS SDDC (US West (Oregon)).

Start Time: October 14, 2019 07:56 UTC
End Time: October 15, 2019 07:08 UTC

10/10/19 VMware Cloud on AWS: SDDC provisioning failures in US East (N. Virginia) region New SDDC provisioning or adding hosts/clusters may fail in US West (Oregon). This incident affected: VMware Cloud on AWS SDDC (US East (N. Virginia), US West (Oregon), Asia Pacific (Sydney)).

Start Time: October 10, 2019 18:00 UTC
End Time: October 10, 2019 15:14 UTC

10/5/19

VMware Cloud on AWS: Unable to access the Network and Security tab Customers will not be able to view Network and Security tab. This incident affected: VMware Cloud on AWS (VMware Cloud on AWS).

Start Time: October 05, 2019 04:20 UTC
End Time: October 05, 2019 11:20 UTC

VMware Cloud on AWS Service Performance Degradation issue User may not be able to access the service or may experience trouble while accessing the service. This incident affected: VMware Cloud on AWS (VMware Cloud on AWS, DRaaS, HCX).

Start Time: October 05, 2019 01:00 UTC
End Time: October 05, 2019 01:34 UTC

10/4/19

-Amazon CloudWatch (Oregon) Elevated Error Rates Starting at 5:27 PM PDT, there were delays delivering scheduled CloudWatch events. Recovery began at 7:36 PM PDT, and by 9:22 PM PDT the backlog of all previously scheduled CloudWatch events was successfully delivered.

-Amazon CloudWatch (N. Virginia) Elevated Error Rates Starting at 5:27 PM PDT, there were delays delivering scheduled CloudWatch events. Recovery began at 7:31 PM PDT, and by 9:28 PM PDT the backlog of all previously scheduled CloudWatch events was successfully delivered.

10/2/19 VMware Cloud on AWS: SDDC provisioning failures New SDDC provisioning or adding hosts/clusters may fail in US West (Oregon). This incident affected: VMware Cloud on AWS SDDC (US West (Oregon)).

Start Time: Tuesday, 1 October 2019, 22:28 UTC
End Time: Wednesday, 2 October 2019, 20:56 UTC

Other Platforms

10/31/19

-Google Cloud infrastructure components Incident #19001 Our engineers have determine issue to be linked to a single Google incident. Incident began at 2019-10-31 16:41 and ended at 2019-11-02 10:00 (US/Pacific).

-Google Cloud Composer Incident #19003 Our engineers have determined that Cloud Composer was impacted by the same underlying issue as the Google Compute Engine (GCE) incident. Incident began at 2019-10-31 16:41 and ended at 2019-11-01 12:41 (US/Pacific).

-Google Cloud Networking Incident #19021 Our engineers have determined that Google Cloud Networking was impacted by the same underlying issue as the Google Compute Engine (GCE) incident. Incident began at 2019-10-31 16:41 and ended at 2019-11-02 10:00 (US/Pacific).

-Google Cloud Machine Learning Incident #19002 Our engineers have determined that Cloud ML Engine was impacted by the same underlying issue as the Google Compute Engine (GCE) incident. Incident began at 2019-10-31 16:41 and ended at 2019-11-01 09:00 (US/Pacific).

-Google Cloud Memorystore Incident #19002 Our engineers have determined that Cloud Memorystore was impacted by the same underlying issue as the Google Compute Engine (GCE) incident. Incident began at 2019-10-31 16:41 and ended at 2019-11-01 07:41 (US/Pacific).

-Google Cloud DNS Incident #19003 Our engineers have determined that Google Cloud DNS was impacted by the same underlying issue as the Google Compute Engine (GCE) incident.Incident began at 2019-10-31 16:41 and ended at 2019-11-01 08:15 (US/Pacific).

-Google Cloud Datastore Incident #19005 We’ve received a report of an issue with Cloud Datastore / Firestore in us-central. Incident began at 15:30 and ended at 16:01 (US/Pacific).

10/27/19 VMware Cloud Services Functionality Issues Intermittent issues with functionality of our VMware Cloud Services.

Start Time: October 27, 2019 21:48 UTC
End Time: October 28, 2019 01:50 UTC

10/24/19 VMware Cloud on AWS: Host Remediation Degradation VMware Cloud on AWS (VMC) experienced an operational issue that caused us to inadvertently remediate hosts that had not actually failed. Customers may have noticed an unusual amount of host replacement activity due to this error.

Start Time (UTC): October 24, 2019, 22:20 hours

End Time (UTC): October 25, 2019, 22:25 hours

Duration: 24hrs 5mins

10/22/19

-SAP Cloud Platform Europe (Netherlands) [cf-eu20] – Service Advisory Since approximately 21:13 UTC on 22 Oct 2019 until 03:04 UTC on 23 Oct 2019 Lifecycle management operations cannot be executed. Application logs in Kibana are unavailable.

-SAP Cloud Platform Europe (Netherlands) [cf-eu20] – Service Advisory Since approximately 23:20 UTC on 21 Oct 2019 until 04:42 UTC on 22 Oct 2019 Data processing for IoT message producers and consumers is not possible

-Google Cloud Bigtable Incident #19001 Our engineers have determined this issue to be linked to the Google Cloud Networking incident in us-west1-b. Incident began at 17:04 and ended at 20:04 (US/Pacific).

-Google Cloud Networking Incident #19020 Google Compute Engine experienced 100% packet loss to and from ~20% of instances in us-west1-b for a duration of 2 hours, 31 minutes. Additionally, 20% of Cloud Routers, and 6% of Cloud VPN gateways experienced equivalent packet loss in us-west1. Incident began at 16:47 and ended at  21:35  (US/Pacific).

-Cloud Memorystore Incident #19001 Our engineers have determined this issue to be linked to the Google Cloud Networking incident in us-west1-b. Incident began at 17:14 and ended at 19:05 (US/Pacific).

-Google Cloud Storage Incident #19008 Incident began at 17:11 and ended at 20:04  (US/Pacific).

10/20/19 VMware Cloud PKS Performance Degradation issue Customers may experience delays or failures in creating or deleting clusters in all regions.

Start Time: October 20, 2019 17:05 UTC
End Time: October 20, 2019 18:33 UTC

10/19/19 SAP Cloud Platform US East (Ashburn) [neo-us1] – Service Advisory Since approximately 07:32 UTC until 08:11 UTC Git Service and HTML5 application deployment are unavailable

10/18/19

-SAP Cloud Platform US East (Ashburn) [neo-us1] – Service Advisory Since approximately 22:03 UTC until 23:02 UTC Forms by Adobe is unavailable

-SAP Cloud Platform US East (Ashburn) [neo-us1] – Service Advisory Since approximately 14:51 UTC until 18:31 UTC A general disruption is impacting the availability of applications and services

10/15/19

VMware Cloud on AWS: SDDC Capacity issue in Asia Pacific Southeast Region New SDDC provisioning may fail for Asia Pacific South East (Singapore).

Start Time: October 15, 2019, 08:32 UTC
End Time: October 18,2019, 07:51 UTC

VMware Skyline Service Availability issue Users may not be able to access the service or may experience trouble while accessing the service.

Start Time: October 15, 2019, 03:48 UTC
End Time: October 15, 2019, 06:54 UTC

10/14/19

VMware Cloud on AWS: SDDC provisioning failures in US West (Oregon) Region New SDDC provisioning may fail in US West (Oregon).

Start Time: October 14, 2019 07:56 UTC
End Time: October 15, 2019 07:08 UTC

-SAP Cloud Platform Japan (Tokyo) [neo-jp1] – Service Advisory Since approximately 02:01 UTC until 03:12 UTC Lifecycle operations for HTML5 applications are not possible. Portal and WebIDE services were unavailable.

10/12/19 VMware Cloud Services Performance Degradation with VMware Log Intelligence Customers can expect to see log ingestion delays ranging from 30 to 90 minutes.

Start Time: October 11, 2019 21:38 UTC
End Time: October 12, 2019 02:15 UTC

10/11/19

SAP Cloud Platform US East (Sterling) [neo-us3] – Service Advisory Since approximately 11:50 UTC until 13:38 UTC A general disruption is impacting the availability of applications and services

SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 05:18 UTC until 10:45 UTC Lifecycle management operations for Java applications cannot be executed. Customers may not be able to deploy, undeploy their application.

10/10/19

VMware Cloud on AWS: SDDC provisioning failures in US East (N. Virginia) region New SDDC provisioning or adding hosts/clusters may fail in US West (Oregon). 

Start Time: October 10, 2019 18:00 UTC
End Time: October 10, 2019 15:14 UTC

-SAP Cloud Platform Europe (Frankfurt) [neo-eu2] – Service Advisory Since approximately 07:30 UTC until 08:32 UTC Lifecycle management operations for Java applications cannot be executed. Customers may not be able to deploy, undeploy their application.

-SAP Cloud Platform Europe (Frankfurt) [neo-eu2] – Service Advisory Since approximately 05:38 UTC until 06:00 UTC Customers may not be able to deploy, undeploy their application. Lifecycle management operations for Java applications cannot be executed

-SAP Cloud Platform Singapore

[cf-ap11]

– Service Advisory Since approximately 00:39 UTC until 04:29 UTC Applications protected by Authorization & Trust Management (XSUAA) are not accessible

10/8/19 SAP Cloud Platform Japan (Tokyo) [neo-jp1] – Service Advisory Since approximately 13:24 UTC until 13:47 UTC A general disruption is impacting the availability of applications and services

10/7/19

VMware Cloud PKS Performance Degradation issue Users might experience issues while creating cluster in US-East Region (North Virginia).

Start Time: October 07, 2019 04:14 UTC
End Time: October 07, 2019 05:06 UTC

-SAP Cloud Platform Brazil (São Paulo) [neo-br1] – Service Advisory Since approximately 12:03 UTC until 13:11 UTC Audit Log retrieval API is not available

10/5/19

VMware Skyline Service Intermittent Availability issue User may not be able to access the service or may experience trouble while accessing the service.

Start Time: October 05, 2019 14:53 hours UTC
End Time: October 05, 2019 16:30 hours UTC

VMware Cloud on AWS: Unable to access the Network and Security tab Customers will not be able to view Network and Security tab.

Start Time: October 05, 2019 04:20 UTC
End Time: October 05, 2019 11:20 UTC

VMware Cloud on AWS Service Performance Degradation issue User may not be able to access the service or may experience trouble while accessing the service.

Start Time: October 05, 2019 01:00 UTC
End Time: October 05, 2019 01:34 UTC

10/4/19 VMware Skyline Service Intermittent Availability issue Some users may not be able to perform any operations on Skyline Advisor Dashboard.

Start Time: October 04, 2019 12:36 UTC
End Time: October 04, 2019 13:17 UTC

10/2/19 VMware Cloud on AWS: SDDC provisioning failures New SDDC provisioning or adding hosts/clusters may fail in US West (Oregon). This incident affected: VMware Cloud on AWS SDDC (US West (Oregon)).

Start Time: Tuesday, 1 October 2019, 22:28 UTC
End Time: Wednesday, 2 October 2019, 20:56 UTC