System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Internap has restored their peering with Cogent.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: We have reactivated our peering with Cogent. Internap has not, though it appears that the connection is back to normal. We will be observing closely.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Internet connectivity is impaired. Cogent peering administratively down. |
Notes: While Cogent reports that the incident is over, we are waiting until after business hours to restore our peering. Internap is doing the same.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Internet connectivity is impaired. Cogent peering administratively down. |
Notes: Cogent reports the following:
Customers Affected: Some Los Angeles Customers
Date: September 19th 2016
Incident Start Time: 11:35 AM PDT
Duration: 4 Hours
Associated Ticket: HD6655601
Dear Cogent Customer:
This letter is in response to the Cogent Communications network event that occurred at 11:35 AM PDT on September 19th 2016 lasting around 4 hours affecting Los Angeles area Customers.
Between 11:30 AM PDT and 3:30 PM PDT on September 19th, 2016 customers in our Los Angeles market experienced severe packet loss and latency. The issue was due to a fiber cable cut experienced by our longhaul fiber vendor in close proximity to our main Hub site in Los Angeles. This was due to unscheduled work being performed by the vendor. The Cogent NOC stayed in contact with our fiber provider, along with having our local Field Engineers on site until the fiber cut was resolved. Our vendor resolved the issue by splicing new fiber for the in-building risers.
We apologize for any impact you experienced. Please call us at 1-877-726-4368, option 2 if you have any additional questions concerning this event. Our NOC ticket number on this incident was HD6655601.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Internet connectivity is impaired. Cogent peering administratively down. |
Notes: After discussions with Internap and observing customer traceroutes, we have determined the cause of the outage is Cogent. Infinitely Virtual and Internap have deactivate our peerings with Cogent. We will need to observe if this will resolve the issue.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Internet connectivity is impaired |
Notes: Some customers report slow or inconsistent connectivity. This appears to be related to Internet connectivity.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Internet connectivity is impaired. Cogent peering administratively down. |
Notes: After taking down the Cogent peering, the majority of customers are normalized. A handful of customers are experiencing slower performance due to suboptimal paths over the Internet.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: The Los Angeles Firewall cluster is up and operating normally. The upgrade was successful.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Offline | Firewall Clusters | LA cluster down due to planned maintenance; Boston cluster operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: The firewall cluster is down in Los Angeles due to planned maintenance. We expect the system to be back online in 20 to 30 minutes.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: The Boston Firewall cluster is up and operating normally. The upgrade was successful.