System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Impaired | Virtual Machines | Affected VMs are down. All other VMs are operating normally. |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Impaired | SAN | Multiple drive failures in one aggregate. All other SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes: The equipment took longer than expected to arrive by courier. The data has now been restored, and we are booting up the servers now.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Impaired | Virtual Machines | Affected VMs are down. All other VMs are operating normally. |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Impaired | SAN | Multiple drive failures in one aggregate. All other SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes: At approximately 11:47am Pacific time one of our SANs suffered multiple drive failures, causing data loss. We are restoring data from our offsite backup. We expect all servers to be operational by 6pm. Servers are expected to be restored to their snapshot from 11:00am.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes: The IP address targetted by the attack has been null routed on all ISPs. Traffic should be normalized on all interfaces.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | One service provider is currently not being routed to our network. |
Notes: At 5:05pm Pacific time we detected a DOS attack designed to take down the network. We have been able to stop the traffic from all but one service provider. We have stopped routing traffic from that location until further notice. All connectivity appears to be back to normal.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes:
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes: The upstream provideer reported 12 hours ago that all traffic should be back to normal. After testing, we have begun again to send production traffic over this link. We will keep an eye on this for the next 12 hours.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | All customer traffic is operating normally. |
Notes: One of our upstream providers is experiencing a hardware issue. We have diverted all traffic away from their network.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Circuit errors to one upstream provider. |
Notes: We have returned production traffic to the suspect circuit. There seems to be an issue with bgp updates. This may cause sporadic, but limited, traffic loss. We are working with our upstream provider to get this resolved as soon as possible.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Impaired | Internet Connectivity | Circuit errors to one upstream provider. |
Notes: We are currently taking errors on one interface to a single upstream provider. We have isolated the issue, and no production traffic is currently using this circuit. We will bring it back online once the circuit has been normalized.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally. |
Normal | Virtual Machines | All VMs operating normally |
Normal | VMware Hosts | All vmware hosts are operating normally. |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewalls operating normally |
Normal | Internet Connectivity | Internet connectivity is normal. |
Notes: All VMs/servers are back online.