System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Impaired | Virtual Machines | Restarting or otherwise troubleshooting issues with individual VMs. |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | stcl01-node01 is back online. Root cause has been tracked back to bug in Ontap. |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewally clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Admin staff is communicating with customers who have opened up cases. Website still down; waiting on developers to determine cause. NPM shows all VMs up. Some customers still reporting issues with their individual VMs. Ontap upgrade scheduled for Sunday morning.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Impaired | Virtual Machines | VMs running on storage controller stcl01-node01 are being restarted. |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Impaired | SAN | stcl01-node01 is back online. Logs being reviewed to determine root cause. |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewally clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Operations VMs have been returned to normal operation. Engineers are using network performance monitor to determine which servers are still down. 80% of the VMs origninally reporting down are now responding to NPM. We are finding that, while VMs are showing up in NPM, they still need to be rebooted to ensure proper operation. Efforts to reboot all affected machines is ongoing.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Impaired | vCloud Director | vCloud Director is rebooting. |
| Impaired | Virtual Machines | VMs running on storage controller stcl01-node01 are being restarted. |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Impaired | SAN | stcl01-node01 is back online. Logs being reviewed to determine root cause. |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewally clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: A team is currently assigned to restoring service to Operations VMs to ensure system stability. A second team is assigned to restoring individual customer VMs.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Impaired | vCloud Director | vCloud Director is offline. |
| Impaired | Virtual Machines | VMs running on storage controller stcl01-node01 are offline or paused. |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Impaired | SAN | Los Angeles storage cluster stcl01 has experienced a kernel panic. |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewally clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: In Los Angeles, at approximately 11:54am PDT, one of our storage clusters has experienced an exceptional failure. stcl01-node01 reports a kernel panic, and the secondary node did not take over as designed. We are working to bring the storage components and affected VMs back online.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewally clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: All customer traffic in Boston has normalized. We will work with JTAC to determine root cause.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewall clusters operating normally |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. STRM shows a single customer VM heavily syn flooding. Shut down network interface and traffic normalizing. I have asked JTAC to explain how this is possible, since we have syn flood screens on this interface.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Impaired | Firewall Clusters | Connectivity issues through Boston firewall cluster |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. JTAC is looking through logs and equipment status.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Impaired | Firewall Clusters | Connectivity issues through Boston firewall cluster |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. The packet loss returned after approximately an hour. Performaing a simultaneous reboot and opening a case with JTAC to get to root cause as quickly as possible.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Impaired | Firewall Clusters | Connectivity issues through Boston firewall cluster |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Extreme packet loss observed in Boston. The issue appears to be with packet flow betweeen the firewall cluster and internal network. Performed cluster failover. The issue appearst to have normalized. We are monitoring.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | vCloud Director | vCloud Director is operating normally. |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Impaired | Firewall Clusters | Connectivity issues through Boston firewall cluster |
| Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Extreme packet loss observed in Boston. Looking into the cause.