System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Impaired | Firewall Clusters | Single firewall cluster is showing some instability |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: Seeing some instability with one firewall cluster. We are looking into it now.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: vcenter.infinitelyvirtual.com is back up, and the upgrade is complete.
System Status: Operational with Exceptions
| Status | System | Notes |
| Down | vCenter | vCenter is down for maintenance |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: We are currently upgrading vcenter.infinitelyvirtual.com to vCenter 5.0. The server will be down for several hours.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: At roughly 3pm Pacific time, our upstream provider finished their move to a different router, and after testing, we activated our peering session. So far we have not detected any packet loss. We are currently watching this carefully.
System Status: Operational with Exceptions
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Impaired | Internet Connectivity | Internet connectivity is normal. One peering is shut down. |
Notes: Our upstream has determined the cause of the problem, and they are preparing to move us to a new router. In the meantime, we have shut down our peering with them.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: At 9:20 am Pacific time we lost our bgp peering with one of our upstream ISPs. This caused an outage for customers who were being routed by way of that ISP for between 1 and 4 minutes. The peering came back up. We are working the the ISP to determine the cause.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: At 1:16pm Pacific time we lost our bgp peering with one of our upstream ISPs. This caused an outage for customers who were being routed by way of that ISP for between 1 and 4 minutes. The peering came back up. We are working the the ISP to determine the cause.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: It was determined that the cause of the outages on 8/25, 8/29, and 9/31 was due to the anti-virus UTM on our Juniper firewalls. While there was no indication in any logs that this was causing the issue, we prevented recurrence by disabling the anti-virus UTM. Juniper recommended a newer level of Junos, and therefore we performed an upgrade on 9/3 and enabled the anti-virus UTM. We are still monitoring the firewalls for any indication that the issue will return now that the anti-virus UTM is enabled.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: Traffic via the firewall03/04 cluster has normalized.
System Status: Operational
| Status | System | Notes |
| Normal | vCenter | vCenter is operating normally |
| Normal | Virtual Machines | All VMs are operating normally |
| Normal | VMware Hosts | All hosts are operating normally |
| Normal | VMware Clusters | All clusters operating normally |
| Normal | SAN | All SAN components operating normally |
| Normal | L3 Switches | All switches operating normally |
| Normal | Firewall Clusters | All firewalls operating normally |
| Normal | Internet Connectivity | Internet connectivity is normal |
Notes: Firewalls have finished rebooting. We are currently monitoring to determine if all traffic is normalized.