System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally. |
Normal | Internet Connectivity | Internet connectivity operating normally. |
Notes:Planned upgrade of firewall cluster has been completed successfully.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally. |
Normal | Internet Connectivity | Internet connectivity operating normally. |
Notes:Firewall cluster in Los Angeles has been upgraded successfully. Post upgrade procedures still being run. Will update again shortly.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Impaired | Firewall Clusters | Firewall cluster in Los Angeles down due to planned upgrade. |
Impaired | Internet Connectivity | Internet connectivity in Los Angeles down due to planned upgrade. |
Notes:Beginning planned firewall upgrade in Los Angeles. Connectivity will be disrupted no more than 1 hour. All VMs will remain in their current state for the duration of the upgrade.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes:Upon review, JTAC recommends upgrading to a newer version of JUNOS. We will be sending out upgrade notifications shortly.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes:Juniper JTAC: We are working on the logs. The primary analysis is that the flowd on node0 has terminated and hence the issue has arised. Ideally flowd crash also should trigger the failure however it did not happen. We are trying to find out why the flowd was crashed in first place.
We shall keep you updated.
Aug 12 12:26:25 firewall03 init: remote-operations (PID 89801) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: class-of-service (PID 89802) terminate signal 15 sent
Aug 12 12:26:25 firewall03 ppmd[51290]: PPMD_READ_ERROR: Read error on pipe from RPM: premature EOF (connection closed)
Aug 12 12:26:25 firewall03 init: jflow-service (PID 89803) terminate signal 15 sent
Aug 12 12:26:25 firewall03 jflowd[89803]: LIBJUNOS_SDK_RECEIVED_SIGTERM: Shutting down on SIGTERM.
Aug 12 12:26:25 firewall03 init: sampling-route-record (PID 89804) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: dhcp (PID 89805) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: ppp (PID 89817) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: l2cpd-service (PID 89831) terminate signal 15 sent
Aug 12 12:26:25 firewall03 l2cpd[89831]: JTASK_SIGNAL_TERMINATE: first termination signal received
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes:Failed back to primary node on firewall03. Cluster operating normally. Since we are still waiting on Root Cause Analysis (RCA), we failed back to node 1 manually.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Impaired | Firewall Clusters | Firewall03 has been manually failed over to secondary |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes: Manually failed over to secondary firewall. Working with Juniper to determine source of failure, both in the primary firewall and in the cluster failover.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Impaired | Firewall Clusters | Firewall03 in Los Angeles is not reachable |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes: Firewall cluster in Los Angeles is not operating as expected. Primary firewall is not passing traffic, and secondary has not taken over.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Normal | Internet Connectivity | Internet connectivity operating normally |
Notes: Our DDOS provider’s peerings are back up. Their Los Angeles pop went down, and the Dallas peerings did not come up as expected. They are investigating further.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN clusters operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Impaired | Internet Connectivity | BGP peerings with DDOS provider are down |