System Status: Operational

Notes:Planned upgrade of firewall cluster has been completed successfully.

System Status: Operational

Notes:Firewall cluster in Los Angeles has been upgraded successfully. Post upgrade procedures still being run. Will update again shortly.

System Status: Operational with Exceptions

Notes:Beginning planned firewall upgrade in Los Angeles. Connectivity will be disrupted no more than 1 hour. All VMs will remain in their current state for the duration of the upgrade.

System Status: Operational

Notes:Upon review, JTAC recommends upgrading to a newer version of JUNOS. We will be sending out upgrade notifications shortly.

System Status: Operational

Notes:Juniper JTAC: We are working on the logs. The primary analysis is that the flowd on node0 has terminated and hence the issue has arised. Ideally flowd crash also should trigger the failure however it did not happen. We are trying to find out why the flowd was crashed in first place.

We shall keep you updated.

Aug 12 12:26:25 firewall03 init: remote-operations (PID 89801) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: class-of-service (PID 89802) terminate signal 15 sent
Aug 12 12:26:25 firewall03 ppmd[51290]: PPMD_READ_ERROR: Read error on pipe from RPM: premature EOF (connection closed)
Aug 12 12:26:25 firewall03 init: jflow-service (PID 89803) terminate signal 15 sent
Aug 12 12:26:25 firewall03 jflowd[89803]: LIBJUNOS_SDK_RECEIVED_SIGTERM: Shutting down on SIGTERM.
Aug 12 12:26:25 firewall03 init: sampling-route-record (PID 89804) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: dhcp (PID 89805) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: ppp (PID 89817) terminate signal 15 sent
Aug 12 12:26:25 firewall03 init: l2cpd-service (PID 89831) terminate signal 15 sent
Aug 12 12:26:25 firewall03 l2cpd[89831]: JTASK_SIGNAL_TERMINATE: first termination signal received

System Status: Operational

Notes:Failed back to primary node on firewall03. Cluster operating normally. Since we are still waiting on Root Cause Analysis (RCA), we failed back to node 1 manually.

System Status: Operational with Exceptions

Notes: Manually failed over to secondary firewall. Working with Juniper to determine source of failure, both in the primary firewall and in the cluster failover.

System Status: Operational with Exceptions

Notes: Firewall cluster in Los Angeles is not operating as expected. Primary firewall is not passing traffic, and secondary has not taken over.

System Status: Operational

Notes: Our DDOS provider’s peerings are back up. Their Los Angeles pop went down, and the Dallas peerings did not come up as expected. They are investigating further.

System Status: Operational with Exceptions