System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is is operating normally |
Notes:As evidence that this week’s Distributed Denial of Service (DDos) attack is waning, we at Infinitely Virtual experienced two incidents this morning, Pacific time – the first quite small, the second somewhat larger. No other attacks of note were recorded during the course of business on Friday, and we anticipate that activity will soon abate completely. Our understanding is that every customer is back up.
Anyone encountering a problem should open a new ticket or reply to an existing ticket, using our customer support email (support@infinitelvirtual.com).
We want to stress how much we value your understanding, your patience and your confidence in IV during this period. As we do throughout the course of business in times of success and stress alike, we remain a learning organization, and you can trust that we will continue to up our game to support our customers every step of the way.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | Performing emergency upgrade to stcl03 in Boston. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Impaired | Internet Connectivity | Internet connectivity is being targeted by the DDOS attack |
Notes: We at Infinitely Virtual wanted to take a moment to recap some important takeaways from this week’s DDoS attack.
The good news is that there was never data involvement of any kind. By their nature, Distributed Denial of Service attacks aim at disrupting communications. As of Thursday afternoon, the vast majority of our customers are online and working; the only exceptions are those using PPTP VPN and customers in select foreign countries. DDoS mitigation from Oracle has succeeded in enabling customers to continue to work, even though the attack is continuing. Complementing that effort, IV engineers are following up with every customer to verify that each is indeed up and running.
By any measure, this qualifies as a substantial attack. The organization that attacked us used a multitude of attack vectors to bring us down. We responded to each successfully until the attacker discovered a vector that worked; Oracle subsequently found an appropriate countermeasure. At the height of the attack, the organization sent 200 gb/sec of attack traffic – an assault from 135,000 IP addresses, sustained for roughly an hour.
We will continue to update you as events warrant. Thanks again for choosing Infinitely Virtual and for your continued support.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | Performing emergency upgrade to stcl03 in Boston. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Impaired | Internet Connectivity | Internet connectivity is being targeted by the DDOS attack |
Notes: As the Nov. 12 DDoS attack continues into the 13th, we at Infinitely Virtual are working closely with Oracle to limit its impact on our customers and arrest further escalation. Because we have taken aggressive measures to mitigate the attack, connectivity has been affected for some customers who appear to the mitigation system to be part of the attack. We can now whitelist IP addresses to circumvent this situation. If you would like your IP addresses to be whitelisted, please go to WhatsMyIP.com and email those addresses to support@infinitelyvirtual.com.
We want to reiterate that this was not a ransomware attack. At no time was any customer data at risk and our system was not breached. The attack was limited to disrupting connectivity.
Once again, thank you again for choosing Infinitely Virtual and for your continued support.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | SAN operating normally |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Impaired | Internet Connectivity | Internet connectivity is being targeted by the DDOS attack |
Notes: In the interest of full transparency, we at Infinitely Virtual wanted to let you know that today, Nov. 12, we experienced a Distributed Denial of Service attack (DDoS) – an attack that has continued into the evening. Connectivity was quickly restored after the initial attack, followed by an additional outage that lasted three hours. Significantly, there has been no data breach and no data loss.
At this hour, our DDoS provider, Oracle/DYN, is attempting to mitigate. We will keep you apprised of additional developments, as appropriate.
Thank you again for choosing Infinitely Virtual and for your continued support.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | Performing emergency upgrade to stcl03 in Boston. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewall clusters operating normally |
Impaired | Internet Connectivity | Internet connectivity is being targeted by the DDOS attack |
Notes: Beginning at 9am Pacific Time, our engineers observed a volumetric DDOS attack coming from 1000s of IP address in many countries. These attacks started by targeting single a single IP address in Los Angeles. As our DDOS provider mitigated the attack, the number of IP address in both LA and Boston increased. Early disruptions were small, however we have had several outages lasting more than an hour. Our team is working hand—in-hand with the our DDOS vendor to do the best we can to weather the attack. We have received a ransom request.
We will try to keep this blog updated, however these outages do affect our website and this blog. Additionally, our communications firm will send out emails with information about the attack. We appreciate your patience as we work through this difficulty.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | All SAN components operating normally. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewally clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Non-disruptive upgrade to stcl01 and stcl03 is complete.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Impaired | SAN | Performing emergency upgrade to stcl03 in Boston. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewally clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Non-disruptive upgrade to stcl01 is complete. Beginning upgrade on stcl03 in Boston.
System Status: Operational with Exceptions
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Impaired | SAN | Performing emergency upgrade to stcl01 in Los Angeles. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewally clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Beginning Ontap upgrade on stcl01.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | Emergency upgrade scheduled for stcl01 and stcl03. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewally clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Website normalized. Ontap upgrade scheduled for Sunday morning.
System Status: Operational
Status | System | Notes |
Normal | vCenter | vCenter is operating normally |
Normal | vCloud Director | vCloud Director is operating normally. |
Normal | Virtual Machines | All VMs are operating normally. |
Normal | VMware Hosts | All hosts are operating normally |
Normal | VMware Clusters | All clusters operating normally |
Normal | SAN | Emergency upgrade scheduled for stcl01 and stcl03. |
Normal | L3 Switches | All switches operating normally |
Normal | Firewall Clusters | All firewally clusters operating normally |
Normal | Internet Connectivity | Internet connectivity is operating normally |
Notes: Operations and customer systems have been normalized. Website still down; waiting on developers to determine cause. Ontap upgrade scheduled for Sunday morning.