What Happened
COPA’s hosting provider, Managed.com suffered a Ransomware attack which impacted their entire datacenter, and their satellite data centers starting on the 16th of November. At that time, they took down their entire environment to stop the spread, which resulted in 2/3’s of the COPA website infrastructure being disabled.
The impacted resources were the www.cirruspilots.org website and our identity solution passport.cirruspilots.org, which is used to manage your username & password to access COPA.
Managed.com did have off-site backups, however, they were unable to access these backups due to their network being shut-down.
COPA Initial Reaction
The COPA IT team quickly worked with other vendors, most specifically, Discourse which hosts the forums for COPA and implemented temporary changes allowing direct login to the forms using Email rather than our traditional process. This provided a level of stability allowing the COPA forum to continue to function, albeit without the ability to sign up new members during the outage.
The Recovery
After just over 5 days of outage, Managed.com was able to provide COPA with one of the off-site backups that was archived prior to their environment being attacked. We have since taken those backups, reviewed them for integrity, applied regularly scheduled security patches, and taken additional precautionary measure to further validate the security & integrity of our data.
We have been assured by the investigators with Managed.com that our data was not transmitted outside of their environment. Additionally, all Credit Card & address information is stored only at Chargify our third-party service that manages payments and unaffected byt the ransomware attack.
The Future
COPA was working with Managed.com to provide a turn-key, and hands-off approach to the hosting & IT Infrastructure. This has proven to not be adequate enough, so the displaced infrastructure was migrated to Microsoft Azure and a comprehensive backup & disaster recovery protocol has been implemented.
- Regular website snapshots are managed in a Regionally Redundant Manner by Microsoft going back 30 days
- Additional website snapshots are managed daily by automated processes and moved into a Geographically redundant storage mechanism as a secondary failsafe. (Automatically by Microsoft)
- Application databases have point-in-time restoration available for a period of 35 days, with additional weekly & monthly snapshots preserved in a geo-redundant manner.
In addition to this, the Azure environment is more significantly robust from a security & redundancy standpoint which will further reduce the risk of a similar style incident impacting COPA. We have tested these backup protocols and should the entire data center at Azure be compromised we could restore within a few hours.
We regret any inconvenience this event caused for our members, please rest assured that the privacy and security of our members data is of paramount concern to the COPA IT team, and we will continue to evaluate our IT Infrastructure and adapt as necessary to provide our members with the highest standard of service.
