Details of the UltraSeedbox outage that began on June 20, 2020
As most of you are aware, bulk of UltraSeedbox’s infrastructure was summarily taken offline on June 20, 2020 due to a situation for which very little information has been publicly disseminated.
We understand how frustrating it must have been to suddenly find your servers offline, and more so, not knowing the reason behind it. To that end, we would first like to sincerely apologize, allay your concerns, address unfounded speculations and rumors that have been making rounds.
Over the years, we have worked with multiple independent security researchers who have advised us of bugs and potential exploits within our systems, and we’ve privately paid out bounties to compensate for their time and the effort that they put in to discover these issues.
On June 19th, one such individual came to us with an exploit that allowed for a standard seedbox user to gain escalated system privileges. This means, theoretically, an existing seedbox user could gain sudo privileges and effectively perform any action on certain servers within our infrastructure — excluding offsite services that communicate via secured APIs, such as WHMCS (which houses client data such as contact information, invoices, support tickets, etc). This was obviously a considerable security concern that needed to be proactively addressed.
We were advised that shutting down every seedbox node would be the first crucial step towards limiting the attack surface, while we systematically patch the vulnerability across our infrastructure. While we could have easily chosen to be silent about this issue, while slowly rolling out patches one server at a time, we felt it would be extremely irresponsible to leave our systems vulnerable, and a disservice to the valued members of our community who trusted us over the years to be able to provide stable and reliable service. In the interest of your security, we made the extremely difficult decision to shut the seedbox nodes down and address this newfound vulnerability without any further delay.
Our failure through this ordeal was not properly informing and communicating with you.
From the moment we decided to take the bulk of our infrastructure offline, we should have maintained proper communication with you. This, unfortunately, did not happen, and we were unable to get the emails sent out in a timely manner. To make things worse, the initial notice posted on our Discord server included an inaccurate timeline. For this, we are extremely sorry.
To make matters worse, while we were busy patching bulk of our infrastructure, we were targeted with a distributed denial-of-service (DDoS) attack that took down our main website and control panels. Frustratingly enough, we had DDoS protection provided by Leaseweb, which was unable to properly filter the traffic - resulting in the website and WHMCS portals becoming unavailable for a while.
We can’t entirely blame Leaseweb for this, as the DDoS was performed sporadically, across multiple layers, with packets of data being sent via thousands of bots at low speeds and connections, effectively disguising bulk of it as legitimate traffic.
However, within a few hours, we were able to move our infrastructure to a different provider and managed to get things under control, with the help of Cloudflare.
WHMCS Defamatory Accusation
While our services were offline, in addition to the DDoS attacks a nasty rumour was circulated around social media, spreading false information about our servers getting "hacked" and our clients personal data being compromised.
This is completely false. There is not an iota of truth to these unsubstantiated statements devoid of proof. This malicious misinformation campaign was probably designed to take advantage of our situation and try to damage our reputation.
We would like to reiterate that our seedbox infrastructure and website / client data are on completely different hosts, and that you may rest assured knowing that we have NOT had any sort of breach, and client data has NOT been compromised.
To those who carried out these malicious acts; we forgive you.
For what it's worth, we have come out stronger, having used this as an opportunity to reflect and suitably enforce certain changes to address our shortcomings with regards to the way we communicate with you.
As we slowly return to normalcy, we are faced with the monumental task of moving forward from this whole event. Our initial plan is to compensate you for the downtime and the inconvenience caused. At the same time, we’ve begun working on a top-down audit of our app management systems, while subsequently laying groundwork to rebuild sections of our infrastructure.
Additionally, we’ll be rolling out:
- comprehensive standard operating procedures for mass client communications
- implementing a data handling policy
- establishing an official service-level agreement (SLA)
- publicizing our bug bounty program
- and a few other policies and ideas to provide to you some peace of mind
Regarding this particular situation, we have rolled out 15 day credit to every active service prior to June 20, 2020. This was accomplished by pushing the due date of your previously upcoming invoice by 15 days, effectively providing 15 days of free service.
In addition, we have awarded all active services prior to June 20, 2020 a one-time use traffic reset token that you can execute in order to reset your month’s traffic usage at any time of your choosing. We felt this was the best way to execute this award and it makes it fair for every client to reset your traffic when needed. This will not expire until used or the service expires.
As of right now the reset must be executed from SSH. The following article will help you should you not know how to access your services shell access method then please CLICK HERE
The command to execute is app-traffic reset but please note that this is a one-time use token so use it wisely. app-traffic info will display information post reset.
Not tech savvy or want help? No problem. Drop support a ticket when you want to use your token and a member of support will apply it for you.
We have already begun a comprehensive audit of our app management systems, doing our best to ensure there are no additional vulnerabilities of this nature and scale. For the immediate future, no new apps will be added to our one-click installable apps until the audit is completed.
Establishment of Robust Standard Operating Procedures (SOP) Regarding Mass Client Communications
This whole situation has definitely shown us that we were, quite frankly, unprepared to handle and exigency of this nature. We will be revising and rebuilding many of our SOPs regarding client communications while adding comprehensive and useful notices that can be effectively distributed through our website and control panels, as well as via email and social media channels.
Data Retention Policy
We will be introducing a transparent policy detailing the information we collect, where we keep it and why. We are currently remapping our Policies structure, this is one of the policies that is a work in progress.
Official Service Level Agreement (SLA)
This policy will serve to detail what clients can expect when there is downtime and when such downtime may occur.
Public Bug Bounty Program
We will be publishing our previously private bug bounty program which will detail the types of bugs we are looking for, the financial reward for such findings and the agree ments we have in place to protect security researchers acting in the best interest of UltraSeedbox.
At the end of the day, our major failure was with client communication. The actions we took to shut everything down and implement a patch for the discovered vulnerability were all done in the interest of keeping our clients’ data safe and to keep our systems secure. While we feel we acted swiftly and effectively in this area, we neglected to extend these qualities when it came down to effectively communicating with you.
Our goal, moving forward, is to do our best to ensure such a lapse in customer support does not reoccur, and we hope to rebuild the trust that was affected by the manner in which we handled this situation.
If you’d allow us, we like to offer our sincere apologies for how this played out, and would very much like to hear how else you feel we might continue to improve our relationship moving forward.
Please feel free to send us any thoughts/suggestions/concerns you may have to [email protected]
Thank you very much for your time.
Saturday, July 4, 2020