Well, crap.
First of all, thank you everyone for putting up with the partial outage last week, especially during the last weekend. Everything is coming back to normal now.
This is a post to sum up what’s been happening.
Hosting Breakdown
One week ago on Friday, our hosting provider (a large one), who I shall not name, got an infrastructure breakdown in the EU region, and is pretty much done for.
This resulted in partial damage to our databases behind AudioNodes, as well as complete undoable damage to the website, because it also affected weekly backups we stored there.
Dear customer, we had an huge outage in our EU datacenter last night. As the result, few of our database servers were badly damaged along with backup data of those servers. We deeply apologize for this incident. We never had such problem before for the past 20 years and we are now in the process of making changes to prevent this from ever happening again.
Canned response from hosting tech support
New Servers & Website
Because we cannot trust this provider to handle your data anymore, AudioNodes moved to a new, much faster, and more advanced server infrastructure. We are now at DigitalOcean, which has an outstanding reputation when it comes to data safety, performance, and uptime. This is the standard we want to stick to.
In addition to the new servers, we rebuilt the website from scratch (we had to, as the previous site is gone). This probably doesn’t change much when it comes to AudioNodes itself though. Hope you’ll like the new look, although it’s still a little raw probably. The new site is still a work in progress.
Safety of Your Data
Registered users only
When the breakdown began, we’ve taken immediate steps to ensure no emails or other user data leaked. So in short, you won’t get spammed because of us. On a similar note: for passwords, which weren’t leaked either, we use industry-standard hashing to ensure they would be useless even if they were leaked.
Luckily, we were still actively working on AudioNodes when the breakdown occurred on Friday. As a result, we noticed the downtime literally seconds after stuff hit the fan. Unfortunately though, the actual damage already happened by then.
What’s Gone
We did our best to contain damage, and it should not affect your regular use of AudioNodes. Affected stuff as of now:
- Most account passwords were reset – please use the forgot password link to restore access to your account
- Lifetime AudioNodes HD subscriptions have been disrupted – if you donated earlier and got a lifetime unlock key, please redeem the key again and everything should be back to normal – if you lost your key, please drop us a message to support at audionodes dot com and we’ll recover it for you, the keys were not lost and continue to remain valid
- There is a good chance some inactive user data was wiped permanently, but the extent of this damage should be minimal (presumably less than a dozen, mostly inactive accounts) – please let us know via the above address if you are affected and we’ll fix it for you
- Blog posts are gone – while we could theoretically restore these from web archive services, this is not something we’ll spend time doing; instead, we are focusing on the next major update of AudioNodes
Docs are not currently loading from within AudioNodes – temporary, working on it- Update: fixed
Most links from within AudioNodes pointing to pages on the website are not working – temporary, working on it- Update: fixed
Closing Words
Again, thanks for putting up with the disruption last week. While this was a serious breakdown, it shouldn’t affect things going forward. Luckily, there was no serious damage to your data.
We want to emphasis that the core AudioNodes app (both web and desktop versions) was not affected by the slightest, and we are working on the next release as before.
Thanks everyone, please stay tuned for the next update.