Server issue on Amazon Cloud on Jan 4th made WikiMaster go down
WikiMaster app was down the first week of January due to a Server issue on Amazon Clouds out of our control. We are now moving forward with lots of learning made by this 60Dzone incident.
During the investigation so far we have concluded: Out of the blue Amazon Clouds initiated a Server update and wiped out the running database we were running. We were completely shocked by the incident. In a combination of the fact that our backup service had some flaws, we faced an obstacle huge enough for evaporating 2 months of data. We apologize to the Community for the inconvenience caused by this issue. The huge amount of learning experience has been made and routines changed.
The incident of Jan 4-6 is called 60D-zone
What we now know happened in bullets:
- On Jan 4 at 10.00 am GMT something made WOK Server go down. WikiMaster did not work. Our team worked for 50 hours to restore and get the Server up and managed to go live after 50 hours on Jan 6, 1 pm GMT.
- When we returned to the state of WOK and WikiMaster running again, we had reinstalled a backup version from Nov 2. All other backup versions after were lost with the last version erased.
- The cause of the evaporation of WOK was investigated by Amazon Cloud Service (AWS) and WOK team during and after we came back. The conclusion of this investigation is that both parties don't exactly know what caused the damage done. We have theories. But its guesses.
- The Jan 4 version of files from WOK is now missing like the MH370 plane gone missing in the Indian Ocean in 2015.
The implications of 60Dzone
- 32000 questions created, edited and indexed were imported during November and December: These questions are in files for re-import. The Qs are not lost forever but need to be reinstalled. We will do this over a time period of 14 days reenter the 400K MCQ we had when 60Dzone happened.
- Compensation for WOKbits will be made for existing WOKers losing the WOKbits during November and December. We will address this loss for the active community members accordingly to earlier efforts. (like many other examples from other game provider prior to this)
- The image added by the community was around 13000 during the last two months of December is lost. Countless granular tagging. We are urging for Community help to illustrate the Questions with appropriate images from Wikimedia Commons to help us move forward and repair quicker.
- No code for App Development was affected. The version sent to AppStore and Google Play are up to date so bugs fixed and API issues (speed) are not going back in time and need to be redone= The app's functionality is same as it was Jan 4 and improved since then-> "Only" the content of questions, quizzes, results, wokbits, chat, and accounts in the database of WOK were hit. Bad enough, but could be worse.
- Our Backup routines are changed forever. The combination of three (3) parties not following normal procedure made 60Dzone worse. We are making sure the implications in the future will be held to a minimum.
Again: We apologize to the Community for the inconvenience caused by this issue. We feel overrun by a train. We feel stupid. We feel angry. We are not down and out, but bruced. We now raise and move forward to make the best of what can be done. This is now part our journey. It's nothing we can just swipe left. We will deal with this best of ability. We like to be transparent. We will keep you updated. We will keep up. We will repair. We apologize for the backup file failure. You fall. You learn. You raise. Back on the Saddle. Thanks for being part of our Journey. We love you all.
Read more about Server related issues affecting WOK here