The search index is finally complete and we've upgraded it to be far more robust moving forward. Thank you everyone for your patience.
Posted Dec 12, 2019 - 14:56 PST
Update
Recovery has been going smoothly. We estimate another 24 hours before completion.
Posted Dec 09, 2019 - 09:48 PST
Update
We've had to do another reset of the indexing due to sharding issues but optimized it dramatically for faster recovery and querying. Will be another 1-2 days but should be a better search experience afterward. Thank you for your patience.
Posted Dec 08, 2019 - 10:38 PST
Update
We've run into some memory limitations and reconfigured the cluster to scale better. We expect this job to run into the weekend but will be actively monitoring.
Posted Dec 06, 2019 - 08:41 PST
Update
Recovery is going smoothly and we estimate approximately 24-48 more hours until completion. Upon completion we will check the data for integrity and ensure that nightly analytics jobs are rerun during the time of data recovery. No data will be lost at the end of this process.
Posted Dec 05, 2019 - 09:07 PST
Monitoring
We have brought the search index back up and are rebuilding it from scratch. You may notice analytics counts not correct in the meantime, we will do everything in our power to accelerate the data restoration process which may take several days. Please use this page for updates.
Posted Dec 05, 2019 - 02:37 PST
Identified
Our analytics search index needed a rolling restart at approximately Dec 5 12:23AM EST. In the meantime, certain search queries querying analytics will fail. We are monitoring the situation and will update when the index is back.