Search Indexes Degraded over Weekend
Incident Report for Pixlee
Resolved
The morning of Monday June 10, we received reports that content search within widgets and Pixlee dashboard in certain situations would not yield correct results - usually, it would claim that no search results were present when it in fact was there just fine.

During some search index maintenance operations the weekend, we ran a routine, temporary operation to un-alias our production elasticsearch index to replace it with a new mapping. An unseen error caused the index not to re-register, which meant that search results over the weekend were coming back empty.

We have immediately rectified this issue as of Mon Jun 10 9:34AM PST. All search results are restored.

At no time was any content lost, deleted or miscollected. Rather the search results were turning up incorrect.

We have added alerting to prevent this possibility in the future and are working to communicate with individual affected stakeholders.

Apologies for the inconvenience.
Posted Jun 10, 2019 - 09:50 PDT