Site incident 23rd Dec


#1

Service went offline at 11am this morning due to a technical fault on the SQL service.
Our data processing server died at around midnight so new posts were not showing up on feeds and posts also did not get pulled or pushed to external instances.
This morning when we brought the processing server back online, the SQL service (provided by Google) did not want to allow any connections to it.
We have turned this service off and are running a quadcore SQL server to handle the database now.
Currently working through over 2 million backlogged tasks on the data processing servers.
We have brought on an additional 12 CPU cores to work through this backlog.

This topic will be updated often so check back.


#2