← Go back to SeismoCloud Status

Major outage on all systems

July 1, 2020 at 1:34 AM

Website API MQTT broker Controller Statistics controller My dashboard Database Database replica 2 Legacy HTTP backend

Resolved after 10h 58m of downtime. July 1, 2020 at 12:32 PM

Full report

Around 2020-07-01 01:31:00 UTC the server colocation losts the AC power. UPS batteries were in place, however the AC power wasn’t restored in time, and by 2020-07-01 01:34:17 UTC all systems were powered off by the UPS monitor software.

At 9 a.m. the power was partially restored, however the IT was working for fixing the network connection (which apparently had a fault at the same time during the night). Around 2020-07-01 12:00:00 UTC the network connection was restored, and the rest of the building was powered on, including SeismoCloud servers. At 2020-07-01 12:32:36 all services were back on.

Future improvements

In last 6 months we already planned to move critical parts of the SeismoCloud network ouside for redundancy. We’re working on it.