Outage on Wednesday 14th June at 2pm until 5pm
As our customers are aware, Dreamtilt suffered a major outage from just after 2pm on Wednesday. This outage affected all customers and services.
We responded within 20mins of being aware of the issue and determined that the problem was a rebooting main core router that connects to the fiber that provides internet access to our network.
Unfortunately this router was faulty and could not be repaired and after several attempts to try and restore the router, we then determined to use a backup router that we had at the same location.
Unfortunately there were some configuration issues with doing this and it took a while to get the backup router back online, rebooted and working.
Even after it was working, there was also an issue with the capacity of this router to handle the connections and routing required – which caused some of the network to be online for some customers, whilst others still suffered no internet connection.
In the end we then quickly set up some virtual routers to handle the main internet connection and routing and configured the backup router (that was still connecting customers) to run through the virtual routers.
This restored services shortly around 5pm.
We realise that any outage of any duration is an impact, and as such, we will be reviewing the incident to determine how we can mitigate such issues happening again. We do our best to provide a great service and any outage we try to keep to a minimum and restore as quickly as possible- but we really like to have none at all.
We are investing in a new fiber connection and routing network in the next few months that will mitigate the above issues, and we will look at ways to make the main routers more reliable and robust. There is more work we are doing in the background to also make sure that we keep our services as reliable as possible for our customers.
We do apologise again for the outage, and we are working on making it better and more reliable.