Dear Valued Customer,

Yesterday (3rd of August 2011) at approximately 02h00 CAT, Absol Internet Business Solutions experienced a failure on the core backbone fibre link to our primary data centre and hosting facility in Centurion. Our network engineers began troubleshooting the failure immediately and escalated the problem with our data centre technical team.  This outage affected all services including: Email, Hosting and DNS services.

While Absol have invested heavily and maintain backup facilities, failover hardware and have implemented fault-tolerant virtualised systems, and the facility housing our equipment is classed as a Tier-4 (highest calibre, enterprise) data centre with a redundant fibre ring connection, and dual redundant 1870KW generator failover systems, the fault was severe and a significant outage was experienced. 

Notification board:

02:15 CAT-  We received confirmation from DFA (Dark Fibre Africa) that there is a fibre break on the fibre route through the Midrand area. Cause of breakage is  due to vandalism. DFA speculates that the problem should be resolved in the next three hours. We will update on any progress.

08:52 CAT - DFA have repaired the fibres at the break site which has NOT restored services at our Centurion node. DFA technicians are now tracing the fault to additional break sites.

12:27 CAT -  You are advised that the cable break is extremely complex, as it involves 12 cables. We have dispatched all our maintenance teams to assist with the task of restoring the affected links. The course of action currently, is to trace out each affected link and to splice it through.
 
14:10 CAT - The tracing process is time consuming and we expect to have all the links restored by 6pm. A descriptive incident report will be compiled detailing the event, once all links have been restored. We apologise for the inconvenience caused.

16:49 CAT -  Unfortunately due to the extent of this problem they are also going around to each customer site to do individual tests on their circuits. Technicians are now being dispatched to our Midrand node for further investigation. I am awaiting an ETR from engineers regarding this matter. We do apologise for any inconvenience caused.

18:15 CAT -  All connectivity restored and tested as up and running.

The cause of the vandalism is being investigated and there is speculation that, since as many as 12 individual break points were found, the cause might have been more malicious in nature. Several major network providers and data centres are connected via this fibre ring and were affected by this outage, this includes Cell C who had total voice and data failure over their networks in this area and the African Internet Exchange (AfricaINX).

This is the most significant outage Absol has ever experienced in over 15 years of business and we humbly apologise for the inconvenience caused. We extend every assurance that we will continue to investigate and pursue a more robust and fault tolerant solution to offer our clients and will carefully review and scrutinize the incident report from DFA with our data centre team.

Please feel free to contact Mac Anamourlis or Alex Shiells directly should you have any further questions or concerns.

Regards,
Absol Technical.