Update - After a technical conversation with our upstream vendor we are confident this evening that the network will remain stable.
We will leave this incident in monitoring status and advise further about steps already in progress to improve resiliency for the POP.
Apologies to all customers affected by todays disruption.
Jan 22, 20:03 ACDT
Update - Access has been restored - we are continuing checks and discussing this issue with our upstream provider. Further updates will be posted as soon as possible. As of 18:30 AEST we ask if customers are still experiencing a problem to please submit a traceroute for us to diagnose further.
Jan 22, 19:00 ACDT
Update - We are working through this issue but the cause is currently outside of our network with an upstream provider. Network engineers are aware of the problem and fully working on a resolution. Further updates to be posted as more information becomes available.
Jan 22, 18:34 ACDT
Update - Unfortunately we are seeing issues for some outside networks reaching our Sydney services again. We are aware of the issue and investigating it now fully. Update to be posted as soon as possible or when we have further information. Apologies for the unreliable connectivity today.
Jan 22, 18:15 ACDT
Monitoring - We are seeing full recovery of connectivity and our upstream vendor has reported all issues related to this incident fully resolved.
Jan 22, 13:55 ACDT
Identified - We are aware of connectivity issues affecting some domestic and international networks. Our upstream vendor is working on a resolution.
Jan 22, 13:30 ACDT
Update - As of 13:00 Sydney time all services are restored.
Due to a BGP failover issue our backup path in Sydney did not promptly come into service today when our primary upstream port had a network issue due to a hardware failure outside of our network. We had to communicate with our upstream partner and manually fall back to another path to restore full connectivity. We are continuing to review the redundancy arrangement in Sydney as this issue was highly disruptive and we feel improved network stability is expected and desired for all our customers. Apologies for the inconvenience caused and we will continue to provide updates to customers impacted.
Jan 22, 12:42 ACDT
Update - We are continuing to monitor for any further issues.
Jan 22, 12:02 ACDT
Monitoring - Full access has been restored; we are monitoring situation further and rechecking all access ports. We are also awaiting further update from upstream vendor about the cause of the issue. Further updates to be provided as soon as possible.
Jan 22, 11:44 ACDT
Update - We are seeing partial recovery to services from many networks and are continuing on working towards a full resolution. Further updates will be provided as soon as possible.
Jan 22, 11:37 ACDT
Update - We are working with upstream vendors to resolve this issue; there has been a hardware fault and a backup path provisioning issue which has extended the downtime. We have all resources working on the issue now.
Jan 22, 11:36 ACDT
Update - We are continuing to work on a fix for this issue.
Jan 22, 10:31 ACDT
Identified - We are aware of connectivity issues affecting Sydney and some Eastern Seaboard services now - at this stage an upstream vendor is also investigating a potential hardware fault. We will provide further updates as information becomes available.
Jan 22, 10:22 ACDT
Investigating - We are investigating reports of connectivity issues from outside of Australia to our services in Sydney.
Jan 22, 10:13 ACDT