Network stability has returned since our last update. We have since added a Megaport link in Sydney and added a secondary carrier to our network via a virtual cross connect for additional capacity and diversity. Our previous configuration had dual paths to a single upstream and due to a mix of issues with the configuration failover not functioning as expected and BGP sessions not falling back correctly some weaknesses with this setup were exposed. A complex hardware fault in our upstreams network meant the backup capacity was not perfectly reliable which unfortunately extended the duration of the outage and network instability.
Now with a secondary carrier connected we will have improved redundancy and resiliency with the network as well as an improved path and capacity into the Sydney network especially for the domestic Telstra network in Australia.
All Australian and New Zealand locations now have full carrier diversity as well as existing peering arrangements.
We apologise to customers affected by the outage and thank you for your patience and ongoing business.
Posted Jan 28, 2020 - 17:43 ACDT
Update
After a technical conversation with our upstream vendor we are confident this evening that the network will remain stable.
We will leave this incident in monitoring status and advise further about steps already in progress to improve resiliency for the POP.
Apologies to all customers affected by todays disruption.
Posted Jan 22, 2020 - 20:03 ACDT
Update
Access has been restored - we are continuing checks and discussing this issue with our upstream provider. Further updates will be posted as soon as possible. As of 18:30 AEST we ask if customers are still experiencing a problem to please submit a traceroute for us to diagnose further.
Posted Jan 22, 2020 - 19:00 ACDT
Update
We are working through this issue but the cause is currently outside of our network with an upstream provider. Network engineers are aware of the problem and fully working on a resolution. Further updates to be posted as more information becomes available.
Posted Jan 22, 2020 - 18:34 ACDT
Update
Unfortunately we are seeing issues for some outside networks reaching our Sydney services again. We are aware of the issue and investigating it now fully. Update to be posted as soon as possible or when we have further information. Apologies for the unreliable connectivity today.
Posted Jan 22, 2020 - 18:15 ACDT
Monitoring
We are seeing full recovery of connectivity and our upstream vendor has reported all issues related to this incident fully resolved.
Posted Jan 22, 2020 - 13:55 ACDT
Identified
We are aware of connectivity issues affecting some domestic and international networks. Our upstream vendor is working on a resolution.
Posted Jan 22, 2020 - 13:30 ACDT
Update
As of 13:00 Sydney time all services are restored.
Due to a BGP failover issue our backup path in Sydney did not promptly come into service today when our primary upstream port had a network issue due to a hardware failure outside of our network. We had to communicate with our upstream partner and manually fall back to another path to restore full connectivity. We are continuing to review the redundancy arrangement in Sydney as this issue was highly disruptive and we feel improved network stability is expected and desired for all our customers. Apologies for the inconvenience caused and we will continue to provide updates to customers impacted.
Posted Jan 22, 2020 - 12:42 ACDT
Update
We are continuing to monitor for any further issues.
Posted Jan 22, 2020 - 12:02 ACDT
Monitoring
Full access has been restored; we are monitoring situation further and rechecking all access ports. We are also awaiting further update from upstream vendor about the cause of the issue. Further updates to be provided as soon as possible.
Posted Jan 22, 2020 - 11:44 ACDT
Update
We are seeing partial recovery to services from many networks and are continuing on working towards a full resolution. Further updates will be provided as soon as possible.
Posted Jan 22, 2020 - 11:37 ACDT
Update
We are working with upstream vendors to resolve this issue; there has been a hardware fault and a backup path provisioning issue which has extended the downtime. We have all resources working on the issue now.
Posted Jan 22, 2020 - 11:36 ACDT
Update
We are continuing to work on a fix for this issue.
Posted Jan 22, 2020 - 10:31 ACDT
Identified
We are aware of connectivity issues affecting Sydney and some Eastern Seaboard services now - at this stage an upstream vendor is also investigating a potential hardware fault. We will provide further updates as information becomes available.
Posted Jan 22, 2020 - 10:22 ACDT
Investigating
We are investigating reports of connectivity issues from outside of Australia to our services in Sydney.
Posted Jan 22, 2020 - 10:13 ACDT
This incident affected: Sydney, Australia (Sydney Dedicated Servers, Sydney VPS Services), Billing & Support System, VPS Control Panel, Melbourne, Australia (Melbourne Dedicated Servers, Melbourne VPS Services), and Brisbane, Australia (Brisbane Dedicated Servers, Brisbane VPS Services).