Sydney connectivity issues
Incident Report for Ransom IT
Postmortem

This incident was caused by a faulty fibre optic patch/transceiver in the core of our SY3 network. The faulty path has been shutdown on both sides and we will review all other internal patches to ensure no errors are going to cause problems.

We will also perform maintenance soon to better isolate portions of our network to try and reduce the severity of physical network issues where possible. It is expected this maintenance can be completed without disruption but if any brief planned outages are required we will notify customers with as much notice as possible.

Apologies for the disruption caused; and thank you to customers for your patience during this incident.

Posted Sep 21, 2018 - 22:44 ACST

Resolved
This incident has been resolved.
Posted Sep 21, 2018 - 17:40 ACST
Monitoring
We have moved a physical fibre patch to a backup link between some devices on our SY3 network. We expect this to resolve the problem and are monitoring the network carefully to ensure full stability. If customers are still seeing packet loss issues after 3:20PM Sydney time please update your existing ticket with us about this to investigate it further.

Apologies for the significant disruption to service stability today.
Posted Sep 21, 2018 - 14:49 ACST
Update
We are working on getting a fibre patch between switches replaced as some physical port issues appear to be disrupting network stability; please stand by for further updates.
Posted Sep 21, 2018 - 14:20 ACST
Update
We are continuing investigation as it appears some packet loss has returned to network. Apologies for inconvenience; please be assured we are attempting to resolve this issue as high priority.
Posted Sep 21, 2018 - 14:16 ACST
Update
We have been working through some network devices systematically to find the cause and have isolated a switch load issue; this is being addressed now and connectivity should be improving. We are monitoring tickets submitted in the last 20 minutes and replying to confirm access for customers concurrently.
Posted Sep 21, 2018 - 13:49 ACST
Investigating
We are investigating reports of intermittent connectivity issues/packet loss to some Sydney services. If you are experiencing a problem please submit a support ticket with bi-directional traceroutes as this issue appears isolated in nature.
Posted Sep 21, 2018 - 13:21 ACST
This incident affected: Sydney, Australia (Sydney Dedicated Servers, Sydney VPS Services).