Overview

Servers Australia Engineers received reports at 6:18 AM that isolated subnets were unreachable from specific upstream providers and commenced investigations.

It was identified that a small number of prefixes were experiencing return path reachability issues. Due to the complex nature of identifying affected external paths into the Servers Australia network, as well as the limited impact, remediation works were planned and conducted with the utmost care to ensure operational paths were not affected and the issue did not become more widespread.

After identifying a temporary workaround as well as a permanent solution, a team of engineers were assigned to work on implementing temporary fixes to bring services back online ASAP, whilst another team were working on the more permanent fix, to restore reachability to affected services.

At 3:45 PM, Engineers completed restorative works and implementation of a permanent fix for the unreachability issue and confirmed that impacted prefixes were no longer seeing any return path issues.

Timeline

06:18 - Initial reports received regarding routing issues for specific subnets
07:00 - External checks performed that returned normal results confirming external reachability from available endpoints - Investigations continued into specific affected prefixes.
08:00 - SAU engineers reverted some changes that were made in the earlier scheduled maintenance window, initially suspected to be contributing to the reachability issues.
08:00 - 08:45 - While monitoring and testing showed some improvement to reachability, it was present on select prefixes this was a false positive result and works were completed to re-instate the earlier planned changes.
08:45 - Reverted changes determined not to be a contributing factor and are re-instated.
08:45 - 12:15 - Engineers ran environment simulations with upstream providers to help identify and eliminate potential contributing factors.
12:15 - Prefix from peering upstreams dropped for testing
12:50 - Prefix re-advertised after testing complete to peering upstreams
13.20 - Engineers identify potential temporary interim fixes to bypass reachability issues for known problem prefixes.
13:50 - Temporary fixes started to be deployed
15:15 - Permanent fixes began to be deployed
15:45 - Permanent fixes completed deployment
15:50 - Services restored to regular operation after permanent fix applied

Resolution

Throughout the incident, while engineers were investigating a fix, a temporary measure was discovered and implemented to restore reachability to the impacted prefixes.

The root cause of the unreachability was confirmed to be a communication issue between 2 iBGP speakers, within the Sydney routing environment. As this was isolated to specific prefixes, and due to all other iBGP learnt paths remaining operational, identifying this as the root cause was a time-consuming process. The permanent resolution was to bring forward planned works to remove iBGP from the impacted infrastructure, as previous maintenance and upgrades had removed the requirement on that network section.

Further Action

Servers Australia Engineers will continue to strengthen the core SAU network, as well as remove legacy infrastructure and configurations throughout the process. Recent upgrades and redesigns have made these changes more streamlined, with less service impact, and we look forward to delivering even better experiences in the future.