M247 Core routing incident - intermittent service impact nationwide - Issue resolved
Configuration changes made as part of the change over the weekend have triggered a bug in the forwarding logic on some of the core network routers, causing traffic to be misdirected across the network. We have spent the past 24 hours triaging and investigating this, with our third party, and through a staged configuration process have isolated and reloaded the affected devices, which we believe has resolved the issue.
If you believe you still have an issue, please raise this with our support desk via ticket@m247.com or 0161 822 2580. A full post incident report (PIR) will be available within 5 working days.
Regards
M247 Support
Configuration changes made as part of the change over the weekend have triggered a bug in the forwarding logic on some of the core network routers, causing traffic to be misdirected across the network. We have spent the past 24 hours triaging and investigating this, with our third party, and through a staged configuration process have isolated and reloaded the affected devices, which we believe has resolved the issue.
If you believe you still have an issue, please raise this with our support desk via ticket@m247.com or 0161 822 2580. A full post incident report (PIR) will be available within 5 working days.
Regards
M247 Support
M247 Infrastructure team have advised that the core routing issue now appears to be resolved, the major incident will remain active until we have conclusive evidence to confirm this.
If any issues are still being experienced then forward to ticket@m247.com with relevant Ping / Traceroutes to impacted destinations. Further information on the initial RCA summary will follow.
Regards
M247 Support
Hello,
Infrastructure engineers have made a change to bring back in one of the devices that looked to be causing an issue earlier today
Early signs show that the issue has not returned.We are still monitoring it.
Next update - 20:00 or sooner
Regards
M247 Support
Hello,
Infrastructure engineers have made a change to isolate a device and seem to have gained some stability.
They are seeing traffic being incorrectly label switched across the core and not in the expected manner.
We are still investigating the issue
Next update - 19:20 or sooner
Regards
M247 Support
M247 Infrastructure team have now isolated and restarted multiple core nodes. This process is still in progress.
Next update will be at 18:00
Regards
M247 Support
M247 Infrastructure team are in the process of isolating individual core nodes. Changing the customer traffic / data paths will allow to team to rule out pieces of core infrastructure.
We have supressed traffic which has been generated by routing changes, in recent maintenance windows (ruling out recent configuration amendments).
Next update: 16:30
Regards
M247 Support
Hi
Additional tools are being deployed by our infrastructure team to provide more in depth traffic analysis
Next update: 15:30 or sooner.
Regards
M247 Support
The M247 infrastructure department have identified routing issues across our core network. The team are working towards implementing a temporary fix.
This incident impacts all services which use M247's network for IP connectivity (VOIP, Cloud, Firewall etc)
Next update: 14:00
Regards
M247 Support
Hi,
Our engineers are still working to resolve the issue at hand.
Apologies again for any inconvenience caused during this outage.
Regards
M247 Support
Hi,
Engineers are still investigating the issue.
Apologies again for any inconvenience caused during this outage.
Next update: 11:30 or sooner
Regards
M247 Support