Telstra is Australia's biggest internet service provider (ISP), so when its connection stops working, lots of people notice. An as yet unidentified problem with its network knocked out most customers for at least half an hour today and impacted other providers. Story developing: we've got further comment from Telstra and iiNet — and confirmation of Dodo's involvement — and we'll keep adding as we learn more.
Telstra initially seemed unaware of the problem when reports began surfacing on Twitter and in online forums. At 2:46PM, Telstra media spokesperson Craig Middleton acknowledged the issue online:
Telstra internet is back after 35 min outage. Cause not known at this stage. Appears to have affected all states.
A statement issued to Gizmodo shortly afterwards gave a little more detail:
We can confirm that Telstra customers lost access to the Internet at approximately 1.50pm. The cause of the outage is unclear at this stage but affected the whole country. Telstra is working quickly to identify the cause of the problem and will keep customers regularly updated on progress.
Users also reported outages with other ISPs, though it's not clear whether that's because they're reselling Telstra's service.
We'll update as we find out more (my guess is that there were some major DNS dramas). Did you experience problems on your computer or phone?
Update 1: As a few commenters have suggested, oft-maligned ISP Dodo is being fingered as part of the problem. A posting on the mailing list for the Australian Network Operators Group suggests that inaccurate network data via Dodo was at least part of the issue.
Update 2: Here's a more detailed statement from Telstra on the problem at of 3:20pm:
Telstra lost connectivity progressively to its international data network at around 1.40 this afternoon meaning customers could not access international websites from either their mobile or fixed services. Customers who couldn’t access international sites were then overloading domestic sites causing major issues in accessing Australian sites as well. The vast majority of customers have been progressively coming back on line since 2.25 this afternoon. Telstra’s engineers are continuing to monitor the network and identify the cause. We are also experiencing a high volume of calls to our customer service centres which means some customers may be having difficulties getting through at this time. Customers accessing our online services may also be experiencing difficulties.
Update 3: An iiNet spokesperson just confirmed to us that it was also impacted by the problem for around an hour, since it uses Telstra for some international routing. Once the problem became apparent, it stopped using Telstra for accessing international sites and switched temporarily to using other connections, which made some sites slower but ensured they could be accessed. The issue has now been resolved.
Dodo And Telstra Investigating
Update 4: (4:03PM): Gizmodo just spoke with Dodo CEO, Larry Kestelman who confirms that Dodo was involved in the outage.
He told Gizmodo that an "issue with one of the minor routers was causing an issue between ourselves and Telstra, and that’s had larger ramifications than it should have with other providers. Even though everything is up and running now, the two companies are still investigating the root cause of the problem."
Update 5 (9PM): It doesn't really change what we already knew, but here's the official statement from Dodo on the issue:
Dodo experienced a hardware issue with a Cisco border router. This issue caused Dodo to broadcast network routes to Telstra. In normal circumstances, this would not result in a network outage. However, it appears that these routes were accepted by Telstra and propagated to Telstra’s downstream customers rather than Telstra simply filtering the routes. This caused major issues for Telstra and its customers which should have been avoided.