Telstra Lost The Internet, Not Sure Why

Telstra is Australia's biggest internet service provider (ISP), so when its connection stops working, lots of people notice. An as yet unidentified problem with its network knocked out most customers for at least half an hour today and impacted other providers. Story developing: we've got further comment from Telstra and iiNet — and confirmation of Dodo's involvement — and we'll keep adding as we learn more.

Telstra initially seemed unaware of the problem when reports began surfacing on Twitter and in online forums. At 2:46PM, Telstra media spokesperson Craig Middleton acknowledged the issue online:

Telstra internet is back after 35 min outage. Cause not known at this stage. Appears to have affected all states.

A statement issued to Gizmodo shortly afterwards gave a little more detail:

We can confirm that Telstra customers lost access to the Internet at approximately 1.50pm. The cause of the outage is unclear at this stage but affected the whole country. Telstra is working quickly to identify the cause of the problem and will keep customers regularly updated on progress.

Users also reported outages with other ISPs, though it's not clear whether that's because they're reselling Telstra's service.

We'll update as we find out more (my guess is that there were some major DNS dramas). Did you experience problems on your computer or phone?

Update 1: As a few commenters have suggested, oft-maligned ISP Dodo is being fingered as part of the problem. A posting on the mailing list for the Australian Network Operators Group suggests that inaccurate network data via Dodo was at least part of the issue.

Update 2: Here's a more detailed statement from Telstra on the problem at of 3:20pm:

Telstra lost connectivity progressively to its international data network at around 1.40 this afternoon meaning customers could not access international websites from either their mobile or fixed services. Customers who couldn’t access international sites were then overloading domestic sites causing major issues in accessing Australian sites as well. The vast majority of customers have been progressively coming back on line since 2.25 this afternoon. Telstra’s engineers are continuing to monitor the network and identify the cause. We are also experiencing a high volume of calls to our customer service centres which means some customers may be having difficulties getting through at this time. Customers accessing our online services may also be experiencing difficulties.

Update 3: An iiNet spokesperson just confirmed to us that it was also impacted by the problem for around an hour, since it uses Telstra for some international routing. Once the problem became apparent, it stopped using Telstra for accessing international sites and switched temporarily to using other connections, which made some sites slower but ensured they could be accessed. The issue has now been resolved.

Dodo And Telstra Investigating

Update 4: (4:03PM): Gizmodo just spoke with Dodo CEO, Larry Kestelman who confirms that Dodo was involved in the outage.

He told Gizmodo that an "issue with one of the minor routers was causing an issue between ourselves and Telstra, and that’s had larger ramifications than it should have with other providers. Even though everything is up and running now, the two companies are still investigating the root cause of the problem."

Update 5 (9PM): It doesn't really change what we already knew, but here's the official statement from Dodo on the issue:

Dodo experienced a hardware issue with a Cisco border router. This issue caused Dodo to broadcast network routes to Telstra. In normal circumstances, this would not result in a network outage. However, it appears that these routes were accepted by Telstra and propagated to Telstra’s downstream customers rather than Telstra simply filtering the routes. This caused major issues for Telstra and its customers which should have been avoided.


Comments

    This was affecting a lot of people outside of Telstra too - just check Whirlpool where almost every provider had issues, for example, iiNet: http://forums.whirlpool.net.au/forum-replies.cfm?t=1871825&p=-1&#bottom

      iiNet uses the Telstra network...

      Here we go with the iinet bashing. I have been on all morning and didnt have a problem at all on my iinet connection apart from fewer trolls in the internet.

    I hear it was to do with routing tables and dodo advertising incorrect ones.

    Is this the internet?

      No, you must still be offline. Call your provider asap!

    I have had troubles with my iinet (naked), but it seems to be back up now. Don't know if it is related or not, but was quite unusual to be offline. Panic started to set in ;)

    I did notice DNS issues but pretty much palmed them away once I had some IP connectivity issues as well. Apparently DODO advertised the full internet table directly back to Telstra

    Experienced issues with all Telstra services including Telstra Internet Direct, Next G, Telstra Business and Home DSL. IPwan was still working through out outage.

    iiNet uses Telstra as an upstream provider for international connectivity.

      beat me to it - thanks for the info.

        Bugger. beat me too!

    http://lists.ausnog.net/pipermail/ausnog/2012-February/012202.html

    Further to my last comment, it may just have been that the sites I tried to visit (domestic & international) including Giz, may have been routed via telstra's cables??

    The only pages i could get to load were google search pages - couldn't get any links to send me to other pages, even Bing search pages wouldn't load....only google result pages...

    wondering what the hell telstra did.....

    Yeah I'm with iiNet and apart from MSN and forum boards everything was down. Should note (correct me if I'm wrong though) that iiNet service is also linked to Telstra's... Will be interesting to see what went wrong.

    where i work we use a fair amount of telstra 3g cards so i knew straight away

    also the Commonwealths bank's website ( and the Commbiz portal ) was down at the same time but their outage lasted an extra 10 minutes

    lucky our office is connected with Fibre ( not through telstra obviously)

    yep, thought my mobile service had been cut off coz I couldnt get on the net....then found out work PC busted too......weird. years ago, a BIG gov department I worked for, our email system went down when a tradie unplugged something to plug in a drill....turns out it was our Domino server....maybe Telstra had some blokes in today doing work :-P

    It went down last night too. For hours.

    everyone was down with pretty much all international down.

      No - Optus and ServiceNet were up through out the whole outage.

      where i work the main fiber line didn't have any problems we could load all international site without problem the only sites where telstra, bigpond, commbank,

      trust me i knew straight away because of 3g cards gotta love being tech support on a day like today

    Our company (HQ + all satellite stores) was definitely affected. Still having email issues, some take 5-10 mins to reach our mail server if sent from an external source (e.g hotmail), some have yet to come through at all. Anyone else?

    my internet was down (domestic and local), but skype was working fine

    Acting like some sort of disaster level : first world

    Yep We lost connectivity too ! Lots of sites called-in to ask "whats wrong" !

    Healthcare sector is criticle and an outage like this does affect a lot to everyone in some or the otherway...

    being said that, its "networks" and it can happen.....

    im with iinet and i could only get to a handful of sites. facebook was down. iinet tool box was down. google search was down but gmail worked..

    with bigpond, i couldn't access any US based sites, only EU and some Aust servers.

    Ok so who googled google??

      lol i did. I googled google and i googled bing but i coudn't bing google...

    I was down about 1400 today, cable not working, Telstra 3G on iPad or iPhone not working, call son in Werribie, same problem
    Tried vodafone pocket wifi2 with iPhone, this seemed to work ok.
    I'm in Bayswater Vic. Telstra cable back about 14:30 approximately

      Similar problem in Adelaide. No contact with clients etc. Finally dragged an old 3 sim card out of the drawer and put it in a pocket WiFi and emailed from iPod. Rotten old 3 to the rescue! Who would have thunk it?

    TPG was fine for me - both TPGs own DSLAMS and there 'offnet' products, had most of my clients call up telling me there internet wasnt working and we need to fix it ASAP, after the first 2 calls I suspected Telstra failed somewhere, good to know they resolved it quickish though.

    Actually, my internet service through internode has been cutting out all day, i thought it was a problem with my router but this could be the cause.

      Nope- Internode was not affected.

        Damn, i guess it is my router then playing up.

          lol i find it quite funny how you deleted my comment about the exact cause as i work for telstra

Join the discussion!

Trending Stories Right Now