Everything You've Been Told About Data Retention Is Wrong

No, your online activity is not being monitored. Data retention does not work that way, goodnight.

Hello, my name is Lance E. McDonald. I spend most of my time on twitter yelling about computers, anime, and video games, but I actually get paid to create and implement software solutions at an internet provider in Australia. The most recent project I had to spend time on was a script that scrapes through account logs and archives the information required to meet the government’s new data retention laws. You’ve probably heard a lot about these laws in the news lately, and I’m guessing almost the entirety of what you heard has been clickbait-fuelled trash. I thought I’d show you what the data actually looks like, and talk about how this whole thing works.

I’m going to just open with a photo of my own personal data that has been retained by my internet provider over the past 10 days. I have accessed thousands of websites, downloaded about 30GB of data and basically used the internet like a typical five-person family would over a ten day period. I have manually rebooted my modem once in this time, while writing this piece. This is the entirety of the data that the government has access to about my usage over the past ten days:

(Disclosure: As well as removing identifying information, the IP address field and the “data volume” field have been removed from this screenshot. Data volume shows how much data measured in bytes I downloaded in the past 10 days; it was around 30GB).

Does this look a bit low detail compared to what you would expect? There’s nothing here other than “Lance turned on his modem, and then turned it off 10 days later” and the next line is “Lance turned his modem back on a few seconds later.” But this is actually what the attorney general’s guidelines describe the data as being expected to look like for internet providers. Data items should be “hours to several days, weeks, or longer apart”.

I’ve seen lists online with titles like, “Here’s what you need to do to avoid the new data retention legislation” consisting of VPN services, recommendations to use Tor, and a bunch of other arbitrarily selected pieces of advice that have zero impact on the data that is actually being retained. I’ve even seen a few anti-virus companies leveraging the public fear to try to sell some kind of encryption services. Perhaps if all the garbage being spread about your ISP recording what you do online were actually true, then sure, using a VPN would definitely hide that. But using a VPN to avoid the new data retention laws is like tinting your car windows to stop speed-cameras from hearing conversations inside your car: cars don’t work that way, conversations don’t work that way, and speed cameras don’t work that way; you’re not even close.

Recently we’ve been flooded by popular news reports making claims such as, “The government can tell you’ve been using Facebook Messenger, they just can’t read your conversations”. This simply isn’t true; your internet provider isn’t retaining anything about what services you use online as this isn’t part of the legislation. The ISP will only retain data about services they directly provide to you: they’re providing you a link to the internet, so they need to record the time that link was connected, and then the time it was disconnected. No data about what you’re using that link for is retained, no metadata, nothing.

The government’s new data retention laws require internet providers to remember, for two years, what IP address is assigned to a customer every time that customer’s modem is turned on, and what times that same IP address is released from the customer when their modem is turned off again. Also recorded is the location of whatever radio tower/telephone exchange/fibre node to which your modem is actually connected. If you’ve ever looked at a Telstra Detailed Bill, you can see how your internet sessions typically say what town you were in when you were using the internet on your phone. This shows the tower to which your phone’s modem was connected at the time.

The other aspects of telecommunication data retention revolve around telephone calls, SMS messaging, and email transmission. Not much is changing in regards to phone-calls and SMS; your provider will continue to keep a list of every phone number you call and how long you speak to those people, as has always been the case. The same goes for SMS, every time you send a message, the number to which you sent it is retained in a database. The only new requirement is that the data is now kept for two years. Previously it did not need to be retained, and providers only did so for billing purposes.

I will say this, though: email data retention is changing quite a lot, and is far more aggressive. The legislation hasn’t been completely clear on the matter, but it’s likely that it will be treated similarly to SMS, and every time you send an email your provider will record the transaction for two years, albeit discarding the body of the email. Please don’t use your internet provider’s email service if you have privacy concerns. Use Gmail or Outlook.com if you’re not using business class services already.

A huge part of the misconception about data retention equating to internet surveillance is the fact that the legislation requires that your telecommunications service provider retain data on “the destination of a communication”, and this is indeed one of the key data points being recorded by all service providers… except internet providers:

(I like the mysterious extra bracket before the question-mark at the start, professional.)

So, as is mentioned above, it’s worth taking a quick look at section 187A of the recently distributed Telecommunications (Interception and Access) Act 1979 where we can see that the intention has never been to perform surveillance.

For internet providers, the “destination of a communication” (which can be argued to mean “the websites you visit” or “people to whom you send messages”) is strictly not required to be monitored or retained. If an internet provider does choose to retain this information, that is their own prerogative, and the government would require a warrant to access that kind of information (again, this is if it was even being stored in the first place, as it is outside of this legislation). Most of this data is impossible to retain, though, as most communication services online now are encrypted with SSL, through which your provider can’t see.

The whole thing might bring to mind recent cases where end-users have downloaded copyrighted materials and the rights-holders have managed to subpoena customer information from the internet provider. How does this work? Well, rights-holders tend to hang out in public torrent swarms watching people seeding their intellectual property, and they take note of every IP address engaging in the illegal activity. Then they send annoying emails to the ISP who owns those IP addresses, insisting they forward email warnings to their customers.

Most ISPs put these emails in the trash, the logic being that if the rights-holder wants legal action they should be speaking to the police, not an internet provider. The rights-holders aren’t approaching internet providers and saying, “Tell us everyone who pirated our movie”, because the internet provider doesn’t retain data about what their customers do online; they’re saying, “We saw these people pirating our movie and we want you to tell them to stop.” As has always been the case, if you’re seen breaking the law, you’ll probably be identified. If you break the law but no one sees it happen, data retention won’t help anyone catch you (the moral grounds for pirating Game of Thrones are obviously a whole different kettle of fish).

Eventually, one rights-holder, someone to do with the movie Dallas Buyers Club, got sick of internet providers throwing their emails in the trash and took the providers to court. The court decided that, in this case, the rights-holder should be allowed to speak to the customers directly.

In the end, nothing much came of it. Things might be changing on this matter in the near future as providers will likely soon be required to send customer details directly to the rights-holder on a 3-strike system so the rights-holder can send a scary email directly to the customer. This is an unrelated legislation, though. And besides, you probably use private trackers anyway, don’t you?

Data retention is like the TAC/VicRoads knowing what your license plate is, and how long you’ve had that license plate, but not where or when you drive each day. If your licence plate is spotted at the scene of a crime, the police can ask VicRoads, “Who owned this licence plate on this day?” But the police can’t go to VicRoads and say, “Here is a list of illegal car crimes, please tell me every driver who did these crimes in the past two years.” It’s just not possible to catalogue or index the data that way. The police need to find the crime, then VicRoads can help identify the criminals. The information kept under the new legislation can’t be used to proactively fight crime, it can only be used to react to a crime after it’s already been done, and as long as the crime was witnessed by someone, or captured in a server log somewhere.

So what’s the point of this data that’s being retained? Does it have anything to do with terrorism? Probably not. In my experience, the data is only used in child pornography cases. Typically the process goes that the police will raid an illegal pornography server and get physical access to the machine. Inside the machine, they find a list of every IP address that has ever connected to it, thus they have a list of every IP address that committed the crime of accessing that pornography server. The police contact the internet providers that own those IP addresses, and the internet providers look in their data retention logs to see which customers were assigned those IP addresses at those times. The internet provider then hands that list of customers to the police.

This actually happens, and has been happening for years. Most internet providers have already been retaining this data the whole time.

You might have heard that a number of internet providers have been granted an 18-month extension on their data retention obligations. This is typically due to the bureaucratic process more than anything else. The majority of internet providers already met their data retention obligations years ago, and now we’re just seeing the government finally put a strict rule set on exactly how this is meant to be done.

It can be very exciting to imagine that the world works in a way where the government is some malevolent, all-powerful force capable of seeing and attempting to control what you do. But the internet is still primarily outside the government’s reach, despite what rival political parties will pin on each other or what the media will say to trick you into clicking on their ads. Even your provider doesn’t have the technology to control what you do with the internet. When was that internet filter coming, again? Was it six months ago, or seven years ago? There’s been a few now, hasn’t there? The government doesn’t understand the internet and is doing enough terrible things every day that we don’t have to make up any extra stuff.

And please stop saying “metadata”, this isn’t CSI: Cyber.

You can follow Lance E. McDonald on Twitter here.



    I so love this guy it's not funny.

    Great summary with only a couple of grey spots!!!

      I agree, I thought this article was very clear and even cleared up a few misunderstandings of my own.

        Yeah, the post did help me in clearing up few underlying facts about the data retention law. There is only a certain set of data that is required to be stored by telecom companies and ISP. However, there are still few areas I am concerned about.

        Australian government says it does not promote companies to record user’s web history but as far as I see it, any Australian based ISP is required to log their users IP address, internet sessions and the websites they visit. There is a big gap in providing clarity for which data can be stored and which should not. Although the post says encryption tools such as VPNs won’t be of any use; I am afraid they will be. Especially if these VPN services are not based in Australia, they can certainly protect your internet activity from being recorded (as explained here: http://www.bestvpnprovider.com/australia-vpn/).

        Even if the government does not record my phone calls or the conversations that take place, they still are monitoring me, aren't they. My phone number, call duration, and the person with whom I talk are still recorded, which leaves me very vulnerable. Same is the case with internet activity. One thing I would disagree with is that in the post, the author said that the data retention regime is implemented to stop child pornography. I think that is one of the reasons but terrorism is definitely amongst them as well.

        The Australian Governments website (https://www.ag.gov.au/dataretention) states that "Metadata is used in almost every serious criminal or national security investigation, including murder, counter-terrorism, counter-espionage, sexual assault and kidnapping cases." This shows that the government has implemented this law to identity anyone that they thing is involved in suspicious activities. Also, if you remember about the information that was leaked by Edward Snowden about Five Eye nations showed that Australian intelligence agencies have working for a long time in the background collecting user information (http://www.news.com.au/technology/online/snowden-leaks-five-eyes-alliance-australian-involvement-detailed/story-fnjwnj25-1227191761395). The only difference now is that things have become official with the data retention law.

        I'm afraid you're being taken for a ride, here...
        Lance E. McDonald is very naive... or very cunning. Either way, I'm not buying it.

        His arguments are shallow and simplistic.
        "It can be very exciting to imagine that the world works in a way where the government is some malevolent, all-powerful force capable of seeing and attempting to control what you do."
        Obviously, Lance lives in a world of his own...

        Even my 3-year-old knows who Edward Snowden is, and that governments are breaching the privacy of millions of innocent non-crime-committing people! So personally, I reckon that running a VPN connection on all my devices is definitely a good idea. Judging by the way governments are behaving themselves all around this crazy world I happen to live in (as opposed to Lance's world...), I think that I'll trust my common sense on this one.

        By the way, whatever VPN you are using or planning on using, there is one very important point that no one ever mentions, yet it is absolutely paramount!
        Don't use the Chrome browser!!! It leaks your DNS address, and there is currently no fix. Firefox is a much wiser choice, but you'll still need to perform a small technical task to prevent the DNS leak.

        In the Firefox browser, type in "about: config" in the URL bar. When the page comes up, enter "media.peerconnection.enabled" into the search bar. When it appears, set that entry to "false" (this can be done by double clicking it, or by right clicking and selecting "Toggle").

        The other important point to remember is the fact that most VPN providers are renting their servers from another company. So this means that the VPN provider doesn't actually have control over the servers, nor over the speed of the servers, nor over what is truly being logged by the owners of these servers.

        I'm running a VPN on all my devices from a provider that does own all of its servers. There are not many like them out there. https://blog.ipvanish.com/why-mass-data-retention-will-make-you-question-everything/?a_aid=OVRPLY
        I'd suggest trialing out several providers, because there is no such thing as "the best VPN provider" - everyone has different needs and expectations, but there are definitely common issues that are relevant to all and do need mentioning.

        Last edited 10/11/15 6:51 am

      Except it's full of inaccuracies. For example, reseller ISPs are only required to retain the "high-level" session data mentioned by the author, but wholesalers are in fact still required to retain all the data around source and destination addresses and communication types as previously discussed.

      The reason for this differentiation is to reduce duplication of data being retained - if the wholesaler is retaining the detailed data, it's not necessary for the reseller to retain the same data. Similarly, the wholesaler will not necessarily have access to customer session information, so this responsibility falls onto the reseller.

      There's a detailed "Industry FAQ" that describes all of this here: https://www.ag.gov.au/NationalSecurity/DataRetention/Documents/DataRetentionGuidelinesForServiceProviders.pdf There's a matrix of ISP types and the class of data they're required to retain - the author's analysis is over simplistic at best.

      The author is not a lawyer, and he's not an expert in the data retention scheme.

        Hmmm.... this is not really correct. Wholesalers are still not required to record destination IP addresses for internet traffic. Only certain services like email (which the wholesaler probably doesn't supply anyway). The matrix in question doesn't outline things for the sake of reducing duplication, in fact, it's quite clear that duplication will occur. The reason it exists is to outline the responsibilities of all parties as the data needing to be recorded is only visible to certain parties (which you mentioned).
        I agree that the author is not a lawyer but he has a pretty good grasp of the data retention changes - more so than any other author of similar articles I have read. The legislation changes are in fact much more straight forward than the media would have you believe. The issue more lies in the fact that it's written in a language that easily confuses people outside of the telecommunications industry.

          I've worked in telecommunications and I struggle to understand some of the intricacies of the legislation. The fact that so many carriers have failed to implement the changes by the required date is indicative of the complexity involved.

          There are many accounts of network professionals struggling to convert the deliberately vague and non-specific language of the legislation into a practical implementation. Why do you think the AG has created so many resources attempting to explain what's required?

          I have much more trust in Mark Newton as an authority on these things, and he seems to disagree with several aspects of the author's interpretation: just because the author's view is articulated clearly and simply doesn't mean it's correct or authoritative.

            What part of the legislation do you have trouble with?
            I've read a lot of articles about the industry having trouble interpreting the changes. Off the top of my head though, none of the articles I've seen have had any spokespeople from ISP's to quote from. They've all been from journalists or people on the fringe of the industry.

            Also, the AG's office is available for contact by all ISP's if they do need assistance.

              Well, the resources on the Attorney General's site do go a long way to clearing some of the confusion, but the amendment itself is still very ambiguous and open to interpretation. Such an important piece of legislation shouldn't be so vague and non-specific that it requires the Attorney General to publish a list of FAQ and provide specific implementation advice to carriers.

              By way of example, email remains in scope for metadata retention, but IMAP traffic is exempt. What about SMTP traffic? What if a service transparently proxies SMTP traffic on a carrier network?

              Out of curiosity, are core routers exempt from the act? Core routers don't exist exclusively to provide an internet service to a subscriber, yet they're typically operated by carriers (who hence are under the force of the act) and are used to send and receive communications?

              Last edited 20/10/15 5:08 pm

                Well, email servers use SMTP to send emails between each other so in that sense, yes - emails count as a communication to be recorded. What is being recorded is not the SMTP traffic itself though, just the who, what, when etc recorded by the server (and happens to use that protocol). IMAP is your device talking to that server so it's not really relevant to the communication in question.
                I think that you are confusing the subject with core routers - not all communications are required to be recorded. Internet traffic recording is limited to your session ie, if the core router doesn't provide an internet session ie, it's directing traffic, it doesn't need to record anything.

                  An Application Server doesn't provide an internet session either, yet application services (referred to as OTT services in the legislation) provided by carriers are required to retain metadata.

                  Can you point to the section of the legislation that stipulates that, as far as email metadata is concerned, SMTP is within scope, but IMAP isn't? What if I use IMAP to "transmit" an email by means of writing to a shared mailbox? Is IMAP then in scope?

                  If a carrier runs a transparent proxy server, which is technically an application, is that within scope for data retention? What if it's an opt-in proxy?

                  Where does it say that not all communications are not required to have details recorded? How does a core router not fall under 187A(3), which states:

                  This Part applies to a service if: 23
                  (a) it is a service for carrying communications, or enabling 24 communications to be carried, by means of guided or 25 unguided electromagnetic energy or both; and 26
                  (b) it is a service: 27
                  (i) operated by a carrier;

                  There is also nothing that I can see in 187B which would exclude a carrier that provides a backhaul network.

            Lee, could you please explain what it is you think is "vague and non-specific" about the following passages from the Act:
            "(4) This section does not require a service provider to keep, or cause to be kept:
            (a) information that is the contents or substance of a communication; or
            Note: This paragraph puts beyond doubt that service providers are not required to keep information about telecommunications content."
            "Note: This paragraph puts beyond doubt that service providers are not required to keep information about subscribers’ web browsing history."

            It all seems very clear to me: A Communication belonging to a Service provided by an Australian Service Provider is subject to Data Retention.
            For telephony, this means they log the information provided when attempting to setup a call: CallerID, DNIS, FACs.
            For internet services, this means they log information provided when connecting to the service: name, address, IP address assigned to you by the ISP, duration and data volume.

              Because there are other sections that explicitly mention that metadata must be retained for Instant Messaging, VoIP, Email and other "OTT" services. There is very little differentiation between what constitutes activity related to "web browsing" (the web to me means HTTP) and non-browsing activity.

              I gave an example in another comment of an SMS being sent via a HTTP SOAP gateway. Does this fall under the "telephony" requirements, or the "internet services" requirements? Does this constitute "web browsing" activity even if the data is being sent via "the web"?

                Reading between the lines, I am guessing you are misinterpreting the import of s187AA.
                s187AA does not "explicitly mention that metadata must be retained", all it does it attempt to define - what is a Relevant Service, what is a Communication, etc...
                As a definitional piece of text, it certainly does not contradict s187A,4 where OTT is quite explicitly excluded.
                You do seem awfully confused, and yet the legislation is very clear:
                "Note: This paragraph puts beyond doubt that service providers are not required to keep information about subscribers’ web browsing history."
                The bottom line is that if it is a service provided by a service provider , and where both service and service provider are subject to the Act, then the communication made on that service needs to be recorded, and not communications made using different services that pass over the top of that service.

      It's a shame that you enjoy and article that so wrong it isn't funny.

    Finally!!! Some sanity in this debate. You can all remove your tin-foil hats now and get on with your lives.

      Buy why does the RSPCA need access?

        In 1979 when the Commonwealth legislation passed the RSPCA was on the list of approved agencies because they are, in some circumstances, the lead investigatory agency for certain types of crimes - mainly animal cruelty and animal welfare. This might antiquated however if you work in the country away from the city these laws are quite appropriate.

        However since the amendments that came into affect this year the RSPCA and a lot of other agencies have since been removed. So no, the RSPCA does not have the power to compel access to telco data.

          Cheers for clearing that up. They just seem a bit odd, since all the others are govenment departments and law enforcement etc.

            Australia Post used to be on the list too (since removed), as is the ATO and a few other agencies that were concerned with collecting public revenue which was one of the three pillars of the access to telco data:

            1) Enforce criminal law
            2) Protect public revenue
            3) Locate missing persons

            RSPCA Inspectors are also typically NSW Police Special Constables.
            Aust Post workers don't get that luxury, no guns for them!

              @robb I wouldn't say they all were, but certainly some of them were. As did some rangers who worked for the various NSW Councils, but since changes to the Police Act a few years ago Special Constable's are no longer appointed to any agency and are used solely as armed security guards at NSW Government (including Police) buildings.

              Special Constables appointed before the change continue on as before but they will no longer be replaced.

              Inspectors, like Council Rangers no longer need to be Special Constables because their legislation has been updated to include the word "Authorised Officers" which include Police and Inspectors in whatever Act they're talking about. It is a far more sensible way to go about business.

    Does any telco record conversations anyway? I was under the impression that Telstra did, at least in the past.

      There are two things to show that it wouldn't matter either way:
      1. Most sites (especially questionably legal ones) use SSL and hide the content of the connection. There is also VPN services if you are really sceptical.
      2. People are acting as if the government is going to sent swat teams after them if they have downloaded something illegal. The Government and ISP's don't have the resources or funding available to monitor every connection for specific content. Even if they miraculously did manage to set this up, there is also the situations where people are found but not charged , hence a waste of money.

      Basically, this would be a huge waste of time.

      As mentioned in the article, rights owners are cloaking themselves in torrents to catch out illegal downloaders, and that has been happening for years, and that method is obviously ineffective as they are still complaining.

      This falls back to the simple fact that you cannot regulate illegal downloads without abusing privacy rights, it just wont happen.

      No mate, it's pretty much always illegal to record a telephone conversation without a warrant.


      Also, in NSW it is illegal to record audio of a private conversation without every single party in the conversation giving consent or obtaining a warrant (called 'two party consent'). This law varies from State to State though. Some States (and a lot of the US - so, what you see on TV) have 'one party consent' which says that as long as the audio is being recording by at least one person in the conversation, then it's fine. This still prohibits 'eaves dropping' or 'bugging' by a third party without warrant.

        I have read (at least in VIC) that the law allows any party to a conversation to record for legal reasons without needing consent.

        So if the recording might possibly be used to prove the facts of the communication, then recording (without consent is OK).

        It was a long while ago so forgive me if I have that wrong.

          Some Australian States have 'one party consent' laws, Victoria isn't one.

          In Victoria every party to a private conversation must give consent to the audio recording. However, Victoria Police can record audio without your consent if they think it will protect someone's safety. From an operational stand point, I don't know how that is interpreted by VicPol.

    The annoying thing is though, if the data is mostly just to combat child pornography but has already been happening this way for years, then I don't really see the point in these changes. Unless their current method has been insufficient to capture these people and there are hard statistics to prove this then I would agree, but seeing as these statistics are not available then it's a bit hard to see the reasoning behind spending all this money for this scheme.
    It's been done in Europe already and declared invalid 8 years later so it's still confusing as to why it was thought of as a good idea by our politicians. Waste of taxpayer money subsidising changes and also a waste of money for ISPs who may either pass the costs on to consumers or redirect funds that could have been used to improve the service.

    Last edited 19/10/15 11:37 am

      In retort to defences based on terrorism, it's not like it would stop Lone wolf or random acts that aren't connected to a network in any way. It really just makes the assumption that everyone is potentially guilty of crime and I see this scheme as the start of a bad slope if it continues.

        From the data, at least internet, I don't think it would stop anything. They can't see what site you visit of anything of interest. But as you said, now that this is in place, that level of monitoring will come.

        FOI requests could be interesting.

        It is interesting that you assume guilt because the way most criminal investigations work is by eliminating innocent people so that what's left is your guilty party. So evidence gleaned from this source is far more likely to exonerate than to implicate, given that the overwhelming percentage of users are guilty of very little.

      By the looks of things, these new laws are not the end result, just the foundation for future laws in Internet regulation. Although they seem pointless now, they may allow for a more annoying law in the near future.

        I definitely agree. Without turning this into a purely political debate, bipartisan support is very disappointing.

        Actually, the laws are simply updates to the existing regulartory framework which has been in place since 1979.

        The laws were based on a request from the Law Enforcement Advisory Committee which is a standing committee that comprises of law enforcement, defence, and telecommunications companies that advise the Australian Communications Authority on law enforcement issues.

        One of the things that it looked at in recent years is the updating of the existing laws which, as I've said, have been reasonably unchanged since 1979. I'm not sure if you're aware but this new thing called The Internet came into existence since then and it's been kinda a big deal that our former laws weren't really equipped to deal with.

        They proposed an update, which included mandating exactly what was to be kept, to standardise how long that data is to be kept and to bring ISPs into the regulatory loop,

        Then a standing intelligence parliamentary committee looked at LEAC's proposal and green-lit it with a few finishing touches, and handed it off to the Government to consider. Then the government agreed, and put it to a vote, which the parliament passed and now it's law.

        It's not the beginning of a new world order.

    this article is a bs
    for the professional these proofs look like a kong-fu fighting in bolivood movie
    it's so FAKE, I cant believe that data retention db had been designed by a junior

    Last edited 19/10/15 11:43 am

      Agreed. It is a big salad.

      You've never worked with online legislation, have you? As one of the driest and most boring topics you'll find anywhere, the presentation of it on the net is so fundamentally basic the best resources generally look like that. Go to http://www.austlii.edu.au/ and see how basic the site is, and ask yourself why someone is going to bother to clean that up wen cutting and pasting. Thats pretty much THE go to resource for legal info in Aus, and when you get to what you want its pretty much just Times New Roman font size 12, and thats it.

      Theres nothing professional about screenshotting online legislation, its looked this amateur as long as I can remember, and I've been looking at it since 1990. Once you grab a screeny, and add your own emphasis (a circled area, or a note about a line), and what the story shows is how its going to look most of the time.

      As a test, go hunt the legislation out yourself and see what it says before you call it fake.

        I don't care about fonts and margins, let UI designers care about it. I just don't see it.
        As developer and DB architect i'm talking about DB schema. And it looks worse then my school-time experiments. Same time I can see how much more information is hidden just 1 level deeper from this two records, which are meaningless by themselves indeed. I can only guess what details are stored as "Source", "Subscriber", and "Session"!!! and what linked to this retainer???
        This is only 1 level out from the data provided. What we could see deeper?

        For people not having DB background i can tell that it looks like attempt to make conclusions about your daily routine by google maps view of your suburb. It's hilariously stupid and inaccurate, but same time you can see that there is a street view of your house, photos and plenty of tags. You don't know what's on photos and tags, but you can see folders and folders of them marked with you house address.

        Last edited 19/10/15 12:55 pm

          Fair enough. I misread what you were trying to say, and what I wrote did come across harshly. My bad.

          There's no way of knowing whats held in any related dB though, so all we can go on is what the OP says, and his general reputation. I dont know him, but as every other story just shouts that the sky is falling, and repeats the same fear mongering stuff, at least theres something being put forward to justify this story.

          It could be a primary dB, and the session ID be the primary key, and really thats all that would be needed in a nested set of relationship tables based around that session ID and the IP's of the various sites, so I see your point, but it could also be exactly as claimed - the total amount of data thats collected by the ISP.

          Up to the individual to decide what they want to believe.

          End of the day, I have nothing to hide anyway, and if all the data retention captures is that I look at www.gizmodo.com.au far too often, then so be it.

    Brilliant. Now that its been put to bed, we can all get along with it.

    You sir, are a very naive man. And we'll continue saying metadata, thanks. Perhaps look up what meta means.

      We all know what "meta" means, we just aren't stupid enough to assume that because the prefix is used in the legislation that it will apply wholesale to every piece of metadata generated. The name of the Bill is not the totality of it.

        Er....do a search of the text of the Act.
        The word "metadata" does not appear in it.

    Very good read! :) TY Friend... TY...

    If all this is true, then why is the Government spending millions to implement it?

      My Local Government just spent 10 Million Dollars to improve a Bike Lane...nothing fancy, just an overpass basically to stop them from having to stop at a set of lights. You really think they'd think twice about throwing a few million at something like this?

      You're thinking of Millions from your own point of view and what you could do with it, you have to look at it from the Governments point of view where they usually deal in tens of billions. Drop in the ocean for them.

    Gizmodo has run a lot of articles about metadata retention (and ways to combat it). To all the people saying "thanks" and "finally" to the author - why didn't you put these views forward in one of the many previous articles?

    This is the entirety of the data that the government has access to about my usage over the past ten days
    This is an incredibly misleading statement. You're looking at the best part and not including the worst part of the data retention program.

    Records of every call and sms, your location during these events, your location every time a 'data connection' is made (which is a hell of a lot more common than on a fixed connection).
    For smartphone users, that means the government has a history of everywhere you / your phone has been.
    THATS the part that worries me.

    Last edited 19/10/15 1:07 pm

      Going to go out on a limb here and assume that when it says "Location" it isn't going to be giving your exact GPS location down to the nearest foot, it'll be more likely the Suburb and even then it might not be entirely correct, how many times has your phone given you weather information for an adjacent suburb?

      At best they could get a very limited idea of your movements, but nothing so precise as to determine if you're making dodgy back alley deals!!

      And he is simply looking at the part that He deals in, it would be far more misleading for him to try and deal fully with something he doesn't have a part in.

      Plus as mentioned, the new laws are simply extensions of the old ones, just increasing the time that these records must be kept so far as I can see. But sure, you go ahead and keep that Tin Foil hat on, I'm sure the Guvmint are real interested in the seedy locations you've been visiting!

      Location was addressed in the article - it means mobile phone tower location - as @ixixly mentioned its not GPS, plus it has always been collected and known and recorded - check any mobile bill for the past 20 years!

        This would be more than just call records though. For most smart phones, it will essentially record the coarse location of every time your phone switched from cell data to wifi or back (e.g. when you leave home or get to work). It'd also record points where you temporarily lost connectivity (going through a tunnel?).

        That's a lot more data than what you'd get from coarse location at the time of phone calls.

          Sure, if someone specifically sits down and takes all the data and analyses it properly they can probably get a pretty good idea of where you've been. But if they're going to that hassle chances are you've probably done some fairly naughty for them to divert these resources into tracking you and I'm certain they have better methods.


    News flash: They've been logging mobile stuff for ages, especially call and sms and location. Not sure about data connection, that will now be logged, but essentially if you are using a phone anytime in the last 20 years you can be tracked.

    Last edited 19/10/15 1:34 pm

    This article reads like government shilling. They were already collecting information about the times we connect. Now they're collecting info about our emails, the addresses we send to, etc. If it was just connection time info then why would they need millions of dollars in server upgrades to store all the extra data?

      Because they are now being required to store it for two years. Storing it for a billing period is one thing. Storing 24 billing periods worth of data takes a bit more gear.

      Also, all the telcos will be getting about $130 mil in funding over 3 or 4 years. With several hundred service providers (the main players, plus all the resellers), that money isn't going to go far.

      Assume 10 million odd devices (rough number given quite a few people have more than one device tablet, phone, landline etc), that works out to be $13 per customer, or less than $4.50 per customer per year (if it is over 3 years).

      Not a lot really.

      Last edited 20/10/15 1:24 am

        Try 30 million devices. Australia (and many countries) have been well over 100% penetration for many years. WiFi-only devices not included, plus fixed-line connections.

    Great article - Finally someone with some common sense OR WAS IT !!
    Possibly a wonderful but of disinformation created by the government on behalf of the movie industry to get everyone to relax and stop using VPNs ??? :)-

    HAHAHA Probably the former not the latter.

    Thanks Lance, It's been great to see an article like this from someone actually dealing with what is going on at the operational end rather than all the doom and gloom journos just trying to push click bait on us all!

    It would be great if you had any contacts showing the same side of what is happening with SMS, Calls and/or Emails as well if you aren't also involved in those.

    Given how vague the legislation is on several key areas, the author seems resolute that his organisation's interpretation of the requirements fulfil the requirements of the act. In particular, the bill has been written to be deliberately agnostic of any one particular technology, using terms such as "communication", "address", "type of communication".

    In the example the author gives, although data has been collected about the source and destination addresses of the PPPoE session, no data has been collected pertaining to the address to which data is being forwarded:

    Identifiers of the account, telecommunications device or relevant service to which the communication:
    (a) has been sent; or
    (b) has been forwarded, routed or transferred, or attempted to be forwarded, routed or transferred.

    The act also specifically lists the following of examples of data that must be retained:

    187AA Information to be kept
    The type of a communication or of a relevant service used in connection with a communication: Voice, SMS, email, chat, forum, social media.

    Can the author explain how the data collected in the table he provided corresponds with the requirements listed in table 187AA?

    Finally, the amendments give the Minister the power to change the scope of what data is collected at any time. Whilst an ISP may believe that recording PPPoE session details only is sufficient to satiate the requirements of the act, this may not always be the case. From the explanatory memorandum:

    The telecommunications industry is highly innovative and increasingly converged. Sophisticated criminals and persons engaged in activities prejudicial to security are frequently early adopters of communications technologies that they perceive will assist them to evade lawful investigations. As such, a declaration is required to ensure the data retention regime is able to remain up-to-date with rapidly changes to communications technologies, business practices, and law enforcement and national security threat environments.

    The Attorney General's "Proposed Data Set" document, from which the author's second table was lifted, also states that the following source and destination address details should be retained:
    identifying details (such as username, address or number) of the account, service or device which receives a text, voice or multi-media communication (examples include email, VoIP, instant message or video communication)

    My question is: if only session information needs to be retained, as per the author's assertion, then where and when does the logging of email, VoIP, IM and text addresses come into play?

    The AG's memo on data to be retained can be accessed here:

    Last edited 19/10/15 2:07 pm

      The author's example is about the internet service, and Data Retention affects it in exactly the way he describes.
      Other services covered by the Act work in different ways, so the retained data is different.
      email: Australia has had very tough anti-spam laws for 15 years, so SMTP traffic is logged, and has been for a long time, the new Act doesn't alter this.
      VoIP: same as legacy telephony: the data included in your connection attempt is logged: CallerID, DNIS and FACs. Nothing new here either, just updated to specify VoIP calls to be treated as any other calls have been since the 1979 Act.
      IM/text: As above - this technology was not even imagined in 1979 and has no 1970's analog, so up until now, when the police are investigating a complaint, the data they can usefully retrieve is entirely dependent on the ISPs' interpretations of how these new technologies are covered by the old legislation. The new Act updates this to bring everybody onto the same page.

        My issue with the author's post is that it gives the impression that the total of all data collected as a result of using his internet connection is limited to what he posted. It doesn't mention that, had he used his carrier's SMTP gateway or FTP service, or VoIP service, that there would be more data recorded. It doesn't mention that other carriers will have retained data on his usage.

        I'm not saying that the new metadata retention laws are necessarily big and scary - I'm saying that the article attempts to minimize them and is just as sensational as the articles he lambasts.

          The author was giving the example of his internet service. A very simple and clear explanation that cuts through the FUD.
          Data Retention in relation to any email service he uises is not being retained by virtue of being accessed through his internet service, it would be retained by the email service.
          Everybody uses an internet service. The other services you mention are not so ubiquitous.

        The author's example is about the internet service, and Data Retention affects it in exactly the way he describes.

        So if a basic internet service was delivered to a customer that provided a pure IP service that didn't require encapsulation, what data would need to be retained by the provider?

    You're telling me that an Australian Telco uses a database schema like that? With redundancies all over the place, and long strings of mixed info instead of individual fields.

    There is something fishy about this story.

      It's actually not even valid SQL. The quotes are all messed up :)

      Yes, he works for one, and that's what he is telling you.
      Are you telling us that you consider fact-free paranoia a better source than the horse's mouth?

    This article is so misleading and over simplistic, it serves only to raise concerns over how individual ISPs are interpreting the deliberately vague langue in the Act.

    Whilst the Attorney General's notes do stipulate that

    For internet sessions this is when a device or account connects to a data network and ends when it disconnected – those events may be a few hours to several days, weeks, or longer apart, depending on the design and operation of the service in question.

    nowhere does it say that this is the only information that needs to be retained for an internet service.

    For example, whilst it may be necessary for a reseller ISP to only collect session data, the Data Retention Industry FAQs specifically state that a wholesaler must retain:

    The service identifier of the receiving party(ies), such as destination e-mail address(es), IM, chat or telephone number (SMS) , including the IP address(es) and port(s).
    Details for each of these where there are multiple parties to the message, such as cc: and bcc: for e-mail.
    Destination details where a message is being forwarded, such as an SMS or another voicemail box.

    So whilst Mr McDonald's company may only be required to log session data, upstream wholesale networks ARE required to log more detailed information about the type and addressing of communications.

    It seems the author should be included in the scope of the headline ;)

    Last edited 19/10/15 3:03 pm

      How many times have you said the same thing in a different way? Who are you replying to?

      Readers wanna know.....

        Probably about the same number of times the legislation and explanatory memorandum says the same thing? Especially since my quotes are taken from those sources ...

        I'm not replying to anyone in particular - I'm commenting on an article, which is what I'd always assumed the comments section after an article is for.

          Yep I got your point the first time you made it, but thanks for the extra clarification. I was also using the comments section to clarify just that.

            Well, I'm pleased you got my point :) My work here is done!

        No, "Readers" don't "wanna know". You do though.

    Two simple truths:

    1: The IP information which the author claims is a triviality, is in truth an indispensable part of enabling entities to determine your identity.

    2: History incessantly demonstrates that power corrupts, and that an excellent way to keep powerful entities beneficent is to limit their power (which means in part, their ability to blackmail).

      The 1979 Act has caused this type of data to be retained in this way for the last 36 years, with law enforcement bodies having access to it during that time.
      Please detail with examples of real-world events the "corrupt power" that has misused this retained data since 1979?

        I'll do my best to answer if you can clarify:
        Who are you referring to as the "corrupt power"?

        I mean, if you want examples of how that metadata has been used corruptly, that's pretty easy.

        But if you want information specific to the "corrupt power" you'll have to at least name them. :)

    Hey Lance, I like you as a person (even met you at the first PAXAUS!), but your post is seriously naive, as outlined by others here and reads as a giant mansplain.

    Your implementation may not have concerning metadata, but you're only speaking for one ISP. Additionally you've made a ton of assertions with absolutely no evidence about what other ISP's are doing.

    Also I'm not sure why you don't like the term metadata - it's not from CSI and is used in the industry.

      ...maybe because the term "metadata" has been used to confuse by the FUD-purveyors who've been running a scare campaign against this much-needed update to the 1979 Act?

      It might interest you to note the legislation contains precisely zero instances of the use of the term "metadata".

        Just, what? The Attorney-General excessively uses the terminology when referring to his own legislation, and used it before the legislation was even released. We're just using the language of the government to engage in the conversation.

          Well, you can choose to occupy the same level of understanding and competence as Brandis if you like, the rest of us would like to discuss this in a factual way.
          And the facts are: Brandis did not write the legislation, and the legislation does not use the word "metadata", a word that has been used by some people to completely confuse others as to what the legislation actually contains.

        ...maybe because the term "metadata" has been used to confuse by the FUD-purveyors who've been running a scare campaign update to the 1979 Act?

        There, I fixed your first statement for you buddy.

    This is what the laws are NOW... Some time down the track they get changed quietly.

    Governments, especially conservative ones, understand you can't do big things all at once anymore. They keep falling over. So now it's by degrees.

    Well I for one am pretty sure in fact almost certain that my ISP is using an invisible proxy to record a lot more than Lance states. Never thought I'd be living in a police state.

      Your belief is non-sensical.
      ...unless a warrant has been served on them under the Interception legislation, in which case your data may well be being hoovered-up.
      You would have to be up to something pretty seriously dodgy for such a warrant to be issued, though...
      Still, this has nothing to do with Data Retention, which clearly does not authorise let alone require any such use of "an invisible proxy".

        Ah, you're nonsensical.

        Unless there's a justifiable reason to suspect that someone is "up to something pretty dodgy" there is absolute no reason the government, or any private organisation, should be collecting this data period.

          They've been collecting it since 1979. They sky hasn't caved in yet.

          It is essentially billing data. Far less personal and intrusive than the data Google, Facebook, and Amazon collects and stores about what you are up to.

            Companies collect your data *with your consent* for making profit. They don't make amendment to laws, nor send the police after you based on that data. Governments on the other hand will collect your data *without your consent* and with the express purpose of using it against you. If they were not going to do exactly that, then a warrant should still be required for access to the data, which requires *proof* that you're suspected of a crime. As it stands - they just want a haystack of data to sift through at their leisure.

            Yeah but I don't use Amazon or Facebook so it's not really a good comparison to make.

            Last edited 30/10/15 11:06 am

    geez I am so glad the government doesn't have a hidden agenda for data retention... i feel much more at ease now... they really are all such jolly good fellows... now I can sleep easy knowing they are just collecting pretty useless data and have no intention of using it against anyone who isn't a child pornographer. Wait was that a flock of pigs I saw going past my window

    Actually, they do know your location to within a couple hundred meters, and the longer you stay, the more accurate it gets. Telcos routinely provide emergency services immediate access to your location when you call 000. They're logging it constantly, and now they have to keep it, and worse - hand it over to random agencies without a warrant. That slippery slope - well the foot has slipped, we just haven't landed on our backside yet, but it's coming. And so is the slide.

    Inch by inch, legislation "updates" are made that take away our existing freedoms. Not everything has to be regulated, documented, and categorised. The fact that they're trying means there's pressure from some interest group. Virtually every time, that means it's only good for that specific interest group. No prizes for guessing which groups are pushing this. This is supporting legislation to prepare for the implementation of the TPP treaty, and it isn't the only piece - there are many more "updates" that go hand-in-hand.

    Hi Lance,

    I too work in the sector and play with usage all the time.

    Could you please comment on IP usage reports. Some ISP's refer to this by the name of their collector, "netflow".

    Are these IP Usage reports captured and disseminated to the government as a part of the data retention act? Clearly IP, PORT, SOURCE, DESTINATION, NETFLOW (intra>inter) are extremely important and could easily expose people's privacy.

    Was this in scope in your project?

    Also just as a point, regarding mobile data you made no mention of GPS coordinates. Could you confirm that this information will also be provided to the government?

    One last matter if you don't mind. If you could comment on this if you are aware of this practice I would appreciate your opinion. At certain telco's that I have worked for I have been advised that mediation data is being uploaded to FTP servers, millions of CDRs and disseminated via the LEU FTP server. Seeing section 282 and 283 of the Telecommunication Act is clear that information given by a ISP must be in relation to a crime it would appear that the government is breaching the privacy of millions of people who have not been involved whatsoever in a crime.



      "GPS Coordinates"??? Crikey!
      Where do I even begin.....?
      GPS is a passive service: satellites transmit data and the coordinate data generated by GPS is generated locally on your device.
      In other words, the Data Retention Act, which requires Australian providers of telecommunications services to retain details of whom they have provided services to (ie, billing data, basically) has absolutely nothing whoatsoever to do with GPS data.

      And then there is... "section 282 and 283 of the Telecommunication Act...". These would be two sections that were repealed in 2007..... I'm actually loling as I type this, it's hilarious how silly conspiracy-theories can be sometimes....

Join the discussion!