A friend of mine, a very well-connected professor of computer sciences — he knows who he is, and so, if he’s right, do the CIA, the FBI, and so on — writes about the Google piece last week:
Had only to add that, if one thinks that it’s only commercial stumbles that are now letting this kind of info out — one would be badly off the mark… Like the NSA phone listening, since 2002 or so, the US has been accumulating and using just this kind of detail — all of it, as far as I understand things. Every email. Every link, recorded. Every jot and tittle. We know because we see the grants coming in to [where I work] from the DHS, and other places – it might be that as much as 40% of all computer science research funding is in this area, now that DARPA has slowed down. You can imagine the AOL scenario, magnified a million times.
It’s all happening, right now, even with this email.
The way this is done, he says, is essentially the same way that the US government fdoes its phone mining. The law there said that telephone companies, obviously, where allowed to share data with each other, so the NSA built an entire telephone network of its own, which could then peer with the public one. This gives the government complete access to everything on anyone’s telephone line.
Similarly, he says, the US government has built, like Google, a shadow internet, containing copies of everything it can find – though with the added twist that it also has access to all the traffic passing through the network if it wants. Think Slitscan in the hands of Dick Cheney.
If there is a ‘shadow Internet’, then it will be detectable by the crawlers it sends out. The various public search engines are very visible to web admins through the pitter-patter of spider feet. They also obey robots.txt, and anything that doesn’t will be much more visible – either the ‘shadow net’ obeys as well, in which case it’s incomplete, or it doesn’t, in which case it’ll be obvious.
R
yes, this has been the case for a very very long time.
it IS nice that now when i mention it, people are becoming far less likely to roll their eyes and make “conspiracy nut” noises. it’s not something that keeps me awake at nights, but it DOES mean i am rather cautious about the individual micro-phrasings of things i transfer via the internet– essentially seeking to minimise the risk of things being taken out of context.
i’m hoping TOR and similar will essentially turn into insulated subnets, so far as the monitoring/traceability is concerned.
oh, and your mate might remember the rather amusing stories hitting the papers in the late 90s, re the UK Police unilaterally turning down the sensitivity, or rather the automatic nature of the response, of the neural nets they were running over the UK’s SMS traffic. they had one too many episodes of the antiterrorist teams being called out en masse after the computer threw a red alert over a particular phrase or word being sent by one kid to another.