Harvesting google profiles
Some minutes ago, I saw an interesting tweet from Mikko H. Hypponen saying that he found out that all (yes, as in ALL – 35,513,445 ) google profiles addresses can be retrieved from a single XML file . Looked through it and , yeap, he was quite right.
Well , all these information is going to be useful somehow ,right? Right. In case it’s going to be removed here is a simple way to harvest them before that happens :
#!/usr/bin/env python import urllib from BeautifulSoup import BeautifulStoneSoup as bs xml = bs(urllib.urlopen('http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml').read()) for i in xml.findAll('loc'): try: urllib.urlretrieve(i.text, i.text[35:]) print 'Downloaded %s' % i.text[35:] except Exception, err: print '%s could not be retrieved' % i.text print 'All done'
That’s it, save it , run it and wait 🙂 Not that I used it, but I calculate that you get around 1.7 GB worth of profile links .
Well , the juicy part is obviously the harvesting of the information from the profiles themselves. People are mentioning on twitter that Google is aware for a long time, or at least should be. Thoughts about the potential implications from that harvesting, on a blogpost to come .