Tuesday, December 27, 2011

Web scraping with Python - the dark side of data

In searching for some information on web-scrapers, I found a great presentation given at Pycon in 2010 by Asheesh Laroia. I thought this might be a valuable resource for R users who are looking for ways to gather data from user-unfriendly websites. The presentation can be found here:

http://python.mirocommunity.org/video/1616/pycon-2010-scrape-the-web-stra

Highlights (at least from my perspective)
  • Screen scraping is not about regular expressions. It is just too hard to use pattern matching for these tasks, as the tags can change regularly and have significant maintenance issues.
  • BeautifulSoup is the go-to html parser for poor quality source. I have used this in the past and am pleased to hear that I was not too far off the money!
  • Configuration of User Agent settings is discussed in detail, as well as other mechanisms that websites exploit to stop you from scraping content
  • Good description of how to use the Live HTTP Headers add-on for Firefox.
  • A thought-provoking discussion about APIs, and comments that suggest that their maintenance and support is woefully inadequate. I was interested to hear his views, as they imply that scraping may be the only alternative when you really need data that is highly inaccessible.
Other notes

The mechanise package features heavily in the examples for this presentation. The following link provides some good examples of how to use mechanise to automate forms:
http://wwwsearch.sourceforge.net/mechanize/forms.html

There was also some mention of how Javascript causes problems for web scrapers, although this problem can be overcome via the use of web-drivers such as Selenium (see http://pypi.python.org/pypi/selenium) and Watir. I have used safari-watir before, and from my experience it can perform many complex data gathering tasks with relative ease.

Please feel free to post your comments about your experiences with screen scraping, and other tools that you use to collect web data for R.

14 comments:

  1. Very interesting information, will have to find time to go through it all and watch that video. Thanks for posting this.

    ReplyDelete
  2. Nice posting,thanks for share the nice blog with us and i read the full blog and this blog is the informative and i have to sure bookmark this blog..

    data extraction services

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. I have been exploring for a little for any high quality articles or blog posts in this kind of area . Exploring in Yahoo I ultimately stumbled upon this web site. Reading this information So i¡¦m glad to show that I have an incredibly good uncanny feeling I came upon exactly what I needed. I such a lot indubitably will make certain to don¡¦t forget this web site and give it a look a relentless basis. https://python.engineering/time-process_time-function-in-python/

    ReplyDelete
  5. Instantly this web site will undoubtedly frequently end up being notable regarding all weblog consumers, simply because painstaking reviews as well as checks. dark web links

    ReplyDelete