BeautifulSoup for web scraping

Ever wondered how you get so many spam e-mails to your inbox with in minutes of posting your e-mail address on a popular website/blog? Most of those spam e-mails are sent to you by bots, whose job is to search(scrape) the web pages for text that match the signature of e-mail addresses. This is accomplished by using powerful scraping tools. One such tool is BeautifulSoup

Forget e-mail spamming for a moment, there are a lot of other things that you can do with these tools. e.g Scraping ebay.in's facebook community 'wall' for all the posts. The following python code shows how this can be done with the help of BeautifulSoup.

If you observe the source code for web page, you'll see that each and every post on the wall is under the tag span class="UIStory_Message. So we have to parse the page for finding out all the 'span' tags, which have the 'class' attribute set to 'UIStory_Message'. The method 'bs.findAll', shown below, does exactly that. Also we may want to print the names of the post's author before every post. From the HTML source, you can see that these names are available as text under the tag span class="UIIntentional_StoryNames". This tag is just before our post tag span class="UIStory_Message. As we already have references to all the post tags, we can find out the name tags by calling 'findPreviousSibling' and the name is available one level deep, under the 'a' tag. Finally, we can call 'getText' method to get the name of the post's author.


If this is confusing, please see the official BeautifulSoup documentation

#!/usr/bin/env python
__author__ = "Kasi Viswanadh Yakkala"
 

import os
import re
import sys
from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup

def ebayin_fb_parse():
    frontpage = urlopen("http://www.facebook.com/ebaydotin?v=wall").read()
    bs = BeautifulSoup(frontpage)
    fbstories = bs.findAll(name='span', attrs={'class':'UIStory_Message'})
    for s in fbstories:
        fbprofile_name = s.findPreviousSibling(name='span',attrs={'class':'UIIntentionalStory_Names'}).a.getText()
        print fbprofile_name,':'
        try:
            print s.getText()
        except:
            print s.find(text=True)

# HOW TO USE
""" Main Function """
if __name__ == "__main__":
    ebayin_fb_parse()