What does Scraping is good for?

The current issue of IRW features Web Scraping as a vehicle for conducting Web Mining.

As mentioned in the newsletter, there are so many things that can be done with scrapers. For instance, the below is a comparative of the number of script tags (<script …>…</script>) and link tags  (<link …./ >) declared in several index pages and extracted with two scrapers mentioned in the IRW article: the Script and Link Tag Scrapers. As expected, pages with a lot of content are prone to have  more scripts.

Search Engines	Script Tags	Link Tags
Yahoo.com *	15	2
Bing.com	12	1
Ask.com	10	0
Google.com	4	0
Gigablast.com	1	0

 

Socially-oriented Sites	Script Tags	Link Tags
Searchenginewatch.com	38	5
Twitter.com	9	3
Seomoz.org	7	13
Facebook.com	5	4
Wikipedia.com **	1	6

 

* At the time of the analysis, Yahoo.com redirects to the m.yahoo.com alias, but same results are obtained.

** Wikipedia.org and Wikipedia.com return same results.

On the other hand, Web Scraping can unveil potential Web Vulnerabilites in an architecture, so there is a positive side to the story.  

In the good hands, scrapers can do great things. In the wrong ones, they can be a nightmare.

Unfortunately, hackers know well that scrapers can be embedded into malware and get their hands on source codes. Ask victims of such scrapers like Google and other companies (http://www.wired.com/threatlevel/2010/01/google-hack-attack/).

Besides legal issues and an unfriendly landscape (censorship), it appears they got tired of chinese hackers picking on them so they are pulling out of China  -or treatening to do so.

http://online.wsj.com/article/SB10001424052748704362004575000440265987982.html?mod=rss_Today’s_Most_Popular

Beaten in their own game: brain power.

IRW:Web Scraping – Web Mining with Scrapers

The current issue of IR Watch – The Newsletter is out. Featuring article’s abstract follows.

“Web Mining is a subfield of Data Mining where patterns are derived from the Web. If scraping tools are used for Web Mining this is referred to as Web Scraping (WS).

A scraper is a program designed to extract information from online documents. Scrapers work by matching document source codes against regular expression libraries.

WS is widely used, in part due to the rising popularity of scripting technologies like Asynchronous JavaScript and XML (AJAX), which allows users to retrieve source codes and manipulate the Document Object Model (DOM). WS is a form of Information Extraction where tools, not necessarily scrapers, and repositories, not necessarily the Web are used.

For the last 10 years we have been developing scrapers to simplify the collection and analysis of intelligence from the Web or local machines. For the last 4 years these were slowly converted to AJAX. In this issue of the newsletter, we want to share with readers our experience using several scrapers.”

Fractal CSS Design as a Row Primary Technique

A new article, Fractal CSS Design as a Row Primary Technique is now available. This one is a follow up work. The abstract is given below. Enjoy its reading.

“This paper describes fractal css design as a row primary technique. It is demonstrated that this approach helps to minimize the user-machine relevance perception mismatch. The impact of design on the parsing of documents is demonstrated. It is shown that replacing table-based layouts with CSS tableless design not necessarily produces equivalent layouts.”