Staying Away from Spamked Domains

I call spamked domains those parked domains that spam your eyes and search experience across the web.

How pervasive this problem is and how do you fight the war against web spam?

In a recent check of URLs using MUST (1) we found that the domain http://www.metalib.com is a spamked domain that resolves to the IP 69.172.201.208

Checking this IP with MHM (2), we found that 50,673 domains share the same IP. Random sampling these reveals that these are either spamked domains or not founds.

So how do you avoid them all? Good question.

A partial solution consists in building a collection of spamked IPs and URLs which can then be queried with a candidate IP/URL. Using a dedicated crawler to recrawl these records should help you update the collection as a result of any change in IPs/URLs, something you would expect from the usual suspects.

This might not be a bullet-proof solution as may result in some inocent victims having a ‘bad company’ (host provider) and being in the ‘wrong place’ (server) at the ‘wrong time’ (crawl timestamp), but hey inocent casualties are part of any war.

Update: Our MUST and MHM tools found that the URL http://ww1.theslammer.com/ resolves to IP 72.52.4.90 and is also shared by 57,451 domains.  Also the URL http://www.bestedeal.com resolves to IP 208.73.211.250 which is shared by 10,742 domains.

Thus, with few initial IPs, we were able to identify more than 100,000 cases. Building a collection of spamked domains is not that hard to do after all.

References

1. MUST (http://www.minerazzi.com/tools/must/must.php).

2. MHM (http://www.minerazzi.com/tools/mhm/mhm.php).

Chikungunya Virus Resources

A new miner about the nasty Chikungunya Virus is now available at http://www.minerazzi.com/chikun/.
Conduct research on this hot topic. Search the scientific literature for Chikungunya, a virus disease transmitted by mosquitoes. Read newspaper articles and research on CHIKV from CDC, NIH, WHO, and other sources.
This is a must needed miner built with Minerazzi. Great for students, researchers, and those that were hit by this nasty virus,
Researchers and students: Have a field day.

Finding Financial Data and News Stories with Minerazzi

Now you can find financial and news stories with these tools:

(1) The Financial Search Engine – This is a public miner for searching and mining financial web sites. It is available at http://www.minerazzi.com/mystocks

(2) The News Center – This is a free service that eventually will be available to all miners built with the platform. So far the service returns stock market summaries and news relevant to a miner. A user can submit multiple ticker symbols or use the default symbols. News sources are randomly displayed when the page is refreshed.

Crimes Against Meta Data: Sec.gov and Investor.gov sites

During the course of building a financial miner, we found sites committing a lot of crimes against meta data. The most recent are courtesy of the SEC.gov and Investor.gov sites. Perhaps the result bad copy rewritten by software or humans?

These are great sites for finding financial and business information, but some of their pages contain poorly written meta tag data that make indexers go ga-ga gu-gu.

To illustrate, check the meta description tags of the pages at the following URLs:

http://www.sec.gov/News/Page/List/Page/1356125649507

http://www.investor.gov/introduction-markets/role-sec/how-submit-comments-sec#.U-E3frR0yyZ

Links and CSS style instructions declared as meta description data? Great!

Finally…Enjoy the Ride!

Finally, Minerazzi is here and open for business, after 1 year in beta.
Enjoy the ride without registration at

Sentiment Viz: A Great Tweet Sentiment Visualization Tool

While developing new tools for our platform (http://www.minerazzi.com), we came across Prof. Christopher Healey’s work on data mining and visualization. For those interested on these subjects, Prof. Healey has an incredible site, full of superb resources for data mining the Web. For instance, he has developed several great tools for analyzing sentiment from tweets. These are available at

  1. Healey, C., Ramaswamy, S. (Accessed on 6-17-2014). Visualizing Twitter Sentiment.
  2. Healey, C. (Accessed on 6-17-2014). Sentiment Viz. Tweet Sentiment Visualization Tool.