An In-Context Topic Crawler


, , , ,

I have completed a breadth-first in-context crawler that traverses the Web, recursively discovering links in two modes:

1. continuous mode: without stopping.
2. discrete mode: controlled by the user.

The first mode quickly fills a small database, but still makes the crawler to act like a vacuum cleaner, collecting all kind of garbage; i.e. links that might be irrelevant to a topic-specific database are also grabbed.

The second one is slower, but lets me decide whether to continue or stop the crawling at a given recursion level and based on topic criteria, reducing the amount of garbage gathered. Relative urls are automatically resolved into absolute ones, a bit tricky task when recursion is involved.

In addition to links, the crawler selectively extracts strings that match specific patterns like email addresses, phone numbers, zip codes, etc. Contextual keywords surrounding the patterns are also collected so these can be reciprocally mapped. The goal is the development of a service that consumes in-context pattern-specific databases–great for people searches, intelligence, and marketing. So far the project simplifies even more the building of topic miners.


Mapping IPs: The Wikipedia Way?


, , , , , ,

Some services apparently used by Wikipedia for mapping IPs to other resources. Great for business intelligence, tracking users, spammers, marketing research,…

We used them to test’s IP (

Replace any instances of with the one you want to test.

MUST Tool Enhancements


, , , , ,

We have enhanced the MUST tool, available at

This is a redirection checker tool that upon url redirections reports initial and final status codes, URLs, and IP addresses.

The tool now:

1. accepts 500 urls per submission.
2. summarizes broken and active URLs.

This is one of several tools that we use in-house for re-indexing databases and cleaning up crawl results, except without url limitations.

Shopping & Shopper Miner


, , , , , , , ,

This miner has been reindexed with new content, ahead of time the Black Friday and Holiday Season special offers & deals.

It is available at

The miner now features an SPP-powered news service with consumer alerts, recalls, and fraud reports

Use it to find coupon deals, price comparisons, holidays offers, consumer reports, and more.

E-Paper Miner


, , , ,

This is a new miner, available at

More than appropriate miner on electronic paper and ink technologies, now that the news is out about Microsoft’s sticky notes using e-ink powered by ambient light.

E-Paper and E-Ink technologies are set to simplify our life:

Mobile Technology Miner


, , ,

This is a new miner available at

Find apps software and vendors for mobile devices, smartphone technology, and more. Search by vendors, makers, models, or keywords. Submit web pages relevant to these topics.

Includes an RSS News section powered by SPP to help you find industry-related news, releases, events, and more so far from AT&T, Samsung, Sprint, and Verizon. Feel free to submit for inclusion a relevant RSS url

Excel File for Quantile-Quantile Analysis


, , ,

This is an Excel .xlsx file for reproducing Table 1 of our tutorial on Quantile-Quantile Plots. Now anyone with Excel installed can play and explore this simple technique aimed at determining if a data set is normally distributed.

To download the Excel file, access the most recent update of the tutorial, available at

We also removed few extra “)” typos that were undetected in previous copies.

Have a great Q-Q day! 🙂

Text Streamer Tool


, ,

The Text Streamer is a new tool available at

Streamline text by removing non-printable or encoded characters and multiple spaces.

The tool converts non-printable characters, including tabs, returns, newliners, and multiple spaces into single spaces. User can opt to remove all encodes. These are characters encoded in %, decimal, and hexadecimal notation.

To use the tool, just enter your input text and submit form. To remove all encodes, check the form checkbox. Click the output text to select it. Copy/paste it as usually you would.

It comes handy for users that need to copy/paste streamlined text (plain text) from one file type to another or post it through html forms residing in blogs, discussion forums, and social network sites, or any site for that matter.


BM25F Model Tutorial


, ,

We have restored, expanded, and updated our tutorial on the BM25 Extension to Multiple Weighted Fields Model, best known as BM25F. It is now available at

Active links were also added to the References section.

Enjoy it.