Minerazzi Web Crawler: Now Individually Crawling Form Fields

These are additional updates made over the weekend to our crawler at http://www.minerazzi.com/labs/crawlinker.php

Changes made include the ability to detect information contained inside selection menus, textareas, and input fields. These changes simplify the examination of name/value pairs used by forms. Very useful when a document contains multiple forms, when query mechanisms of target databases must be identified, or when one need to assess whether a database is susceptible to query injections or script infections. In the latter, a security component is more than obvious.

We also moved both the Markup and Robots Text File Reports to the Source Code section. This is a new section that is now listed as the last reports of the application.

Minerazzi Web Crawler: Spidering Forms

This is an update on our Minerazzi Web Crawler. (http://www.minerazzi.com/labs/crawlinker.php)

Our crawler is now detecting form tags. Changelogs are given below:

06-23-11: Form tags detection capabilities added.

05-08-11: New user interface.

05-06-11: Log file access.

05-01-11: Copy changes.

04-29-11: Script tags detection capabilities added.

04-22-11: DocType, Base, and Link tags detection capabilities added. HTML parsing changes.

04-19-11: Robots Text File detection capabilities added. Meta data parsing changes.

04-18-11: Title and Meta Tags detection capabilities added. Layout changes.

04-16-11: User Environment detection capabilities added.

04-14-11: Timer capabilities added.

04-10-11: Deduplication capabilities added.

04-09-11: Color palette reporting capabilities added.

04-05-11: DNS and MX reporting capabilities added.

04-03-11: Source code reporting capabilities added.

03-30-11: Relative URL resolving capabilities added.

03-28-11: Hypertext wrapping, ip, and headers reporting capabilities added.

 

IFLA World Congress

The International Federation of Library Associations and Institutions (IFLA) is the leading international body representing the interests of library and information services and their users.

They are having their world congress here in Centro de Convenciones de Puerto Rico from 13-18 of August. See http://www.ifla.org/

See also conference program at http://www.ifla.org/en/news/ifla-wlic-2011-final-programme

We will see you there.

Federated Searches and The Search Paradox

The problem with federated searches (aka, parallel, broadcast, or meta searches) is that it is implemented under the assumption that the more it is  searched at one time, the better for the user. This is not necessarily the case. Easy search results does not equate to right results. Submitting a query on a given topic to dissimilar databases with dissimilar content or about dissimilar topics often produces off-topic or irrelevant results. Indeed, easy searching not necessarily translates into a relevance experience.

And there is also the question of how to rank the results. Two strategies are frequently used: (a) data appending and (b) data fusion.

Some of the early forms of federated search engines for the Web used to do (a) and were soon called meta search engines. These search tools simply returned a long list of results by appending the top N ranked results from each databases in a tandem fashion. Obviously this strategy failed to recognized the top M relevant results from this huge list and soon was phased out in favor of (b).

In (b), arithmetic or weighted averages from the top N results from the different databases are computed. The problem with this approach is that is very subjective. In order to compute an arithmetic or weighted relevance score, who decides how much weight should be assigned to a given ranked result from a given database? No matter which weighting criteria are used, at the end it is still a subjective score and one that not necessarily improves the end-user search experience. Just the opposite.

This leads to what I call “The Search Paradox”: Information gateways as information roadblocks.

To learn more about this, visit this old link: http://www.accessmylibrary.com/article-1G1-182034526/federated-search-101-alexis.html