Tags

, , , ,

We have added a new algorithm to the MUST tool, available at

http://www.minerazzi.com/tools/must/must.php

The tool now automatically detects bogus http status code responses. These types of response codes are frequently designed, though not always, to game crawlers and automated header request tools; i.e., to believe that a resource is not accessible.

For instance, test the following with our tool and as given:

arxiv.org
ask.com
webcrawler.com
metacrawler.com

etc…

Have fun!

Advertisements