URL Cleaner: Clean URLs from search results and websites

Tags

Data Mining, Search, search engines, tools, URL Cleaner, URL Tools

The URL Cleaner (http://www.minerazzi.com/tools/url-cleaner/muc.php) is our most recent tool.

Clean URLs from search engine result pages and websites, including Google, Bing, Yahoo, Yandex, Wikipedia, and others.

Introduction

• The Problem
  Sometimes collection curators and content developers use web scrapers (Wikipedia, 2018a) to extract URLs from websites and search result pages.If a web scraper is not available or the target search engine reacts against the scraping (Wikipedia, 2018b), URL extraction can still be possible by installing a browser add-on like Copy Selected Links or a similar plugin. Once installed, users can right-click selected text and copy the URL of any links it contains. To copy all links from a page, they just need to press Ctrl + A to select the entire page text, right-click the selected text, and copy all available URLs at once.Regardless of how URLs are collected (with or without web scrapers), the end result might be a list of dirty, ugly records with obscure attribute-value pairs appended by the search engine.Sometimes the list of URLs include entries with:
  • • URLs pointing to social networks. These URLs are often viewed by collection curators as “plastic contamination” in search results suppose to be “organic”. Typical examples are results from Google and similar search engines.
    • URLs about self-promotion. The same search engine might include URLs pointing to unrequested content like its own products, services, partners/ads, links to additional content, etc. Typical examples are results from Google and URLs extracted from Wikipedia webpages.
    • URLs with special characters. For instance, those defining queries (?), fragment identifiers (#), and hash-bangs (#!), among others (Wikipedia, 2018c; 2018d).
    • URLs with some characters encoded.
    • URLs containing mailto:, javascript:, or data:
    • 6-22-2018 Update: URLs obfuscated by shortening services: e.g., bit.ly, goo.gl, is.gd, t.co, and many more. Regardless of their merits, shortened URLs can open the door to all sort of problems (Wikipedia, 2018e). These are frequently viewed by collection curators as unnecessary noise.
• The Solution
  Would it be nice to have a tool that lets users generate by default a list of clean, sorted, and deduplicated URLs, with options for selectively include/exclude some of the above contaminants? This is precisely what our Minerazzi URL Cleaner (MUC) does.
• Unlike other URL cleaners, MUC cleans multiple URLs at once from search engines and websites, and can be used free of charge. Before proceeding any further, lets explain what MUC is and is not. The tool is a data cleaner and a lightweight version of our popular Editor and Curator tool. It is not a web scraper, URL validator, or URL shortener resolver, but can be used to clean results from these.
• In the next section, we describe some uses for MUC, its features and limitations.

What is computed?

• Searches Support
  MUC was designed to edit search results from the following.
  • Google, Bing, Yahoo, Yandex, and DuckDuckGo
  • 100searchengines, HotBot, Ask, and textise.net
  • Google Scholar, and Wikipedia

  The tool is compatible with individual sites and might be so with other search engines. Whenever possible, we are open to add support for other search engines as suggested by users.

• Editing Features
  The tool implements the following edits by default.
  • Social networks
    URLs pointing to Linkedin, Facebook, Twitter, Myspace, Instagram, Pinterest, Snapchat, Youtube, Vimeo, and Tumblr are removed.
  • Self-promotions
    URLs about the supported search engines and pointing to their products, services, and partners/ads, or any additional content are removed.
  • Special characters
    Sections of a URL that start with ? # [ ] @ ! $ & ‘ ( ) * , ; = are removed. Trailing forward slashes (/) are also removed.
  • Special strings
    URLs with mailto:, javascript:, data: are removed.
  • Encoded characters
    URL %-encoded characters are replaced by their unencoded versions.
  • Shorteners (6-22-2018 Update)
    URLs obfuscated by shortening services (nearly 600 of these services) are removed.
  • One or more of the above edits can be disabled by properly checking the corresponding checkboxes.

  Since these features are enabled by default, if a run produces no results it means that either all URLs are fully contaminated or there are no URLs to edit.

• First time users
  We recommend first time users install the Copy Selected Links , or a similar add-on, before proceeding any further. Then do a search in Google and, with the add-on installed, clean URLs, first selectively and then at full blast, MUC.
• Tool limitations
  Up to 5,000 URLs can be submitted at once. We arbitrarily imposed this limit to (a) provide fast responses, (b) minimize browser crashes, and (c) minimize abuses.
• Last but not least, the tool might fail to remove non English, obfuscated, or encrypted characters.

 

Updating Several Miners

Tags

data miners, minerazzi, miners, Search

We have reindexed the following miners:

http://www.minerazzi.com/webspam
http://www.minerazzi.com/zika
http://www.minerazzi.com/usc
http://www.minerazzi.com/seominer
http://www.minerazzi.com/religions
http://www.minerazzi.com/shopper
http://www.minerazzi.com/rblogs
http://www.minerazzi.com/quantum

 

 

Python Search Engine

Tags

Algorithms, Data Mining, Mathematics, open source, Python, R, Search, tutorials

Here is a python-based search engine with an implementation inspired on one of our papers at the old Mi Islita.com site, now a search engine on Puerto Rico.

http://nullege.com/codes/show/src@g@e@gensim-HEAD@gensim@test@test_miislita.py

This module replicates the miislita vector spaces from

"A Linear Algebra Approach to the Vector Space Model -- A Fast Track Tutorial"

by Dr. E. Garcia...

 

Great and positive accomplishment!

That tutorial is no longer at miislita.com, but was long ago moved to minerazzi.com. Find it here:

For other resources do a search for python in our IR miner at

http://www.minerazzi.com/irc

For inquiries about that implementation, contact its author.

For other inquiries, applications, suggestions, drop me a line.

PS. Please note that Nullege.com itself is a search engine for finding python code. Here is a good example: http://nullege.com/codes/search/wx.calendar.CalendarCtrl

Minerazzi: Allowing Users to Recrawl Search Results

Tags

Crawls, Data Mining, Recrawling, Search

Effectively immediately Minerazzi (http://www.minerazzi.com) allows users to recursively recrawl search results.

Why is recrawling so important?

The purpose of allowing users to recrawl URLs is to expose them to new content, to involve them in learning through discovery. To turn their searches into a mining activity. This makes more sense than limiting their search experience to inspecting zillion of cached records from a search engine index. The problem with the latter is that frequently those records are either outdated or irrelevant, not to mention that in that scenario the users are simply passive expectators.

Allowing users to recrawl search results has many advantages and possibilities. For instance, users can use the discovered URLs to build curated collections, self-guide investigative work, or gather link intelligence from sites, directories, blogs, forums, or social networks. In general, recrawling allows users to discover hidden paths to fresh, new, or rich content.

Considering that the total number of primary and secondary URLs defines the reach of a microindex, in theory recrawling should result into an endless reach.

At this time, we do not recrawl .css and .pdf files, but we recrawl the most common file formats (.php, .asp, .aspx, .html, .htm, .js, etc). However, if the content of a file is dynamic, obfuscated, or poorly coded more likely it will return garbage or nothing.

Having said that, we invite you to try the recrawling experience with our public miners.