• About IR Thoughts
  • Minerazzi Tools
  • Minerazzi Tutorials

IR Thoughts

~ Thoughts on Information Retrieval, Search Engines. Data Mining, and Science & Engineering

IR Thoughts

Category Archives: Web Mining

A Simple Example of Phonetic Similarity vs. Text Similarity

09 Monday Sep 2019

Posted by egarcia in Data Mining, minerazzi, Programming, Scripts, Software, Web Mining

≈ Leave a comment

Tags

levenshtein distance, levenshtein similarity, minerazzi tools, phonetic similarity, tools

The followings sound the same so their phonetic similarity is 1.

(a) r u?
(b) ar u?
(c) are u?
(d) r you?
(e) ar you?
(f) are you?

However, the Levenshtein Distance (LD) and Levenshtein Similarity (LS) of (a) with the other strings differ:

LD(a, b) = 1; LS(a, b) = 0.5
LD(a, c) = 2; LS(a, c) = 0.33
LD(a, d) = 2; LS(a, d) = 0.33
LD(a, e) = 3; LS((a, e) = 0.25
LD(a, f) = 4; LS(a, f) = 0.2

Can you find LD and LS results for other possible combinations?

For i = j, LD(i, j) = 0 and LS(i, j) = 1 so you may want to ignore this case.

Note: LD and LS results were computed with our tool at http://www.minerazzi.com/tools/levenshtein/levenshtein-distance-calculator.php

References
http://www.minerazzi.com/tutorials/levenshtein-distance-tutorial.pdf
http://www.minerazzi.com/tutorials/distance-similarity-tutorial.pdf

Zillman’s 2019 Directory of Directories

29 Thursday Aug 2019

Posted by egarcia in Curated Collections, Data Mining, directories, Directories, information retrieval, IR Tools, miner, minerazzi, News, Public Databases, Web Mining

≈ Leave a comment

Tags

directories, Directory of Directories, miner, minerazzi

Zillman’s 2019 Directory of Directories is a handy resource for those interested in finding specialized gateways to the Web.

Glad to know that this summer Minerazzi was included in the Academic/Education section.

(http://columns.virtualprivatelibrary.net/2019_Directory_Of_Directories_June19_column.pdf)

See also http://www.2019directoryofdirectories.com/

Since 2018 we are listed in the Bot and Intelligent Agent Resources category and few other sections, but did not realize that in that one too.

(http://www.zillman.us/minerazzi-your-search-and-mine)

Cool!

Extracting Topic-Specific Wikipedia Links

10 Saturday Aug 2019

Posted by egarcia in Data Mining, IR Tools, miner, minerazzi, Web Mining

≈ Leave a comment

Tags

miner, minerazzi, wiki links, wikipedia

Below is an illustrative example of a miner for extracting specific links from Wikipedia.

http://www.miislita.com/wikilinks

This miner provides entry points to Wikipedia links relevant to Puerto Rico.

Recrawling a search result discovers new topic-specific links, allowing for further exploration and mining.

Recent News on Predatory Journals

20 Monday May 2019

Posted by egarcia in miner, Predatory Journals, social pulse parser, Web Mining

≈ Leave a comment

Tags

deceptive publishers, fake news, fake research, miner, Predatory Journals

Here are some recent news on predatory journals, obtained through the news channel of our Predatory Journals miner at

http://www.minerazzi.com/predatory-journals/

How I became easy prey to a predatory publisher – Science Magazine
Thu, 09 May 2019 07:00:00 GMT
How I became easy prey to a predatory publisher  Science MagazinePressure to publish and an unfortunately timely email led this assistant professor astray.

Academics Raise Concerns About Predatory Journals on PubMed – The Scientist
Thu, 09 May 2019 07:00:00 GMT
Academics Raise Concerns About Predatory Journals on PubMed  The ScientistThe National Library of Medicine has quality control procedures in place, but some researchers believe additional scrutiny is necessary.

The crisis of predatory publishers sucking the blood of science – Science Friction – ABC News
Sun, 05 May 2019 07:00:00 GMT
The crisis of predatory publishers sucking the blood of science – Science Friction  ABC NewsScience journalist John Bohannon was shocked by the results of his sting operation to expose predatory publishers. Academic librarian Jeffrey Beall found …

Mechanism to accredit academic journals in the pipeline – University World News
Fri, 17 May 2019 13:21:20 GMT
Mechanism to accredit academic journals in the pipeline  University World NewsKenya’s Commission for University Education (CUE) has proposed the establishment of a mechanism for accrediting academic journals, with a view to …

The problem of predatory journals – AAMCNews
Tue, 09 Apr 2019 07:00:00 GMT
The problem of predatory journals  AAMCNewsRoughly 12 times a day, Kurt Albertine, PhD, deletes emails from suspicious journals inviting him to submit articles. The invitations are from predatory journals: …

Flaws in Academic Publishing Perpetuate a Form of Neo-Colonialism – The Wire
Sat, 11 May 2019 00:44:38 GMT
Flaws in Academic Publishing Perpetuate a Form of Neo-Colonialism  The WireMaking the scientific literature open access from both the production and the consumption perspectives is essential to make knowledge more democratic.

Medical Journals: Is Milton Packer Crying Wolf? – MedPage Today
Mon, 29 Apr 2019 07:00:00 GMT
Medical Journals: Is Milton Packer Crying Wolf?  MedPage TodayAcademic publishing: is it as bad a Milton Packer says? In a recent blog post — Medical Journals: a Sluggish Form of Twitter — Packer exposes the many flaws of …

University of Pisa research reveals the cost of false science – Science Business
Wed, 24 Apr 2019 07:00:00 GMT
University of Pisa research reveals the cost of false science  Science BusinessItalian researchers and professors have spent over 2.5 million dollars to publish articles in predatory journals, that is journals which boast scientific standards …

EDITORIAL: Fake research in the halls of learning blight academia – Business Day
Thu, 09 May 2019 07:00:00 GMT
EDITORIAL: Fake research in the halls of learning blight academia  Business DayA group of unscrupulous academics at local universities have joined the long list of unsavoury people adept at identifying weaknesses in government rules in …

The Price for ‘Predatory’ Publishing? $50 Million – The New York Times
Wed, 03 Apr 2019 07:00:00 GMT
The Price for ‘Predatory’ Publishing? $50 Million  The New York TimesThe Federal Trade Commission accused Omics International, a publisher in India, of operating hundreds of questionable scientific journals. A federal judge …

A Unicode Mnemonic for Vowels with Diacritics

09 Thursday May 2019

Posted by egarcia in Data Mining, News, Web Mining

≈ Leave a comment

Tags

Mnemonic, Unicode, Unicode Mnemonic, Vowels with Diacritics

A mnemonic for easily recalling the unicode entities of vowels with grave accents, acute accents, and circumflexes is available at

http://www.minerazzi.com/tutorials/unicode-mnemonic.pdf

We developed the mnemonic after some work on mining the unicode system. It might come handy for those that need to encode said vowels without looking at reference books, tables,… It is also a great example for those teaching mnemonics.

For those interested, we have a tool for exploring unicode entities at

http://www.minerazzi.com/tools/unicoder/unicoder.php

Update: I found a typo in the “o” lower case circumflex and also an error in the 194 entity. Fixed both. Sorry.

W3C Miner

13 Wednesday Feb 2019

Posted by egarcia in Curated Collections, Data Mining, IR Tools, miner, minerazzi, News, RSS/Atom Feeds, Software, Web Mining

≈ Leave a comment

Tags

Curated Collections, miner, minerazzi, W3C, W3C Miner

W3C Miner. http://minerazzi.com/w3c/

W3C public resources, news, standards, work groups, and more.

Use its news channel to easily access from a single place several rss news feeds relevant to World Wide Web Consortium.

Recrawl search results and build your own curated collection of resources.

Document tree flattening as an exploration technique for data mining .xml files (sitemaps, feeds, inventories, raw data, etc)

25 Monday Jun 2018

Posted by egarcia in Crawlers, Data Mining, Data Structures, Feed Tools, information retrieval, IR Tools, minerazzi, Programming, RSS/Atom Feeds, Sitemaps, Software, URLs Mining, Web Mining

≈ Leave a comment

Tags

Data Mining, Feeds, Flatteners, minerazzi, Sitemaps, tools, URLs Extraction, web feeds, XML Mining

Two of our tools, Web Feed Flattener and Feed URLs Extractor, were updated and now accept files with the .xml extension so we changed their names to indicate this. These tools are available at

http://www.minerazzi.com/tools/flattener/feed-flattener.php

http://www.minerazzi.com/tools/feed-urls/extractor.php

These updates take the tools to a whole new level. Now you can flatten the tree structure of files like sitemaps.xml and similar files and extract URLs. Just submit a  target web address and you are good to go.

I know there are tools out there that can scrape .xml files in order to extract specific pieces of data like URLs, but found them too cumbersome. A major drawback of said design alternatives is that frequently one must know in advance how the document tree was constructed, with all of its tags and nuances, before coding a tool. To top off, if the author of the file changes or edits tags, probably the tool won’t work as expected.

Our approach is different and very flexible. The key here is the flattening of the document tree structure embedded in XML files without even having to know how it was designed or edited. Document tree flattening will unveil this information before you can say: “Give me some soup!”

Of course, we assume that the document tree has no orphan or broken tags (and better, pass validation) which is something to be expected from trusted sources. If it is not valid, well, there are ways of fixing it or ignore the offenders.

With the proposed technique, we can mine all sort of .xml files and build customized tools on top of the flattened results, like derivative tools for mining sitemaps, inventories, raw data, recipes, etc… No need to know in advance anything about the document tree, resource to additional scripting technologies, software, or reinvent the wheel.

Right now we can mine sitemaps all over the Web, including sitemaps hosted at Google, W3C, company sites, etc, and then recrawl the output to grow a microindex. See “Suggested Exercises” sections of the tools for interesting examples. This is a value-added approach for our Maps2Miners ongoing project.

Considering that there are government agencies and organizations facilitating data in .xml format for developers to mine, flattening .xml files and build on top of these is one of those “ah-ha!” ideas.

 

URL Cleaner: Clean URLs from search results and websites

21 Thursday Jun 2018

Posted by egarcia in Curated Collections, Data Mining, information retrieval, IR Tools, Marketing Research, minerazzi, News, Programming, Scripts, Software, URLs Mining, Web Mining

≈ Leave a comment

Tags

Data Mining, Search, search engines, tools, URL Cleaner, URL Tools

The URL Cleaner (http://www.minerazzi.com/tools/url-cleaner/muc.php) is our most recent tool.

Clean URLs from search engine result pages and websites, including Google, Bing, Yahoo, Yandex, Wikipedia, and others.

Introduction

  • The Problem
    Sometimes collection curators and content developers use web scrapers (Wikipedia, 2018a) to extract URLs from websites and search result pages.If a web scraper is not available or the target search engine reacts against the scraping (Wikipedia, 2018b), URL extraction can still be possible by installing a browser add-on like Copy Selected Links or a similar plugin. Once installed, users can right-click selected text and copy the URL of any links it contains. To copy all links from a page, they just need to press Ctrl + A to select the entire page text, right-click the selected text, and copy all available URLs at once.Regardless of how URLs are collected (with or without web scrapers), the end result might be a list of dirty, ugly records with obscure attribute-value pairs appended by the search engine.Sometimes the list of URLs include entries with:

      • URLs pointing to social networks. These URLs are often viewed by collection curators as “plastic contamination” in search results suppose to be “organic”. Typical examples are results from Google and similar search engines.
      • URLs about self-promotion. The same search engine might include URLs pointing to unrequested content like its own products, services, partners/ads, links to additional content, etc. Typical examples are results from Google and URLs extracted from Wikipedia webpages.
      • URLs with special characters. For instance, those defining queries (?), fragment identifiers (#), and hash-bangs (#!), among others (Wikipedia, 2018c; 2018d).
      • URLs with some characters encoded.
      • URLs containing mailto:, javascript:, or data:
      • 6-22-2018 Update: URLs obfuscated by shortening services: e.g., bit.ly, goo.gl, is.gd, t.co, and many more. Regardless of their merits, shortened URLs can open the door to all sort of problems (Wikipedia, 2018e). These are frequently viewed by collection curators as unnecessary noise.
  • The Solution
    Would it be nice to have a tool that lets users generate by default a list of clean, sorted, and deduplicated URLs, with options for selectively include/exclude some of the above contaminants? This is precisely what our Minerazzi URL Cleaner (MUC) does.
  • Unlike other URL cleaners, MUC cleans multiple URLs at once from search engines and websites, and can be used free of charge. Before proceeding any further, lets explain what MUC is and is not. The tool is a data cleaner and a lightweight version of our popular Editor and Curator tool. It is not a web scraper, URL validator, or URL shortener resolver, but can be used to clean results from these.
  • In the next section, we describe some uses for MUC, its features and limitations.

What is computed?

  • Searches Support
    MUC was designed to edit search results from the following.

    • Google, Bing, Yahoo, Yandex, and DuckDuckGo
    • 100searchengines, HotBot, Ask, and textise.net
    • Google Scholar, and Wikipedia

    The tool is compatible with individual sites and might be so with other search engines. Whenever possible, we are open to add support for other search engines as suggested by users.

  • Editing Features
    The tool implements the following edits by default.

    • Social networks
      URLs pointing to Linkedin, Facebook, Twitter, Myspace, Instagram, Pinterest, Snapchat, Youtube, Vimeo, and Tumblr are removed.
    • Self-promotions
      URLs about the supported search engines and pointing to their products, services, and partners/ads, or any additional content are removed.
    • Special characters
      Sections of a URL that start with ? # [ ] @ ! $ & ‘ ( ) * , ; = are removed. Trailing forward slashes (/) are also removed.
    • Special strings
      URLs with mailto:, javascript:, data: are removed.
    • Encoded characters
      URL %-encoded characters are replaced by their unencoded versions.
    • Shorteners (6-22-2018 Update)
      URLs obfuscated by shortening services (nearly 600 of these services) are removed.
    • One or more of the above edits can be disabled by properly checking the corresponding checkboxes.

    Since these features are enabled by default, if a run produces no results it means that either all URLs are fully contaminated or there are no URLs to edit.

  • First time users
    We recommend first time users install the Copy Selected Links , or a similar add-on, before proceeding any further. Then do a search in Google and, with the add-on installed, clean URLs, first selectively and then at full blast, MUC.
  • Tool limitations
    Up to 5,000 URLs can be submitted at once. We arbitrarily imposed this limit to (a) provide fast responses, (b) minimize browser crashes, and (c) minimize abuses.
  • Last but not least, the tool might fail to remove non English, obfuscated, or encrypted characters.

 

The URL Query Parser

14 Thursday Jun 2018

Posted by egarcia in Data Mining, IR Tools, Marketing Research, minerazzi, Programming, Queries, Software, URLs Mining, Web Mining, Web Security

≈ Leave a comment

The URL Query Parser is our most recent tool for mining URLs. It is available at

http://www.minerazzi.com/tools/url-query/parser.php

What is a URL query?

A URL query is the trailing text after the question mark (?) found in a URL. It consists of attribute-value pairs delimited by ampersands (&). These are also called name-value, key-value, or field-value pairs.

What this tool does

This tool parses URL queries and extracts its name-value pairs.

The tool helps users identify and filter URL queries from a collection or build collections consisting exclusively of URL queries.

With minor modifications, the tool can be converted into a massive URL cleaner. We are currently building another tool that does this, precisely. In this way we may be able to clean up URLs found in Google and Bing search result pages and safely use them in data mining studies.

What is computed

  • Up to 5,000 URLs can be parsed. If no query is found in a URL, that record is ignored.
  • We have arbitrarily imposed the 5,000 limit for several reasons: to (a) provide fast responses, (b) minimize browser crashes, and (c) minimize abuses.
  • Users can opt between two query result modes:
    • individual results (useful for comparing individual URL queries).
    • combined results (useful for comparing specific name-value pairs).

    The latter is the default mode. Since in this mode results are alphabetically sorted, users can easily identify the most common or popular name-value pairs.

Implications to Web Security

This tool can be used by those interested in mining URL queries or conducting studies relevant to Web Security. Why? Please keep reading.

URL queries are used to transmit small pieces of data in the form of name-value pairs. The transmission can be of three types: (a) between web pages, (b) between a web page and a database, or (c) between databases. Real-world applications include access to web services, social profiling, and cloud computing, among others (Kantarcioglu, 2013).

In addition, URL queries are frequently used as vehicles for transmitting session parameters, form data, tracking mechanisms, user names, email addresses, and other data considered sensitive by users.

In a 2014 study, West & Aviv, from Verisign and the US Naval Academy, analyzed over 892 million user-submitted URLs containing 1.3 billion name-value pairs. They found over a quarter-billion plain text pairs involving referral tracking, with more than 10 million pairs potentially revealing some form of demographic, identity-based, or geographical information. Extreme cases involved the facilitation of password authentication credentials, email addresses, and user names (West & Aviv, 2014).

Thus, the development of tools designed to mining URL queries is something relevant to Web Security.

Suggested Exercises

  • Do a search in several search engines or public databases. Collect a set of URL queries and submit this set to our tool. Compare results.
  • Analyze a set of URL queries obtained from a public forum, social networks, or groups (e.g. Google Groups).
  • For this exercise you need to install a browser add-on to facilitate collection of URLs. In Firefox for instance you can install an add-on plugin that lets you selects multiple links and copy their URLs. Do a search in Google or similar search engines and with said add-on collect search result URLs. Submit these URLs to our tool. Compare results. This is a nice way of grabbing intelligence from URL queries relevant to specific search terms. In addition, since Google lets you do advanced field-specific searches (e.g., inurl, intitle, etc), this is a nice way of mining URL queries driven by advanced searches.

References

  • Kantarcioglu, M. (2013). BigSecret: A Secure Data Management Framework for Key-Value Stores. See also these Researchgate or Github entries.
  • West, A. G. & Aviv, A. J. (2014). On the Privacy Concerns of URL Query Strings. In W2SP’14: Proceedings of the 8th Workshop on Web 2.0 Security and Privacy. San Jose, CA, USA. May 2014.
December 2019
M T W T F S S
« Nov    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Favorite Sites

  • Minerazzi.com

Pages

  • About IR Thoughts
  • Minerazzi Tools
  • Minerazzi Tutorials

Categories

  • 4D Printing
  • AIRWeb Course
  • Algorithms
  • Amazon Alexa
  • Android
  • Arithmetic Geometry
  • Best Match Models (BM)
  • Big Data
  • BioDesign
  • bioinformatics
  • Blogroll
  • calculators
  • Cancer
  • Cancer Research
  • Causality & Determinism
  • Chaos
  • chemical mining
  • Chemist Biographies
  • chemistry
  • Chemometrics
  • Clinical Trials
  • Cloud Computing
  • Conferences
  • Correlation Coefficients
  • Cortana
  • Crawlers
  • Curated Collections
  • Data Conversion
  • Data Mining
  • Data Structures
  • Deep Neural Networks
  • directories
  • Directories
  • Docker
  • Dynamics
  • Favorite Sites
  • Feed Tools
  • Fisher Transformations
  • Fractal Geometry
  • Fractal Patterns
  • google
  • Graduate Courses
  • Hacking
  • Homeland Security
  • Human-Computer Interaction
  • Image Compression
  • information retrieval
  • Internet Engineering
  • Internet Standards
  • inverted index
  • ir
  • IR Quizzes
  • IR Tools
  • IR Tutorials
  • Kubernetes
  • Latent Semantic Indexing
  • Legacy Posts
  • LIGO
  • Machine Learning
  • Marketing Research
  • Mathematics
  • Medical Cannabis
  • meta-analysis
  • Mind Retrieval
  • miner
  • minerazzi
  • Miscellaneous
  • National Laboratories
  • New Information Retrieval Paradigms
  • News
  • Newsletters
  • Nonlinear Dynamics
  • One-to-Many (O2M)
  • Open Source Projects
  • PageRank
  • Patents
  • PCA
  • People Searches
  • Perfectoid Spaces
  • PHP
  • Physiology
  • Poems
  • political networks
  • Predatory Conferences
  • Predatory Journals
  • Programming
  • Public Databases
  • Public Records
  • Quack Science
  • Quantum Computing
  • Quantum Information Retrieval
  • Quantum Searches
  • Quantum Theory
  • Queries
  • Ranking Results
  • Research Centers
  • RSS/Atom Feeds
  • Scripts
  • search engines
  • Search Engines Architecture Course
  • Search Modes
  • self-weighting
  • SEO Myths
  • Sitemaps
  • social mining
  • social pulse parser
  • Software
  • Spam
  • Statistics and Mathematics
  • SVD
  • Technology Inventions
  • Theoretical Physics
  • Theses
  • twitter
  • URLs Mining
  • Vector Space Models
  • Voice Assistants
  • Web Mining
  • Web Mining Course
  • Web Security
  • Xamarin

Recent Posts

  • On the Myth of d Orbitals Hybridization
  • The Bond Order Calculator: Updates
  • Semantic Similarity of Healthcare Data
  • Going the multidisciplinary way
  • CUNY Computational Chemistry Tools
  • Why I chose to be a multidisciplinary scientist?
  • A Simple Example of Phonetic Similarity vs. Text Similarity
  • A Simple News Hub
  • Zillman’s 2019 Directory of Directories
  • More on Perfectoid Spaces
  • Lymphomas Miner
  • Extracting Topic-Specific Wikipedia Links
  • Programming Languages Miner
  • Beware of Chemistry Heuristics
  • New IANA Miners

Archives

  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • October 2011
  • September 2011
  • August 2011
  • July 2011
  • June 2011
  • May 2011
  • April 2011
  • February 2011
  • January 2011
  • December 2010
  • November 2010
  • October 2010
  • September 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • January 2010
  • December 2009
  • November 2009
  • October 2009
  • September 2009
  • August 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009
  • February 2009
  • January 2009
  • December 2008
  • November 2008
  • October 2008
  • September 2008
  • August 2008
  • July 2008
  • June 2008
  • May 2008
  • April 2008
  • March 2008
  • February 2008
  • January 2008
  • December 2007
  • November 2007
  • October 2007
  • September 2007
  • August 2007
  • July 2007
  • June 2007
  • May 2007
  • April 2007

Algorithms calculators chemical mining Conferences Data Mining Hacking Homeland Security Human-Computer Interaction IR Tools IR Tutorials Latent Semantic Indexing Machine Learning Marketing Research Mathematics miner minerazzi Miscellaneous New Information Retrieval Paradigms News Newsletters Programming Queries Scripts Search Engines Architecture Course SEO Myths Software Spam Statistics and Mathematics Vector Space Models Web Mining Course

Create a free website or blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy