The URL Query Parser

The URL Query Parser is our most recent tool for mining URLs. It is available at

http://www.minerazzi.com/tools/url-query/parser.php

What is a URL query?

A URL query is the trailing text after the question mark (?) found in a URL. It consists of attribute-value pairs delimited by ampersands (&). These are also called name-value, key-value, or field-value pairs.

What this tool does

This tool parses URL queries and extracts its name-value pairs.

The tool helps users identify and filter URL queries from a collection or build collections consisting exclusively of URL queries.

With minor modifications, the tool can be converted into a massive URL cleaner. We are currently building another tool that does this, precisely. In this way we may be able to clean up URLs found in Google and Bing search result pages and safely use them in data mining studies.

What is computed

  • Up to 5,000 URLs can be parsed. If no query is found in a URL, that record is ignored.
  • We have arbitrarily imposed the 5,000 limit for several reasons: to (a) provide fast responses, (b) minimize browser crashes, and (c) minimize abuses.
  • Users can opt between two query result modes:
    • individual results (useful for comparing individual URL queries).
    • combined results (useful for comparing specific name-value pairs).

    The latter is the default mode. Since in this mode results are alphabetically sorted, users can easily identify the most common or popular name-value pairs.

Implications to Web Security

This tool can be used by those interested in mining URL queries or conducting studies relevant to Web Security. Why? Please keep reading.

URL queries are used to transmit small pieces of data in the form of name-value pairs. The transmission can be of three types: (a) between web pages, (b) between a web page and a database, or (c) between databases. Real-world applications include access to web services, social profiling, and cloud computing, among others (Kantarcioglu, 2013).

In addition, URL queries are frequently used as vehicles for transmitting session parameters, form data, tracking mechanisms, user names, email addresses, and other data considered sensitive by users.

In a 2014 study, West & Aviv, from Verisign and the US Naval Academy, analyzed over 892 million user-submitted URLs containing 1.3 billion name-value pairs. They found over a quarter-billion plain text pairs involving referral tracking, with more than 10 million pairs potentially revealing some form of demographic, identity-based, or geographical information. Extreme cases involved the facilitation of password authentication credentials, email addresses, and user names (West & Aviv, 2014).

Thus, the development of tools designed to mining URL queries is something relevant to Web Security.

Suggested Exercises

  • Do a search in several search engines or public databases. Collect a set of URL queries and submit this set to our tool. Compare results.
  • Analyze a set of URL queries obtained from a public forum, social networks, or groups (e.g. Google Groups).
  • For this exercise you need to install a browser add-on to facilitate collection of URLs. In Firefox for instance you can install an add-on plugin that lets you selects multiple links and copy their URLs. Do a search in Google or similar search engines and with said add-on collect search result URLs. Submit these URLs to our tool. Compare results. This is a nice way of grabbing intelligence from URL queries relevant to specific search terms. In addition, since Google lets you do advanced field-specific searches (e.g., inurl, intitle, etc), this is a nice way of mining URL queries driven by advanced searches.

References

Advertisements

Document Linearization Tools

Tags

, , , , , ,

Document Linearization (DL) is aimed at creating a pseudo document from which words are extracted; e.g., for vocabulary construction/indexing.

DL involves removal of markup tags, punctuations, stopwords, upper cases, and some times stemming (reduction of words to common roots) and code/decoration filtering (removal of code & style lines). The end result can be used as part of analyses aimed at exploiting semantics and user intent.

Frequently, a pseudo document is represented as a stream of tokens or lowercased terms without any punctuation in it.

In addition to information retrieval specialists, end users can benefit from DL. For instance, collection curators can use DL to identify keywords representative of a refined collection. Search and digital marketers can use DL to find relevant keywords to be placed in ads and optimized content.

We added three tools to all of our Minerazzi miners (productivity search engines) to help users do DL on the fly for a single search result: The Plain Text Extractor, The Tokens Extractor, and the Words Extractor. These are found in the “Crawlers & Extractors” section displayed under each search result of a miner.

To access them, just do a search in any miner at http://www.minerazzi.com and, under a search result, click file icons labeled “text”, “tokens”, or “words”. For instance, try with our most recent RSS/Atom Feeds miner at

http://www.minerazzi.com/feeds

Support for PDF files & graphical analysis will be available in the near future.

 

RSS/Atom Feeds Miner

Tags

, , , , ,

Having problems finding research focused news feeds? Try our RSS/Atom Feeds miner (http://www.minerazzi.com/feeds).

Use it to find search engines, directories, and sites listing topic-specific news feeds. Then recrawl the miner search results and build your own list or collection of news feeds. Supports the popular RSS and Atom formats, among others.

Use the results from this miner in combination with other tools we developed for data mining URLs, or if you prefer submit your own page listing news feeds and improve the web visibility of your news.

This miner comes handy, particularly now that Google News has disabled its RSS subscription features, breaking tons of feeds and, in the process, affecting their presence on the web. (https://www.seroundtable.com/google-news-rss-feed-gone-25795.html).

Theoretical Physics Miner

Tags

, , ,

The Theoretical Physics Miner, available at

http://minerazzi.com/theoretical-physics/

is our most recent search solution.

Use its recrawling capabilities under a given search result to start building your own curated collection.

Use its news section at

http://www.minerazzi.com/theoretical-physics/spp.php

to access all ARXIV and MIT news feeds relevant to theoretical and experimental physics.

The figure below is for illustration purposes. It was generated through affine transformations that include reflection operations within an n-gon. Any resemblance with a black hole at its center is pure coincidence.

 

Feed URLs Extractor

Tags

, , , ,

Extract all URLs from web feeds, the easy way, with this new tool: The Feed URLs Extractor. Access the tool at the following URL:

http://www.minerazzi.com/tools/feed-urls/extractor.php.

The tool is derived from a previous one: The Web Feed Flattener, also available at

http://www.minerazzi.com/tools/flattener/feed-flattener.php

Enjoy them!

Cancer Research Miner

Tags

, , ,

The Cancer Research MinerĀ  is our most recent specialty search solution, available now at

http://www.minerazzi.com/cancer-research/

Find research centers, news, clinical trials, discoveries, informatics, & more.

Due to the magnitude and impact of cancer-related topics, we are contemplating adding an exclusive section of miners for different types of cancers so researchers can build even more focused curated collections.

The figure below is a nature-inspired shape. It corresponds to a computer generated cluster pattern obtained via affine transformations that include reflection operations. Similar clusters resembling the progression of cancer can be generated in this way.

Fractals Miner

Tags

, , , ,

Fractals Miner: Fractal Patterns and Growth Phenomena – Theory, Experiments, & more. Available now at

http://minerazzi.com/fractals/

Research the fractal geometry literature. Use the images tool below a result to view beautiful patterns or recrawl search results to build your own curated collection.

Note: Image below was created with Manglar, an experimental tool under development.

 

On Fractal Topography and Scale-Invariance

Tags

, , , ,

Closing a gap in the fractal geometry theory: Definition of fractal topography to essential understanding of scale-invariance.

https://www.nature.com/articles/srep46672.pdf

One year old, but very relevant these days. Very important and enlightening research article.

A better understanding of scale-invariance by means of defining fractal topography opens the door to many practical applications. This was something loosely suggested in the literature, but not fully addressed. Great job!

BioDesign Miner

Tags

, , , ,

The BioDesign Miner is a new search solution available now at

http://www.minerazzi.com/biodesign

Find nature-inspired solutions, discoveries, techniques, innovations, & more.

Use it to build curated collections on bio-inspired research or find news on biodesign.

Math Bios: Mathematician Biographies

Tags

, , , , , ,

Math Bios is a new miner available at

http://www.minerazzi.com/mathbios

Find biographies of famous mathematicians from ancient to modern times.

Interested in the life of Euler, Fermat, Poincare, Blackwell, and other great mathematicians? Want to build a curated collection about math people from the ancient and modern times? This is the place to start.

A great tool for math history researchers and students in general.