05 Wednesday Sep 2018
This is a great tool from Google
https://www.blog.google/products/search/making-it-easier-discover-datasets/
I will try to see how the feed flattener can benefit from it.
http://www.minerazzi.com/tools/flattener/feed-flattener.php
25 Monday Jun 2018
Tags
Data Mining, Feeds, Flatteners, minerazzi, Sitemaps, tools, URLs Extraction, web feeds, XML Mining
Two of our tools, Web Feed Flattener and Feed URLs Extractor, were updated and now accept files with the .xml extension so we changed their names to indicate this. These tools are available at
http://www.minerazzi.com/tools/flattener/feed-flattener.php
http://www.minerazzi.com/tools/feed-urls/extractor.php
These updates take the tools to a whole new level. Now you can flatten the tree structure of files like sitemaps.xml and similar files and extract URLs. Just submit a target web address and you are good to go.
I know there are tools out there that can scrape .xml files in order to extract specific pieces of data like URLs, but found them too cumbersome. A major drawback of said design alternatives is that frequently one must know in advance how the document tree was constructed, with all of its tags and nuances, before coding a tool. To top off, if the author of the file changes or edits tags, probably the tool won’t work as expected.
Our approach is different and very flexible. The key here is the flattening of the document tree structure embedded in XML files without even having to know how it was designed or edited. Document tree flattening will unveil this information before you can say: “Give me some soup!”
Of course, we assume that the document tree has no orphan or broken tags (and better, pass validation) which is something to be expected from trusted sources. If it is not valid, well, there are ways of fixing it or ignore the offenders.
With the proposed technique, we can mine all sort of .xml files and build customized tools on top of the flattened results, like derivative tools for mining sitemaps, inventories, raw data, recipes, etc… No need to know in advance anything about the document tree, resource to additional scripting technologies, software, or reinvent the wheel.
Right now we can mine sitemaps all over the Web, including sitemaps hosted at Google, W3C, company sites, etc, and then recrawl the output to grow a microindex. See “Suggested Exercises” sections of the tools for interesting examples. This is a value-added approach for our Maps2Miners ongoing project.
Considering that there are government agencies and organizations facilitating data in .xml format for developers to mine, flattening .xml files and build on top of these is one of those “ah-ha!” ideas.
28 Monday May 2018
Posted in Data Mining, Feed Tools, information retrieval, IR Tools, miner, minerazzi, News, RSS/Atom Feeds, social mining
Having problems finding research focused news feeds? Try our RSS/Atom Feeds miner (http://www.minerazzi.com/feeds).
Use it to find search engines, directories, and sites listing topic-specific news feeds. Then recrawl the miner search results and build your own list or collection of news feeds. Supports the popular RSS and Atom formats, among others.
Use the results from this miner in combination with other tools we developed for data mining URLs, or if you prefer submit your own page listing news feeds and improve the web visibility of your news.
This miner comes handy, particularly now that Google News has disabled its RSS subscription features, breaking tons of feeds and, in the process, affecting their presence on the web. (https://www.seroundtable.com/google-news-rss-feed-gone-25795.html).
