RBlogs is a new miner built with Minerazzi and accessible at http://www.minerazzi.com/rblogs.
This is a huge collection of blogs from and for the R community. It comes with R-specific rss news gathered with our SPP service.
We are testing a random rss parser that works as an unfocused news aggregator. We are already working on a second version to make it focused, but it will take quite a bit.
The goal is to develop a tool that not just will deliver topic-specific RSS feeds to end users, but one capable of data mining said RSS news readers.
In the meantime, you can try the unfocused version by visiting any of our current miners, available at http://www.minerazzi.com.
The US Army Miner is now available at http://www.minerazzi.com.
Mine Army divisions, units, battalions, breaking news, military social media, and more. Build US Army specific collections. Search by military resource or location.
PS. Amazing news obtained through the News Center of this miner
Imagine controlling things or communicating to each others through our brain waves! Coming soon?
Wikiminer is a new miner built with Minerazzi ( http://www.minerazzi.com/wikiminer ) exclusively for mining Wikileaks.
It allows users to find secret information, news leaks, and classified media from anonymous sources by mining Wikileaks link graph. Search by keywords or location.
Query [ cablegate ] in this miner. Then locate the result whose URL is https://wikileaks.org/cablegate.html and click the Search Inside tool icon below said result. By recursively searching inside a result you will be walking a portion of Wikileaks link graph.
Open Source Projects is a new miner available at http://www.minerazzi.com/osp. It allows you to find or submit all kind of open source projects. Access open source community resources. Search by software, hardware, or project name.
Looking for open source projects relevant to Apache, Linux, or Windows? Need to be more specific in your search (e.g. search for Weka, JQuery, Aptana, JNode, Ubuntu, Mozilla, etc…)? Want to build your own open source collections? If so, this miner is for you.
Wikileaks.org is one of those huge sites where researchers and investigative reporters can feel like in heaven.
That is, provided that they have a way to move across Wikileaks complex link structure. Simply put, they need a tool that allows them to understand the relationships between links and quickly move in and out of specific link paths of interest. This need to be done at different levels of the link graph, while current resources are pulled out of said structure and in almost real time.
That is hard to do by just searching or by crafting site, command, or custom searches–not even by using Wikileaks own search engine.
Fortunately, you can do the above with Minerazzi recrawling features–at least to some degree.
Although Minerazzi technology is evolving and not perfect, moving from searching indexes to mining user-driven recrawls is a right step in the right direction.
However, there might be a broad spectrum of starting experimental conditions, each one requiring of different crawling strategies.
The purpose of this post is not to discuss solutions for all possible experimental conditions. It is assumed that users are familiar with Minerazzi’s Recrawl It (RI) and Search Inside (SI) complementary tools. To simplify, the recrawls are done with SI
Example 1: Initial URL is not given.
Search for [wikileaks] in the Investigative Journalism miner (http://www.minerazzi.com/journalism). Find a result that might interest you.
A good starting point is the result whose URL is https://www.wikileaks.org/wiki as it contains links to latest leaks and recent analyses. Click the Search Inside tool icon below this result.
That should retrieve all links from this result with the tool icon now at the right of each of the new results. You should see three output sections. The first one logs the current URL being crawled. The other two’s are the External and Internal Links sections.
You can now recursively recrawl results by clicking their SI icon and, again, check how the above sections are updated. That is, you will be walking a portion of Wikileaks link graph. At any given step you can walk backward or forward the link graph by clicking the SI icons from the above sections.
This mechanism works as expected with the latest versions of Firefox, Opera, Safari, and Chrome browsers. However, sometimes the state of the logged section is not preserved in IE. We are working on fixing this anomaly.
Example 2: Initial URL is given.
If the initial URL is given or obtained through a search or previous crawl, recrawl its links as in Example 1. A good starting point is https://www.wikileaks.org/the-spyfiles.html
You can always submit for indexing in the above miner a particular Wikileaks URL. Once indexed, you can use it as a starting point.
Example 3: What if I still want to combine searching with recrawling?
You can always do that. Wikileaks link graph has many URLs with the pattern [keyword].wikileaks.org which can be easily mined.
For instance search for [file wikileaks] and recrawl with SI the result whose URL is https://file.wikileaks.org. Next from the results page recrawl the result whose URL is https://file.wikileaks.org/file. You will be presented with over a thousand of interesting results. Have a field day!
What is next?
Because Wikileaks is so huge, perhaps it is time for us to start building a miner exclusively for mining Wikileaks.org site. Such a miner will help us to address initial starting point and link walk issues.
The news miner (http://www.minerazzi.com/news) was built for indexing and mining newspapers. However, you can use it to mine news aggregation sites like HuffingtonPost, DrudgeReport, Topix, Google News, Yahoo News, Bing News, and many more. Just visit the above link and search for any of those sites.
After that you can recursively crawl these with Minerazzi’s Search Inside and Recrawl It tools. These are complementary tools so if one returns no results, try the other one.
To illustrate, the HuffingtonPost and DrudgeReport are two of the best user-friendly and content-rich news sites on the Web. These are great sources for building news collections about relevant topics like politics.
By searching for [ huffingtonpost ] or for [ drudgereport ] you can discover additional news services and even follow specific authors and their posts. You can then start building curated collections of news services, authors, and their posts.
When building collections from news services, if a remote host is busy you may want to retry it at another time. However, if the remote host denies you service you are out of luck. This is not really a drawback. As there are zillion of friendly hosts out there that will provides you with rich content, the ones that eventually refuse connection are expendable.
At the time of writing, curated collections of topics and authors can be easily done by mining Google Scholar results with Minerazzi. This can be illustrated with the following examples.
Mining Topics Example
1. Search [ pagerank ] with the Information Retrieval Collection miner at http://www.minerazzi.com/irc
2. Locate the result whose URL is http://scholar.google.com.pr/scholar?q=pagerank and click the Search Inside tool icon located below said result.
3. Note from step 2 output that for Google Scholar some of the links discovered by the tool are about co-authors discovered by Google. Locate a co-author and click the Search Inside tool icon, this time located the right of said result.
4. You will be presented with a new list of results each with the Search Inside icon. Some of these include co-authors.
By recursively using Search Inside you can build a curated collection on the pagerank topic or a curated collection of co-authors, without having to resubmit the query.
This approach assumes that the initial Google Scholar URL to be mined is already in the IRC microindex. For other queries, you need to query Google Scholar and submit for indexing in IRC the search results URL. Once indexed, it can be mined as described above.
However, if a user discovers a Google Scholar URL when using the Search Inside tool on a previous result, said URL can be recrawled and mined as described above, so it no need to be in the IRC microindex at all.
In general, any URL searchable with Search Inside can be mined, unless the tool hits a dead end (no links accessible or to follow).