On the Nonadditivity of Correlation Coefficients Part 2: Fisher Transformations

Tags

Correlation Coefficients, Data Mining, Mathematics, Pearson Correlation, Spearman Correlation, statistics, tutorials

This is Part 2 of a tutorial series on the nonadditivity of correlation coefficients. This time we discuss Fisher r-to-Z and Z-to-r transformations and the risks of arbitrarily implementing these.

Misusing the transformations can have a detrimental impact on significance testing, confidence intervals, reported standard errors of correlations, and meta-analysis.

The tutorial is available at

http://www.minerazzi.com/tutorials/nonadditivity-correlations-part-2.pdf

 

 

On the Nonadditivity of Correlation Coefficients Part 1: Pearson and Spearman Coefficients

Tags

Correlation Coefficients, Data Mining, Mathematics, statistics, tutorials

This is Part 1 of a tutorial series on the nonadditivity of correlation coefficients. We demonstrate why it is not possible to arithmetically add, subtract, and average Pearson’s r or Spearman’s r_s.

The article is available at

http://minerazzi.com/tutorials/nonadditivity-correlations-part-1.pdf

07-04-2017 update: In page 1 the line for the Beta₁ should read “is the slope of a simple linear regression model”. My fault. Fixed today along with few other nuances.

Enjoy it!

Python Search Engine

Tags

Algorithms, Data Mining, Mathematics, open source, Python, R, Search, tutorials

Here is a python-based search engine with an implementation inspired on one of our papers at the old Mi Islita.com site, now a search engine on Puerto Rico.

http://nullege.com/codes/show/src@g@e@gensim-HEAD@gensim@test@test_miislita.py

This module replicates the miislita vector spaces from

"A Linear Algebra Approach to the Vector Space Model -- A Fast Track Tutorial"

by Dr. E. Garcia...

 

Great and positive accomplishment!

That tutorial is no longer at miislita.com, but was long ago moved to minerazzi.com. Find it here:

http://www.minerazzi.com/tutorials/term-vector-linear-algebra.pdf

For other resources do a search for python in our IR miner at

http://www.minerazzi.com/irc

For inquiries about that implementation, contact its author.

For other inquiries, applications, suggestions, drop me a line.

PS. Please note that Nullege.com itself is a search engine for finding python code. Here is a good example: http://nullege.com/codes/search/wx.calendar.CalendarCtrl

Excel File for Quantile-Quantile Analysis

Tags

Data Mining, Mathematics, statistics, tutorials

This is an Excel .xlsx file for reproducing Table 1 of our tutorial on Quantile-Quantile Plots. Now anyone with Excel installed can play and explore this simple technique aimed at determining if a data set is normally distributed.

To download the Excel file, access the most recent update of the tutorial, available at

http://www.minerazzi.com/tutorials/quantile-quantile-tutorial.pdf

We also removed few extra “)” typos that were undetected in previous copies.

Have a great Q-Q day! 🙂

BM25F Model Tutorial

Tags

Algorithms, Data Mining, tutorials

We have restored, expanded, and updated our tutorial on the BM25 Extension to Multiple Weighted Fields Model, best known as BM25F. It is now available at

http://www.minerazzi.com/tutorials/bm25f-model-tutorial.pdf

Active links were also added to the References section.

Enjoy it.

Box-Cox Power Transformations Tool

Tags

Data Mining, Mathematics, statistics, tools, tutorials

This is a new statistical and data mining tool, available now at

http://www.minerazzi.com/tools/power-transformations/box-cox.php

It greatly simplifies the work of those dealing with data transformation problems.

Enjoy it.

About the tool:

• This tool lets you transform a data set by applying one or more Box-Cox Power Transformations. The research articles given in the References section of the tool cover this topic.
• To use the tool enter one data set value per line. End each line by hitting the Enter key so these are recognized as individual entries.
• To apply multiple transforms, check preset field.
• To apply a single transform, uncheck preset field and enter a p value (p ∈ [-2,+2]).
• Submit or reset form as needed.

10-02-2016 Update:

We added a new feature to the tool so it now lets users return all non-negative transforms.

OKAPI BM25 Tutorial

Tags

Algorithms, ir, minerazzi, tutorials

We have restored, refined, and updated this tutorial and added some historical background.

Abstract

This is a light tutorial on OKAPI BM25, a Best Match model where local weights are computed as parameterized frequencies and global weights as RSJ weights. Local weights are based on a 2-Poison model and the verbosity and scope hypotheses and global weights on the Robertson-Spärck-Jones Probabilistic Model.

Introduction

In the early 80s Gillian Venner, Nathalie Mitev, and Stephen Walker (1985, 1987) conducted research work that led to the design and evaluation of online public access catalogs (OPACs) at Polytechnic of Central London (PCL).

The project initial phases spanned from November 1982 to May 1985. The prototype was named OKAPI (Online Keyword Access to Public Information). As Mitev (1985) wrote:

“Designing an online public access catalogue [OPAC]: Okapi, a catalogue on a local area network [LAN] is the final report of a two-year research  project ”Microprocessor networking in libraries” which was funded by the British Library and the Department of Trade and Industry, and based at the Polytechnic of Central London.”

“The aim was to produce an OPAC on a LAN, that would be readily usable without training or experience, without sacrificing effectiveness or being tedious for experienced users.”

“The result was a functioning prototype OPAC called Okapi, which has a number of distinctive features: use is eased by coloured keys and a lack of jargon; the system uses search decision trees to select a suitable action at each stage of a search, and it performs automatic Boolean and hyper-Boolean functions where appropriate. The OPAC was installed and evaluated in one of the Polytechnic site libraries.”

Want more? Read the tutorial at

http://www.minerazzi.com/tutorials/okapi-bm25-model.pdf

Probabilistic Model Tutorial

Tags

information retrieval, ir, minerazzi, tutorials

This is an updated version of a tutorial on the Robertson-Spärck-Jones Probabilistic Model.

It is available now at

http://www.minerazzi.com/tutorials/probabilistic-model-tutorial.pdf

The model computes global weights, known as RSJ weights, based on Independence Assumptions and Ordering Principles for probable relevance. The model subsumes IDF and IDFP as RSJ weights in the absence of relevance information.

Enjoy it.

09-26-2016 Update: A new section was added to the tutorial before the Conclusion section. References were added accordingly. Few lines were edited.

PS: I corrected the original publication date to read “Published: 03-30-2009” which is the correct date. My fault.

72 Binary Similarity Measures

Tags

Algorithms, bioinformatics, chemical mining, chemistry, Data Conversion, data miners, Data Mining, information retrieval, ir, minerazzi, miners, mining, news, social mining, statistics, tools, tutorials, Vector Space Models

We have expanded the number of similarity measures that our Binary Similarity Calculator computes from 30 to 72 (and counting…)

Same measures with different names have been consolidated into a single record, and different measures with same name have been enumerated as necessary.

These similarity coefficients have many applications across disciplines: from bioinformatics to chemistry, chemometrics, statistics, data mining, information retrieval, marketing research, etc.

The tool is available at

http://www.minerazzi.com/tools/similarity/binary-similarity-calculator.php

We have also included the new similarity measures proposed by Consonni & Todeschini (2012), and Todeschini, et al (2012).

Our Tutorial on Distance and Similarity was also updated, accordingly. Check it out at

http://www.minerazzi.com/tutorials/distance-similarity-tutorial.pdf

 

References

Consonni, V. and Todeschini, R. (2012). New Similarity Coefficients for Binary Data. MATCH Commun. Math. Comput. Chem. 68, 581-592.

Todeschini, R., Consonni, V., Xiang, H., Holliday, J., Buscema, M., and Willet, P. (2012). Similarity Coefficients for Binary Chemoinformatics Data: Overview and Extended Comparison Using Simulated and Real Data Sets. J. Chem. Inf. Model. 52 (11).

BM25IR (Updated Version)

Tags

Algorithms, Data Conversion, Data Mining, information retrieval, ir, miner, minerazzi, news, text mining, tutorials

An updated version of the BM25IR paper is available at

http://www.minerazzi.com/tutorials/bm25ir.pdf

Essentially we corrected few typos and added one line to explain even more the transformation of (a) into (b) in Figure 1 of the paper.

Other than that, its content and thesis remains the same.

For the sake of transparency, we keep the older version accessible through the following services:

ArXiv.org: http://arxiv.org/abs/1608.01573

ScienceWISE: http://sciencewise.info/articles/1608.01573/

SAO/NASA ADS: http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1608.01573

SciRate: https://scirate.com/arxiv/1608.01573

Science.io: http://science.io/paper/4005211/local-term-weight-models-from-power

MathPubs: http://www.mathpubs.com/detail/1608.01573v1/Local-Term-Weight-Models-from-Power-Transformations-Development-of-BM25IR-A-Best-Match-Model-based-o

Sparrho: https://www.sparrho.com/item/local-term-weight-models-from-power-transformations-development-of-bm25ir-a-best-match-model-based-on-inverse-regression/961979/

Paperspire: https://www.paperspire.com/?paper_id=17197