New similarity measures

We have added a new round of similarity measures to our Binary Similarity Calculator, for a total of 30.

http://www.miislita.com/binary-similarity-calculator/binary-similarity-calculator.php

We plan to add few more in the future, so more data miners, researchers, and students can benefit from it. We plan to make this a comprehensive project on similarity analysis. For this purposes, several doctoral theses are deployed. Any help, correction, feedback is appreciated.

The Binary Distance Calculator – a tool for comparing binary sets

The Binary Distance Calculator, a new tool for computing the distance or dissimilarity (lack of resemblance) between any two binary sets of same size is available now at http://www.miislita.com Its FAQs section includes a clear definition of distance in the context of Information Retrieval and Mathematics.

This tool was developed to complement The Binary Similarity Calculator, one of our popular tools.

The Binary Similarity Calculator

We have just launched The Binary Similarity Calculator. This is a new tool for computing binary-based similarity measures that is available now.

What it is

The Binary Similarity Calculator (BSC) can be used to compare binary sets, groups consisting of only two types of items or states. These are item sets that can be represented as sequences of 1′s and 0′s.

Who can benefit from it

• Marketing analysts that need to examine Yes/No-type questionnaires about products and services.

• Teachers and examiners that must score Yes/No-type exams or assess plagiarism cases.

• Engineers, mathematicians, and physicists that must evaluate On/Off-type records.

• Statisticians, bioanalysts, and others involved with sequencing analysis.

• To sum up, anyone that uses binary sets.

A Quantile-Quantile (Q-Q) Plot Tutorial in Excel

Although no data set is exactly normally distributed, most statistical analyses require that the data be approximately normally distributed for their findings to be valid. One way of testing for normality is through a quantile-quantile (q-q) plot, a technique for determining if data sets originate from populations with a common distribution.

In this tutorial, you will determine if a data set is normally distributed by comparing its quantiles against those of a theoretical normal distribution. You will also learn how to make a data set nearly normally distributed.

http://www.miislita.com/statistics/quantile-quantile-plot-tutorial.pdf

Oct 3, 2012 Update: I’ve added a new figure to the article and reworked few lines.

Correlation coefficients: How many decimal places to use?

That is an interesting question that Jacob Cohen’s addressed in
Things I have learned (So Far)

He writes:

“A less profound application of the less-is-more principle
is to our habits of reporting numerical results. There
are computer programs that report by default four, five,
or even more decimal places for all numerical results.
Their authors might well be excused because, for all the
programmer knows, they may be used by atomic scientists.
But we social scientists should know better than to
report our results to so many places. What, pray, does an
r = .12345 mean? or, for an IQ distribution, a mean of
105.6345? For N = 100, the standard error of the r is
about . 1 and the standard error of the IQ mean about
1.5. Thus, the 345 part of r = .12345 is only 3% of its
standard error, and the 345 part of the IQ mean of
105.6345 is only 2% of its standard error. These superfluous
decimal places are no better than random numbers.
They are actually worse than useless because the clutter
they create, particularly in tables, serves to distract the
eye and mind from the necessary comparisons among the
meaningful leading digits. Less is indeed more here.”

End of the quote.

So, r = 0.12 is good enough.

Some statistical tables use more than 2 decimal places, but then is up to the writer to use two decimal places in the actual experimental results.

Hey SEOs. Read This: Is a Sample Size of 30 all You Need?

That was the question that Dr. Chuck Chaprakani addressed in this interesting article.

It reminds me of Jacob Cohen’s Things I have learned (So Far) article.

Yes, beware of the so-called sample-size-of-30 “magic” number.

Things to come: Q-Q Normality Test Tutorial

I’ve been very busy putting together an article to be submitted to a research journal on Statistics. I hope soon to be free to blogging again.

Under the pipeline, I’m planning to publish online a tutorial on Q-Q Normality tests using just Excel. The procedure is pretty much straightforward and can be used to optimize power transformations (Tukey, Box-Cox, etc).

How the Media Misunderstood Shouryya Ray’s Solution

Beware of the media when they report solutions to problems allegedly posed by Newton.

A current claim is that a 16-year old kid that goes by the name of Shouryya Ray solved a 350-year old problem allegedly formulated by Sir Isaac Newton.

An article putting in the right perspective Ray’s solution is given here.

Still, Ray must be praised for solving what old school-trained mathematicians and physicists were not able to solve.

On the Additivity, or Lack of it, of Variables

Ignoring the additivity nature, or lack of it, of a variable can invalidate any statistical treatment. This is part of a research paper on the Self-Weighting Model (SWM) that I’m writing. I presented a sneak preview on this during a recent seminar before Fundación de Investigación, a local research company dedicated to clinical trials.

In general, if x is a non-additive variable, we cannot:

• obtain a mean (arithmetic average) x value from a set of x’s.
• calculate a standard deviation from a set of x’s.
• mean-center a set of x’s by subtracting a mean score from each x.
• standardize a set of x’s by subtracting a mean score from each x and dividing each by a standard deviation score (i.e., convert each x into a z score) .
• compute the L1-norm (Manhattan, taxicab distance) from a vector whose elements are a set of x’s.

[Added on 6-21-2012] In addition for said variable, we cannot:

• take the difference between any two pairs of x values.
• compute a coefficient of variation (standard deviation/mean), from a set of x values.
• compute a mean difference from any two sets of x values.
• compute a pooled standard deviation from any two sets of x values.
• compute a Cohen’s d (mean difference/pooled standard deviation) from any two sets of x values.

In a math/statistics scenario, the following are non-additive: slopes, cosines, sines, tangents, and ratios (e.g., standard deviations, correlation coefficients, beta coefficients, coefficients of variations, Cohen’s d, etc).

In an experimental sciences scenario, the following are non-additive: any intensive property, any ratio of extensive properties (e.g., density = mass/volume), any dissimilar ratio, etc.

The information presented in the aforementioned seminar is applicable to many dissimilar fields, including web analytics, data mining, information retrieval, and almost any research field that requires of numerical analysis of experimental variables.

Unfortunately, from time to time we see research articles published wherein the additivity/non-additivity nature of variables is ignored and data crunching and analysis is arbitrarily carried out. The result: sloppy approximations, invalid models, and erroneous forecasts.

PS. There might be counterexamples to the notion of classifying properties as intensive or extensive. Very few properties are neither one. There are also some cases wherein instrumental lectures (not properties) are subtracted in order to compute signal responses or signal-to-noise ratios, derivatives, numerical analysis, etc, but for the most part the above holds in the physical world.

When Big Data leads to Big Errors

The hilarious picture above shows how some SEOs look when playing to be scientists. This often occurs when interpreting big data.

Few specific scenarios:

1. Applying the statistical theory of small samples to extremely large samples, like …
2. …using large amount of data to force very small correlation coefficients to become statistically significant.
3. Trying to arithmetically average ratios (like correlation coefficients, standard deviations, slopes, and cosine similarities).
4. Mistaking Cauchy Distributions for Normal Distributions.
5. Adding together intensive properties.

Fortunately, I know of good folks that are doing a great job at educating their search marketing peers (Mike Grehan, Bruce Clay, Danny Sullivan, etc) without playing to be scientists.