Hey SEOs. Read This: Is a Sample Size of 30 all You Need?

That was the question that Dr. Chuck Chaprakani addressed in this interesting article.

It reminds me of Jacob Cohen’s Things I have learned (So Far) article.

Yes, beware of the so-called sample-size-of-30 “magic” number.

When Big Data leads to Big Errors

The hilarious picture above shows how some SEOs look when playing to be scientists. This often occurs when interpreting big data.

Few specific scenarios:

1. Applying the statistical theory of small samples to extremely large samples, like …
2. …using large amount of data to force very small correlation coefficients to become statistically significant.
3. Trying to arithmetically average ratios (like correlation coefficients, standard deviations, slopes, and cosine similarities).
4. Mistaking Cauchy Distributions for Normal Distributions.
5. Adding together intensive properties.

Fortunately, I know of good folks that are doing a great job at educating their search marketing peers (Mike Grehan, Bruce Clay, Danny Sullivan, etc) without playing to be scientists.

Which separators to use with title tags?

More and more SEOs are using separators like pipes, dashes, commas, etc when writing title tags. (Exclude underscores from the list, which are used as concatenators.)

Which of these separators perform better?

From time to time some SEO “experts” and their cheerleaders promote the idea that a particular separator performs better over others. See the following link:

http://searchenginewatch.com/article/2154469/How-to-Write-Title-Tags-For-Search-Engine-Optimization

And that is despite the fact that Google’s Matt Cutts has mentioned in two different videos that there is no real significative advantage of using one over the other. These videos are available at the following links:

http://www.youtube.com/watch?v=T2_7PTio3Qc

http://youtu.be/jHSqLYUPq8w

My take?

I would like to see an analytical study supporting the facts. I believe this is reasonable to ask. Don’t you think so?

Starting 2011 with a great research

Nothing better than starting 2011 with more research work.

Check this blog tomorrow as there will be a value-added good news for those interested in conducting research at the interface of information retrieval, statistical analysis, and applied mathematics. You’re welcome to grab a copy of this four-month investigation, for use in your own research, as a teaching tool, or to chase away SEO snakeoil.

Understanding Correlation

I came across Professor R.J. Rummel page on Understanding Correlation. This is an old, but still relevant book-like Web page on how to interpret properly correlation coefficients.

In Chapter 4 he discusses on the proper way of looking at correlation coefficient values. He writes and quote (emphasis added in boldfaces):

“As a matter of routine it is the squared correlations that should be interpreted. This is because the correlation coefficient is misleading in suggesting the existence of more covariation than exists, and this problem gets worse as the correlation approaches zero. Consider the following correlations and their squares.”

“Note that as the correlation r decrease by tenths, the r² decreases by much more. A correlation of .50 only shows that 25 percent variance is in common; a correlation of .20 shows 4 percent in common; and a correlation of .10 shows 1 percent in common (or 99 percent not in common). Thus, squaring should be a healthy corrective to the tendency to consider low correlations, such as .20 and .30, as indicating a meaningful or practical covariation. “

Rummel’s page is very relevant these days where SEOs from SEOMOZ and few other snakeoil marketing sites are buying the bogus discourse from Fishkin and Hendrickson that low correlation coefficients in about that range are evidence of LDA scores and Google ranks being “highly” correlated.

As mentioned before at this blog, SEO marketers are good at selling that kind of snakeoil or “quack” science.

Statistical significance does not equate to high correlation. For large enough sample sizes even very low r values (0.1, 0.01, etc) eventually become significant, but these do not equate to high correlation.

On a side note, I’m reading an IR thesis wherein Spearman’s and Kendall’s coefficients are used. Quite interesting.

First PS

According to a Sloan Consulting article at ISIXSIGMA.COM site and quote (emphasis added)

“As a rule of thumb a strong correlation or relationship has an r-value range of between 0.85 to 1, or -0.85 to -1. In a moderate correlation, the r-value ranges from 0.75 to 0.85 or, -0.75 to -0.85. In a weak correlation, one that is not a very helpful predictor, r ranges from 0.60 to 0.74 or -0.60 to 0.74. Though an entirely random relationship equals, 0.00, any relationship that has a correlation r-value that is 0.59 and below is not considered to be a reliable predictor.”

According to this Intel Teach Program a correlation between 0 and 0.19 is a very weak one while one between 0.2 and 0.39 weak enough.

True that there are many correlation charts out there and some do not agree in specific degrees or ranges, but they all tend to agree in one thing: that a correlation value below 0.20 is a very, very weak correlation, never deemed as evidence of variables being “highly correlated” as claimed by SEOMOZ in their LDA fiasco posts.

Second PS

Here is a list of reference links wherein these marketers make correlation claims based on quite weak correlation values and in the process keep misleading naive peers and the public. “Highly correlated”? “Remarkably well correlated?” Evidently Statistics is a Loss for SEOs.

http://www.seomoz.org/blog/lda-correlation-017-not-032

http://www.seomoz.org/blog/lda-and-googles-rankings-well-correlated

http://www.seomoz.org/blog/google-vs-bing-correlation-analysis-of-ranking-elements

On Statistical Significance and SEO Statistical “Studies”

Back in 2008, Jan M. Hoem wrote an interesting reflexion paper titled “The reporting of statistical significance in scientific journals” (VOLUME 18, ARTICLE 15, PAGES 437-442; 03 JUNE 2008 http://www.demographic-research.org/volumes/vol18/15/18-15.pdf . The piece was an expanded version of a previous paper (http://www.demogr.mpg.de/papers/working/wp-2007-037.pdf )

He wrote (and I quote):

“Scientific journals in most empirical disciplines have regulations about how authors should report the precision of their estimates of model parameters and other model elements. Some journals that overlap fully or partly with the field of demography demand as a strict prerequisite for publication that a p-value, a confidence interval, or a standard deviation accompany any parameter estimate. I feel that this rule is sometimes applied in an overly mechanical manner. Standard deviations and p-values produced routinely by general-purpose software are taken at face value and included without questioning, and features that have too high a p-value or too large a standard deviation are too easily disregarded as being without interest because they appear not to be statistically significant. In my opinion authors should be discouraged from adhering to this practice, and flexibility rather than rigidity should be encouraged in the reporting of statistical significance. One should also encourage thoughtful rather than mechanical use of p-values, standard deviations, confidence intervals, and the like. Here is why:”

Hoem then dissects five points related with misusing statistical significance results and automatic software solutions. I’m listing these below.

1. The scientific importance of an empirical finding depends much more on its contribution to the development or falsification of a substantive theory than on the values of indicators of statistical significance.
2. Measures of statistical significance may be misleading. When a model has been developed through repeated use of tests of significance to include and exclude covariates, to split or combine levels on categorical covariates, and to determine other model features, the user often loses control over statistical-significance values, and the values computed by standard software may be completely misleading.
3. Standard p-values can be insufficiently precise indicators of statistical significance, particularly if their values are given only in grouped levels, which are often indicated by asterisks beside parameter estimates (“* = p<0.1, ** = p<0.05, *** = p<0.01”, and so on).
4. It may be more important for an understanding of demographic behavior or other phenomena studied to know whether the inclusion of a categorical covariate in its entirety contributes significantly to an improvement of the model than to know the significance indicators of each of its levels.
5. Standard deviations, when used, should be reported for interesting contrasts, not for features selected automatically by statistical software.

I completely agree with Hoem.

SEOMOZ and their statistical “studies”

These days search engine optimization marketers (SEOs/SEMs) keep misinterpreting statistical results spitted from software without stopping and thinking about the significance-behind-the-significance, especially when it comes to a correlation coefficient, r (Pearson, Spearman, etc).

When one reads SEO hearsays and urban legends at SEOMOZ about very small correlation coefficients (0.17, 0.32, etc) derived from large sample sizes, as evidence that variables are “highly correlated” or “well correlated”, it is time to stop and put into question such “studies”. For reference see the following links

http://www.seomoz.org/blog/lda-and-googles-rankings-well-correlated
http://www.seomoz.org/blog/lda-correlation-017-not-032
http://irthoughts.wordpress.com/2010/04/23/beware-of-seo-statistical-studies/  

Fortunately, leading search marketers like Danny Sullivan has put into question those “studies” at a recent search engine conference

http://outspokenmedia.com/internet-marketing-conferences/evening-forum-with-danny-sullivan/ ,

and that was even before SEOMOZ admitted the 0.32 result has to be recanted as 0.17.

Sean Golliher, founder and publisher of the Search Engine Marketing Journal (SEMJ.org) has also put into question their results (http://www.seangolliher.com/2010/uncategorized/185/ ), which Hendrickson from SEOMOZ still insists in defending.

Since then they have never disclosed the source of the mistake, dismissing it just as a programming error. Unfortunately, they are still claiming that a 0.15 – 0.30 range validates their “studies” (http://www.seo.co.uk/seo-news/seo-tools/the-seomoz-lda-tool-%E2%80%93-our-disappointing-findings.html) .

Small and Large Sample Sizes

When William Sealy Gosset (aka “Student”) proposed, and Ronald A. Fisher expanded on, the test later termed Student’s t-test of significance, the test was meant to be used to assess information from small samples, not from large samples. I have discussed the case of small sample sizes in another post (http://irthoughts.wordpress.com/2010/10/18/on-correlation-coefficients-and-sample-size/ ).

In order to apply a t-test (and other small sample analysis tests) to large samples, divide-and-conquer techniques, like stratification, were eventually developed. In the case of correlation and regression, the reason for doing this is that applying something like a t-test to, for instance, a single correlation coefficient coming from a huge sample can produce misleading results. Let see why.

For large enough sample sizes eventually any correlation coefficient, even the smaller ones, will always be significant (t-observed > t-table). At that point it might be tempting to assume that the variables in question are highly correlated. Wrong assumption!

The fact is that statistical significance does not necessarily equate to variables being highly correlated and vice versa. Let address this point in two parts: (1) the question of statistical significance and (2) the question of high correlation.

Statistical Significance: Bigger not always is better

As noted in a Wikipedia entry, “given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero. (http://en.wikipedia.org/wiki/Effect_size ). I have discussed effect size and power analysis in a previous post (http://irthoughts.wordpress.com/2010/10/21/on-power-analysis-and-seo-quack-science/  ).

The reason for the above effect has a lot to do with the definition of statistical significance itself. Statistical significance is the confidence one has in a given result and that such a result is not by random chance.

In mathematical terms, the confidence that a result is not by random chance is given by the following formula by Sackett (http://en.wikipedia.org/wiki/Statistical_significance , http://www.cmaj.ca/cgi/content/full/165/9/1226 ):

Confidence = (Signal/Noise)*Sqrt[Sample Size]

This simple expression or derivatives of it appears in many different scenarios and disciplines. It describes a generic Confidence Function, F, in terms of a Signal, a Noise, and a Sample Size; that is, F(Signal, Noise, Sample Size). In general, such a generic function tells us that:

• Confidence is proportional to a Signal source (S).
• Confidence is inversely proportional to a Noise source (N).
• Confidence is proportional to a Signal-to-Noise ratio (S/N).
• Confidence is proportional to a Sample Size.

Let’s apply a version of this expression to correlation. To do this, let Y be the dependent variable and let X be the independent variable. Let also make the following substitutions:

• Confidence: expressed as t²
• Signal: expressed as r²; i.e., fraction of explained variations in Y (due to X).
• Noise: expressed as 1 – r²; i.e., fraction of unexplained variations in Y.
• Sample Size: expressed as degrees of freedom; i.e., n – 2 for a two-tailed test.

F(Signal, Noise, Sample Size) = t-observed² = [r²/(1 – r²] [n – 2]

Taking the square root (Sqrt) at both sides, we obtain the so-called formula for a two-tailed t-test.

t-observed = r*Sqrt[(n – 2)/(1 – r²)]

Evidently for a given r value, t-observed increases when n increases. By rearranging this expression, it is possible to compute for a large enough sample size a critical value above which r values will be significant. For very large samples at a 95% confidence level, t-table= 1.96 (http://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values ). Replacing arbitrarily this value in the above expression (t-observed = t-table = t = 1.96) and solving for r, we obtain that the critical r value is given by

r = t/Sqrt[(n – 2) + t²]

The following table lists values for very small r values and huge sample sizes. I’m intentionally using several decimal places and ignoring significant figure rules since I want to make a point on the small values used. I’m also using a 0.95 confidence level for illustration purposes, but for the large samples I could and should use other confidence levels as well.

n	n – 2	t	r	S = r*r	N = 1 – r*r	S/N
1,000	998	1.96	0.0619	0.003835	0.996165	0.003849
10,000	9998	1.96	0.0196	0.000384	0.999616	0.000384
100,000	99998	1.96	0.0062	0.000038	0.999962	0.000038
1,000,000	999998	1.96	0.0020	0.000004	0.999996	0.000004

For a sample size of 10,000 observations the critical r is 0.0196 or about 0.02, meaning that for such a huge sample size any r value above this small and critical r value will be significant. However, something interesting is observed from this table: (PS See footnote update)

When one moves to large sample sizes the Noise becomes greater than the Signal. For instance, at n = 10,000 the amount of Signal is very small (0.000384) while the amount of Noise is above 0.9996… or 99.96…%, giving a quite trivial S/N ratio. A similar reasoning can be applied for r = 0.17 (S = 0.0289, N = 0.9711) and r = 0.32 (S = 0.1024, N = 0.8976). The corresponding S/N ratios are trivial.

One can also solve the above expression for n to find sample sizes for some small r values and arbitrary t as shown in the following table (PS See footnote update).

t	r	S = r*r	N = 1 – r*r	S/N	n	n – 2
1.96	0.0200	0.000400	0.999600	0.000400	9602	9600
1.96	0.1500	0.022500	0.977500	0.023018	169	167
1.96	0.1700	0.028900	0.971100	0.029760	131	129
1.96	0.3000	0.090000	0.910000	0.098901	41	39
1.96	0.3200	0.102400	0.897600	0.114082	36	34

Still, note that the amount of Noise completely overcomes the Signal, producing trivial S/N ratios. In general for small r and large n values significance is achieved at the cost of Noise masking the Signal. When this occurs the statistical significance is not a practical guideline for drawing useful conclusions from the data at hand.

This drives the present discussion to the substantive part of the problem missed by SEOs, and that is …

Statistical Significance Does Not Necessarily Mean Highly Correlated Results

Simply stated, statistical significance does not necessarily imply that the X, Y variables are highly correlated.

A simple scatterplot will convince anyone that for the above small r values there will be no pattern or trend in the data set. The corresponding regression model will be useless for forecasting or inferring anything of value, except that the data spreads so wildly that it has no method to its chaos. What else is to be expected from a data set with a large Noise and small S/N ratio?

This is something that SEOs/SEMs still don’t seem to understand: t-observed > t-table not necessarily means high correlation, and vice versa. I don’t have any personal stake (or take) against them, but when folks like Hendrickson, Fishkin, and others from SEOMOZ ignore Signal-to-Noise ratios and start referring to small r values as evidence that experimental variables are “highly” or “well” correlated, it is more than fair to call such “studies” Quack “Science”. That label might sound harsh, but in this case is appropriate.

Search engine marketers might be good at selling snakeoil, publishing sloppy “studies”, or recanting on overhyped statements, but not at doing real Science. They should know better; i.e., that

• “significance” does not mean “correlation”.
• “significance” does not mean “important”.
• “insignificance” does not mean “unimportant”.

Statistical “significance” only means that any confidence in the data is not by random chance. Therefore, a significant correlation does not necessarily mean a “high”, “well”, or “strong” correlation between variables.

To understand all this we need to distinguish between statistical significance and practical significance.

Statistical Significance vs. Practical Significance

As stated at this Wikipedia entry (emphasis added in boldfaces) http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Criticism ):

A common misconception is that a statistically significant result is always of practical significance, or demonstrates a large effect in the population. Unfortunately, this problem is commonly encountered in scientific writing. Given a sufficiently large sample, extremely small and non-notable differences can be found to be statistically significant, and statistical significance says nothing about the practical significance of a difference.

Use of the statistical significance test has been called seriously flawed and unscientific by authors Deirdre McCloskey and Stephen Ziliak. They point out that “insignificance” does not mean unimportant, and propose that the scientific community should abandon usage of the test altogether, as it can cause false hypotheses to be accepted and true hypotheses to be rejected.

Some statisticians have commented that pure “significance testing” has what is actually a rather strange goal of detecting the existence of a “real” difference between two populations. In practice a difference can almost always be found given a large enough sample. The typically more relevant goal of science is a determination of causal effect size. The amount and nature of the difference, in other words, is what should be studied. Many researchers also feel that hypothesis testing is something of a misnomer. In practice a single statistical test in a single study never “proves” anything.

That pretty much settles the question of discerning between statistical significance and practical significance of correlation coefficients, but does not tell us how to quantitatively discern between the two concepts. In an upcoming article, I will derive expressions that might help to quantitatively assess these.

Since the tutorial on correlation coefficients http://www.miislita.com/information-retrieval-tutorial/a-tutorial-on-correlation-coefficients.pdf has been updated several times and is getting too long, I will put that upcoming material on a separate pdf file. As a sneak preview, we will be examining extreme cases (too high/low r values, too high/low sample sizes, and too high/low signal-to-noise ratios, etc.).

PS. I updated this post to fix some little typos.

Footnote. I found erroneous including the entries for n = 10 and n = 100 in the first table so I removed these altogether and limited the discussion to the large n values. A reader asked why I used t-table = 1.96 for all entries.  I thought it was clear from the discussion that the above tables are meant to show calculations for arbitrarily set t-values.  In a real test, you would need to use the actual t values from statistical tables. For instance, for n = 10 you would have to use a t-table value of  t = 2.306 at the 0.95 level. You should get

n	n-2	t	r	S = r*r	N = 1 – r*r	S/N
10	8	2.306	0.6319	0.399293	0.600707	0.664705

On Power Analysis and SEO Quack Science

One of the trickiest aspects of publishing statistical studies is the sample size to be used. Not stipulating a valid procedure for estimating a proper sample size can hurt, for instance, a grant proposal. Ethical committees are concerned about the right number of observations in a study, asking submitters to justify on statistical grounds how they arrived at a given sample size. Research projects with too few or too many observations or no sample size methodology at all often get rejected. This is something those conducting SEO quack “science” don’t seem to understand or are not aware of.

Too small samples are unethical, because the researcher cannot be specific enough about the size of, for example, the effect of a drug in a population. Too large samples are also unethical, because represent a waste of funding. True that a large sample improves precision, but it might involve an unjustified cost. Stratification is preferred, but it gets too complicated with huge sample sizes, not to mention that statistical significance not necessarily scales between samples.

As Rahul Dodhia from RavenAnalytics (http://ravenanalytics.com/Articles/Sample_Size_Calculations.htm ) indicates: a 2000-sample might not be very different from a 20000-sample, but a 200-sample maybe very different from a 2000-sample even when in each case the sample ratio is 10. So, a large sample not always is justified, even if such a sample size improves statistical significance and precision.

Consider the case of search engine ranking results. Upon a query, search engines are capable of finding many results, frequently in the range of thousand or million results per query. Still search engines and retrieval systems show to users a limited answer set. For instance, Google limits its viewable answer set to a maximum of 1,000 results (100 pages, 10 results/page).

Like in most retrieval systems, relevant results are accumulated at the first few result pages forming clusters. This is in agreement with Rijsbergen’s Cluster Hypothesis, which states that documents that cluster together have a similar relevance to a given query. Moving down the list of search results one often find cluster transitions wherein the quality and aboutness of documents is polluted with off-topic content.

Documents buried in a list of results often contain content irrelevant to the initial query or full of spam techniques. If one wants to conduct a statistical study of ranking results versus a particular document feature, one can do better by considering a sample from the first few result pages than from the entire answer set of 1,000 results.

In general, in a non-search engine scenario one cannot just arbitrarily select large samples to “force” the statistical significance of very low correlation coefficients and then use those values to draw conclusions. Furthermore, what is the selection criterion for using 1,000 or 10,000 results?

Simply stated: If 10,000 observations are arbitrarily selected, why not use 100,000 or 1,000,000 instead? We already know that very small correlation coefficients between any two arbitrary pair of random variables will be significant at those huge sample levels, anyway. And?

As noted in a Wikipedia entry, “given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero. (http://en.wikipedia.org/wiki/Effect_size ).

For example, a correlation coefficient of r = 0.04 would be significant at a 95% confidence level if coming from a 10,000-sample (t-calc = 4.003 >> t-table = 1.96) while a correlation coefficient of r = 0.01 would be significant at a 95% confidence level if coming from a 100,000-sample (t-calc = 3.162 >> t-table = 1.96). And? This proves nothing, especially when the magnitude of a “signal” approaches the magnitude of its “noise”.

As noted at the above Wikipedia entry, a correlation coefficient of 0.1 is strongly statistically significant when sample size is 1000, (t-calc = 3.175 >> t-table = 1.96) but reporting only the small p-value from this analysis could be misleading if a correlation of 0.1 is too small to be of interest in a particular application. (http://en.wikipedia.org/wiki/Effect_size ).

Statistical significance of extremely small r values is not surprising as is just a mathematical consequence of the fact that a t-value is a function (F) of a weighted ratio: the ratio of explained-to-unexplained variations weighted by the number of degree of freedoms:

F(r, n) = t = SQRT[(r²/(1 – r²))*(n – 2)]
F(r, n) = t = r*SQRT[((n – 2)/(1 – r²))]

For a given r value, increasing n increases t. No surprise here. One thing is what a math equation tells you and another different thing is what the nature and obvious boundaries of a physical system tell you.

At trivially low r values any claim with regards to the statistical significance or strength of some results proves nothing and one cannot do much with such trivial r values. For instance for r = 0.04, r² = 0.0016, meaning that 1 – r² = 0.9984 or 99.84% of the variations in the dependent variable (y) are not explained by variations in the independent variable (x).

In such a scenario, assessing the effect of x on y is a futile exercise. Such a model would be useless for drawing conclusions or predicting anything. And here is the point that many SEOs at SEOMOZ (http://www.seo.co.uk/seo-news/seo-tools/the-seomoz-lda-tool-%E2%80%93-our-disappointing-findings.html , Fishkin, Hendrickson, and others elsewhere) don’t seem to grasp:

When a correlation coefficient is useless for all practical purposes.

If the raw data constantly changes, that’s another “Chaos Layer” that compounds the problem.

Enters Cohen’s Power

According to Cohen’s work, when conducting a sample size study of correlation coefficients, one needs to consider the required confidence level and power of the test, the desired probability for Type I and Type II Errors, and the hypothesized or anticipated correlation coefficient (http://www.medcalc.be/manual/correlation_coefficient.php ). One cannot just use an arbitrary sample size for testing things.

In general, given any three of the following, the fourth one can be determined (http://www.statmethods.net/stats/power.html ):

1. sample size
2. effect size
3. significance level = P(Type I error) = probability of finding an effect that is not there
4. power = 1 – P(Type II error) = probability of finding an effect that is there

One also needs to consider what is the statistical parameter that is undergoing the power analysis. One needs to ask questions like the following:

Are we testing means from a given group? http://www.nss.gov.au/nss/home.nsf/pages/Sample+Size+Calculator+Description?OpenDocument

Are we testing means from different groups? http://www.ncbi.nlm.nih.gov/pmc/articles/PMC137461/

Are we testing correlation coefficients? Read Simon’s take on the impact of sample size on the desired level of precision in correlation coefficients (http://www.childrens-mercy.org/stats/weblog2005/CorrelationCoefficient.asp ).

Are we interested in significance level, effect size, sample effect, or power?

When conducting an effect size analysis one must keep in mind that effect sizes estimate the strength of a possible relationship, rather than assigning a significance level. However, effect sizes do not determine significance levels, or vice-versa.

So, how do we go about implementing Power Analysis?

For those interested in implementing power analysis written in the R Language, I recommend the libraries at http://www.statmethods.net/stats/power.html

Software for conducting power analysis is also available elsewhere, as shown in the following table. My favorites are G*Power and SPSS SamplePower (http://www.spss.com/software/statistics/samplepower/).

Power Analysis SoftwareSource: http://www.epibiostat.ucsf.edu/biostat/sampsize.html
Software	Remarks
G*Power License: Free	Uses both exact and approximate methods to calculate power. It will deal with sample size/power calculations for t-tests, 1-way ANOVAs, regression, correlation, and chi-square goodness of fit. For t-tests and ANOVAs you find the effect size by supplying mean and variance information. For correlation coefficients the effect size is a function of r2. http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/
PC-Size License: Free	Deals with sample size/power calculations for t-tests, 1-way and 2-way ANOVA, simple regression, correlation, and comparison of proportions. http://www.esf.edu/efb/gibbs/monitor/usingDSTPLANandPCSIZE.pdf ftp://ftp.simtel.net/pub/simtelnet/msdos/statstcs/size102.zip
DSTPLAN License: Free	Uses approximate methods to calculate power. It will calculate sample size/power for t-tests, correlation, a difference in proportions, 2xN contingency tables, and various survival analysis designs. http://biostatistics.mdanderson.org/SoftwareDownload/SingleSoftware.aspx?Software_Id=41
PS License: Free	Performs sample size/power calculations for t-tests, Chi-square, Fisher’s exact, McNemar’s, simple regression, and survival analysis. http://biostat.mc.vanderbilt.edu/twiki/bin/view/Main/PowerSampleSize
Tibco Spoffire S+ License: Paid	The only commercially-supported statistical analysis software that delivers a cross-platform IDE for the award-winning S programming language, the ability to analyze gigabyte class data sets on the desktop, and a package system for sharing, reuse and deployment of analytics in the enterprise and in validated environments. Used widely in validated production environments (e.g., 21 CFR Part 11).http://spotfire.tibco.com/products/s-plus/statistical-analysis-software.aspx
NQuery Advisor License: Paid	Performs sample size/ power calculations for t-tests, 1 and 2 way ANOVAS, tests of contrasts in 1-way ANOVAs, univariate repeated measures designs, regression (simple, multiple and logistic), correlation, difference of proportions, 2XN contingency tables, and survival analyses. http://www.statsol.ie/nquery/nquery.htm
PASS License: Paid	Performs sample size/power calculations for z-tests, t-tests, 1, 2, and 3-way ANOVAs, univariate repeated measures designs, regression (simple, multiple and logistic), correlations, difference in proportions, 2xN contingency tables, survival analyses and simple non-parametric analyses.  http://www.ncss.com/pass.html
Stata License: Paid	It has some simple built-in power and sample size functions. http://www.stata.com/
SPSS SamplePower License: Paid	If your sample size is too small, you could miss important research findings. If it’s too large, you could waste valuable time and resources. Finds the right sample size for your research in minutes and test the possible results before you begin your study, with IBM SPSS SamplePower. Strikes the right balance among confidence level, statistical power, effect size, and sample size using IBM SPSS SamplePower. Compares the effects of different study parameters with its flexible analytical tools. http://www.spss.com/software/statistics/samplepower/

On Correlation Coefficients and Sample Size

Today I updated my Tutorial on Correlation Coefficients to include a new section on the effect of sample size on the significance of correlation coefficients. This was motivated by some comments from search engine marketers on correlation strengths. (http://searchenginewatch.com/3641002). The new material might help those interested in learning whether a reported correlation coefficient is statistically different from zero. It is given below. Enjoy it.

The problem with correlation strength scales is that these say nothing about how the size of a sample impacts the significance of a correlation coefficient. This is a very important issue that is now addressed.

Consider three different correlation coefficients: 0.50, 0.35, and 0.17. Assume that we want to test that there is no significant relationship between the two variables at hand. The null hypothesis (H0) to be tested is that these r values are not statistically different from zero (rho = 0). How to proceed?

As recommended by Stevens (17), for rho = 0, H0 can be tested using a two tailed (i.e.,two sided) t-test at a given confidence level, usually at a 95% level. If t_calculated ≥ t_table, H0 is rejected. However, if t_calculated < t_table H0 is not rejected and there is no significant correlation between variables.

Here t_calculated is computed as r/SEr = r*SQRT[((n – 2)/(1 – r²))] while t_table values are obtained from the literature (http://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values ). Table 2 summarizes the result of testing the null hypothesis at different sample size values.

Table 2. H0 tests at different sample sizes; two-tailed, 95% confidence.
n	df = n – 2	r	SEr	t(calc)	t (0.95)	Reject (H0 : rho = 0)?
5	3	0.50	0.50	1.000	3.182	don’t reject
10	8	0.50	0.31	1.633	2.306	don’t reject
12	10	0.50	0.27	1.826	2.228	don’t reject
14	12	0.50	0.25	2.000	2.179	don’t reject
20	18	0.50	0.20	2.449	2.101	reject
30	28	0.50	0.16	3.055	2.048	reject
40	38	0.50	0.14	3.559	2.024	reject
50	48	0.50	0.13	4.000	2.011	reject
5	3	0.35	0.54	0.647	3.182	don’t reject
10	8	0.35	0.33	1.057	2.306	don’t reject
12	10	0.35	0.30	1.182	2.228	don’t reject
14	12	0.35	0.27	1.294	2.179	don’t reject
20	18	0.35	0.22	1.585	2.101	don’t reject
30	28	0.35	0.18	1.977	2.048	don’t reject
40	38	0.35	0.15	2.303	2.024	reject
50	48	0.35	0.14	2.589	2.011	reject
5	3	0.17	0.57	0.299	3.182	don’t reject
10	8	0.17	0.35	0.488	2.306	don’t reject
12	10	0.17	0.31	0.546	2.228	don’t reject
14	12	0.17	0.28	0.598	2.179	don’t reject
20	18	0.17	0.23	0.732	2.101	don’t reject
30	28	0.17	0.19	0.913	2.048	don’t reject
40	38	0.17	0.16	1.063	2.024	don’t reject
50	48	0.17	0.14	1.195	2.011	don’t reject

The table addresses at which size level an r value is high enough to be statistically significant.

For n = 14, all three r values (0.50, 0.35, and 0.17) are not statistically different from zero.

For n = 30, r = 0.50 is statistically different from zero while r = 0.35 and r = 0.17 are not.

Conversely, r = 0.50 is not statistically different from zero when n is equal or less than 14 while r = 0.35 is not different from zero when n is equal or less than 30.

Finally, r = 0.17 is not statistically different from zero at any of the sample sizes tested.

Understanding Accuracy and Precision

Students often have hard time understanding the difference between accuracy and precision, particularly when they read quack “science” “studies” when surfing  the Web. This post might help them to grasp these concepts.

What is Accuracy?

Accuracy is a term describing deviation of an experimental value from a target value. A target value is a value accepted as ‘true’. Constants, fundamental quantities, and theoretical values are considered ‘true values’. Thus, accuracy is proximity to a true value.

To illustrate, assume that a quantity x is measured. Its true value is x_t =1.00 and we report an experimental value x_e of 0.90. The absolute error of this observation is | x_e – x_t | = 0.10 and its relative error is (| x_e – x_t |/ x_t)*100 = 10%. The accuracy is the ratio between the experimental to true value. When expressed as a percent, it is called relative accuracy. In this case, x_e/ x_t = 0.90/1.00. This corresponds to a 90% accuracy.

What is Precision?

Precision has been loosely defined as how reproducible experimental results are. However, modern convention makes a careful distinction between reproducibility (between-run precision) and repeatability (within-run precision). Furthermore according to Freiser (1992),

• Repeatability is the closeness of agreement between individual experimental results obtained with the same method on identical test material or samples, under the same conditions (same operator, same apparatus, same laboratories, and same intervals of time).
• Reproducibility is the closeness of agreement between individual experimental results obtained with the same method on identical test material or samples, but under different conditions (different operator, different apparatus, different laboratories, and different intervals of time).

Note that the source of dispersion and errors in the experimental results is different in each case. Therefore arbitrarily expressing the precision of results in terms of standard deviations without considering how the data was collected (within- or between-run precision) should be avoided.

Similarly, comparing any two standard deviations, or standard errors for that matter, without regard for how the data was collected (experimental conditions, number of degrees of freedom, different sampling times, etc) should also be avoided. In particular, estimates of precision or comparisons of precisions from data set that constantly change within sampling times is a futile exercise.

Last but not least, the precision of a measurement depends on the measuring scale used. For instance, saying “He is about 55 years old.” is less precise than saying “He is 660 months old.” or than saying “He is 20,075 days old”.

References

Freiser, H. (1992). Concept Calculations in Analytical Chemistry. Chapter 12, p. 203. CRC Press, Boca Raton.

Miller, J. C. & Miller, J. N. (1984). Statistics for Analytical Chemistry. Chapter 1, p.19. Wiley, New York.

PS. I misplaced repeatability and reproducibility and fixed few more typos. Well and done. Thanks Dr. J. C. for pointing that out.

On the SEOMOZ LDA Fiasco

 LDA and Google’s ranks well correlated?

After the hilarious example of this guy with the SEOMOZ LDA tool (http://smackdown.blogsblogsblogs.com/2010/09/09/proof-that-the-new-seomoz-tools-is-at-least-half-accurate/ ) I can only laugh out loud. Have anyone tried something like that?

Regarding the new fiasco with their LDA tool. Oh, no, another one… (http://www.seomoz.org/blog/lda-correlation-017-not-032) : What can I said? They sound pathetic and apologetic. The words overhyped, shitty, sloppy, flawed, etc are not enough to describe their “research work”.

What will happen now with those Mute Speakerphones that were misled? Those that listen to fools become one.

I don’t feel any sympathy for their 15 minutes of “honesty”. The damage was done already to naïve readers.

Also, note that this latest flaw was discovered by them. It was not the result of any peer review process from external referees, as those throwing a towel at them would like to believe.

As mentioned before, beware of SEOs statistical “studies” and their quack “science”  (http://irthoughts.wordpress.com/2010/04/23/beware-of-seo-statistical-studies/  ), especially if coming from SEOMOZ.

Probably their snakeoil will make a comeback soon. (Oh, no. Again?)

If they still think they have a valid LDA implementation, why not announce it at David Blei’s Topic-Models werein a community of LDA experts will review  it  and compare it against other implementations?

Two things can happen:

(a) It will be reviewed.

(b) it will be ignored.

I “invite” them to do so.

Please, just don’t show up with your snakeoil, yellow shoes, your seo mom, paid cheerleaders, vested investors, overhyped claims, etc, etc.

PS.

More on their hype machine here: http://skitzzo.com/archives/seomoz-hype-machine.php

 It appears that even Danny Sullivan is not buying SEOmoz’s “research” on LDA. Accordingly, “He didn’t think it was the remarkable change that SEOmoz was making it out to be.” (http://outspokenmedia.com/internet-marketing-conferences/evening-forum-with-danny-sullivan/). He even confronted and put into question their “highly correlated” numbers. And that was even before they recanted.