• About IR Thoughts
  • Minerazzi Tools
  • Minerazzi Tutorials

IR Thoughts

~ Thoughts on Information Retrieval, Search Engines. Data Mining, and Science & Engineering

IR Thoughts

Category Archives: Fisher Transformations

On the Non-Additivity of Correlation Coefficients Part 3: The Bias & Nature of Correlation Coefficients

01 Thursday Nov 2018

Posted by egarcia in Data Mining, Fisher Transformations, Mathematics, Statistics and Mathematics

≈ 1 Comment

Tags

bias, bivariate normality, Correlation Coefficients, normal distribution, shifted up cosines

statistical relationships
This the third and last part of a tutorial series on the non-additivity of correlation coefficients.
http://www.minerazzi.com/tutorials/nonadditivity-correlations-part-3.pdf

Their bias & nature, transformations, and approximations to normality are discussed. The risks of blindly transforming scores to ranks or arbitrarily converting r-to-Z values/Z-to-r values (Fisher Transformations) are discussed. Shifted up cosine approximations to normality are also covered.

Not all researchers know that score-to-rank transformations can change the sampling distribution of a statistic (e.g. a correlation coefficient) and that Fisher transformations are sensitive to normality violations. Combining both types of transformations is a recipe for a statistical disaster.

Alas, some meta analysis and data analytic folks are guilty of that.

On Men and Ideas: Fisher vs. Pearson

05 Friday Oct 2018

Posted by egarcia in Correlation Coefficients, Data Mining, Fisher Transformations, Mathematics, Statistics and Mathematics

≈ Leave a comment

Tags

Fisher, Fisher vs. Pearson, Pearson, Royal Statistical Society

Ronald Aylmer Fisher was considered an outsider by the statistical establishment of his time.

The links below (1-3) show his struggles & nuances with Karl Pearson, his son Egon, Bowley, their followers, and the Royal Statistical Society (RSS). His life was a story of accomplishments and noise (deceptions and nasty RSS politics). He was too ahead of his time.

That reminds me of the struggles of another maverick: Benoit Mandelbrot. Eventually and like Mandelbrot, Fisher greatness was recognized. Also like Mandelbrot, he was able to boost the signal-to-noise of his career and life.

Most statisticians consider Fisher the Father of Modern Statistics (https://en.wikipedia.org/wiki/Ronald_Fisher), even when he was not allowed to teach Statistics at the University of Cambridge (they tried to silence Fisher).

Yes, scientists too can be demeaning to other scientists, more for personal reasons than for ideas and the Scientific Method. After all, they are also mostly carbon units called “humans”.

1. Fisher in 1921 https://projecteuclid.org/download/pdfview_1/euclid.ss/1118065041

2. Fisher vs Pearson: A 1935 Exchange from Nature
http://physics.princeton.edu/~mcdonald/examples/statistics/inman_as_48_2_94.pdf

3. Fisher: The Outsider
R. A. Fisher: how an outsider revolutionized statistics

hashtag#Fisher
hashtag#Pearson

Regression & Correlation Calculator: Updates and Improvements

14 Friday Sep 2018

Posted by egarcia in Algorithms, calculators, Data Mining, Fisher Transformations, Mathematics, Statistics and Mathematics

≈ 7 Comments

Tags

Correlation Coefficients, Pearson Correlation, Spearman Correlation

Regression & Correlation

We have updated and improved our Regression & Correlation Calculator to demonstrate, as shown in the above figure, that a Spearman’s Correlation Coefficient is just a Pearson’s Correlation Coefficient computed from ranks.

The tool uses an algorithm that converts values to ranks and averages any ties that might be present before calculating the correlations. This comes handy when we need to compute a Spearman’s Correlation Coefficient from ranks with a large number of ties.

We have explained in the “What is Computed?” section of the page’s tool that as the number of ties increases the classic textbook formula for computing Spearman’s correlations

Spearman's Correlation Coefficient

increasingly overestimates the results, even if ties were averaged.

By contrast, computing a Spearman’s as a Pearson’s always work, even in the presence or absence of ties.

To illustrate the above, consider the following two sets:

X = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Y = [1, 1, 1, 1, 1, 1, 1, 1, 1, 2]

using Spearman’s classic equation rs = 0.6364 ≈ 0.64.
By contrast, rs = 0.5222 ≈ 0.52 when computed as a Pearson coefficient derived from ranks. This is a non trivial difference.

Accordingly, we can make a case as to why we should ditch for good Spearman’s classic formula.

We also demonstrate in the page’s tool why we should never arithmetically add or average Spearman’s correlation coefficients. The same goes for Pearson’s.

Early articles in the literature of correlation coefficients theory failed to recognize the non-additivity of Pearson’s and Spearman’s Correlation Coefficients.

Sadly to say, this is sometimes reflected in current research articles, textbooks, and online publications. The worst offenders are some marketers and teachers that, in order to protect their failing models, resist to consider up-to-date research on the topic.

PS. Updated on 09-14-2018 to include the numerical example and to rewrite some lines.

Beating a dead horse, again.

23 Monday Jul 2018

Posted by egarcia in Correlation Coefficients, Data Mining, Fisher Transformations, Quack Science, self-weighting, SEO Myths, Statistics and Mathematics

≈ Leave a comment

Tags

Correlation Coefficients, Fisher Transformations, SEO Myths, statistics

Happy to see that Bruce J. Ladewski’s PhD thesis, Expanding a Path Analytic Model of Quality Management to Include the Management of Safety, at

https://scholarworks.wmich.edu/cgi/viewcontent.cgi?article=4107&context=dissertations

cited the Self-Weighting Model Tutorial Part 1
http://www.minerazzi.com/tutorials/self-weighting-model-tutorial-part-1.pdf

and stated what we all know: that correlation coefficients are not additive.

I don’t understand why some search marketers still believe the contrary. Stay away from dumb analytic from dumb SEOs, their myths, and nonsense.

Well: what can I say? Beating a dead horse,…again.

December 2019
M T W T F S S
« Nov    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Favorite Sites

  • Minerazzi.com

Pages

  • About IR Thoughts
  • Minerazzi Tools
  • Minerazzi Tutorials

Categories

  • 4D Printing
  • AIRWeb Course
  • Algorithms
  • Amazon Alexa
  • Android
  • Arithmetic Geometry
  • Best Match Models (BM)
  • Big Data
  • BioDesign
  • bioinformatics
  • Blogroll
  • calculators
  • Cancer
  • Cancer Research
  • Causality & Determinism
  • Chaos
  • chemical mining
  • Chemist Biographies
  • chemistry
  • Chemometrics
  • Clinical Trials
  • Cloud Computing
  • Conferences
  • Correlation Coefficients
  • Cortana
  • Crawlers
  • Curated Collections
  • Data Conversion
  • Data Mining
  • Data Structures
  • Deep Neural Networks
  • directories
  • Directories
  • Docker
  • Dynamics
  • Favorite Sites
  • Feed Tools
  • Fisher Transformations
  • Fractal Geometry
  • Fractal Patterns
  • google
  • Graduate Courses
  • Hacking
  • Homeland Security
  • Human-Computer Interaction
  • Image Compression
  • information retrieval
  • Internet Engineering
  • Internet Standards
  • inverted index
  • ir
  • IR Quizzes
  • IR Tools
  • IR Tutorials
  • Kubernetes
  • Latent Semantic Indexing
  • Legacy Posts
  • LIGO
  • Machine Learning
  • Marketing Research
  • Mathematics
  • Medical Cannabis
  • meta-analysis
  • Mind Retrieval
  • miner
  • minerazzi
  • Miscellaneous
  • National Laboratories
  • New Information Retrieval Paradigms
  • News
  • Newsletters
  • Nonlinear Dynamics
  • One-to-Many (O2M)
  • Open Source Projects
  • PageRank
  • Patents
  • PCA
  • People Searches
  • Perfectoid Spaces
  • PHP
  • Physiology
  • Poems
  • political networks
  • Predatory Conferences
  • Predatory Journals
  • Programming
  • Public Databases
  • Public Records
  • Quack Science
  • Quantum Computing
  • Quantum Information Retrieval
  • Quantum Searches
  • Quantum Theory
  • Queries
  • Ranking Results
  • Research Centers
  • RSS/Atom Feeds
  • Scripts
  • search engines
  • Search Engines Architecture Course
  • Search Modes
  • self-weighting
  • SEO Myths
  • Sitemaps
  • social mining
  • social pulse parser
  • Software
  • Spam
  • Statistics and Mathematics
  • SVD
  • Technology Inventions
  • Theoretical Physics
  • Theses
  • twitter
  • URLs Mining
  • Vector Space Models
  • Voice Assistants
  • Web Mining
  • Web Mining Course
  • Web Security
  • Xamarin

Recent Posts

  • On the Myth of d Orbitals Hybridization
  • The Bond Order Calculator: Updates
  • Semantic Similarity of Healthcare Data
  • Going the multidisciplinary way
  • CUNY Computational Chemistry Tools
  • Why I chose to be a multidisciplinary scientist?
  • A Simple Example of Phonetic Similarity vs. Text Similarity
  • A Simple News Hub
  • Zillman’s 2019 Directory of Directories
  • More on Perfectoid Spaces
  • Lymphomas Miner
  • Extracting Topic-Specific Wikipedia Links
  • Programming Languages Miner
  • Beware of Chemistry Heuristics
  • New IANA Miners

Archives

  • November 2019
  • October 2019
  • September 2019
  • August 2019
  • July 2019
  • June 2019
  • May 2019
  • April 2019
  • March 2019
  • February 2019
  • January 2019
  • December 2018
  • November 2018
  • October 2018
  • September 2018
  • August 2018
  • July 2018
  • June 2018
  • May 2018
  • April 2018
  • March 2018
  • February 2018
  • January 2018
  • September 2017
  • August 2017
  • July 2017
  • June 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • October 2011
  • September 2011
  • August 2011
  • July 2011
  • June 2011
  • May 2011
  • April 2011
  • February 2011
  • January 2011
  • December 2010
  • November 2010
  • October 2010
  • September 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • January 2010
  • December 2009
  • November 2009
  • October 2009
  • September 2009
  • August 2009
  • July 2009
  • June 2009
  • May 2009
  • April 2009
  • March 2009
  • February 2009
  • January 2009
  • December 2008
  • November 2008
  • October 2008
  • September 2008
  • August 2008
  • July 2008
  • June 2008
  • May 2008
  • April 2008
  • March 2008
  • February 2008
  • January 2008
  • December 2007
  • November 2007
  • October 2007
  • September 2007
  • August 2007
  • July 2007
  • June 2007
  • May 2007
  • April 2007

Algorithms calculators chemical mining Conferences Data Mining Hacking Homeland Security Human-Computer Interaction IR Tools IR Tutorials Latent Semantic Indexing Machine Learning Marketing Research Mathematics miner minerazzi Miscellaneous New Information Retrieval Paradigms News Newsletters Programming Queries Scripts Search Engines Architecture Course SEO Myths Software Spam Statistics and Mathematics Vector Space Models Web Mining Course

Create a free website or blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy