Still, from time to time some “experts” and pseudo “scientists” do that.

Want to know why this is not mathematically and statistically possible? This is the subject of a paper I wrote and that is about to be published in Communications in Statistics – Theory and Methods (by Taylor & Francis).

Incidentally, I will provide a preview of the topic to the search marketing community. Thanks to my dear friend, Mike Grehan, this will be the topic I’ll be speaking about at the March, 2012 SES, NY.

Not even mathematicians agree on what the result should be. It has been argued that the answer is a matter of convenience, an element controversial -if not contrary- to Mathematics. What’s your take on the issue?

To learn more about this rule and when it should be applied, see Wikipedia or WolframMathWorld.

I’ve posted these in my facebook page, posing with back then IPAM director and with world-recognized LSI expert Dr. Michael Berry and his former students. To learn more about the workshop and the speakers, follow this link http://www.miislita.com/ipam/ipam-document-space-workshop.pdf

Once published, you will understand the SEO (* SEOmoz, I should say) non-sense of computing arithmetic averages of correlation coefficients and why some meta-analysis studies published in the past (* Hunter-Schmidt; Hedges-Olkin) are flawed and invalid.

It took me several meals and research hours to figure it out. I hope that IRs, dataminers, and statistics colleagues find new applications for the model.

The model can be applied to many fields, including marketing, business, risk analysis, data mining, signal processing, engineering, clinical trials, and almost any field or knowledge domain that involves the calculation of weighted statistics. I look forward to discuss it online once it get published.

Happy New Year.

PS. (*) I’ve edited this post to make these points obvious. So, the issue of arithmetically averaging correlations has been raised and killed for good before the scientific and statistical community.

PS. Just in: Last night (Jan-03-2012) I received news from one of the editors of the journal that the paper was assigned to issue 41 (8). Check for its title: The Self-Weighting Model (in Spanish is something like “El Modelo de Autoponderacion“. I forget to mention that this journal is published biweekly; so, things are moving fast. What a way of ending 2011 and starting 2012!!!

index.of
index.of/php
index.of/pswd
index.of/db
index.of/mda
index.of/pgp

or check the list at http://www.thenetworkadministrator.com/googlesearches.htm These types of searches will spit out directory trees.

There are many “smart cookies” posting derivatives of these lists all over the Web.

And how about typos?

Try filetype command searches with extra characters in extensions like

0php
1php
phps
php.

etc….

Servers will spit out entire php codes.

The great offenders are large sites like those belonging to .edu, .gov, .org, not to mention large .com and .net sites.

Ho, Ho, Ho, Merry Christmas, Santa.

In this Thanksgiving Day I thank that all these myths have been debunked to no end: LSI-rank correlations, LDA-rank correlations, KD-rank correlations, additiveness of correlation coefficients, blah, blah, blah…  I thank also that along came this: http://infolab.stanford.edu/~sergey/349/

LSI?

Known from the onset by Google.

A cost effective implementation in a large scale and dynamic environment as the Web is?

Nope.

The Whois Miner is getting smarter. It now gives you the alternate name of a whois server service. It also has network mining features, enabling you to mine registrant DNS lookups, Headers, NS/MX records, and contact emails.

In addition, we keep expanding our index of whois servers. Follow link below.

Use it for free while you can.

http://www.minerazzi.com/labs/whois.php

Play with it and send some feedback.

Added support to:

1. generic third-level domains (gTLDs).

2. country-code TLDs.

3. subdomain TLDs.

4. status persistency of form fields (without using cookies, sessions, JavaScript, but just pure PHP).

As we keep improving and adding new TLDs and whois servers to its index, we expect this to become a destination for our regular users.

The tool was designed in such a way that even support to the upcoming dotBrand Revolution is possible.

Enjoy it

Apparently they don’t get it. And I thought SEO “statistical studies” was something out of this world. SEO myth listers are worse.

However…

Here I would like to reference research at both sides of the Scope Hypothesis.

In the abstract of ”Revisiting the relationship between document length and relevance” (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.3786&rep=rep1&type=pdf), Losada, D.E., Azzopardi, L. and Baillie, M. (2008) state:

“The scope hypothesis in Information Retrieval (IR) states that a relationship exists between document length and relevance, such that the likelihood of relevance increases with document length. A number of empirical studies have provided statistical evidence supporting the scope hypothesis. However, these studies make the implicit assumption that modern test collections are complete (i.e. all documents are assessed for relevance). As a consequence the observed evidence is misleading. In this paper we perform a deeper analysis of document length and relevance taking into account that test collections are incomplete. We first demonstrate that previous evidence supporting the scope hypothesis was an artefact of the test collection, where there is a bias towards longer documents in the pooling process. We evaluate whether this length bias affects system comparison when using incomplete test collections. The results indicate that test collections are problematic when considering MAP as a measure of effectiveness but are relatively robust when using bpref. The implications of the study indicate that retrieval models should not be tuned to favour longer documents, and that designers of new test collections should take measures against length bias during the pooling process in order to create more reliable and robust test collections.”

Really….?

However in the abstract of “Enhancing ad-hoc relevance weighting using probability density estimation” (http://www.sigir2011.org/papershow.asp?PID=104), Zhou, Huang, and He (2011) state:

“Classical probabilistic information retrieval (IR) models, e.g. BM25, deal with document length based on a trade-off between the Verbosity hypothesis, which assumes the independence of a document’s relevance of its length, and the Scope hypothesis, which assumes the opposite. Despite the effectiveness of the classical probabilistic models, the potential relationship between document length and relevance is not fully explored to improve retrieval performance. In this paper, we conduct an in-depth study of this relationship based on the Scope hypothesis that document length does have its impact on relevance. We study a list of probability density functions and examine which of the density functions fits the best to the actual distribution of the document length. Based on the studied probability density functions, we propose a length-based BM25 relevance weighting model, called BM25L, which incorporates document length as a substantial weighting factor. Extensive experiments conducted on standard TREC collections show that our proposed BM25L markedly outperforms the original BM25 model, even if the latter is optimized.”

My take…

I haven’t reviewed BM25L vs. BM25F, yet. Still the question on the Scope Hypothesis is intriguing. For what I can tell (and this is my sole opinion), if an author writes more about a topic or several topics in a given document, more likely he will be using more instances of index terms. A cluster of the top index term density values (IDs) spreaded over said document should give some insight about its scope. We have developed a tool that computes these clusters. We are testing now whether that would translate into an improved relevance.

Assuming that Web IR systems out there (e.g,, search engines) use these algorithms or derivatives of these: What would be the implications for content writers trying to understand algos based on the Verbosity and Scope Hypotheses? Hello, copywriters, SEOs, etc. This puppy is nice to watch.

Filed under: Data Mining, IR Tools, IR Tutorials, Queries ]]> http://irthoughts.wordpress.com/2011/08/13/the-scope-hypothesis-in-ir-who-is-right/feed/ 7 irthoughts http://irthoughts.wordpress.com/2011/08/04/bm25-and-bm25f-implications-to-seo-and-web-design/ http://irthoughts.wordpress.com/2011/08/04/bm25-and-bm25f-implications-to-seo-and-web-design/#comments Thu, 04 Aug 2011 14:15:01 +0000 egarcia http://irthoughts.wordpress.com/?p=1655 Continue reading »

Yesterday we published two great tutorials on the BM25 and BM25F algorithms.

The “take away home” from the theory behind these algorithms:

1. A term (e.g., a keyword) has more information gain when it occurs for the very first time.

2. More likely, a term weights more in a title field than in other fields.

3. The weight of a term and its ocurrence frequency are not linearly related.

4. A linear combination of field scores that destroys term dependencies is contraindicated (See BM25F).

Most SEOs know well about 1 and 2.

As a term has more information gain during its first occurrences, a document about specific terms should mention these at the beginning, particularly in the title tag. For testing purposes and since end user assume that a large headline is the actual title of a document (which is not)  we like to repeat the title tag content in an h1 header that is placed prominently at the beginning of the copy. Keywords from the title are then repeated early in the document body. In this way, one can write for both end users and search engines. If a search engine uses some form of the above algorithms (which we don’t know if they do), that base is covered, too. You don’t have to adopt this strategy, unless you want. It is just our way of conducting tests, but is a flexible approach.

Filed under: IR Tutorials ]]>

http://www.miislita.com/information-retrieval-tutorial/okapi-simple-bm25f-tutorial.pdf

Unlike the BM25, this model (known as Simple BM25F) incorporates the structure of documents into the scoring process.

In addition, we’ve uploaded a new, improved, and expanded version of the Okapi Best Match 25 tutorial.

http://www.miislita.com/information-retrieval-tutorial/okapi-bm25-tutorial.pdf

Have a great IR day!

07-05-11: Email address extraction, deduplication, and sorting capabilities added.
07-04-11: Design and copy changes.
07-03-11: Navigation menu restored and bug fixed.
07-03-11: Navigation menu removed to test bug.
07-02-11: Top-bottom quick navigation menu added.
07-02-11: Day/Time Stamp, Reverse DNS, and IPv4 List capabilities added.
07-02-11: Integration to Whois Tool.

The Whois Database Retriever (http://www.minerazzi.com/labs/whois.php) now features suffix/prefix stripping capabilities. This means that users only need to enter a candidate domain name without any alias or extension and the tool scans multiple registrar databases. We expect to add some additional features to this time-saving application.

In the meantime, we keep beta testing the engine. Our staff of ‘miners’ are doing just a great job.

This is a tutorial on the classic Okapi Best Match 25.

Enjoy it.

Changes made include the ability to detect information contained inside selection menus, textareas, and input fields. These changes simplify the examination of name/value pairs used by forms. Very useful when a document contains multiple forms, when query mechanisms of target databases must be identified, or when one need to assess whether a database is susceptible to query injections or script infections. In the latter, a security component is more than obvious.

We also moved both the Markup and Robots Text File Reports to the Source Code section. This is a new section that is now listed as the last reports of the application.

Our crawler is now detecting form tags. Changelogs are given below:

06-23-11: Form tags detection capabilities added.

05-08-11: New user interface.

05-06-11: Log file access.

05-01-11: Copy changes.

04-29-11: Script tags detection capabilities added.

04-22-11: DocType, Base, and Link tags detection capabilities added. HTML parsing changes.

04-19-11: Robots Text File detection capabilities added. Meta data parsing changes.

04-18-11: Title and Meta Tags detection capabilities added. Layout changes.

04-16-11: User Environment detection capabilities added.

04-14-11: Timer capabilities added.

04-10-11: Deduplication capabilities added.

04-09-11: Color palette reporting capabilities added.

04-05-11: DNS and MX reporting capabilities added.

04-03-11: Source code reporting capabilities added.

03-30-11: Relative URL resolving capabilities added.

03-28-11: Hypertext wrapping, ip, and headers reporting capabilities added.

They are having their world congress here in Centro de Convenciones de Puerto Rico from 13-18 of August. See http://www.ifla.org/

See also conference program at http://www.ifla.org/en/news/ifla-wlic-2011-final-programme

We will see you there.

And there is also the question of how to rank the results. Two strategies are frequently used: (a) data appending and (b) data fusion.

Some of the early forms of federated search engines for the Web used to do (a) and were soon called meta search engines. These search tools simply returned a long list of results by appending the top N ranked results from each databases in a tandem fashion. Obviously this strategy failed to recognized the top M relevant results from this huge list and soon was phased out in favor of (b).

In (b), arithmetic or weighted averages from the top N results from the different databases are computed. The problem with this approach is that is very subjective. In order to compute an arithmetic or weighted relevance score, who decides how much weight should be assigned to a given ranked result from a given database? No matter which weighting criteria are used, at the end it is still a subjective score and one that not necessarily improves the end-user search experience. Just the opposite.

This leads to what I call “The Search Paradox”: Information gateways as information roadblocks.

To learn more about this, visit this old link: http://www.accessmylibrary.com/article-1G1-182034526/federated-search-101-alexis.html

According to their FAQs page at http://www.intute.ac.uk/faq.html,

Why have JISC made this decision?

As stated in the JISC statement about the Intute review, when services “reach the end of their existing funding cycle it is always intended, wherever possible, that they move from being fully funded to being part-funded or fully sustained by other sources”. Unfortunately in the current economic climate no realistic alternative funding model for Intute as it currently stands has been identified.

However, we are working to ensure that the legacy of Intute lives on, and we are working with other organisations in the sector to find a new home for Intute content.

Why can’t Intute continue without JISC funding?

Over the last three years, we have investigated alternative funding models for Intute, including alternative grant funding, subscription and advertising/sponsorship, and we have spoken to librarians, academics and students to find out what they think. Unfortunately, we have been unable to find a model that will sustain Intute in its current form into the future.

and

Can you open up Intute for community updating and contributions? This model may be a better fit now with the rise of social /community web 2.0 ways of working.

We have looked at the possibility of facilitating a community generated resource catalogue, and investigated exporting all of our resources to Delicious. However, in December 2010 reports circulated that Yahoo will be shutting down Delicious. With Intute funding ending in July 2011, the uncertainty surrounding Delicious means that further investigations are unlikely.

Is there any way to save Intute? What about an internet fundraising drive or trying to raise funds from institutions, foundations or advertising?

In principle – maybe, but in practice we have investigated alternative funding models for Intute, including alternative grant funding, subscription and advertising/sponsorship, and we have been unable to find a model that will sustain Intute in its current form into the future. Intute as it stands costs over 1 million a year to run excluding the contributions associated with housing staff at our different partner institutions.

 Intute was created by a consortium of seven universities, working together with a whole host of partners.

The Intute consortium was:

• University of Birmingham
• University of Bristol
• Heriot-Watt University
• The University of Manchester
• Manchester Metropolitan University
• University of Nottingham
• University of Oxford

Amazing that with so much human resources talent their fate is as described above.

Unlike similar tools, ours can access multiple registrar databases.

Feel free to play with it at

http://www.minerazzi.com/labs/whois.php

04-29-11: Script tags detection capabilities added.
04-22-11: DocType, Base, and Link tags detection capabilities added.

If you are a Web developer, these new features of our crawler can help you to mine programming “gems” by examining, isolating, and collecting script lines that have been embedded in the source code of documents. And by crawling or accessing URLs of external files or link tags, programmers can view and dissect hidden scripts.

To learn more about these new features or previously added features, visit

http://www.minerazzi.com/labs/crawlinker.php

We mentioned that the online version will undergo several changes, all intended to provide an online ”web crawler for the masses”. The idea is to put users in control of the crawling process since current crawlers lack of human intuition with regard to the next URL to crawl from a to-do list.

We are pleased to announce the following changes.

Changelogs
04-19-11: Robots Text File detection capabilities added. Meta data parsing changes. (*)
04-18-11: Title and Meta Tags detection capabilities added. Layout changes.
04-16-11: User Environment detection capabilities added.
04-14-11: Timer capabilities added.
04-10-11: Deduplication capabilities added.
04-09-11: Color palette reporting capabilities added.
04-05-11: DNS and MX reporting capabilities added.
04-03-11: Source code reporting capabilities added.
03-30-11: Relative URL resolving capabilities added.
03-28-11: Hypertext wrapping, ip, and headers reporting capabilities added.

(*) Just added.

There is something for everyone here.

Web Designers: Want to use a color palette from another site or tweak yours? Easy. Launch a crawl to a css file already discovered by the crawler.

Web Developers: Want to see diamonds? View the source of any file, including PDFs.

Data miners:  Need to mine links? Crawl a document. Better: launch a crawl to a site map file already discovered by the crawler.

Researchers: Want to check system configurations? Check IPs, DNS, MX and header traces (including data from cookies/sessions, etc)

More updates are coming soon!

We are excited to announce several updates to the minerazzi crawler. This is the online version of the indexing crawler used by the minerazzi search engine (beta).

The long-term goal is to turn this version into a multifunctional mining platform and a crawler for the IT masses; i.e., a crawler to be used by IR researchers, data miners, webmasters, developers, etc. That is, a crawler that even Web designers and the average public can use.

You’re welcome to give it a try. Keep in mind the tool is still in beta. While you are there, feel also free to test the multiple whois domain name tool.

The current issue of IRW is out and should arrive to subscribers inboxes today.

In this issue of IRW, we introduce the minerazzi project and two useful tools available at minerazzi.com

Web Crawler and Link Checker Tool (http://www.minerazzi.com/labs/crawlinker.php).

Multiple Whois Domain Name Tool (http://www.minerazzi.com/labs/whois.php).

This is a research project conducted in association with several scholars and the private sector.

Enjoy it.

In Windows Vista, you need to install the Telnet Client:

1. Navigate to Start > Control Panel > Programs > Programs and Features > Turn Windows features on or off.  If you are prompted for an administrator password or confirmation, type the password or provide confirmation.

2. In the Windows Features dialog box, select the Telnet Client check box.

3. Click OK. The installation might take several minutes.

Other nifty installs available are

RIP Listener

SNMP

Simple TCPIP services

Telnet Server

TFTP Client

The current issue of IRW should reach subscribers inboxes during the day.

This is Part Two of the series on statistical analysis of n-grams. This is a text mining analysis technique widely used in information retrieval and data mining in general. In this issue we cover the implementation of association measures derived from contingency tables.

The QA section explains how to conduct a Chi Square Test for tables with many items; i.e., beyond the usual 2 x 2 contingency tables.

Enjoy it.

Lang raises the question based on the following bullet points:

1. Forty-page Articles Are Dead.
2. Survey Articles Are Dead.
3. Journal Issues Are Dead.
4. Page Numbers Are Dead.
5. Copy Editing Is Dead.
6. Peer Reviewing Might Be Dying Too.
7. The Article as a Unit of Publication Is Dead.

Lang then concludes with a question and call to action.

A New Beginning for Scholarly Publishing?

“So let’s abandon all the 20th-century baggage of traditional journals, and move to a more rational model for scholarly publication, with no copy editors, no reviewers, no redundancy, and no unnecessary delays. A concrete step would be to give each ACL member a DOI for a unipaper, and then ask them to non-redundantly populate this with a sequence, or a tree, of numbered paragraphs that consolidate all their work on a topic. Then, to get things moving, the present journal could insist that some proportion of citations be to paragraphs within these unipapers, with hyperlinks embedded right there in the citations. What are we waiting for?”

Feel free to take issues with any of the above points.

My opinion? Lang has very good arguments. However, …. I would say that due to the changing times -read here smart phones, I-tablets, blogs, social networks, etc- many hardcopy scholarly journals are actually evolving while the weakers or unfit to changes are dying as a natural e-phenomenon observed in online ecosystems. This is not unique of scholarly journals. Actually the same is true for any piece of hardcopy journal, newspaper, magazine, newsletters.

With more retailers giving discounts and even freebies just for showing a tweet about their products or services at their store, who knows what will be the fate of  flyers, coupons, etc.

Publishers that don’t adjust their business models to the changing times are deemed to become the next  LPs, 8-tracks, cassette tapes, etc.   

Lang, N. (2010) Are We Near the End of the Journal. Computational Linguistics Volume 36, Number 4.  Retrieved from http://www.mitpressjournals.org/doi/pdf/10.1162/coli_a_00019  

So this might explain why the January IRW newsletter is delayed. It should arrive subscribers inboxes during the day. The February issue will be out in about one week. These are back to back issues on Statistical Analysis of N-Grams.

Part 1:N-Grams & Contingency Tables

Part 2: N-Grams & Association Measures

On other matters, a PhD student published few years ago an excellent application of the Vector Space Model applied to Protein Analysis. You can revisit the post at http://irthoughts.wordpress.com/2008/11/12/vector-space-model-and-protein-retrieval/ .

If others have other applications of VSM in other disciplines, let me know. I’m interested in multidisciplinary stuff.

Computing weighted averages of correlation coefficients depends on the weighting strategy used: unit weights, sample size, optimal weights, and within/between study variances, etc. Most text books advocate the use of Fisher’s Z Transformation for, for instance compute confidence intervals and average correlations.

One thing that has been bothering me for a long time now is this: what would be the discriminatory power of such weighting strategies if we are in the presence of identical data sets of correlation values, but coming from samples with different variabilities effects in the dependent variable?

Research conducted for the last four months let me to realize an alternate approach to the above weighting strategies.

At that time I was putting together a new tutorial series on meta-analysis, so this problem diverted my attention and was always in the back of my head.

So after many meals and long nights, I finally decided to include my research findings as Part 1 of the tutorial series, which you can read here: On the Non-Additivity of Correlation Coefficients.

I hope you like it. Since this is relevant to many research areas, please send your feedback through private, confidential email and not through this blog.

PS. If others are interested in testing how the proposed approach compares with other weighting strategies, feel free to contacting me. I’m interested in testing with real data (non-simulated) from any field: science, engineering, education, behavioral & social sciences, allied health, literature, politics, marketing, etc.)

Check this blog tomorrow as there will be a value-added good news for those interested in conducting research at the interface of information retrieval, statistical analysis, and applied mathematics. You’re welcome to grab a copy of this four-month investigation, for use in your own research, as a teaching tool, or to chase away SEO snakeoil.

The current issue of the Information Retrieval Newsletter is out! Due to the Holiday Season, it is a short issue.

The article section is dedicated to Fisher’s Z Transformation, its origins, advantages, and limitations.

We have included a less known visualization of it. Using a geometrical interpretation helps one to understand how the transformation works.

Enjoy it and Happy New Year!

Doz.me Can Launch DDoS Attacks Using Shortened URLs

URL Shortener Is Also A DDOS Tool

Beware of clicking on links with URLs shortened.

Ho, Ho, Ho.

Happy Holidays.

In Chapter 4 he discusses on the proper way of looking at correlation coefficient values. He writes and quote (emphasis added in boldfaces):

“As a matter of routine it is the squared correlations that should be interpreted. This is because the correlation coefficient is misleading in suggesting the existence of more covariation than exists, and this problem gets worse as the correlation approaches zero. Consider the following correlations and their squares.”

“Note that as the correlation r decrease by tenths, the r² decreases by much more. A correlation of .50 only shows that 25 percent variance is in common; a correlation of .20 shows 4 percent in common; and a correlation of .10 shows 1 percent in common (or 99 percent not in common). Thus, squaring should be a healthy corrective to the tendency to consider low correlations, such as .20 and .30, as indicating a meaningful or practical covariation. “

Rummel’s page is very relevant these days where SEOs from SEOMOZ and few other snakeoil marketing sites are buying the bogus discourse from Fishkin and Hendrickson that low correlation coefficients in about that range are evidence of LDA scores and Google ranks being “highly” correlated.

As mentioned before at this blog, SEO marketers are good at selling that kind of snakeoil or “quack” science.

Statistical significance does not equate to high correlation. For large enough sample sizes even very low r values (0.1, 0.01, etc) eventually become significant, but these do not equate to high correlation.

On a side note, I’m reading an IR thesis wherein Spearman’s and Kendall’s coefficients are used. Quite interesting.

First PS

According to a Sloan Consulting article at ISIXSIGMA.COM site and quote (emphasis added)

“As a rule of thumb a strong correlation or relationship has an r-value range of between 0.85 to 1, or -0.85 to -1. In a moderate correlation, the r-value ranges from 0.75 to 0.85 or, -0.75 to -0.85. In a weak correlation, one that is not a very helpful predictor, r ranges from 0.60 to 0.74 or -0.60 to 0.74. Though an entirely random relationship equals, 0.00, any relationship that has a correlation r-value that is 0.59 and below is not considered to be a reliable predictor.”

According to this Intel Teach Program a correlation between 0 and 0.19 is a very weak one while one between 0.2 and 0.39 weak enough.

True that there are many correlation charts out there and some do not agree in specific degrees or ranges, but they all tend to agree in one thing: that a correlation value below 0.20 is a very, very weak correlation, never deemed as evidence of variables being “highly correlated” as claimed by SEOMOZ in their LDA fiasco posts.

Second PS

Here is a list of reference links wherein these marketers make correlation claims based on quite weak correlation values and in the process keep misleading naive peers and the public. “Highly correlated”? “Remarkably well correlated?” Evidently Statistics is a Loss for SEOs.

http://www.seomoz.org/blog/lda-correlation-017-not-032

http://www.seomoz.org/blog/lda-and-googles-rankings-well-correlated

http://www.seomoz.org/blog/google-vs-bing-correlation-analysis-of-ranking-elements

The current issue of IRW is out and should reach subscribers during the day. 

In this issue we delve into our Tables of Correlation Features.  

The QA section addresses the question on how university administrators can allocate faculty to programs using an interesting formula.

The Who’s Who in CS is dedicated to one of my heroes: Wesley A. Clark.

Enjoy it and happy Holiday Season.

1. How-to instructions for reproducing the tables.
2. Additional statistics theory.
3. Working examples on how to use the tables.
4. An example on how results compare with G*Power software.

Enjoy it and Happy Thank-Day.

These are presented for the convenience of analysts and for use with statistical and practical significance tests. Readers requiring additional theory or statistical data corresponding to confidence levels and/or degrees of freedom not covered in the tables are referred to the literature.

In the future I will provide some practical applications of these. I hope you like the tables.

He wrote (and I quote):

“Scientific journals in most empirical disciplines have regulations about how authors should report the precision of their estimates of model parameters and other model elements. Some journals that overlap fully or partly with the field of demography demand as a strict prerequisite for publication that a p-value, a confidence interval, or a standard deviation accompany any parameter estimate. I feel that this rule is sometimes applied in an overly mechanical manner. Standard deviations and p-values produced routinely by general-purpose software are taken at face value and included without questioning, and features that have too high a p-value or too large a standard deviation are too easily disregarded as being without interest because they appear not to be statistically significant. In my opinion authors should be discouraged from adhering to this practice, and flexibility rather than rigidity should be encouraged in the reporting of statistical significance. One should also encourage thoughtful rather than mechanical use of p-values, standard deviations, confidence intervals, and the like. Here is why:”

Hoem then dissects five points related with misusing statistical significance results and automatic software solutions. I’m listing these below.

1. The scientific importance of an empirical finding depends much more on its contribution to the development or falsification of a substantive theory than on the values of indicators of statistical significance.
2. Measures of statistical significance may be misleading. When a model has been developed through repeated use of tests of significance to include and exclude covariates, to split or combine levels on categorical covariates, and to determine other model features, the user often loses control over statistical-significance values, and the values computed by standard software may be completely misleading.
3. Standard p-values can be insufficiently precise indicators of statistical significance, particularly if their values are given only in grouped levels, which are often indicated by asterisks beside parameter estimates (“* = p<0.1, ** = p<0.05, *** = p<0.01”, and so on).
4. It may be more important for an understanding of demographic behavior or other phenomena studied to know whether the inclusion of a categorical covariate in its entirety contributes significantly to an improvement of the model than to know the significance indicators of each of its levels.
5. Standard deviations, when used, should be reported for interesting contrasts, not for features selected automatically by statistical software.

I completely agree with Hoem.

SEOMOZ and their statistical “studies”

These days search engine optimization marketers (SEOs/SEMs) keep misinterpreting statistical results spitted from software without stopping and thinking about the significance-behind-the-significance, especially when it comes to a correlation coefficient, r (Pearson, Spearman, etc).

When one reads SEO hearsays and urban legends at SEOMOZ about very small correlation coefficients (0.17, 0.32, etc) derived from large sample sizes, as evidence that variables are “highly correlated” or “well correlated”, it is time to stop and put into question such “studies”. For reference see the following links

http://www.seomoz.org/blog/lda-and-googles-rankings-well-correlated
http://www.seomoz.org/blog/lda-correlation-017-not-032
http://irthoughts.wordpress.com/2010/04/23/beware-of-seo-statistical-studies/  

Fortunately, leading search marketers like Danny Sullivan has put into question those “studies” at a recent search engine conference

http://outspokenmedia.com/internet-marketing-conferences/evening-forum-with-danny-sullivan/ ,

and that was even before SEOMOZ admitted the 0.32 result has to be recanted as 0.17.

Sean Golliher, founder and publisher of the Search Engine Marketing Journal (SEMJ.org) has also put into question their results (http://www.seangolliher.com/2010/uncategorized/185/ ), which Hendrickson from SEOMOZ still insists in defending.

Since then they have never disclosed the source of the mistake, dismissing it just as a programming error. Unfortunately, they are still claiming that a 0.15 – 0.30 range validates their “studies” (http://www.seo.co.uk/seo-news/seo-tools/the-seomoz-lda-tool-%E2%80%93-our-disappointing-findings.html) .

Small and Large Sample Sizes

When William Sealy Gosset (aka “Student”) proposed, and Ronald A. Fisher expanded on, the test later termed Student’s t-test of significance, the test was meant to be used to assess information from small samples, not from large samples. I have discussed the case of small sample sizes in another post (http://irthoughts.wordpress.com/2010/10/18/on-correlation-coefficients-and-sample-size/ ).

In order to apply a t-test (and other small sample analysis tests) to large samples, divide-and-conquer techniques, like stratification, were eventually developed. In the case of correlation and regression, the reason for doing this is that applying something like a t-test to, for instance, a single correlation coefficient coming from a huge sample can produce misleading results. Let see why.

For large enough sample sizes eventually any correlation coefficient, even the smaller ones, will always be significant (t-observed > t-table). At that point it might be tempting to assume that the variables in question are highly correlated. Wrong assumption!

The fact is that statistical significance does not necessarily equate to variables being highly correlated and vice versa. Let address this point in two parts: (1) the question of statistical significance and (2) the question of high correlation.

Statistical Significance: Bigger not always is better

As noted in a Wikipedia entry, “given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero. (http://en.wikipedia.org/wiki/Effect_size ). I have discussed effect size and power analysis in a previous post (http://irthoughts.wordpress.com/2010/10/21/on-power-analysis-and-seo-quack-science/  ).

The reason for the above effect has a lot to do with the definition of statistical significance itself. Statistical significance is the confidence one has in a given result and that such a result is not by random chance.

In mathematical terms, the confidence that a result is not by random chance is given by the following formula by Sackett (http://en.wikipedia.org/wiki/Statistical_significance , http://www.cmaj.ca/cgi/content/full/165/9/1226 ):

Confidence = (Signal/Noise)*Sqrt[Sample Size]

This simple expression or derivatives of it appears in many different scenarios and disciplines. It describes a generic Confidence Function, F, in terms of a Signal, a Noise, and a Sample Size; that is, F(Signal, Noise, Sample Size). In general, such a generic function tells us that:

• Confidence is proportional to a Signal source (S).
• Confidence is inversely proportional to a Noise source (N).
• Confidence is proportional to a Signal-to-Noise ratio (S/N).
• Confidence is proportional to a Sample Size.

Let’s apply a version of this expression to correlation. To do this, let Y be the dependent variable and let X be the independent variable. Let also make the following substitutions:

• Confidence: expressed as t²
• Signal: expressed as r²; i.e., fraction of explained variations in Y (due to X).
• Noise: expressed as 1 – r²; i.e., fraction of unexplained variations in Y.
• Sample Size: expressed as degrees of freedom; i.e., n – 2 for a two-tailed test.

F(Signal, Noise, Sample Size) = t-observed² = [r²/(1 – r²] [n – 2]

Taking the square root (Sqrt) at both sides, we obtain the so-called formula for a two-tailed t-test.

t-observed = r*Sqrt[(n – 2)/(1 – r²)]

Evidently for a given r value, t-observed increases when n increases. By rearranging this expression, it is possible to compute for a large enough sample size a critical value above which r values will be significant. For very large samples at a 95% confidence level, t-table= 1.96 (http://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values ). Replacing arbitrarily this value in the above expression (t-observed = t-table = t = 1.96) and solving for r, we obtain that the critical r value is given by

r = t/Sqrt[(n – 2) + t²]

The following table lists values for very small r values and huge sample sizes. I’m intentionally using several decimal places and ignoring significant figure rules since I want to make a point on the small values used. I’m also using a 0.95 confidence level for illustration purposes, but for the large samples I could and should use other confidence levels as well.

For a sample size of 10,000 observations the critical r is 0.0196 or about 0.02, meaning that for such a huge sample size any r value above this small and critical r value will be significant. However, something interesting is observed from this table: (PS See footnote update)

When one moves to large sample sizes the Noise becomes greater than the Signal. For instance, at n = 10,000 the amount of Signal is very small (0.000384) while the amount of Noise is above 0.9996… or 99.96…%, giving a quite trivial S/N ratio. A similar reasoning can be applied for r = 0.17 (S = 0.0289, N = 0.9711) and r = 0.32 (S = 0.1024, N = 0.8976). The corresponding S/N ratios are trivial.

One can also solve the above expression for n to find sample sizes for some small r values and arbitrary t as shown in the following table (PS See footnote update).

Still, note that the amount of Noise completely overcomes the Signal, producing trivial S/N ratios. In general for small r and large n values significance is achieved at the cost of Noise masking the Signal. When this occurs the statistical significance is not a practical guideline for drawing useful conclusions from the data at hand.

This drives the present discussion to the substantive part of the problem missed by SEOs, and that is …

Statistical Significance Does Not Necessarily Mean Highly Correlated Results

Simply stated, statistical significance does not necessarily imply that the X, Y variables are highly correlated.

A simple scatterplot will convince anyone that for the above small r values there will be no pattern or trend in the data set. The corresponding regression model will be useless for forecasting or inferring anything of value, except that the data spreads so wildly that it has no method to its chaos. What else is to be expected from a data set with a large Noise and small S/N ratio?

This is something that SEOs/SEMs still don’t seem to understand: t-observed > t-table not necessarily means high correlation, and vice versa. I don’t have any personal stake (or take) against them, but when folks like Hendrickson, Fishkin, and others from SEOMOZ ignore Signal-to-Noise ratios and start referring to small r values as evidence that experimental variables are “highly” or “well” correlated, it is more than fair to call such “studies” Quack “Science”. That label might sound harsh, but in this case is appropriate.

Search engine marketers might be good at selling snakeoil, publishing sloppy “studies”, or recanting on overhyped statements, but not at doing real Science. They should know better; i.e., that

• “significance” does not mean “correlation”.
• “significance” does not mean “important”.
• “insignificance” does not mean “unimportant”.

Statistical “significance” only means that any confidence in the data is not by random chance. Therefore, a significant correlation does not necessarily mean a “high”, “well”, or “strong” correlation between variables.

To understand all this we need to distinguish between statistical significance and practical significance.

Statistical Significance vs. Practical Significance

As stated at this Wikipedia entry (emphasis added in boldfaces) http://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Criticism ):

A common misconception is that a statistically significant result is always of practical significance, or demonstrates a large effect in the population. Unfortunately, this problem is commonly encountered in scientific writing. Given a sufficiently large sample, extremely small and non-notable differences can be found to be statistically significant, and statistical significance says nothing about the practical significance of a difference.

Use of the statistical significance test has been called seriously flawed and unscientific by authors Deirdre McCloskey and Stephen Ziliak. They point out that “insignificance” does not mean unimportant, and propose that the scientific community should abandon usage of the test altogether, as it can cause false hypotheses to be accepted and true hypotheses to be rejected.

Some statisticians have commented that pure “significance testing” has what is actually a rather strange goal of detecting the existence of a “real” difference between two populations. In practice a difference can almost always be found given a large enough sample. The typically more relevant goal of science is a determination of causal effect size. The amount and nature of the difference, in other words, is what should be studied. Many researchers also feel that hypothesis testing is something of a misnomer. In practice a single statistical test in a single study never “proves” anything.

That pretty much settles the question of discerning between statistical significance and practical significance of correlation coefficients, but does not tell us how to quantitatively discern between the two concepts. In an upcoming article, I will derive expressions that might help to quantitatively assess these.

Since the tutorial on correlation coefficients http://www.miislita.com/information-retrieval-tutorial/a-tutorial-on-correlation-coefficients.pdf has been updated several times and is getting too long, I will put that upcoming material on a separate pdf file. As a sneak preview, we will be examining extreme cases (too high/low r values, too high/low sample sizes, and too high/low signal-to-noise ratios, etc.).

PS. I updated this post to fix some little typos.

Footnote. I found erroneous including the entries for n = 10 and n = 100 in the first table so I removed these altogether and limited the discussion to the large n values. A reader asked why I used t-table = 1.96 for all entries.  I thought it was clear from the discussion that the above tables are meant to show calculations for arbitrarily set t-values.  In a real test, you would need to use the actual t values from statistical tables. For instance, for n = 10 you would have to use a t-table value of  t = 2.306 at the 0.95 level. You should get

Too small samples are unethical, because the researcher cannot be specific enough about the size of, for example, the effect of a drug in a population. Too large samples are also unethical, because represent a waste of funding. True that a large sample improves precision, but it might involve an unjustified cost. Stratification is preferred, but it gets too complicated with huge sample sizes, not to mention that statistical significance not necessarily scales between samples.

As Rahul Dodhia from RavenAnalytics (http://ravenanalytics.com/Articles/Sample_Size_Calculations.htm ) indicates: a 2000-sample might not be very different from a 20000-sample, but a 200-sample maybe very different from a 2000-sample even when in each case the sample ratio is 10. So, a large sample not always is justified, even if such a sample size improves statistical significance and precision.

Consider the case of search engine ranking results. Upon a query, search engines are capable of finding many results, frequently in the range of thousand or million results per query. Still search engines and retrieval systems show to users a limited answer set. For instance, Google limits its viewable answer set to a maximum of 1,000 results (100 pages, 10 results/page).

Like in most retrieval systems, relevant results are accumulated at the first few result pages forming clusters. This is in agreement with Rijsbergen’s Cluster Hypothesis, which states that documents that cluster together have a similar relevance to a given query. Moving down the list of search results one often find cluster transitions wherein the quality and aboutness of documents is polluted with off-topic content.

Documents buried in a list of results often contain content irrelevant to the initial query or full of spam techniques. If one wants to conduct a statistical study of ranking results versus a particular document feature, one can do better by considering a sample from the first few result pages than from the entire answer set of 1,000 results.

In general, in a non-search engine scenario one cannot just arbitrarily select large samples to “force” the statistical significance of very low correlation coefficients and then use those values to draw conclusions. Furthermore, what is the selection criterion for using 1,000 or 10,000 results?

Simply stated: If 10,000 observations are arbitrarily selected, why not use 100,000 or 1,000,000 instead? We already know that very small correlation coefficients between any two arbitrary pair of random variables will be significant at those huge sample levels, anyway. And?

As noted in a Wikipedia entry, “given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero. (http://en.wikipedia.org/wiki/Effect_size ).

For example, a correlation coefficient of r = 0.04 would be significant at a 95% confidence level if coming from a 10,000-sample (t-calc = 4.003 >> t-table = 1.96) while a correlation coefficient of r = 0.01 would be significant at a 95% confidence level if coming from a 100,000-sample (t-calc = 3.162 >> t-table = 1.96). And? This proves nothing, especially when the magnitude of a “signal” approaches the magnitude of its “noise”.

As noted at the above Wikipedia entry, a correlation coefficient of 0.1 is strongly statistically significant when sample size is 1000, (t-calc = 3.175 >> t-table = 1.96) but reporting only the small p-value from this analysis could be misleading if a correlation of 0.1 is too small to be of interest in a particular application. (http://en.wikipedia.org/wiki/Effect_size ).

Statistical significance of extremely small r values is not surprising as is just a mathematical consequence of the fact that a t-value is a function (F) of a weighted ratio: the ratio of explained-to-unexplained variations weighted by the number of degree of freedoms:

F(r, n) = t = SQRT[(r²/(1 – r²))*(n – 2)]
F(r, n) = t = r*SQRT[((n – 2)/(1 – r²))]

For a given r value, increasing n increases t. No surprise here. One thing is what a math equation tells you and another different thing is what the nature and obvious boundaries of a physical system tell you.

At trivially low r values any claim with regards to the statistical significance or strength of some results proves nothing and one cannot do much with such trivial r values. For instance for r = 0.04, r² = 0.0016, meaning that 1 – r² = 0.9984 or 99.84% of the variations in the dependent variable (y) are not explained by variations in the independent variable (x).

In such a scenario, assessing the effect of x on y is a futile exercise. Such a model would be useless for drawing conclusions or predicting anything. And here is the point that many SEOs at SEOMOZ (http://www.seo.co.uk/seo-news/seo-tools/the-seomoz-lda-tool-%E2%80%93-our-disappointing-findings.html , Fishkin, Hendrickson, and others elsewhere) don’t seem to grasp:

When a correlation coefficient is useless for all practical purposes.

If the raw data constantly changes, that’s another “Chaos Layer” that compounds the problem.

Enters Cohen’s Power

According to Cohen’s work, when conducting a sample size study of correlation coefficients, one needs to consider the required confidence level and power of the test, the desired probability for Type I and Type II Errors, and the hypothesized or anticipated correlation coefficient (http://www.medcalc.be/manual/correlation_coefficient.php ). One cannot just use an arbitrary sample size for testing things.

In general, given any three of the following, the fourth one can be determined (http://www.statmethods.net/stats/power.html ):

1. sample size
2. effect size
3. significance level = P(Type I error) = probability of finding an effect that is not there
4. power = 1 – P(Type II error) = probability of finding an effect that is there

One also needs to consider what is the statistical parameter that is undergoing the power analysis. One needs to ask questions like the following:

Are we testing means from a given group? http://www.nss.gov.au/nss/home.nsf/pages/Sample+Size+Calculator+Description?OpenDocument

Are we testing means from different groups? http://www.ncbi.nlm.nih.gov/pmc/articles/PMC137461/

Are we testing correlation coefficients? Read Simon’s take on the impact of sample size on the desired level of precision in correlation coefficients (http://www.childrens-mercy.org/stats/weblog2005/CorrelationCoefficient.asp ).

Are we interested in significance level, effect size, sample effect, or power?

When conducting an effect size analysis one must keep in mind that effect sizes estimate the strength of a possible relationship, rather than assigning a significance level. However, effect sizes do not determine significance levels, or vice-versa.

So, how do we go about implementing Power Analysis?

For those interested in implementing power analysis written in the R Language, I recommend the libraries at http://www.statmethods.net/stats/power.html

Software for conducting power analysis is also available elsewhere, as shown in the following table. My favorites are G*Power and SPSS SamplePower (http://www.spss.com/software/statistics/samplepower/).

The problem with correlation strength scales is that these say nothing about how the size of a sample impacts the significance of a correlation coefficient. This is a very important issue that is now addressed.

Consider three different correlation coefficients: 0.50, 0.35, and 0.17. Assume that we want to test that there is no significant relationship between the two variables at hand. The null hypothesis (H0) to be tested is that these r values are not statistically different from zero (rho = 0). How to proceed?

As recommended by Stevens (17), for rho = 0, H0 can be tested using a two tailed (i.e.,two sided) t-test at a given confidence level, usually at a 95% level. If t_calculated ≥ t_table, H0 is rejected. However, if t_calculated < t_table H0 is not rejected and there is no significant correlation between variables.

Here t_calculated is computed as r/SEr = r*SQRT[((n – 2)/(1 – r²))] while t_table values are obtained from the literature (http://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values ). Table 2 summarizes the result of testing the null hypothesis at different sample size values.

The table addresses at which size level an r value is high enough to be statistically significant.

For n = 14, all three r values (0.50, 0.35, and 0.17) are not statistically different from zero.

For n = 30, r = 0.50 is statistically different from zero while r = 0.35 and r = 0.17 are not.

Conversely, r = 0.50 is not statistically different from zero when n is equal or less than 14 while r = 0.35 is not different from zero when n is equal or less than 30.

Finally, r = 0.17 is not statistically different from zero at any of the sample sizes tested.

In Part 1 we covered different types of inverted indexes: Boolean, non-positional indexes.

In Part 2 we covered some techniques for fast indexing.

In Part 3  we will be covering more on positional inverted indexes, examples, and techniques for fast intersecting posting lists.

What is Accuracy?

Accuracy is a term describing deviation of an experimental value from a target value. A target value is a value accepted as ‘true’. Constants, fundamental quantities, and theoretical values are considered ‘true values’. Thus, accuracy is proximity to a true value.

To illustrate, assume that a quantity x is measured. Its true value is x_t =1.00 and we report an experimental value x_e of 0.90. The absolute error of this observation is | x_e – x_t | = 0.10 and its relative error is (| x_e – x_t |/ x_t)*100 = 10%. The accuracy is the ratio between the experimental to true value. When expressed as a percent, it is called relative accuracy. In this case, x_e/ x_t = 0.90/1.00. This corresponds to a 90% accuracy.

What is Precision?

Precision has been loosely defined as how reproducible experimental results are. However, modern convention makes a careful distinction between reproducibility (between-run precision) and repeatability (within-run precision). Furthermore according to Freiser (1992),

• Repeatability is the closeness of agreement between individual experimental results obtained with the same method on identical test material or samples, under the same conditions (same operator, same apparatus, same laboratories, and same intervals of time).
• Reproducibility is the closeness of agreement between individual experimental results obtained with the same method on identical test material or samples, but under different conditions (different operator, different apparatus, different laboratories, and different intervals of time).

Note that the source of dispersion and errors in the experimental results is different in each case. Therefore arbitrarily expressing the precision of results in terms of standard deviations without considering how the data was collected (within- or between-run precision) should be avoided.

Similarly, comparing any two standard deviations, or standard errors for that matter, without regard for how the data was collected (experimental conditions, number of degrees of freedom, different sampling times, etc) should also be avoided. In particular, estimates of precision or comparisons of precisions from data set that constantly change within sampling times is a futile exercise.

Last but not least, the precision of a measurement depends on the measuring scale used. For instance, saying “He is about 55 years old.” is less precise than saying “He is 660 months old.” or than saying “He is 20,075 days old”.

References

Freiser, H. (1992). Concept Calculations in Analytical Chemistry. Chapter 12, p. 203. CRC Press, Boca Raton.

Miller, J. C. & Miller, J. N. (1984). Statistics for Analytical Chemistry. Chapter 1, p.19. Wiley, New York.

PS. I misplaced repeatability and reproducibility and fixed few more typos. Well and done. Thanks Dr. J. C. for pointing that out.

The implications of 6 and 8 dot-matrix notation systems to IR are many. After reading on the Nemeth Uniform Braille System, you might grasp the point.

Check also the unicode entities for braille at http://unicode.org/book/ch12.pdf

The current issue of IRW is out!

This is Part Two of the series on inverted index architectures, a 3-part series organized as follows:

Part One: Inverted Index Types
Part Two: Fast Indexing Techniques
Part Three: Fast Intersecting and Sharding

Tasks related with indexing, searching and processing are also discussed.

The QA section features short code liners in JavaScript aimed at helping readers understand what is tokenization and how is implemented.

Although not described in the newsletter, it is possible to construct these type of components with scripting languages. As a matter of fact, we have built an entire forward index and inverted index written entirely with JavaScript. Once computed, the inverted index can be written to memory. This work for small collections. For large collections, we read/write it to a text file using ActiveX, which is then posting-lists intersected in the usual way. However, for really large collections this is not effective and a database solution is recommended. The point to be made is that constructing a JavaScript-based search engine at the client and with real components, not a mere over-sized look-up “site search tool”, is possible. Since ActiveX is Microsoft’s land, it is not a universal solution. As a quick enterprise solution for short collections, it is ok, I guess.

After the hilarious example of this guy with the SEOMOZ LDA tool (http://smackdown.blogsblogsblogs.com/2010/09/09/proof-that-the-new-seomoz-tools-is-at-least-half-accurate/ ) I can only laugh out loud. Have anyone tried something like that?

Regarding the new fiasco with their LDA tool. Oh, no, another one… (http://www.seomoz.org/blog/lda-correlation-017-not-032) : What can I said? They sound pathetic and apologetic. The words overhyped, shitty, sloppy, flawed, etc are not enough to describe their “research work”.

What will happen now with those Mute Speakerphones that were misled? Those that listen to fools become one.

I don’t feel any sympathy for their 15 minutes of “honesty”. The damage was done already to naïve readers.

Also, note that this latest flaw was discovered by them. It was not the result of any peer review process from external referees, as those throwing a towel at them would like to believe.

As mentioned before, beware of SEOs statistical “studies” and their quack “science”  (http://irthoughts.wordpress.com/2010/04/23/beware-of-seo-statistical-studies/  ), especially if coming from SEOMOZ.

Probably their snakeoil will make a comeback soon. (Oh, no. Again?)

If they still think they have a valid LDA implementation, why not announce it at David Blei’s Topic-Models werein a community of LDA experts will review  it  and compare it against other implementations?

Two things can happen:

(a) It will be reviewed.

(b) it will be ignored.

I “invite” them to do so.

Please, just don’t show up with your snakeoil, yellow shoes, your seo mom, paid cheerleaders, vested investors, overhyped claims, etc, etc.

PS.

More on their hype machine here: http://skitzzo.com/archives/seomoz-hype-machine.php

 It appears that even Danny Sullivan is not buying SEOmoz’s “research” on LDA. Accordingly, “He didn’t think it was the remarkable change that SEOmoz was making it out to be.” (http://outspokenmedia.com/internet-marketing-conferences/evening-forum-with-danny-sullivan/). He even confronted and put into question their “highly correlated” numbers. And that was even before they recanted.

In this issue of IRW, we cover what some IRs consider the heart of a search engine: its inverted index. This is a 3-part subject, organized as follows:

Part One: Inverted Index Types

Part Two: Fast Indexing Techniques

Part Three: Fast Intersecting & Sharding

Enjoy it.