IRW:2010-9: Inverted Index Architectures Part Two

The current issue of IRW is out!

This is Part Two of the series on inverted index architectures, a 3-part series organized as follows:

Part One: Inverted Index Types
Part Two: Fast Indexing Techniques
Part Three: Fast Intersecting and Sharding

Tasks related with indexing, searching and processing are also discussed.

The QA section features short code liners in JavaScript aimed at helping readers understand what is tokenization and how is implemented.

Although not described in the newsletter, it is possible to construct these type of components with scripting languages. As a matter of fact, we have built an entire forward index and inverted index written entirely with JavaScript. Once computed, the inverted index can be written to memory. This work for small collections. For large collections, we read/write it to a text file using ActiveX, which is then posting-lists intersected in the usual way. However, for really large collections this is not effective and a database solution is recommended. The point to be made is that constructing a JavaScript-based search engine at the client and with real components, not a mere over-sized look-up “site search tool”, is possible. Since ActiveX is Microsoft’s land, it is not a universal solution. As a quick enterprise solution for short collections, it is ok, I guess.

On the SEOMOZ LDA Fiasco

 LDA and Google’s ranks well correlated?

After the hilarious example of this guy with the SEOMOZ LDA tool (http://smackdown.blogsblogsblogs.com/2010/09/09/proof-that-the-new-seomoz-tools-is-at-least-half-accurate/ ) I can only laugh out loud. Have anyone tried something like that?

Regarding the new fiasco with their LDA tool. Oh, no, another one… (http://www.seomoz.org/blog/lda-correlation-017-not-032) : What can I said? They sound pathetic and apologetic. The words overhyped, shitty, sloppy, flawed, etc are not enough to describe their “research work”.

What will happen now with those Mute Speakerphones that were misled? Those that listen to fools become one.

I don’t feel any sympathy for their 15 minutes of “honesty”. The damage was done already to naïve readers.

Also, note that this latest flaw was discovered by them. It was not the result of any peer review process from external referees, as those throwing a towel at them would like to believe.

As mentioned before, beware of SEOs statistical “studies” and their quack “science”  (http://irthoughts.wordpress.com/2010/04/23/beware-of-seo-statistical-studies/  ), especially if coming from SEOMOZ.

Probably their snakeoil will make a comeback soon. (Oh, no. Again?)

If they still think they have a valid LDA implementation, why not announce it at David Blei’s Topic-Models werein a community of LDA experts will review  it  and compare it against other implementations?

Two things can happen:

(a) It will be reviewed.

(b) it will be ignored.

I “invite” them to do so.

Please, just don’t show up with your snakeoil, yellow shoes, your seo mom, paid cheerleaders, vested investors, overhyped claims, etc, etc.

PS.

More on their hype machine here: http://skitzzo.com/archives/seomoz-hype-machine.php

 It appears that even Danny Sullivan is not buying SEOmoz’s “research” on LDA. Accordingly, “He didn’t think it was the remarkable change that SEOmoz was making it out to be.” (http://outspokenmedia.com/internet-marketing-conferences/evening-forum-with-danny-sullivan/). He even confronted and put into question their “highly correlated” numbers. And that was even before they recanted.

IRW:August: Inverted Index Architectures: Part One

The current issue of IRW is out:

In this issue of IRW, we cover what some IRs consider the heart of a search engine: its inverted index. This is a 3-part subject, organized as follows:

Part One: Inverted Index Types

Part Two: Fast Indexing Techniques

Part Three: Fast Intersecting & Sharding

Enjoy it.

On SEOs Latest Deceiving Artifact (LDA)

Back in 2009/04/03 we wrote a nice comparative between LDA, LSI, and Vector Space theory. http://irthoughts.wordpress.com/2009/04/03/vector-space-probabilistic-lsi-and-lda/  LDA was also discussed by its creator (David Blei) at the 2006 IPAM’s Document Space Workshop (http://www.miislita.com/ipam/ipam-document-space-workshop.pdf ). Years before, in 8/25/2006, we wrote in an old asp-based blog a post about warning users against SEOs selling snakeoil in the form of SVD, LSI, and LDA arguments. The problem with these approaches is that they don’t scale well for the Web. I ended up that 2006 post with a prediction:

“At this point I got tired of highlighting more flaws in the claims of these search marketing firms. A sample list of the latest LSI myths is available for your perusal.

Next stop

Next stop for these snakeoil marketers? How about PCA (Principal Component Analysis) or LDA (Latent Dirichlet Allocation)?”

That post was eventually referenced in a rebuttal I posted at that cesspool of quacks known as seomoz and later fully reproduced at this blog in 2007/05/03 (http://irthoughts.wordpress.com/2007/05/03/latest-seo-incoherences-lsi/).

It was a matter of time for johnny-comes-late to “discover” LDA, the Niagara Falls and the Grand Canyon. Oh my God, what a “bombshell”.

Expect a new wave of marketers trying to game naïve cheerleaders and their clients with their latest crap.

Nothing new under the Sun. Will the next stop of these snakeoil marketers disguised as “scientists” be NMF? How about Diffusion Geometries?

PS

One more thing, for those that really want to learn LDA: subscribe to Topic-Models at  https://lists.cs.princeton.edu

This is a list forum on LDA  run by David Blei and others. I’ve being subscribed for many years now and the discussion on the topic is really useful.