How do users and machines assess relevance? That is, how do they determine what is/is not relevant content upon a given information need like a query?

For users, this is mostly a visual experience. By contrast for machine this is mostly a parsing experience. So we end up with a fascinating machine learning problem.

Unless machines acquire human vision and rationalization this inherent mismatch will be there. Consider this:

Users Relevance Experience

User’s judgement of relevancy is a visual experience, influenced when their eyes scan a document and find presentation clues like:

1. apparent distribution of words presented in a preformatted text – on topic words close to one another.
2. font size – large text is associated with text conveying a prominent message.
3. font style – bolded, underlined, or italized text is associated with outstanding meaning.
4. color – most Web users still tend to associate or recognize text in blue as links.
5. HTML/CSS positioning – either relative or absolute positioning of specific portions of content can artificially affect their perception of relevance.

To convince yourself, look at an online newspaper or magazine. Better: do a self-assessment of your own perception experience while reading this document.

Graphics, texture, shades, even sound files can also influence users’ perception of relevance.
Still not convinced? Ask traditional publishers and editors for a second opinion. They are good at mastering the perception of relevancy.

Machines Relevance Experience

Their judgement of relevancy is a parsing experience. Unlike users, their judgement is not influenced by presentation, but by things like:

1. the effective distribution of words in a linearized text stream. This is the pseudo-document obtained after applying markup removal, tokenization, filtration, and stemming (optional) to the original document.
2. weight scores assigned to terms, concepts, and links according to a given scoring scheme.
Indeed, the visual clues that users pay attention to are almost ignored by machines.

There might be other reasons like off-page factors (e.g., in/out links, etc), but these are common things that produce a gap between what users perceive as relevant and what search engines score as relevant.

The problem with visual clues is that these can be manipulated at will by end-users. That’s a good reason for not assigning weights to presentation elements.

The way I see it, SEOs should try to minimize the user-machine perception mismatch. They can do this by finding a happy medium through proper document optimization strategies. Certainly arbitrarily web design, without proper IR knowledge -e.g., how do search engines index and score documents- enhances this gap.

Ironically, some SEOs do not understand any of this or don’t want to hear about IR. Often, these waste their time building “relevance strength tools” while ignoring the most basic IR principles. In my book such tools are just futile exercises of garbage-in-garbage-out. These folks should get some basic knowledge on IR before publishing their non sense on a Web screen or resourcesing to hearsay.