The current issue of IRW arrived last week to subscribers. Look what non-subscribers missed:.

Featuring article: Subnetting and Security

“The purpose of subnetting is to break large networks into smaller networks called subnets by borrowing bits from the network portion of an IP address. This is done by using a variable length subnet mask. As the IP addresses of host computers on each subnet are masked by the network address these are invisible to those outside the network. In this sense, subnetting benefits security. Conversely, the cost of misconfigured subnets creates Internet vulnerabilities. For those working at the intersection of Information Security, IP address masking and subnetting might be relevant when it comes to analyzing spoofing, penetration hacking, and subnet advertisements. Accordingly, in this issue of the newsletter we present a straightforward approach on the art of subnetting.”

Enjoy it.

Nine years ago the great Mike Muuss, inventor of PING and many other ground  breaking software tools, was killed in a car accident. Almost a decade later we pay respect to his memory in the current issue of IR Watch (to be delivered a bit late in few days). The Who is Who column of IRW features the following:

Michael John Muuss (October 16, 1958 – November 20, 2000), a multi-talented computer wizard who helped lay the foundations for the modern-day Internet, was killed at the age of 42 in an automobile accident near his home in Havre de Grace, while returning home from a restaurant, when his car was involved in a multivehicle pileup on Interstate 95. At 9 years from his death, we pay respects to his memory. The following are excerpts from his Obiturary. (Obituary from The Baltimore Sun Company: http://www.ping127001.com/pingpage/muuss.htm).

A graduate of the Johns Hopkins University, Muuss spent his entire career at the U.S. Army Research Laboratory (ARL) at Aberdeen Proving Ground, where he established a reputation as an enthusiastic problem-solver who did groundbreaking work in areas ranging from computer networks to graphics.

He is best known for inventing PING, one of the most widely used IP address retrieval and diagnostic tools for computer networks in the world and used in almost all PCs.

Contrary to popular opinion/urban legends, the PING name was not intended to be a ping pong analogy or to stand for Packet Internet Grouper. According to his own words (http://ftp.arl.mil/~mike/ping.html):

• “I named it after the sound that a sonar makes, inspired by the whole principle of echo-location.”
• “From my point of view PING is not an acronym standing for Packet InterNet Grouper, it’s a sonar analogy.“

In the early 1980s, Muuss helped lay the technological foundation that would transform what was then called the ARPANET, back then an obscure military computer network created in 1969 by the Department of Defense, into the modern-day Internet. (http://ftp.arl.mil/~mike/).  

Muuss also created BRL-CAD, a program that allowed the military to create sophisticated 3-D models. Over the years, BRL-CAD has become one of the Army’s most-licensed technologies and is used to model everything from tanks to brain tumors.

His work in computer security landed him a cameo appearance in Clifford Stoll’s 1989 hacker classic “The Cuckoo’s Egg,” a nonfiction thriller about the hunt for an international band of computer criminals. Muuss was also known for tracking down crackers.

In 1990, he was one of the government’s key witnesses in the case against Robert Tappan Morris, whose “Morris Worm” in 1988 nearly brought down the Internet.
(Case Sentence: http://www.nytimes.com/1990/05/05/us/computer-intruder-is-put-on-probation-and-fined-10000.html?scp=2&sq=robert+tappan+morris&st=nyt
Case Appeal: http://morrisworm.larrymcelhiney.com/morris_appeal.txt).

In 1999 and at the age of 41, Muuss was given the Research and Development Achievement Award, the Army’s highest civilian award for scientific accomplishments. Before his death, he assembled an impressive review on the History of Computing Information (http://ftp.arl.army.mil/~mike/comphist/).

Two new tutorials on Internet Engineering are available now from Mi Islita.com:

IP Packet Fragmentation Tutorial [pdf]

MTU and MSS Tutorial [pdf]

Both are based on lecture material provided in class during the IE Part 1 course. These can be used as reference material for other courses such as Network Security, Internet Engineering, Internet Architecture, etc.

The first one is an introduction to packet fragmentation analysis while the second one delves into experimental techniques on maximum transmission unit and maximum segment size calculations.

The tutorials can be used to understand Ping of Death, Fragmentation Offset Attacks, Tiny Fragmentation Attacks, Firewall Fragmentation Attacks and other forms of hacks.

Enjoy it.

If you are into homeland security oriented data mining, this post is for you.

The University of Maryland has a Global Terrorism Database (GTD; http://www.start.umd.edu/gtd/) with information on over 80,000 terrorist attacks that intelligence researchers can tap into.

GTD is an open-source database including information on terrorist events around the world from 1970 through 2007 (with annual updates planned for the future). Unlike many other event databases, the GTD includes systematic data on domestic as well as international terrorist incidents that have occurred during this time period and now includes more than 80,000 cases.

You can search by keywords or browse by region, country, perpetraror, weapon, attack, or target.

It also has advanced search capabilites. To perform an advanced search you need to select all categories you wish to search. If you do not check any options then your search will include all content from that category, for example, selecting Algeria from the “Country” list will restrict your search to incidents in Algeria, while leaving it blank searches all countries.

Incident searches can be restricted to specific years using several pull-down menus.

I tested by querying [puerto rico] and indeed was able to obtain incident records related with Los Macheteros. The answer set, however, included results not relevant to the Island of Puerto Rico.

The database is pretty small, but can come handy at times. Definitively, I will use it for one of my next graduate courses on search engine architectures.

The october issue of IRW newsletter is out, but late due to my academic duties. It should arrive to subscriber’s inbox today (or at the latest tomorrow). Sorry for the inconvenience.

In the featuring article, I included material from one of my grad exams. The QA column features IP-to-MAC conversions. The Who is Who section features Van Jacobson, creator of UDP-based traceroute and of many other tools.

Hopefully, the november issue will not be that delayed. BTW, it will feature the late Mike Muuss, inventor of Ping plus some super-fast track on the art of subnetting.

Ellen Voorhees, Director of TREC at NIST.Gov sent me this Call for Participation, reproduced below to facilitate its dissemination:

CALL FOR PARTICIPATION

FIRE
(Forum for Information Retrieval Evaluation)
Workshop
DAIICT, Gandhinagar, India
19-21 February 2010

http://www.isical.ac.in/~fire

The success of TREC, CLEF, and NTCIR has clearly established the importance
of building reusable, large-scale standard test collections in Information
Access research. The aim of FIRE is to encourage research in Indian language
Information Access by creating a similar platform for Indian languages that
provides the data and a common forum for comparing models and techniques.

The Tasks:
==========
1) Ad-hoc monolingual document retrieval in Bengali, Hindi and Marathi.

2) Ad-hoc cross-lingual document retrieval
- documents in Bengali, Hindi, Marathi, and English,
- queries in Bengali, Hindi, Marathi, Tamil, Telugu and English.
- Bengali and Hindi topics will also be transliterated and made available
in Roman script. Adhoc monolingual task participants are encouraged to
submit runs using these queries as well.

3) Retrieval and classification from mailing lists and forums.
This is a pilot task being offered by IBM India Research Lab.

4)  Ad-hoc Wikipedia-entity retrieval from news documents
- Entities mined from English Wikipedia
- Query documents from English news website
This is a pilot task being offered by Yahoo! Labs, Bangalore.

Important Dates:
================
Ad-hoc monolingual and cross-lingual document retrieval:
Training data release   Aug 15 ‘09
Test data release       Nov 01 ‘09
Adhoc run submission    Nov 25 ‘09
Results released        Feb 01 ‘10

Retrieval and classification from mailing lists and forums:
Training data release   Oct 16 ‘09
Test data release       Nov 01 ‘09
Run submission          Nov 25 ‘09
Results declared        Feb 01 ‘10

Ad-hoc Wikipedia-entity retrieval from news documents:
Training data release   Oct 15 ‘09
Test data release       Nov 01 ‘09
Run submission          Nov 25 ‘09
Results declared        Feb 01 ‘10

Task Co-ordinators:
===================
Ad-hoc retrieval:
Pushpak Bhattacharyya (pb@cse.iitb.ac.in)
IIT Bombay
Dipasree Pal (dipasree_t@isical.ac.in)
ISI Kolkata

Retrieval and classification from mailing lists and forums:
Debapriyo Majumdar (debapriyo@in.ibm.com)
IBM India Research Lab
Ayan Bandyopadhyay (ayan_t@isical.ac.in)
ISI Kolkata

Ad-hoc Wikipedia-entity retrieval from news documents:
Ashwin Tengli (ashwint@yahoo-inc.com)
Yahoo! Labs, Bangalore
Pabitra Mitra (pabitra@cse.iitkgp.ernet.in)
IIT Kharagpur

Overall co-ordinators:
Prasenjit Majumder (p_majumder@daiict.ac.in)
DAIICT, Gandhinagar
Mandar Mitra (mandar@isical.ac.in)
ISI Kolkata

International Advisory Committee for FIRE:
==========================================
Amit Singhal, Google Fellow, USA
Carol Peters, ISTI-CNR, Italy
Christian Fluhr, CEA, France
Donna Harman, National Institute of Standards and Technology, USA
Doug Oard, University of Maryland, USA
Ee Peng Lim, Nanyang Technological University, Singapore
Ellen Voorhees, National Institute of Standards and Technology, USA
Fabrizio Sebastiani, ISTI-CNR, Italy
Gareth Jones, Dublin City University, Ireland.
Hsin-Hsi Chen, National Taiwan University, Taipei, Taiwan
Hwee Tou Ng, National University of Singapore, Singapore
Iadh Ounis, University of Glasgow, UK
Ian Soboroff, National Institute of Standards and Technology, USA
Jacques Savoy, University of Neuchatel, Switzerland
James Allan, University of Massachusetts Amherst, USA
Krishna Kummamuru, IBM Research Lab, India
Mark Sanderson, University of Sheffield, UK
Mun Kew Leong, Institute for Infocomm Research, Singapore
Norbert Fuhr, University of Duisburg, Germany
Noriko Kando, National Institute of Informatics, Japan
Paul McNamee, Johns Hopkins University, USA
Prabhakar Raghavan, Yahoo! Research Labs, USA
Ricardo Baeza-Yates, Yahoo! Research Labs, Spain
Stephen Robertson, Microsoft Research, Cambridge, UK
Sung Hyon Myaeng, KAIST, South Korea
Tat-Seng Chua, National University of Singapore, Singapore
Tetsuya Sakai, Microsoft Research Asia, Beijing

Today’s Internet Engineering Part 1 course lecture will be on DNS Intelligence and how we can use DNS records to understand virus and worm attacks as well as remote network topologies. Quite handy these days.

Please check Lecture 8

According to a Techcrunch 10-10-09 news a crash at Microsoft’s Danger servers resulted in the lost of all user personal data and they don’t have a backup!

The news says:

T-Mobile and Danger, the Microsoft-owned subsidiary that makes the Sidekick, has just announced that they’ve likely lost all user data that was being stored on Microsoft’s servers due to a server failure. That means that any contacts, photos, calendars, or to-do lists that haven’t been locally backed up are gone.

And there is no backup for the data. Really smart, Microsoft people. That says a lot!

This is gonna be in an information security textbook near you. How about in a textbook on Human-Computer No-Interaction?

If you are enrolled in the IE-Part 1 course, here is some reference material on Email Headers for today’s lecture:

Exposing email headers

http://www.abs-comptech.com/EmailHeaders.htm

Tracking the source of email spam

http://www.rahul.net/falk/mailtrack.html

How to read email headers

http://www.emailaddressmanager.com/tips/header.html

Reading the email header

http://antivirus.about.com/od/windowsbasics/a/emailheaders.htm

Reading email headers

http://www.tinhat.com/email/read_email_headers.html

Spamlinks: Reading email headers

http://spamlinks.net/track-trace-headers.htm

ACCC: Reading Email Headers

http://www.uic.edu/depts/accc/newsletter/adn29/headers.html

E-mail Headers and SMTP Commands

http://www.avolio.com/columns/E-mailheaders.html

All About Email Headers

http://www.stopspam.org/index.php?option=com_content&view=article&id=45&Itemid=56

Security Optimization Strategies in the Workplace

http://www.miislita.com/searchito/security-optimization-strategies.html

If you are a student enrolled in the Internet Engineering I graduate course, check the Lecture 7 update.

We will be covering email protocols such as SMTP, POP3, and IMAP. The exercise section covers email headers intelligence and email crawlers.

If you are a student enrolled in the Internet Engineering I graduate course, check the Lecture 6 update.

I will be covering all about DNS configuration files. For the hands-on exercise section, we will be using nslookup commands to snoop at all relevant records of remote Web domains.

Use nslookup/? to access the options helper
Use nslookup followed by ? in a different line to access the commands helper
To quit nslookup, press ctrl C or either type quit or exit.

Two weeks ago, venerable Vinton Cerf urged the Internet community to migrate from IPv4 to IPv6. According to Cerf, co-designer of the TCP/IP protocols, IPv4 will run out of addresses next year or in early-2011.

However, there is a problem.

Back in March, it was reported of an allegued Fatal Flaw for IPv6: it’s Not Backwards Compatible

Both news are equally intriguing.

IPv6 migration: Your Next Nightmare?

The following are the lecture and exercise topics covered in the PUPR.edu core graduate course Internet Engineering, Part I. Students enrolled in the course might want to revisit this post as it will be updated.

Lecture 0

History of the Internet & Search Engines

Internet Basics

Lecture 1

RFCs (Request for Comments)

Network Types

IP (Internet Protocol)

Exercise 1 – RFCs, Network types, IP calculations

Lecture 2

OSI Reference Model

ARP

ICMP

Exercise 2 – IP-MAC Mapping, Prompt Commands (arp, ipconfig, nslookup)

Lecture 3

Man-in-the-Middle ARP Attacks

IGMP

IP Packets

Exercise 3 – Broadcast & Multicast IPs, Prompt Commands (netstat, ping, tracert, ipconfig, arp, nslookup)

Lecture 4

Fragmentation Offset

FO Overlapping Attacks

FO Gap Attacks

Tiny FO Attacks

TCP Protocol & Buffers

Exercise 4 – TCP buffers, Congestion Windows, Advertised Windows

Lecture 5

PING

PING of Death

Smurfing

TRACEROUTE-based Intelligence

Exercise 5 – Prompt Commands (arp, ipconfig, nslookup, netstat, ping, tracert)

Lecture 6

BIND & WINDOWS DNS (Domain Name Server)

Internet backbone root servers

Configuration Files

DNS Configuration Errors

Forward Lookup (Zone) Files

Reverse Lookup Files

Exercise 6 – Prompt Commands (interactive/non-interactive nslookup modes)

Lecture 7

SMTP

POP3

IMAP

Email Headers

Exercise 7 – Email Intelligence.

Lecture 8

DNS Intelligence

Using DNS records to understand Virus & Worm Attacks

Network Topology Intelligence from DNS records

Exercise 8 – DNS Intelligence

Lecture 9

General Review

Practice Test

Lecture 10

Final Exam, Oct 27

Course Grading System

8 out of 9 hands-on exercises count (worse exercise grade dropped)
1st partial exam = average of first best 4 exercise grades
2nd partial exam = average of last best 4 exercise grades
The average of these two is the same as adding up best 8 grades and dividing by 8. This result amounts to 75% of total grade (course letter grade score).

Final Exam amounts to 25 % of total grade.

After that, course letter grade is curved as shown below.

A (100-89%)
B (88-77%)
C (76-60%)
D (59-50%)
F (49-0%)

where

course letter grade score = (sum of best 8 exercise grades/8)*(0.75) + (final exam grade)*(0.25)

The current issue of IR Watch is out.  Sorry it was a bit delayed. The featuring article is a practice exam on TCP/IP that I’m giving to students enrolled in my Internet Engineering I graduate course.

The test was designed to review what students have learned during the first five lectures. Students need to describe about 10 TCP/IP-related vulnerability/hacking practices. So the test also is a great jump start for those interested in such weaknesses.

I have included an Excel gooddie for making IP conversions (IPv4/hexadecimal/decimal equivalent/binary) as well as some material from Tim Berners-Lee 1989 WWW proposal.

Enjoy it.

As PUPR students know by now, the AIRWeb and Internet Engineering courses have been consolidated into a single course called Internet Engineering I (IE-I), which is on Tuesday’s.

This was a decision made strictly by the administration. 12 graduate students are enrolled –a big number for a grad course. We are now in the fourth week of IE-I and I can tell that is a lot of fun.

This coming Winter semester I’m scheduled to teach a new grad course called Advanced Search Engine Architecture (ASEA). Both, IE-I and ASEA are hands-on. This means students need to get their hands and feet wet, not just learning the theory.

What we are trying to accomplish in IE-I is to understand how hackers and spammers use Internet architectures at the level of TCP/IP and Search Engines to game the system. I’ll open a special blog category for it during the week.

First lecture (Lecture 1) was briefly summarized in the August 2009 issue of IR Watch. BTW. Tonight’s lecture (Lecture 4) covers the following:

IP Protocol (MAC and IP Mapping)

ICMP Protocol

ARP Hacking Attacks

ICMP Hacking Attacks

Firewall’s Fragmentation Offset  Attacks

Meanwhile, ASEA is an expanded version of the previous Search Engine Architecture (SEA) course I’ve taught before. Students interested in registering, can search this blog for the SEA category and check what we have covered in the past. This will give them an idea of what to expect from the Advanced SEA course. One thing I’m planning to do different is to build an inverted index from scratch using AJAX. The most recent version of Terrier will also be used for testing/benchmarking experimentals.

Last but not least, September Issue of IRW will be a bit delayed.

Here is the paper that started all: A Protocol for Packet Network Intercommunication, by Vince Cerf and Bob Kahn.

The current issue of IRW is already out –a bit delayed due to reasons previously mentioned. Enjoy it.

I’ve been asked what is the standard notation for vectors. I normally use loose notation, unless I need to write or review a formal piece, in which case I follow the APS style. See also here.

A vector should be represented by a letter, in boldface or with a right arrow on top.

A caret should be used to indicate a unit vector.

An inner product should be indicated by placing a dot between two letters representing vectors.

Note that Dirac Notation is a different animal.

The APS Style Guide has additional guidelines.

I came across Thesaurus as a complex network, a fascinating 2003 paper written by Adriano de Jesus Holanda, Ivan Torres Pisa, Osame Kinouchi, Alexandre Souto Martinez and Evandro Eduardo Seron Ruiz from Universidade Sao Paulo, Brazil in which they model thesauri using graph theory. The abstracts reads:

“A thesaurus is one, out of many, possible representations of term (or word) connectivity. The terms of a thesaurus are seen as the nodes and their relationship as the links of a directed graph. The directionality of the links retains all the thesaurus information and allows the measurement of several quantities. This has lead to a new term classification according to the characteristics of the nodes, for example, nodes with no links in, no links out, etc. Using an electronic available thesaurus we have obtained the incoming and outgoing link distributions. While the incoming link distribution follows a stretched exponential function, the lower bound for the outgoing link distribution has the same envelope of the scientific paper citation distribution proposed by Albuquerque and Tsallis [1]. However, a better fit is obtained by simpler function which is the solution of Ricatti’s differential equation. We conjecture that this differential equation is the continuous limit of a stochastic growth model of the thesaurus network. We also propose a new manner to arrange a thesaurus using the “inversion method”.”

The study is important because it provides an interesting look at word relationships. They have identified an underlying power law, which in my opinion might be worth to be investigated as to whether it is at core of semantic relationships.

They briefly mentioned the limitations of LSA.:

“However, LSA has been criticized as a poor approach for predicting semantic neighborhood”.

Indeed, LSA (or LSI) not necessarily describes or predicts semantics, as originally thought. In my view, LSA/LSI itself is a misnomer. Research references can be provided to support this view.

I do have one additional comment on the paper. In it, LSA is described as a PCA technique. The authors write:

“Another interesting way to treat data is the Latent Semantic Analysis (LSA) [5] which deals with word covariance in a corpus. LSA is a principal component analysis (PCA) technique , i.e., the covariance matrix is diagonalized and from the most important eigenvalues (around 300) the eigenvectors are considered to span an Euclidean vector space.”

This might not be entirely accurate. Let see why:

1. PCA was invented by Karl Pearson in 1901 so is more than half a century  older than Golub and Kahan’s SVD algorithm which was published in 1965. See G. Golub and W. Kahan, J. SIAM, Numer. Anal. SEr. B, Vol 2, No. 2 (1965). 

2. In 1988 Dumais, et al applied Golub’s SVD to text and called that LSA (LSI). See Proceedings of the Conference on Human Factors in Computing Systems, CHI. 281-286, Dumais, S. T., Furnas, G. W., Landauer, T. K., Deerwester, S. & Harshman, R. (1988). See also, Improving information retrieval using Latent Semantic Indexing. Proceedings of the 1988 annual meeting of the American Society for Information Science. Deerwester, S., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Beck, L. (1988).

3. In LSA (LSI) the SVD algorithm can be applied to matrices that not necessarily are populated with covariance values.

4. It was only later realized that SVD can be applied to a covariance matrix to obtain the PCA components.

5. See the PCA & SPCA Tutorial

6. PCA is not LSI. See http://irthoughts.wordpress.com/2007/05/05/pca-is-not-lsi/

1. The current issue of IR Watch will be out over the weekend–a bit delayed due to getting ready for school, preparing lessons and research projects. If things go as expected, my academic schedule will be a bit busy between teaching and research at two different universities.

2. I’m researching for a manuscript that deals with affine transformations applied to several IR problems. It expands on Vector Space Theory and allows one to think out of the “term-document” box. Great stuff.

3. Here is a great grad project in ppt format: Semantically Motivated Information Retrieval. I thank its author for referencing my  SVD Fast Track Tutorial.

4. Talking about semantically motivated, sentiment analysis, spam, etc… Funny how some folks in the SEO world like to damage the reputation of others without presenting any evidence. This time the trolls took on Kim Krause Berge ( http://cre8pc.com/archives/1489 ). I always admire Kim’s work, consider her an usability icon, and had the privilege of meeting her back in 2005. I was surprised to see these folks having a field day at her expense at Rand’s site. Kim, I feel your pain. However, more than one SEO forum/blog had lose credibility by allowing these folks, most of which think they can be socially “ranked” by attacking whoever is at the “top”. The fact is that most trolls are paper tigers that go hidding at the first Cease & Desist or defamation lawsuit.

I am intrigued with the subject of data mining poetry. This is an interesting topic for a grad student thesis since:

EXACT querying search engines for “data mining poetry” returns a small answer set.

Unlike other type of content, all words, including those considered stopwords, might matter; i.e., these must be counted as might act as content-bearing terms –thus, there is no such thing as stopwords in poetry.

Word statistics (e.g., word counts per lines) and specific tokens matter, unless we talk about the so-called free-style poetry.

Metric makes poetry suitable for building language-specific and writing style-specific parsers.

Any help will be kindly appreciated. Meanwhile, here are some relevant links:

poetry parser, anyone?

Poema Materotico – This is a blog-popular poem written in Spanish

poemas matematicos – Spanish resources

mathematical poems

mathematical poetry

A day ago Michael Arrington’s Techrunch published excerpts from “leaked” documents stolen from the Google Apps account of a Twitter Employee which included over 300 confidential files meant for “internal” Twitter consumption. “Hacker Croll” sent TechCrunch a zip file with 310 private files from inside Twitter.
(http://www.techtree.com/India/News/Leaked_Documents_Twitter_TechCrunch_Faceoff/551-104503-643.html).

It appears HC essentially used a cracker tool of some sort to brute-guess weak passwords. Once inside the first security ring, …

Cloud Programs: A Web Vulnerability Paradise for Hackers

Twitter relies heavily on cloud-based apps (Web-centric programs such as Google Docs or Web-based e-mail), and these services are becoming increasingly interconnected. Even social Web apps are beginning to share data: Facebook Connect and Google Friend Connect, for example, let you log in to multiple sites with a simple Facebook or Google account, raising the vulnerability of your entire online identity.
(http://www.switched.com/2009/07/17/twitter-employee-accounts-hacked-business-documents-leaked/)

The documents coming out of the hacker seem to be pretty significant. The “problem” is that if you have a Google Apps email account compromised, you also have shared calendar, Docs, Contacts, Wikis(Sites), etc.
(http://www.pcworld.com/article/168572/google_apps_security_questioned_after_twitter_leak.html)

This might be a good case study for students planning to take the AIR Web: Web Spam and Internet Vulnerability course.

Google yesterday announced Google Voice for Android and BlackBerry, a cell phone mobile service which brings voicemail transcriptions, the ability to call and text with your Voice number, and cheap international dialing to yourmobile phone. It is like one number to rule them all!

According to Google Mobile Blog:

“The Google Voice app integrates seamlessly with your phone’s native address book, making it even easier to call or text with your Voice number. Voicemail transcriptions are now available, and the app will highlight individual words during playback just like your favorite karaoke song. It also lets you take advantage of Google Voice’s low-priced international call rates, starting at only $0.02/minute.”

It is expected the service to eventually serve as a glue for other Google services like GMail, Web Searches, YouTube, SMS, etc.

These are great news, considering that as mentioned in the current issue of IRW, texting is a new playground for data miners. Imagine then a similar mining playground involving voice!

http://googlemobile.blogspot.com/2009/07/google-voice-for-android-and-blackberry.html

See also

http://searchengineland.com/google-voice-for-mobile-one-number-to-rule-them-all-22417

It is surprising how even serious information retrieval researchers and journals quote papers that were never written!

This is the thesis of David Dubin’s 2004 great article
The Most Influential Paper Gerard Salton Never Wrote

Dubin wrote:

“In giving credit to Salton for the vector model, a number of authors cite an overview paper titled “A Vector Space Model for Information Retrieval,” which some show as published in the JASIS in 1975 and others as published in the Communications of the Association for Computing Machinery (CACM) in 1975. In fact, no such article was ever published, and citations to it usually represent a confusion of two 1975 articles (Salton, Wong, & Yang, 1975; Salton, Yang, & Yu, 1975), neither of which were overviews of the VSM as it is generally understood (see section 5 below). Some of Salton’s own colleagues have been guilty of this mistake: both Cardie et al. and Singhal cite the CACM version, for example (Singhal, 2001; Cardie, Ng, Pierce, & Buckley, 2000). The paper is even cited in a few of the very last articles on which Salton is listed as a coauthor (Singhal, Salton, Mitra, & Buckley, 1996; Singhal & Salton, 1995). These papers were published close to or shortly after the time of his death, and so the errors cannot be blamed on Salton (remembered by his colleagues as a very careful and meticulous writer).”

Somehow far too many IRs misquote Salton’s 1975 paper titled “A vector space model for automatic indexing“. This causes digital libraries to create a spurious record attached to many cross-referenced articles.

I searched Google for “a vector space model for information retrieval” + salton and indeed there are many reputed publications and researchers citing a paper that was never published! What a shame.

That says a lot about researchers, editors, and reviewers that were lazy enough to never bother about the accuracy of the references.

The QA column of the current issue of IR Watch Newsletter has a great question that might help IR, CS, and stats students.

Q: Centering Data with Excel- In Excel, how do you center a data set?

 A: To center a data set, use the STANDARDIZE function which converts x values into z-scores; i.e.

z = (x – a)/s

where a and s respectively are the population arithmetic mean and standard deviation. The following table emulates an Excel spreadsheet.

Rows 2 – 13 contains the data set x(A), x(W), and x(H). In rows 16 – 27 the set was centered by typing in cell A16 the formula

 =STANDARDIZE(A2,AVERAGE(A$2:A$13),STDEV(A$2:A$13))

 Pasting this formula in cells A16 through C27 centers the data set. That was easy!

The current issue of IRW the newsletter is out.

Featuring Article:

Data Mining Texting
TTMD OMG MOS CU

“My parents send email, I text.” This illustrates the obvious: a digital divide between parents and teens. While parents are busy replying to email or blogging at the most, their kids probably are busy developing their own language to alert their peers when mom or dad is trying to figure out what they are texting about. Did you know that MOS  CU means ‘Mother over shoulder’. ‘See you’. And how about PW CUL? (‘Parents watching. See you later’).

Indeed… Texting is not just for teens:

Texting not only is revolutionizing the way businesses are being conducted in 2009, but is an emerging data mining playground. The number of behavioral patterns in connection with texting is on the rise at different diffusion fronts: from sexting and sextcasting (transmission of conversations, videos, photos with sexual content) to dealing (transmission of conversations in connection with illegal drug activities), to encoding conversations about Wall Street transactions, industrial espionage, and so forth.

As the 4th of July weekend approaches, here are some notes before hitting to planet oblivious.

1. Yesterday we had an interesting business entrepreneur meeting with the CIO of the Government of Puerto Rico at El Palacio Rojo, Fortaleza.

2. IRW should be out by Monday. Main article: Data Mining Texting.

3. Only monkeys still believe in KD Myths. Ha, Ha.

Back in April, I mentioned that Microsoft will be co-launching with Interamerican University of Puerto Rico, Metropolitan Campus the Microsoft Innovation Center (MIC) of Puerto Rico.

Well, tomorrow is the official inauguration. the university generously has provided me with lab and office space to start an interesting research project within the MIC building. These are exciting news. I cannot comment much about the project, except to say that it is at the interface of search engines, social networks, and information security.

It looks like I will have my hands full between workig at two universities, blogging, and doing consulting work.

I normally do not put online my lecture notes (ppt, pdf, videos). However, there are two public conferences that event organizers taped. Both last over 1 hour and are in Spanish, but with slides in English. Here are the links. The quality of the videos is so-so.

Since the videos were made available few months later after the events, these are not properly dated. I have included below the actual date of the events. If you don’t know Spanish, you are out of luck.

1. Understanding Search Engines (Entendiendo a los Buscadores), University of Puerto Rico, Bayamon, 4-23-2008

http://video.google.com/videoplay?docid=-653964730907023811

This one last for about two hours. The audience consisted of grad students and researchers. Unfortunately, the video has an audio-visual mismatch of about one slide. If you can coupe with this, I hope you like it.

2. Demystifying LSI (Desmitificando LSI)- OJOBuscador Congress, Madrid, Spain, 3-09-2007.

http://www.ojotube.com/videos/congreso-ojobuscador-2007-ponencia-desmitificando-lsi-de-dr-e-garcia/

This one last for over one hour. Since it was for a non-scientific audience  (most Spanish SEOs)  I tried to talk very slow.

Soon or later CS students, in particularly those in IR, will need to deal with similarity matrices.

In simple terms, any matrix M that exhibits the following five characteristics is a similarity matrix.

Squaredness = M must have the same number of rows and columns.
Non-Negativity = all elements of M must be real, non-negative numbers.
Boundedness = all elements of M must adopt values between 0 and 1.
Reflexivity = all diagonal elements of M (i.e. from left to bottom) must be filled with 1.
Symmetry = all ij elements must be identical to all ji elements.

A matrix that fails to exhibit any of these characteristics is not a similarity matrix.

Accordingly, some matrices found in the literature on LSI and whose elements have been referred to as similarities are not so since the corresponding matrix does not conform to the above definition.

Note. This information will help those that took the IR Quiz on Matrices to realize how well they did.

The QA column of the current issue of IR Watch – The Newsletter features the following question:

Question: In Excel, how do you convert a term-document occurrence matrix into a term-term or document-document co-occurrence matrix?

Answer:

Let A be a matrix populated with term occurrences (frequencies).
Let A^T be its transpose.

Then, T = AA^T is a term-term co-occurrence matrix, and D = A^TA is a document-document co-occurrence matrix.

The following table emulates an Excel spreadsheet.

In the table, T was computed by selecting a destination array, entering in its first empty cell (B7) the formula =MMULT(B2:D4,TRANSPOSE(B2:D4)), pressing the f2 key and then the Ctrl+Shift+Enter keys.

Similarly, D was computed by selecting a destination array, entering in its first empty cell (B12) the formula =MMULT(TRANSPOSE(B2:D4),B2:D4), pressing the f2 key and then the Ctrl+Shift+Enter keys.

That was easy!

Note that none of these are similarity matrices. Can you tell why?

The current issue of IRW should reach subscribers inbox during the day or at the latest, tomorrow.

In this issue:

• Featuring article: Hackers: Taxonomy and Writing Styles
  Due to the increasing interest in developing Information Retrieval and Data Mining courses at the intersection of Information Security, this issue of the newsletter covers a brief taxonomy on hackers and their writing styles.
• QA: Excel Matrix Multiplications: How to convert a term-document occurrence matrix into a term-term or document-document co-occurrence matrix?
• Vacuum Tubes & Transistors Historical
• Who is Who in IR: Thomas K. Landauer
• Top CS Departments: Dartmouth College
• Outstanding Graduate Theses
• Calls and Events
• IR Blogs
• and more…

I’m putting together a piece on several local term weight models. It should be ready in few weeks.

It is a research paper that can be used as a tutorial. It describes a systematic approach for the derivation of any kind of local term weighting model. Students can use it as a recipe for proposing their own candidate models.

The article touches on some aspects of the problem of trusting models that lack of attenuation. Here is one snippet on the subject:

<last nail in KD coffin  style=”intensity:100%;”>

“It should be stressed that term repetition not necessarily satisfies users’ queries nor is evidence of:

 Pertinence (P); e.g., that a term repeated x times is x times more pertinent to the document.

Aboutness (A); e.g., that the document is x times more about the term.

Importance (I); i.e., that there is a term-document relationship of pertinence and aboutness.

Relevance (R);i..e., that a document repeating a term x times is x times more relevant.

Accordingly, fulfilling such ‘PAIR criteria’ on a regular basis is hard to accomplish with any model that lacks of attenuation.”

</last nail in KD coffin>

What is the (^H^H^H) best definition for data mining and database? It depends on who you ask and in which context.

According to Section 126 of the USA Patriot Act,

(1) DATA-MINING- The term `data-mining’ means a query or search or other analysis of one or more electronic databases, where

(A) at least one of the databases was obtained from or remains under the control of a non-Federal entity, or the information was acquired initially by another department or agency of the Federal Government for purposes other than intelligence or law enforcement;

(B) the search does not use personal identifiers of a specific individual or does not utilize inputs that appear on their face to identify or be associated with a specified individual to acquire information; and

(C) a department or agency of the Federal Government is conducting the query or search or other analysis to find a pattern indicating terrorist or other criminal activity.

(2) DATABASE- The term `database’ does not include telephone directories, information publicly available via the Internet or available by any other means to any member of the public, any databases maintained, operated, or controlled by a State, local, or tribal government (such as a State motor vehicle database), or databases of judicial and administrative opinions.

Asking the government or a KDDM researcher the question and using LSI to clusters results for the above question can be a futile exercise.

It is like asking President Obama or Vice President Cheney to agree on: “What is Torture?”

Today, a reader (name removed to protect confidentiality) asked me:

My name is **** ****. I working as a junior research fellow in a project in India. I red the SVD techniques from the web page http://www.miislita.com/information-retrieval-tutorial/svd-lsi-tutorial-3-full-svd.html#right-eigenvectors. I found it is quite satisfactory for me. Now I can understand how SVD works. But I have a query as follows.

query:

As mentioned in this tutorial that we have arrange these eigen-values in descending order. Cold you please tell me if I put these values in ascending order or arbitrary what will be wrong with the SVD.

Looking forward your early kind response.

Thanking you.

With best regards.

*******

My answer follows.

It depends on what you are trying to address.

SVD is used to identify singular values interpreted as dimensions. When used as a dimensionality reduction technique, the largest N singular values are normally retained and thus retaining the smaller singular values is meaningless.  The largest singular values capture most of the information of the original data set and is therefore a noise minimization approach.

If the retention criterion used is reversed (smaller singular values are retained) this implies retaining the more noisy dimensions such that the reconstructed matrix will be a matrix of the hidden (latent) data noise. This is a noise maximization approach.

If the retention criterion is based on a random selection, the resultant reconstructed matrix might be one representing a data structure with randomized noise.

These scenarios depend on the original data under examination. 

In Image Compression, these approaches have been already explored. If the goal is a stability study and not just SVD dimensionality reduction, “the ratio between the highest singular value and the lowest singular value of the Jacobian matrix quantifies the spread of the Jacobian’s singular values, which in practice, reflects the extent of the solution’s instability with respect to small changes in the observation”  (Horesh’s Thesis )

Having said all that, we should not render noise in a data set as something that must be discarded at all cost.

This is intimate linked with the so-called Inverse Problem. Incorporating noise and a priori SVD information can provide the complete information in a linear sense. Qianqian Fang has a beautiful PPT presentation “Look Closer to Inverse Problem” on the subject. If you want to visualize the MATRIX Problem, this presentation is for you.

I’m thinking in putting together a tutorial on the Singular Value Expansion algorithm (SVE), if I ever find the time.

I hope this helps.

According to a report from the British Computer Society (BCS) covering a Security Panel Strategic Forum, “ethical hacking” is an oxymoron.

The report highligths do’s and don’t when it comes to defining terms like “hacker”, “ethical hacking”, “penetration tester”, “white/black hats”, and derivatives terms. These labels are frequently used in the IT industry. The report also underscores which terms should not be used by schools offering IT courses.

The problem with defining and redefining such labels is that there will always be others disagreeing with/circumventing said definitions.

For instance, in the December 1986 issue of MicroTimes, Bob Bickford wrote:

“A Hacker is any person who derives joy from discovering ways to circumvent limitations.”

If we accept this definition then a person that doesn’t derive any joy from discovering ways to circumvent limitations is not a hacker. Similarly a spouse cheater, an SEO, a spammer, a politician, a mobster, or a kid trying to get some candies from mom is a hacker.

I am taking this extreme, off-topic interpretation to illustrate the problem of semantics when it comes to defining things.

Whether you agree or disagree partial or totally with the report, it is a good read. For sure it will be a good piece for students planning to take my AIRWeb graduate course.

According to Ben Edelman, Google is engaged in a conversion-inflation syndyication fraud.

These tactics are nothing new.

In the featuring article of the November 2008 issue of IR Watch, “Fraudulent Web Analytics – Engineering the Fraud“, we covered how in-the-middle mechanisms are part of Web Analytic Frauds and Business Collusion Schemes.

As in man-in-the-middle attacks found in information security settings, the underlying goal is the same: the crafting of deceiving intermediary events.

Expect soon a pr damage control campaign from the useful idiots/moles.

What is next? A class action lawsuit?

Still, I have a little taste of satisfaction in my mouth when crooks disguised as advertisers/search marketers are gamed. Gaming the gamers: Life ironies!

Explain and give example for the following matrices used in IR:

1. Term-document occurrence matrix.

2. Term-term cooccurrence matrix.

3. Term-term correlation matrix.

4. Term-term similarity matrix.

5. Term-term coweights matrix.

6. Term-term distance matrix (*).

7. Covariance matrix (*).

(*) PS. I forgot to list these other matices.

Back in March, we explained how to normalize column vectors with Excel. But, what about normalizing row vectors? This question is addressed in the current QA column of IRW. I think it might be useful sharing the answer with readers since many of these are students struggling with similar questions. So, here we go.

The following table emulates an Excel array consisting of three columns (A, B, and C) and six rows (1-6).

Rows 1, 2, and 3 are row vectors. Rows 4, 5, and 6 are the corresponding normalized vectors, also known as unit vectors because their length is 1. To compute these, do as follows:

1. In cell A4, enter the formula =A1/(SQRT(SUMSQ($A1:$C1))). The result should be as given in this cell.

2. Copy this formula, select cells A5 and A6 and paste the formula in these.

 3. Finally, copy at once cells A4 through A6, select the remaining empty cells of the array, i.e., cells B4 through C6 and paste the formulas in these.

As blogged yesterday, the current issue of IRW should reach subscribers inbox today. The Top CS Departments column features Polytechnic University of Puerto Rico, where I teach graduate courses. As mentioned few days ago, PUPR has been designated a CAE. This is a great news that is making a splash across academic centers within the U.S., the Caribbean Region and Latin America, and whose mission is research relevant to homeland security.

Associate Director for Computer Science, Dr. Alfredo Cruz, sent me an  official announcement, which I am reproducing.

Polytechnic University of Puerto Rico (PUPR) is Designated National Center of Academic Excellence in Information Assurance Education by NSA and DHS. PUPR was recently designated as a National Center of Academic Excellence in Information Assurance Education (CAE/IAE) by the National Security Agency (NSA) and the Department of Homeland Security (DHS) on April 22, 2009. The goal of these centers is to reduce the vulnerability of the national information infrastructure by promoting higher education and research in Information Assurance (IA) and Security through the development of a growing number of professionals with IA expertise in various related disciplines. PUPR will be recognized as the first institution in Puerto Rico to be designated as a CAE/IAE on June 3, 2009 in Seattle, Washington. Dr. Alfredo Cruz from the Department of Electrical & Computer Engineering and Computer Science will be present to receive the designation. He is the Director of the Center of Information Assurance for Research and Education (CIARE) at PUPR. Dr. Cruz is the person responsible for this designation. PUPR is of the very few Hispanic serving institution (HSI) in the Nation to receive this designation, and to become one of the first 100 institutions nationwide; this is a very special recognition. This designation requires that the President of the United States send the Governor of Puerto Rico a certification that should be handed to the president of PUPR designating the Institution as a CAE/IAE at a National level. The Congress and all the respective Congressional Committees are also notified.

Some of the benefits of the CAE/IAE designation are:
• PUPR will receive formal recognition from the U.S. Government as well as opportunities for prestige and publicity for our roll in securing the Nation’s information systems.
• This designation increases collaboration opportunities between designated and aspiring institutions at local and national levels. This includes internships, faculty and student exchange, research, and publications, among other activities.
• With this designation as a CAE/IAE PUPR can obtain scholarships that can help outstanding students to pursue graduate studies in IA, enabling them to work with the Federal Government or other federal institutions and agencies.
• PUPR can compete and benefit from proposal calls (RFP) that are specifically for designated CAE/IAE institutions. These proposals offer millions of dollars from the DoD, NSF, NSA and “Homeland Security”, among others, for research and infrastructure.
• Student scholarships offered under the NSF’s Scholarship for Service (SFS) program. The SFS scholarship offers the following:
–2-year scholarship, includes 8K stipend (12K for graduate students), plus tuition and nominal room and board expenses.
–Paid summer internship in a federal agency.
–Placement in federal government at the end of the scholarship period.

The current of issue of IRW should reach subscribers inbox tomorrow.

In this issue:

Featuring article: RIA Vulnerabilities

This issue of the newsletter discusses how hackers might be exploiting Web vulnerabilities found in Rich Internet Applications (RIAs). As mentioned in our previous issue, some RIAs are based on Adobe’s technologies like Flash, Flex, or AIR. Some are designed to be run online or offline. Their rising popularity has attracted developers and marketers, and -as expected- hackers and spammers.

QA: Excel Vector Normalization: How do I convert a row vector into a unit vector?
Who is Who in IR: C.J. van Rijsbergen
Top CS Departments: Polytechnic University of Puerto Rico
Historical Notes: ENIAC Computer
Outstanding Graduate Theses
Calls and Events
Research Blogs
and more…

Today I feel like giving a piece of advise to spammers, so this will force raising the bar in the “we versus them” in the Spam War. Think of this as a love-hate relationship.

C’mon spammers, I know you can do better. Don’t make our IR life easy at neutralizing your tactics. He, He.

At the recent AIRWeb Workshops, Brian Davison presented the paper Looking into the Past to Better Classify Web Spam, which received high reviews from referees and the audience.

Wannabe spammers, if you are really committed to spamdexing, at least know the how-tos. Don’t leave a temporal fingerprint of your web presence. Try this:

1. Prevent online resources from caching your web pages, like the Wayback Machine and commercial search engines.

2. Use No-Cache and No-Archive.

3. Switch hosts whenever you can.

4. Constantly mutate your link structure.

5. Don’t profile yourself with easy to detect/predictable honeypots, link swapping, strongly-connected component structures, etc.

Why giving these advices? Check current AIRWeb “gems”.

This afternoon, Microsoft in partnership with The Interamerican University of Puerto Rico, Metropolitan Campus (Inter-Metro) will announce that they are officially co-launching the Microsoft Innovation Center (MIC) of Puerto Rico.

This will be the first MIC in the region. A two stores building has been abilitated within the Inter-Metro campus for this project. As member of the MIC steering committee, I have been invited to the presentation by President, Manuel J. Fernos.

They have also provided me with office and lab space in the MIC building to put together the Internet Business Development Center (IBDC). The objectives of the MIC is the development and commercialization of ecommerce-related software tools. Emphasis will be given to egovernment and ebusiness solutions.

It looks like I will split my schedules between being the IBDC principal investigator, MIC meetings, doing research at Inter-Metro, teaching at PUPR, and writing IRWs. These are exciting news. Let see how things go, especially with the other great news  that PUPR’s ECE&CS department has been accredited by NSA as a CAE.

Here are the proceeding papers of AIRWeb 2009, available at http://airweb.cse.lehigh.edu/2009/proceedings.html

OK, SEOs, Spammers, and Hackers: start your engines and let the fun begin.

If you are a PUPR graduate student and are planning to take my AIR course, it might be a good idea to start browsing through these “gems”. Check also previous proceedings of AIRWeb.

Invited Talks

The Potential for Research and Development in Adversarial Information Retrieval — slides

Brian D. Davison

Web Spam Challenges: Looking Backward and Forward — slides

Carlos Castillo

Temporal Analysis

Looking into the Past to Better — slides

Na Dai, Brian D. Davison and Xiaoguang Qi

Classify Web Spam

A Study of Link Farm Distribution — slides

Young-joo Chung, Masashi Toyoda and Masaru Kitsuregawa

and Evolution Using a Time Series of Web Snapshots

Web Spam Filtering in Internet — slides

Miklós Erdélyi, András A. Benczúr, Julien Masanes and

Archives

Dávid Siklósi

Content Analysis

Web Spam Identification — slides

Juan Martinez-Romo and Lourdes Araujo

Through Language Model Analysis

An Empirical Study on — slides

Taichi Katayama, Takehito Utsuro, Yuuki Sato, Takayuki Yoshinaka, Yasuhide Kawada and

Selective Sampling in Active Learning for Splog Detection

Tomohiro Fukuhara

Linked Latent Dirichlet Allocation — slides

István Bíró, Dávid Siklósi, Jácint Szabó

in Web Spam Filtering

and András Benczúr

Social Spam

Social Spam Detection

— slides

Benjamin Markines, Ciro Cattuto and Filippo Menczer

Tag Spam Creates Large Non- — slides

Nicolas Neubauer, Robert Wetzker and Klaus Obermayer

Giant Connected Components

Spam Research Collections

Nullification Test Collections
— slides

Timothy Jones, David Hawking, Ramesh Sankaranarayana and Nick Craswell

for Web Spam and SEO

Web Spam Challenge Proposal for — slides

András A. Benczúr, Miklós Erdélyi, Julien Masanes and

Filtering in Archives

Dávid Siklósi

George M. Zinkhan III, from Terry College of Business at the University of Georgia allegedly went into a killing rampage, killing his ex-wife and two others, and hurting two.

According to his university page (accessible at the time of writing), Zinkhan is a Coca-Cola Company Professor Department of Marketing and Distribution. Zinkhan is well known in the academic marketing research circles, having served as editor of the JOURNAL OF THE ACADEMY OF MARKETING SCIENCE.

His 40-page CV reveals he conducted extensive research on Marketing and Net Advertising.

In 2008 he was part of an American Marketing Association committee that redefined marketing. The new definition reads:

“Marketing is the activity, set of institutions, and processes for creating, communicating, delivering, and exchanging offerings that have value for customers, clients, partners, and society at large.”

According to the AMA committee,

“Marketing is no longer a function — it is an educational process.”.

Zinkhan published extensively with Yue Pan, associate professor of marketing, University of Dayton. He published on the concept of Netvertising (“Netvertising Characteristics, Opportunities and Challenges: A Research Agenda,” International Journal of Internet Marketing & Advertising, 1(3), 283-299.). According to their abstract:

“Netvertising, or “advertising on the internet”, is attracting much attention from advertising and marketing researchers. However, surprisingly little is known about its new features as compared to other forms of advertising and the implications of the new medium for advertisers. Here, we focus on the following issues: the opportunities and challenges associated with internet advertising; the differences of netvertising from other forms of communication; banner ads – the most popular type of netvertising. Applying this framing perspective, we propose a research agenda for the study of netvertising.”

Netvertising is something search marketers do using different out-of-the-thin-air theories/naming conventions.

Read more about the Netvertising Image Communication Model (NICM)

That was then. Today Zinkhan’s name is associated with a Negative Image on the Net. It will be a matter of time before others will dissassociate themselves with such an image. Life ironies!

He didn’t seem to fit the academic stereotype.

Unfortunately as in any profession, some people cannot coupe with their personal misfortunes and end up doing bad things.

It happened again: Thanks to Web vulnerabilities, hackers were able to hit the Pentagon. 

According to CCN (http://www.cnn.com/2009/US/04/21/pentagon.hacked/),

Thousands of confidential files on the U.S. military’s most technologically advanced fighter aircraft have been compromised by unknown computer hackers over the past two years, according to senior defense officials.

The Internet intruders were able to gain access to data related to the design and electronics systems of the Joint Strike Fighter through computers of Pentagon contractors in charge of designing and building the aircraft, according to the officials, who did not want to be identified because of the sensitivity of the issue.

In addition to files relating to the aircraft, hackers gained entry into the Air Force’s air traffic control systems, according to the officials. Once they got in, the Internet hackers were able to see such information as the locations of U.S. military aircraft in flight.

This news is quite relevant to my Fall 2009 Web Vulnerability graduate course (http://www.miislita.com/courses/airweb-web-spam-syllabus.pdf)

BTW. Associate Director of the CS Department at PUPR.edu, also a colleague and friend, Dr. Alfredo Cruz, called me two days ago with some great news: The department has been accredited for 2009-2014 as a National Center of Academic Excellence in Information Assurance Education. Soon they will be listed with members of this exclusive “club” in the National Securing Agency web site (http://www.nsa.gov/ia/academic_outreach/nat_cae/institutions.shtml)

An official press release and formal presentation before the pertinent authorities is being coordinated for within the next few weeks or so.

The next issue of IR Watch – The Newsletter provides additional coverage of such an exciting news.

I have tied these two news in a single post to underscore the need for IR/data mining courses at the intersection of Information Security, which is precisely the mission statement of IRW, reaching now more than 300 investigators/research centers.

According to a McAfee report,

Until now, spam’s impact has been measured in time, money, and aggravation. It turns out there is a massive environmental impact as well. McAfee recently commissioned climate-change consultant ICF International and spam expert Richi Jennings to calculate the environmental impact of spam. The results that came back were startling: The energy consumed in transmitting and deleting spam is equivalent to the electricity used in 2.4 million U.S. homes, with greenhouse gas (GHG) emissions equivalent to 3.1 million passenger cars(http://resources.mcafee.com/content/NACarbonFootprintSpam)

I first learned about these findings through ABC. Essentially,

Anything powered by electricity also emits greenshouse gases. McAfee researchers say each junk e-mail emits 0.3 grams of the greenhouse gas carbon dioxide (CO2). That may not sound like much, but when you consider the volume of global annual spam, it all adds up. (http://abcnews.go.com/Technology/GlobalWarming/story?id=7343518&page=1).

Following that reasoning, spamdexing search engines and any adversarial information retrieval (AIR) practice is also an insult to injury, so as too many things that comes to my mind.

I will tell that to students of my Fall 2009 AIRWeb Course.

Humm, shocking: AIR vs. Environment.

I never thought about such an obvious connection. 

Recently a known SEO (name reserved) inquired me about some aspects of IDF (Inverse Document Frequency). Below are three of his questions.

I am partially reproducing/editing my responses, so it might help other SEOs with similar questions.

Questions 1 and 3 are related so I will answer both now. After that, I will answer question 2.

1) Why is a log function used for calculating IDF?
3) Would it be accurate to describe IDF as “the ratio of documents in a collection to documents in that collection with a given term”? I’m guessing your answer would be, IDF is the [LOG of " the ratio of documents in a collection to documents in that collection with a given term"]? Which brings us back to question, I guess? hehe

These are recurrent questions students asked me before. The reason for using logs is due to two assumptions frequently made in most IR models; i.e.

I. that scoring functions are additive.
II. that terms are independent.

While in some models II might not be present, both (I and II) play well with logs since these also are additive.

These functions and why the use of logs is explained in the recent RSJ-PM Tutorial http://www.miislita.com/information-retrieval-tutorial/information-retrieval-probabilistic-model-tutorial.pdf

Document Frequency (DF) is defined as d/D, where d is number of documents containing a given term and D is the size of the collection of documents. If we take logs we obtain log(d/D).

But since often D > d the log of d/D, that is log(d/D) gives a negative value. To get rid off the negative sign, we simply invert the ratio inside the log expression. Essentially we are compressing the scale of values so that very large or very small quantities are smoothly compared. Now log(D/d) is conveniently called Inverse Document Frequency.

Now going back to d/D, this is a probability estimate p that a given event has occurred. Let the presence of a term in a document be that event. If terms are independent, it must follows that for any two events, A and B

p(AB) = p(A)p(B).

Taking logs we can write

log[p(AB)] = log[p(A)]+ log[p(B)]

It is easy to show that for two terms

log(d12/D) = log(d1/D) + log(d2/D)

Inverting and using the definition of IDF we end up with

IDF12 = IDF1 + IDF2

validating assumption I; that IDF as a scoring function is additive.

That is the IDF of a two term query is the sum of individual IDF values. However, this is only valid if terms are independent from one another. If terms are not independent we would have two possibilities; i.e.,

p(AB) > p(A) + p(B)

or

p(AB) < p(A) + p(B)

and we cannot say that the IDF of a two term query (e.g, a phrase) is the sum of individual IDF values. Assuming the contrary as many SEOs think in order to promote some dumb keyword research tools is plain snakeoil.

2) What do you mean by ‘discriminatory power’ in the phrase “IDF is a measure of the discriminatory power of a term in a
collection.”

This is legacy idea from Robertson and Sparck Jones. The discriminatory power of a term (aka term specificity) implies that terms too frequently used are not good discriminators between documents. If a a term is used in too many documents its use to discriminate between documents is poor. By contrast, rare terms are assumed to be good discriminators since they appear in few documents.

The RSJ-PM Tutorial mentioned above was written to kill for good some misconceptions regarding IDF. In it we explain why IDF is considered by Robertson and Jones a particular RSJ weight in the absence of relevance information.

In a nutshell, IDF is a collection wide estimate and as such the information on whether documents containing the terms being queried are relevant to these is unknown. Similarly, the information on whether documents not containing the query terms are relevant or not is unknown and often remains unscrambled when we just look at the d/D and d/(D – d) collection-wide ratios. All we can say is that relevant documents might have a higher probability of containing query terms in comparison with other documents from the collection as a whole. But we could make such assertion without resourcing to IDF as well.

In the case of Web documents, often these are about multiple topics. Many documents aggregate content from dissimilar sources (news headlines, rss, blogs, etc) and said document content might change in time. The mere mention of a term (regardless of repetition) is not a proof of its relevancy or of its importance with respect to the topics discussed in a document.

Thus, the idea that we can assess if terms are relevant to a document by simply comparing IDF values is missing the whole point and defeats the purpose for which the RSJ-PM model and many of its variants (e.g., BM25) were developed.

I hope this helps to clear up some SEO misconceptions on the topic.

If you search this blog (IRThoughts) for LSI or visit its Latent Semantic Indexing category you will find many posts wherein SEO LSI Myths are debunked. Prior to this wordpress blog I used to maintain a personal blog wherein SEO myths regarding LSI were also debunked.

Over the years, many realized they were taken by the usual agents of misinformation, at least when it comes to “SEO LSI” and “LSI-Friendly” documents.

Recently, I found traffic coming from a blog discussion about a video (http://www.stomperblog.com/warning-advanced-seo-technique-does-not-work/) wherein LSI in relation with Google is debunked.

The video also discusses one flavor of LSI; i.e. one wherein weights are tf-IDF weights. This flavor does not incorporate relevance information or entropy information, like other LSI variants.

The video does a good job at debunking LSI Myths. However, it has at least a factually incorrect argument in relation to how the SVD algorithm works.

The video gives an example implying that SVD works by reducing a large set of words to a few words, such that, for example thousand of words are reduced to, let say 300 words.  This is incorrect and certainly is not a trivial flaw.

SVD does not work by reducing a vocabulary, but by reducing dimensions, and there are as many dimensions as singular values. This is why is called a dimensionality-reduction and not a vocabulary-reduction algorithm.  I should stress that an LSI Space is not like a Term Space wherein each term is a dimension such that there is a 1:1 correspondence.

In LSI, the SVD algorithm is used to reduce the dimensions of a matrix; the number of singular values of the matrix.

For instance in our SVD and LSI Tutorial series at

http://www.miislita.com/information-retrieval-tutorial/svd-lsi-tutorial-5-lsi-keyword-research-co-occurrence.html

we present an LSI problem example consisting of many words and few initial dimensions such that for the initial matrix

#words >> # initial dimensions

more specific, we used 11 words and 3 dimensions

After truncation, we ended up with 11 words and 2 dimensions.

Other than this, the video is fun to watch, but ended up as an introductory promotion for another SEO proposal.

 PS.

After reviewing several times the video, unfortunately I found the video has another incorrect argumentation.

When objecting to that Google might not use LSI, an argument is made in the sense that LSI has to return same results when word variants are used like plurals and tenses. This might be the case if stemming is heavily used in an LSI implementation, but the use of stemming is not a requirement for implementing LSI at all.

When stemming is not implemented, for sure the SVD reduction will return different results since these will be entered in the original term-doc matrix to be undergo decomposition as different tokens.

The video also misses what the power of LSI comes from: higher order co-occurrence connectivity path hidden (latent) in the original matrix. Whether terms have to be synonyms, related terms, or even of non-derivative forms is not a requirement for observing these hidden paths in LSI.

Terms no need to be related terms either to end up clustered with LSI. It is the hidden co-occurrence patterns what is behind the clustering. For example, in our SVD and LSI tutorial above, we intentionally used stopwords and zero synonyms/related terms and these ended-up in their corresponding clusters, without being necessarily semantically related. This simple example shows that in LSI the SVD algorithm produces an output based on crushing numbers, not on making sense out of meaning or intelligence, and contradicts the generalized opinion that LSI works at the level of meaning. 

I have to conclude that while the video is intended to debunk LSI SEO myths (a noble effort), it uses incorrect arguments and hearsays lines from around the Web. Debunking hearsay with more hearsay: What a shame.

The current issue of IRW should be in subscribers inbox today or tomorrow, at the latest.

In this issue of the newsletter we cover Rich Internet Applications (RIAs) and how these can be used for Web/Data Mining. A RIA is a browser-independent application that can be compiled and run from the desktop.

In this issue:

Featuring article: Web & Data Mining with RIAs
QA: Recommended RIAs
Who is Who in IR: Bruce Croft
Top CS Departments: UMass, Amherst
Historical Notes: John von Neumann and Bugs
Outstanding Graduate Theses
Calls and Events
Research Blogs
and more…

IRW currently reaches a fine audience of university and government researchers and their labs. If you are a graduate student or IR practitioner and want to be known within this exclusive circle, submit a short article (2, 3 pages, IRW format, free from marketing and sale pitches) for its consideration