Minerazzi Crawler and Whois Updates: Email Addresses, Reverse DNS, IPv4 Mapping, Navigation

We keep improving the Minerazzi site (http://www.minerazzi.com). We moved all pages to a php format. In addition, here are recent changelogs for the Web Crawler (http://www.minerazzi.com/labs/crawlinker.php):

07-05-11: Email address extraction, deduplication, and sorting capabilities added.
07-04-11: Design and copy changes.
07-03-11: Navigation menu restored and bug fixed.
07-03-11: Navigation menu removed to test bug.
07-02-11: Top-bottom quick navigation menu added.
07-02-11: Day/Time Stamp, Reverse DNS, and IPv4 List capabilities added.
07-02-11: Integration to Whois Tool.

The Whois Database Retriever (http://www.minerazzi.com/labs/whois.php) now features suffix/prefix stripping capabilities. This means that users only need to enter a candidate domain name without any alias or extension and the tool scans multiple registrar databases. We expect to add some additional features to this time-saving application.

In the meantime, we keep beta testing the engine. Our staff of ‘miners’ are doing just a great job.

Matrix Algebra for Search Marketing

Today I feel like giving away a little quiz material on applied linear algebra. The topic is relevant these days wherein some misleading SEOs are playing the we-do-”science” game (quack “science”, after all).

The following is taken from the Search Engines Architecture grad course I lectured back in 2008. I’m providing only one exercise with multiple parts. The quiz with answers might be a great topic for an IRW issue.

1.1 A search engine has three types of revenue channels: pay-per-click (PPC), pay-per-placement (PPP), and pay-for-conversion (PFC). In quarter 1, the million-dollar revenues respectively were: 20, 4, and 9. In quarter 2, PPC revenues were 20% less, PPP revenues doubled, and PFC revenues remained constant.

1.1.1 Write a matrix M1 expressing the revenue and quarter vectors for the first two quarters.

1.1.2 If the goal in quarter 3 is to increase by 20% all revenues earned in quarter 2, update M1 so it reflects such a goal as a new matrix M2.

1.1.3 If the goal in quarter 4 is to meet the average revenues of each of the previous channels in quarter 4, update M2 such that it reflects that goal as a new matrix M3.

1.1.4 Express the above quarters as column unit vectors. Inspecting either rows or columns, construct a nearest neighbor similarity matrix Mnn and construct scalar clusters of quarters. Ignore cosine similarity deviations of 0.02 units or less. How similar the quarters are?

Have fun.

What is a Similarity Matrix?

Soon or later CS students, in particularly those in IR, will need to deal with similarity matrices.

In simple terms, any matrix M that exhibits the following five characteristics is a similarity matrix.

Squaredness = M must have the same number of rows and columns.
Non-Negativity = all elements of M must be real, non-negative numbers.
Boundedness = all elements of M must adopt values between 0 and 1.
Reflexivity = all diagonal elements of M (i.e. from left to bottom) must be filled with 1.
Symmetry = all ij elements must be identical to all ji elements.

A matrix that fails to exhibit any of these characteristics is not a similarity matrix.

Accordingly, some matrices found in the literature on LSI and whose elements have been referred to as similarities are not so since the corresponding matrix does not conform to the above definition.

Note. This information will help those that took the IR Quiz on Matrices to realize how well they did.

IR Quiz: Matrices

Explain and give example for the following matrices used in IR:

1. Term-document occurrence matrix.

2. Term-term cooccurrence matrix.

3. Term-term correlation matrix.

4. Term-term similarity matrix.

5. Term-term coweights matrix.

6. Term-term distance matrix (*).

7. Covariance matrix (*).

 

(*) PS. I forgot to list these other matices.

Answers to IR Quiz

Answers to the IR Quiz are given below:

Term Independence Assumption:

If k1 and k2 are statistically independent they should occur by chance, co-occurring in only

(100)(200)/500 = 40 documents.

Thus, if they occur by chance, the number of documents mentioning the k1 k2 sequence should be unknown, but certainly no greater than 40.

Term Dependence Assumption:

If terms actually co-occur in 70 documents, they are co-occuring more often than by chance (70 > 40). So, terms are statistically dependent and positively correlated. It is a given that the k1 k2 terms sequence is present in 25 out of the 70 documents wherein terms co-occur.

Detailed Results:

Results are given below, rounded off to two decimal places. First/second results respectively are for terms independence/dependence assumptions. You should be able to double check these results.

1. k1 NOT k2: 60, 30

2. k2 NOT k1: 160, 130

3. k1 OR k2 (unconditional OR): 260, 230

4. k1 OR k2 (conditional OR): 220, 160

5. NOT k1: 400, 400

6. NOT k2: 300, 300

7. NOT (k1 AND k2): 460, 430

8. k1 AND k2 NOT (k1 k2): NC, 45

9. EF-Ratio of the k1 k2 terms sequence: NC, 0.36

10. c12-index of the k1 k2 terms sequence: NC, 0.11

11. c12-index of k1 AND k2: 0.15, 0.30

12. IDF of k1: 0.70, 0.70

13. IDF of k2: 0.40, 0.40

14. IDF of k1 AND k2: 1.10, 0.85

15. IDF of k1 k2 terms sequence: NC, 1.30

Additional exercises open to discussion:

Calculate the associated odds, odd ratios, and logits.