You have given me untold hours of interesting reading.

Reg

Thank you for stopping by.

Indeed. And to compute IDF, thus tf-IDF, one must know N, the size of the entire collection of a search engine.

Over the years some IRs have tried to estimate N by resourcing to the term independence assumption. The result are estimated N values that vary so wild that they cannot be trusted; at least not with Web search engines.

The term independence assumption is the source of all kind of inconsitencies in IR scoring functions, including vector space models. When one thinks about it thoroughly, the notion of term specificity itself is inherently divorced from term independence.

On other matters. Here are some links to post somehow related with this thread posts.

http://irthoughts.wordpress.com/2008/07/21/seos-and-their-exhaustivity-search-myths/

http://irthoughts.wordpress.com/2008/07/14/claps-and-slaps/

http://irthoughts.wordpress.com/2008/07/07/understanding-tfidf/

http://irthoughts.wordpress.com/2007/07/09/a-call-to-seos-claiming-to-sell-lsi/

http://irthoughts.wordpress.com/2007/07/19/seos-and-still-their-lsi-misconceptions/

http://irthoughts.wordpress.com/2007/05/03/latest-seo-incoherences-lsi/

When going through the series of posts these things struck me.

1). andyed Says:
July 3, 2008 at 7:33 pm
“Worse than that, your reference to me saying TF-IDF is new is entirely inaccurate. In no way did I suggest this is a new concept. My quote was “The buzzword in IR”, not the new buzzword.”
My eye was caught on the last sentence.
“I was reviewing a new website doing some interesting word frequency analyses.”

As a 12+ year practitioner of SEO, it never ceases to amaze me with what some of the fertile minds can come up with to inflate their own value.

Let me put this in context as I interpreted it.

“I was reviewing a new website doing some interesting word frequency analyses”

And just HOW were you doing this analysis andyed? Using TF-IDF as you imply?
Are you privy to Google’s algorithms?
Using any other technology is simply a waste of the client’s money.

I am not saying KW analysis is a waste of time, but having to dissect each (new) client’s site and weigh the value of the current composition is ludicrous.
ANY practitioner of SEO worth their salt knows how to build the site properly from a results driven “template”. Stop trying to “game” the engines.

2). danthies Says:
July 4, 2008 at 1:21 am
I have to agree with danthies.
andyed is using TF-IDF as “A word or phrase connected with a specialized field or group that usually sounds important or technical and is used primarily to impress laypersons.” With emphasis on “usually sounds important or technical and is used primarily to impress laypersons”.
The more you can impress, the more you can charge. Build the BS mystique.

3). E. Garcia Says:
July 4, 2008 at 11:37 am
In IR we know exactly what tf-idf means, stands for, and what it does.

Of course, but the general population does not. And the article is for the general population.

4). E. Garcia Says:
July 4, 2008 at 2:15 pm

“He has stated that TFIDF is a buzzword in IR and I contend that is not and explained why.”

Of course he would. To have it so would imply more importance to his use of the term for the general market.

5). E. Garcia Says:
July 4, 2008 at 10:06 pm
“Still, I want to be fair. Please go ahead and start discussing the merits of the tool here.”

I used TheRarestWords.com on a couple of my sites and learned nothing useful.
Results were as expected.

6). Going back to your article on understanding TF-IDF I agree strongly that “IDF as the TFIDF product, aij = tfij*IDFi, does not estimate term importance either. The importance of a term, a string, a passage, a message, etc is linked to many things like its meaning (semantics) and amount of information carried (entropy). A TF-IDF product does not evaluate either one.”

The semantics are the key.

Even if TF-IDF were to be the end-all and be-all of evaluating keyword and keyword phrases over the scope of the database, it still means nothing without the exact vectors used by the search engine(s). Using it as a buzzword is simply hyperbole.

Writing for the search engines is not rocket science.
Writing the search engines is.

Reg Charie.

For your information sir, IR Thoughts is a blog wherein we comment on IR and search engines and debunk search marketing myths like the many promoted by SEOs. We dissect these through IR knowledge, not hearsay.

We didn’t go after you, Dan. You came here voluntarily, with a spaghetti-like defense, throwing all sort of false statements to see what sticks to the wall. Are you still bleeding from NY SES 2005?

Refluxing words just shows you are short of your very own. To waste your will over the beautiful Fourth of July weekend in such manner might suggest that you are a very lonely, depressive man. You need a vacation.

Since your comments add no value to the discussion, you lost your chance, so you and your friends are out of here. BTW, more entertaining than chasing rare terms most users don’t care to search about is reading Understanding TFIDF.

Doctor, there’s nothing to “defend” - your “nitpick attack” is without merit. I was just pointing out that it’s as easy to find fault with your own choice of words, and even easier to question your motivation. Readers will easily see that this is so without me taking it any further.

In http://alwaysbetesting.com/abtest/index.cfm/2008/5/24/Term-Frequency-Inverse-Document-Frequency-TDIDF-Exploring-TheRarestWordscom

Edmonds wrote:

“The buzzword in IR is TFIDF, or term frequency inverse document frequency. This is a method for giving more importance to the less common words in a document that match the query. Mid-range frequency words get discounted, but they’re likely key terms, if the page is truly relevant, and often repeated.”

If your read his first post here, he did not clarify anything, but reassured what he said: “The buzzword in IR is TFIDF”.

There is a difference between buzzword and jargon and IR and SEO.

Then alone came you here. i have seen you doing this before to promote you or your associates.

First you dropped by with a definition defense/diversion. Good try.

That didn’t work. So you switched to a different scenario (SEO buzzwords), then to accusations…

Talking about nitpick attacks. I think the nickpicker is you after all. Get a 4th of July life!

So, shall we discuss the merits of the tool? Or if you want to pick something apart, we could take a look at the “auto-SE-wordizer” for fun:

Now on the sustantive part:

Their rare parser looks like a wanderer/sampler matching certain words from documents, nothing of a novelty. How TFIDF plays in the picture and what is the document collection size used to compute IDF, if ever used?

I have seen many of these tools coming, going, and eventually ignored, often because add little or zero value to the bottom line of a business.

Still, I want to be fair. Please go ahead and start discussing the merits of the tool here.

So, shall we discuss the merits of the tool? Or if you want to pick something apart, we could take a look at the “auto-SE-wordizer” for fun:
http://rarestblog.com/2008/05/auto-sewordizer-automatic-search-engines-words-optimizer/

Having expended several hundred of your own words so far arguing the case for “slightly inaccurate word use” by Mr. Edmonds, you haven’t persuaded me that his comments are “false, inaccurate, and shallow.”

Some folks seem to have a need to be “right” all the time. You seem to be such a person, as I’ve watched you change your methods over the years from simply illuminating the subject to going after people.

This time, you’ve ignored the actual content of Andy Edmonds’ post, over the use of the word “buzzword.”

Which, I still contend, is exactly what tf-idf is when it’s “dropped” by SEOs who want to give the impression that they have some sort of secret sauce. LSI is also tossed out as a buzzword.

Frankly, I think you would agree with this, if you weren’t trying so hard to be “right.”

Hey, Dan; this is simple.

He has stated that TFIDF is a buzzword in IR and I contend that is not and explained why. So far he or you haven’t shown any evidence that it is a buzzword in IR.

In your previous post it was you that raised the “definition defense” and I simply show you why that defense simply cannot be sustained to tf-IDF based on the merits of those definitions. That is not going after anyone.

Now you are switching to TFIDF as buzzword in the SEO circles, which are another twenty bucks and topic. I have to agree with you on that one. But again that is a different scenario. If Edomonds wanted to say “the buzzword in SEO” why not saying it?

After that switching defense, you raises the “motivation defense”. Good try and equally wrong about me.

Sorry you are now taking things personal. If I contend that some of the folks within the SEO sphere are incorrect in some assessments or expose some of these guys for selling false claims, misrepresentation, and snake oil, and you don’t like it, I am sorry for you.

What you’re doing here is often interesting, often useful, often helpful. But you’re not always right.

I am not always right, by the way, nor pretend to be always right. Take that accusation you know where…

Should I expend a thousand words dissecting your “false, inaccurate, and shallow” characterization of TheRarestWords.com as a “tf-idf tool,” or stick to the subject?

It is up to you to expect that. You tell me.

As mentioned before, lets discuss the merits of the tool. Shall we?

Some folks seem to have a need to be “right” all the time. You seem to be such a person, as I’ve watched you change your methods over the years from simply illuminating the subject to going after people.

This time, you’ve ignored the actual content of Andy Edmonds’ post, over the use of the word “buzzword.”

Which, I still contend, is exactly what tf-idf is when it’s “dropped” by SEOs who want to give the impression that they have some sort of secret sauce. LSI is also tossed out as a buzzword.

Frankly, I think you would agree with this, if you weren’t trying so hard to be “right.”

What you’re doing here is often interesting, often useful, often helpful. But you’re not always right. Should I expend a thousand words dissecting your “false, inaccurate, and shallow” characterization of TheRarestWords.com as a “tf-idf tool,” or stick to the subject?

If you’re going to go after people, Dr. Garcia, please don’t lose credibility and do your homework. Start here.

One more thing, for definition lovers and pseudo teachers out there. This by virtue of wikipedia and previous definitions:

The statement that “the buzzword in IR is TFIDF” is by all means false, innaccurate, and shallow. In IR tf-idf is not a buzzword or “the buzzword”, nor is obscure, a new concept, trivial, a camouflage for saying nothing in particular, used to impress, etc, etc. In IR we know exactly what tf-idf means, stands for, and what it does.

Thank you for stopping by. Last time we exchanged few words was back in SES, NY 2005, I believe. I hope you are doing well.

Dan, you are more than welcome to disagree.

…a “buzzword” is exactly what tf-IDF is.

Sorry, Dan. I understand what you are coming from, but I have to disagree with you as well. Referring to IDF as a “buzzword” should be avoided. Why?

Buzzword is often used in reference to something that is new and obscure. He might not have literaly used the qualifier, but by using “buzzword” the message that came across was one of presenting tf-idf as something new and obscure.

Dan, since you want to go with the “definition defense”, a definition is as good and accurate as who proposes it -Google in this case.

Merriam-webster dictionary (http://www.merriam-webster.com/dictionary/buzzword) lists this for “buzzword”: (emphasis added to stress intention)

1: an important-sounding usually technical word or phrase often of little meaning used chiefly to impress laymen
2: a voguish word or phrase —called also buzz phrase

tf-idf stands for term frequency*inverse document frequency as a clear meaning. It is not of little meaning used chiefly to impress, nor is a voguish word or phrase. The scoring model it defines is quite simple:

tf = number of times a term appears in a document
idf = log(N/ni)

where N is the collection size and ni number of documents mentioning query term i. It is not possible to compute IDF, nor tf*idf without knowing the number of documents N in the collection. If assuming term independence, any computed estimate of N, IDF, and tf*idf can be meaningless, especially with a dumb tool that does not use large-scale web collections.

Wikipedia has this line about “buzzword” (emphasis added to stress intention)

Buzzwords differ from jargon in that they have the function of impressing or of obscuring meaning, while jargon (ideally) has a well-defined technical meaning, if only to specialists. However, the hype surrounding new technologies often turns technical terms into buzzwords.

Wikipedia also lists these “Reasons for using buzzwords” (emphasis added to stress intention)

Reasons for using buzzwords

With any stipulative neologism, such as “quark,” to describe new concepts, without the danger of over-simplification and confusion that can arise from using words and phrases with previously established, commonplace meanings.

To control thought by being intentionally vague. In management, stating organizational goals by using words with unclear meanings but positive connotations prevents anybody from questioning the directions and intentions of these decisions, especially if many such words are used.[2] (See also newspeak.)
To boost creativity among listeners by compelling them to think of the applications and particulars on their own.
To make something trivial seem to have greater import and stature.
To impress a judge or examiner by seeming familiar with a theory or principle by dint of mere name-dropping, as with “cognitive dissonance” or the “Heisenberg Uncertainty Principle.”
To provide a camouflage for saying nothing in particular.

Indeed, none of the above “definitions” applies to IDF or tf-idf, and quite honest is insulting and a bad descriptor used by someone that pretends to be a teacher, which is why your apparent “definition defense” cannot be sustained.

Let’s discuss now the merits of the tf-idf tool reviewed.

“A word or phrase connected with a specialized field or group that usually sounds important or technical and is used primarily to impress laypersons.”

Isn’t that exactly what you’ve been talking about? And isn’t it, in fact, quite accurate?

For instance, Wikipedia (http://en.wikipedia.org/wiki/Tf-idf) incorrectly describes tf-idf as a measure used to evaluate how important a word is to a document in a collection.

“The tf–idf weight (term frequency–inverse document frequency) is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document.”

It is not true that tf*idf is used to evaluate term importance. A term weight score does not equate to term importance. Personally I don’t know of any current IR colleague that claims such thing.

The other problem with Wikipedia’s definition is that assumes a function f of the form f(term importance) = tf*idf and therefore that “the importance increases proportionally to the number of times a word appears in the document.”

Obviously, this is an incorrect definition. The importance of a term does not necessarily increases proportionally to the number of times a word appears in a document. A term repeated x times is not x time more important. Similarly a document repeating a term x times is not x times more pertinent to the term.

In addition, taking the very same document and placing it in different collections, might change its tf*idf weight in the collection, but the importance of a term to a document remains the same, contradicting the proportionality assumption between term importance and the number of times a word appears in the document.

If that crap is what SEOs are teaching to their peers, they are messengers of misinformation.

Since you first referenced my post on TF-IDF, I’ve read your writings on it, and realized that I did not speak especially clearly in my post. It’s been on my list to revise it, while preserving the original, to make my message more clear.

I hope you do.

I’ll address your original post once I’ve rewritten it to make my points more clear, but it’s obvious you didn’t do your homework on me
Worse than that, your reference to me saying TF-IDF is new is entirely inaccurate. In no way did I suggest this is a new concept. My quote was “The buzzword in IR”, not the new buzzword.

IDF and tf-idf is still not a buzzword in IR. Good try.

Inaccurate and shallow reading like this makes me doubt the quality of all your writings.

Sorry to hear that. I might have to reciprocate your feelings about all your writings as well.

First of all, thanks for your work communicating about the challenges of search in a very detailed way.

Since you first referenced my post on TF-IDF, I’ve read your writings on it, and realized that I did not speak especially clearly in my post. It’s been on my list to revise it, while preserving the original, to make my message more clear.

I tried to provide direct feedback through several channels — but only received email bounces. It’s a tough email world with spam, I get it.

I’ll address your original post once I’ve rewritten it to make my points more clear, but it’s obvious you didn’t do your homework on me

Worse than that, your reference to me saying TF-IDF is new is entirely inaccurate. In no way did I suggest this is a new concept. My quote was “The buzzword in IR”, not the new buzzword. I was reviewing a new website doing some interesting word frequency analyses.

Inaccurate and shallow reading like this makes me doubt the quality of all your writings.

Best Regards,
Andy Edmonds

Thank you for expressing your opinion.

I’m inclined to agree with you, but only partially. The correctness of referring to log(N/n) as Inverse Document Frequency is a legacy from Dr. Stephen Robertson and the late Karen Sparck Jones who invented this measure of term specificity. For details, see The IDF Page over at

http://www.soi.city.ac.uk/~ser/idf.html

I always recommend my students to read link 3 of The IDF Page, to really understand what is/is not IDF; i.e., “Understanding Document Frequency: on theoretical arguments for IDF”.

http://www.soi.city.ac.uk/~ser/idfpapers/Robertson_idf_JDoc.pdf

The N/n is “inverse document frequency”.

In my opinion, there is nothing wrong with rescaling this using log notation and still referring to it as IDF. Honestly this is a matter of taste, in the same way is a matter of taste referring to pH as the negative log of the hydrogen ion concentration, pH = -log[H+], which even many chemists don’t agree with. But you can have your own opinions on this, just like Robertson and me.

Nevertheless, logs are used simply because they are additive and play well to two notions:

1. That document scoring functions tend to be additive.
2. The term independence assumption.

Incidentally, I am finishing the next issue of IRW newsletter which precisely revisits Robertson-Sparck Jones’s IDF concept and presents a new model for term specificity. I think you will like it.

I liked your post, except for the IDF business, and the TF discussion. Specifically something that really bugs me is this line:

IDF = log(N/n), which I have seen quite a few times in the literature.

We cannot do this anymore. Inverse document frequency has to be defined as 1/document frequency. Its really an insult to students with good math/physics background to use the log(N/n) definition.

And taking a log of IDF, we get the information content of the keyword. And in our dark ages very few people are aware of Shannon’s information theory, so a small discussion on logs and entropy is very warranted.

I’m planning to teach my first course on IR and text mining soon, so I got really interested in your blog.

thx
Pavel

P.S. Wikipedia has the same definition, and you can almost read IDF = log( IDF )

By now you should have copy of the practice exam. I am emailing copy of Lab 1 answers, concentrating in those the class found a bit hardy.

Please remember to send us the practice exam and the Lab 1 answers.

Thanks,
Gina

http://www.seomoz.org/blog/building-a-better-spam-detector

In any-case, you continue to be an inspiration. Thanks!

Correct.

I understand we will be recycling the query normalizer from Lab 4. On that exercise, we only removed trailing stops. However, I am guessing that for this exercise, we should completely tokenize the query. Is this right?

Thanks,
Gina

During the last lecture, I gave you a CSS file with block lines ending with semicolons. One of the students (Gina) pointed out to me that this should not be done. She is right as the CSS will not validate nor will be interpreted by Firefox (IE is more lenient in this sense). Thanks, Gina.

Please remove the semicolons and you will see the CSS in action in both browsers. When I wrote the CSSs and companion JavaScript instructions my texts editor was checked to end all lines with a semicolon. This explains the outcome. I forgot to double check the output in both browsers. My fault.

Talking about semicolons, Yahoo!’s Douglas Crockford and creator of JavaScript Object Notation (JSON) recommends ending functions blocks with a semicolon, especially when part of prototype lines. http://yuiblog.com/blog/2007/01/24/video-crockford-tjpl/

Interestingly, he also suggests the use of === instead of == in JavaScript to avoid coercion. My old scripts use ===, but when I run these with the latest version of IE on Windows Vista these seem to trepidate. Using == they seem to work just fine. I’m researching what causes the glitch in my scripts.

2. Reminder:

A reminder that to get full credit for your lab reports you need to turn in both hard and electronic copies of these.

Don’t forget to document all findings, including how-to installations if any is needed.

Try other alternatives.

1. Google for other perl scripts (there are plenty).
2. Try to use a C code like the old Lycos Scoutget code.
http://robot-club.com/lti/lycos/scoutget.html
3. Try any PHP or Java based spider.

Here is a barebone simple pseudocode. The crawler should do this:

1. get a web page via an http request (it can be via AJAX) and send it to a directory.
2. scrape links from that page.
3. pick a link from the page and do step 1.

Modifications:

In step 2, the program sends links to a list of links and in step 3, picks a link from the list instead from the page.

Once we send enough documents to the directory, we can index this with Terrier.

We’ve been working with the perl spider for a while now and while it seems to work (at least it runs!!!) it doesnt seems to be working right. It never get the right results (it doesn’t get any results, that is). I’d document what I do understand about the code but there not much I can do for now.

G. Nieves

I just ran a fresh search for inverted file without problem. This is what I did. I’m using the Windows version of Terrier from a USB removable drive. I tested on Vista and on XP.

1. Ran Terrier, indexing its own documentation.
2. Searched for inverted file as a query.
3. Double clicked on bin/interactive_terrier.bat file
4. Searched for inverted file as a query.

After the standard headers, I got.

Set TERRIER_HOME to be J:\terr
WARNING: The file terrier.prop
rrier\etc\terrier.properties
Assuming the value of terrier
INFO - time to intialise index
Please enter your query: inverted file

Displaying 1-82 result
0 326 112 2.908176030920721
1 753 471 2.8691891125133053
2 426 212 2.776936903771998
3 759 477 2.747474445016238
4 424 210 2.691801832968892
5 745 463 2.478234538724751
6 741 459 2.4410476260123852
7 734 452 2.4026089191907287
8 1000 665 2.351714630117981
9 427 213 2.3437764460259336
10 402 188 2.3294244235277524
11 975 640 2.2693842354980465
12 425 211 2.2625182087305125
13 422 208 2.2102952746378635
14 301 87 2.067004898225863
15 429 215 2.06550056347662
16 548 279 2.024848127420526
17 245 78 2.024848127420526
18 197 75 2.007975485950941
19 76 40 1.9954047945790332
20 404 190 1.979005166160501
21 703 421 1.9562281732712443
22 339 125 1.9536831254119893
23 434 220 1.9235037768768704
24 756 474 1.7579234589850747
25 28 10 1.7505691014173455
26 439 225 1.7188073021593937
27 406 192 1.6978730490041303
28 436 222 1.6698926255833755
29 747 465 1.6273903541804449
30 338 124 1.5987128520302278
31 22 5 1.5618933140413125
32 19 2 1.4967606279394443
33 751 469 1.480498642467794
34 124 63 1.4586748548300594
35 437 223 1.4486166892075465
36 749 467 1.424296235607222
37 468 254 1.3813768603244259
38 752 470 1.3636373922215335
39 304 90 1.2695069276928859
40 707 425 1.2595481602180154
41 968 633 1.2427295999967598
42 340 126 1.238697939749856
43 758 476 1.2361806398396333
44 718 436 1.2171117485149614
45 112 51 1.2044548267644286
46 303 89 1.191849419444905
47 1013 678 1.1900218043921074
48 716 434 1.175694715373152
49 305 91 1.1144461445417837
50 373 159 1.071949456510424
51 350 136 1.0442178750271376
52 120 59 1.032249223156422
53 39 21 1.0289066140518082
54 754 472 1.021245562355564
55 431 217 1.0112196888981333
56 40 22 0.9783128316736401
57 33 15 0.952665942151352
58 130 69 0.9323554684113402
59 401 187 0.9298191911149045
60 606 324 0.9031691289936831
61 662 380 0.8991516469240083
62 736 454 0.8835616760231381
63 323 109 0.882295943944168
64 126 65 0.8768326329975594
65 403 189 0.8603341765453334
66 667 385 0.853341277999745
67 742 460 0.8361417417726925
68 668 386 0.8064130750674726
69 412 198 0.7467488439890779
70 123 62 0.745745069118931
71 1007 672 0.6855982352785376
72 121 60 0.6414169643360372
73 115 54 0.612631867415782
74 717 435 0.5569985846392175
75 117 56 0.5425438654855919
76 128 67 0.5311998902670805
77 118 57 0.4688456765488282
78 127 66 0.44246701888818213
79 113 52 0.42745882075231073
80 125 64 0.34905485374098644
81 75 39 0.20774592741474804
Please enter your query:

Try this:

1. Index some files with Terrier Desktop and then do a search.
2. Keeping this one open, try using the interactive_terrier.bat for the same search.

I’ve been playing around with Terrier, searching the contents of the collection and learned how to index a specific collection. And while the desktop_terrier.bat searches are returning the correct results of my searches, I haven’t been able to do the same with the interactive_terrier.bat. It keeps returning “No results”.

I renamed the /etc/terrier.properties file and modified it to suit my document paths. However, with or without modifying the file, it does not return results.

I have searched the Terrier wiki and forums, and even Google, but I haven’t found a solution. Can you shed any light on this issue?

Thanks,
Gina

Yes.

Los labs tenemos que entregarlos también en formato digital pero, ¿en un archivo comprimido para pasarlo de un flash drive a otro?

Gracias.

Double check the naming conventions. Schlen and Smith also use different naming conventions. We can discuss it tomorrow in class.

Cheers

In order to test the pca1 program given in the papers, I advice you to use the example given in PCA and SPCA Tutorial by Dr. Garcia.

In order to obtain the same results than in the tutorial type:

[signals, PC, V] = pca1(data);

where

1) data is the transpose of X (first matrix in tutorial, where columns are ordered Weight, Height, Age)
2) PC is equivalent to V in the tutorial,
3) V is the diagonal elements of matrix S in the tutorial.

Not sure what signals really mean but sure it aint Step 4’s YV!!!
Can anybody share any light on this matter???

Cheers!!!

G. Nieves

Spectrum := set of all eigenvalues of a given matrix.

Matlab:
[PC, V] = eig(covariance);

Scilab:
[PC,V] = spec(covariance);

In order to use an user-built function you need to load the function first, once the function is written, type in Scilab environment:

getf(”C:\…whatever path…\cov.sce”);

In this case I was trying to load my function cov.

Once loaded simply type:

>>[c] = cov(A);

PS: If there is an error in the code, fix it and save and load the function again!!!

Cheer!!!

G. Nieves

Still having problems with:

1) [PC, V] = eig(covariance) usins scilab, well I found a possible solution for this:

“The Spectrum of a Matrix is the set of all eigenvectors of a matrix”

That is why I suggest you use

[PC, V] = spec(A);

instead. If youre using scilab, this function yields the same results (mathematically speaking, that is) than [PC, V] = eig(covariance)!!!

For more information just type help spec in the scilab environment.

2) If you need to express your results in with two decimals, just write format bank in matlab or format 5 in scilab (thanks A. Paris for this one).

3) All functions need to end with endfunction so be careful with the code, Im still working on this.

More info? Just keep on blogging (is that even a word/verb?).

Cheers!!!

G.Nieves

I did Exercise 4 using the provided codes. I copy and paste them in Matlab and had to do minor corrections to the code in order to use them. I didnt use Scilab for this problem but I will try them tomorrow on Scilab and do corrections if necessary and tell you whatever I find out.

Cheers!!!

G. Nieves

It is easier if you can provide EXCEL files and reference these in the report as doc-1.xls, doc-2.xls, etc. Place all files, docs, etc in a single folder and zip it. Provide also hard copy of the report.

Le escribo para preguntar si el todo el Lab hay que pasarlo usando el template que usted envió incluyendo los que expecifican EXCEL.

Gracias.

* There are extra ’ symbols inserted in the functions that generate syntax errors

* M-file style comments % convention give errors, I had to substitute with //

* All functions must end with endfunction. I had to add these to the code.

* ASCII to number function in Scilab does not exist. The only alternative I found is scanf which in c/c++ can cause lots of problems with overflows.

* Scilex gui is unstable and linking functions to TCL libraries gives errors at compilation time.

These are other errors that I have gotten so far, and I have not been able to solve. Does anybody have a recommendation?

* On Shlens code: the repmat substitute, ones command found in http://www.scilab.org/product/dic-mat-sci/M2SCI.htm, produces errors in matrix multiplication consistency.

* In Lindsey’s program the line: finaleigs = eigenvectors(:,1:dimensions); is giving errors. I looked up the function eigenvectors in scilab, and it does not appear in the site, nor in her code.

In Lab 1 Part 4, you need to provide a modified SciLab code of PCA. You also need to reproduce the example given in Lindsay Smith’s tutorial. Use EXCEL and any SVD calculator.

Be aware that the accepted formula for covariance uses n - 1 in the denominator, not just n. If your version of EXCEL uses n, you need to correct results by multiplying times n/(n-1).

I am putting together a tutorial on PCA to help you.

Cheers

Thanks.

Indeed we don’t have classes that week, according to the academic calendar over at http://www.pupr.edu/academiccalendar/ac-wi05.pdf

Verifique el calendario academico y segun dice en el, este sabado no hay clases, solo para notificarlo a usted y al resto del grupo. Cheers!!!

G. Nieves