Sunday, October 19, 2003

USELESS RANKING

In a desperate attempt to imitate Maclean’s Magazine the Globe and Mail has released their second annual University Report Card. For their part, Maclean’s has been criticized in the past for the methodology they employ in ranking Canada’s universities. But at least they have a methodology.

The Globe Report Card is so flawed as a survey and a ranking system it is unbelievable that anyone could take it seriously or use it as a valuable point of reference for anything. The survey was conducted by Uthink, an on-line youth marketing firm. This should give you some idea of what the real purpose behind this survey is.

The method of the survey is as follows: Uthink asked all of its subscribers to complete a hundred question survey on their university experience. Respondents rated their ‘satisfaction’ with their school’s performance in each category. Then Uthink ranked the results and the Globe and Mail published them.

Here are the problems:
The vast majority of student respondents have little or no way to compare the school they attend to any other single one, let alone all of the rest of the schools in the country. How are students at Trent supposed to know how their school rates in the availability of teaching assistant compared to the University of Alberta? How are students at McMaster supposed to compare their school’s career counselling to that of Dalhousie?
The survey does not account in any way for the fact that students at one school may be judging their situation based on criteria and standards far more rigorous than those at another school.

In his introduction to the survey Allan Gregg writes, "For example, Simon Fraser students rank its on-line library and teaching materials as one of the best in the nation, yet are among the most dissatisfied with their school spirit and the quality of student residences." This tells the reader nothing beyond the fact that SFU students like their library more than their residences. Yet the Globe acts as if they can assume that SFU residences are worse in absolute terms than those at Brock on the basis that Brock students rated their residences better than their public transportation system!

The next problem is the number of students who responded to the survey. Mr. Gregg titled his introduction "Why 26,000 students can’t be wrong." Actually they can be. At least in collective terms. The survey made no attempt to gain a representative sample of students from each of the universities ranked. This is Survey Statistics 101. Even I, the arts student, know this. Any school that received 230 responses was included in the survey. In theory, if only 200 Queen’s students responded their school would not have even been considered. Next we find out that the responses for individual schools ranged between 235 and 1,532. Clearly Uthink put a lot of thought into accurately representing opinion from various schools. If only 200 students from UofT completed the survey and 1500 completed it from McGill are we supposed to believe that we can objectively compare these two schools on a statistical basis?

This survey is so flawed, so methodologically unsound, so utterly useless as a point of comparison that giving it this much analysis is probably a compliment.

I submit that the real reasons behind this survey are the following:
1. To provide Uthink and the Globe and Mail with valuable advertising access to the lucrative 18-25yrs. age bracket.
2. To convince Globe and Mail board members that they have just as important a ‘national publication’ as Maclean’s does.
3. To be used as a point of reference to criticize the administrators of Canadian universities and academia in general
4. To drive me crazy by ranking such pantheons of scholastic achievement as Sherbrooke University, Laurentian University and the University of Victoria higher than McGill on ‘overall quality of education.’ (A slight degree of academic elitism? What did you expect?)

Disclaimer:
Yes, I am a student of McGill University who happens to believe that my school (the one that graduated Wilfred Laurier and financed Ernest Rutherford's Nobel Prize winning work) is the best school in the country if not the whole world. This does not mean that any of the aforementioned criticism is less valid.

Posted by Matthew @ 3:05 a.m.