Tucows Ratings

This question submitted by Jane Owens, Carl Schroder, William Fromius, Linda Jamison, Amy Wilson and numerous others

There has always been much discussion, by everyone involved, about ratings. Whether you're the software author who has a product rated or the user who is looking for a product and utilizes the ratings as a guide, ratings are an issue. The goal here is not to go over the entire Tucows Rating System, which you can read by clicking on this link, but to clear up some misunderstandings about ratings. It's my hope that by addressing this topic we can clear up these questions.

"It's An Honor..."

As a society we like to honor things. We have awards for countless achievements. For movies we have the Academy Awards, for theatre we have the Tony Awards, for books we have Pulitzer Prizes, and I could go on and on. We like to recognize achievement. Lets take the Academy Awards as an example. I don't even want to hazard a guess as to how many movies are produced each year. Or how many actors and actresses have roles in those movies. Or how many "supporting roles" there are in each. And you'll often hear people say, and rightly so, that "It's an honor to even be nominated." I have to agree. If you take all the movies made in a year, and narrow it down to five for "Best Picture", it is indeed an honor to be one of the five nominees. How does this relate to Tucows ratings?Well read the next line carefully.

Over 70 percent of programs submitted to Tucows are REJECTED!

That's right, 70 percent plus are NOT accepted. Only three programs out of10 do well enough in the Tucows ratings to be included on the site. I've been looking at, evaluating, and reviewing software for nearly 25 years.To me, the fact that only three out of 10 make the cut means "It's an honor to even be on the site." And it is! Tucows only selects what we consider to be the best products from those that are submitted.To do that, a "ratings guide" is used, and this guide has evolved and changed over time.

Ratings

The reviewers at Tucows look at each program submitted and try to weigh it both quantitatively and qualitatively. And like any system that tries to "grade" or "rate" something, it's a blend of both opinion as well as absolute functionality that are combined to determine a score. That is to say that each rating is determined using both objective as well as subjective criteria. ALL reviews or ratings contain subjectivity, that personal and individual bias that comes from so many things. No system is ever perfect, but then, nothing is.

Rating Criteria

I'll briefly summarize the criteria each program is evaluated on. If you'd like to read the criteria in detail you can simply click HERE. The criteria are:

• Usability:(37 percent) This criterion is divided into seven separate categories; there are 21 possible points available for the entire criterion.
• Help, documentation and support:(25 percent) This criterion is divided into two separate categories; there are fourteen possible points available for the entire criterion.
• Program enhancements:(18 percent) This criterion is divided into six separate categories; there are ten possible points available for the entire criterion.
• Overall evaluation (20 percent) (11 points possible)

And the cow ratings emerge like so:

• 27 to 33 points = 3 cows
• 34 to 37 points = 4 cows
• 38 points or more = 4 cows

As you can see, the difference between the highest number to get three cows and the lowest number to get five cows is only 5 points. And in a rating system one point can make the difference between four cows and five cows. Or three cows and four cows. What's my point? I'll tell you.

Summing It Up

Ratings are based on both objective and subjective criteria. With the objective criteria things are more straightforward. If the criteria says you get one point for having such and such and two points for having such and such plus that, then the decision is very clear. The program either has one or the other. But, when you get to something like the "reviewer's overall impression", you're getting into something subjective. An area that awards points based on the reviewer's opinion, which is also based on their past experience.

What would I hope that you would get out of this article? A couple of things. First of all I think it's important for both users and software authors to realize that any program we have listed on Tucows is better than 70 percent of the other programs submitted. If they hadn't made the cut they wouldn't be on the site at all.

The next thing I would hope is that both users and authors would realize that there is most often very little difference between a program with three cows and one with five cows. I often get comments from users about reviews.Such as, "You made that program a Pick and it only got three cows!" Sure,but I'm not looking at the same criteria. And next week in this column I'll tell you what my criteria is. A program that gets three cows is not a bad program, it just missed some points in the review process.

Ultimately, as I've said many times before "Software is always a matter of personal taste!" What appeals to one person may not appeal to the next person. Here at Tucows we've spent a great deal of time and effort to sort through the software submitted to us using the ratings criteria. And we've accepted the 30 percent that make the cut and they're here for you to evaluate. Whether three cows or five cows, you can know that each product you look at on Tucows has survived our testing and our ratings system.

I'd like to thank Jane Owens, Carl Schroder, William Fromius, Linda Jamison, Amy Wilson and numerous others for asking this question.

If you have a question on any technology topic that you'd like someone to tell you about you can submit it via email by clicking HERE You will not receive a reply, but all topics will be considered.