Monday, October 10, 2005

Apples to apples: Do comparisons always need to be fair?

I'm becoming increasingly ambivalent about the PC hardware review sites - Tom's Hardware is perhaps one of the better known, but there are a whole bunch of them. This is no doubt partly to do with my changing interests - I'm not quite so involved in hardware as I once was.

But the bigger issue seems to me that the review sites have lost touch with what I do with computers.

Case in point: PC motherboard reviews. I've been playing around with Personal Video Recorder projects for the past 12 or 18 months. In essence, turning a PC into a Tivo workalike. Lots of fun. My latest setup has a big box in a closet with a couple of TV tuners and a big RAID, and a little bitty box beside the TV that has almost no moving parts (there's still one fan I can't seem to shake while still getting decent video output).

It isn't without bugs - but that's what makes it a project.

But one of those bugs seemed, after googling, reading, chatting on email lists and so forth, to be due to DMA issues between a tuner card and the other parts of the system when running large data volume that is generated by a card. in essence, the motherboard or chipset couldn't handle large-volume DMA from multiple sources at once - it caused a reset.

And this got me thinking: if I wanted to pick a great motherboard for home video servers, do the benchmark sites give me any useful information?

The reviews seem to follow a tried and true (and old) formula. Picture of the board. Oh gee, it's green. Isn't that cool. The box is very nice. Look, it has the same slot connectors as everyone. Yup there's room for a large processor heatsink. And so on. Then some benchmarks, which compare this board to others in a raft of performance-critical uses, like spreadsheets, word processing. Okay okay, they do some games as well - very important, the game tests.

But you'll note, no tests at all in which large amounts of bus traffic are driven from an expansion card - all the tests of the system are application-related. And these days, when we're seeing a convergence of home computer and media, application-related use is not the only thing happening.

Another case: I saw a great article - I hate to call it a review - of a couple of desktop PC motherboards designed to take laptop CPUs. The article mentions that these boards are targeting a segment of the market intersted in noise (well, quiet) and power consumption (less of it, that is). Folks like me - I was charmed. "Do processors need to consume over 100 watts?" the author wondered.

Pity was, the article didn't compare any of the boards on the basis of power consumption or noise. The article didn't even report the power consumption or noise of these products. Instead, it included the same old performance comparison.

I think the reason the review folks do this is so they can compare the new product against those they reviewed last week, month, or year. And compare tomorrow's review against this one.

This is nonsensical. In their drive to compare apples to apples, they fail to give the folks for whom these products were built any useful way to select between them.

PS. For those as may be interested, the video server project in question combines MythTV and tuner cards including pcHDTV 3000, AirStar HD5000, and Hauppauge WinTV-PVR-250. The server runs Debian linux and includes a 1.2 TB RAID5 array. The client is a diskless EPIA M10000 based mini-itx system in a Morex 3677 case that boots across the LAN using pxeboot.

[Comments from my previous blog]

1. Sean Kennedy left...
Saturday, 3 December 2005 7:21 am ::
Very good observations. I would be interested to know more about your client setup. Can it display HDTV transport streams? Does it output 5.1 sound?

No comments:

Post a Comment