What does an Amateur Radio Rating or Review really mean?
The annual pre-Hamvention “who is rated highest” games has started it 2019 round with yet another new rig be trumpeted as “the bee’s knees.”
As I write this radio hasn’t been on the market for a month, but the campaign is lauding the rig as so exceptional that you just have to buy one. (The FCC approval for one version was achieved April 12th 2019 despite a Hamvention 2018 May debut, and the higher power version is still waiting for FCC approval.)
It pretty interesting the weight some ham give various other ham’s testing results, especially as there is no assurances of what these tests mean in actual use nor any way to stay clear of the murky world of how all this testing is paid for. It isn’t like every ham pitched in a fiver for a set of 2019 tests to be run, as even the closest labs to that ideal are part of larger organizations who sell advertising to the very same radio manufacturers they are testing their radios.
It is pretty hard to stay clear of the concerns if there is a financial relationship or sponsorship in the mix.
So what do these test mean?
One testing guru recently stated by email that in one of his tests despite his ranking radios, that anything better than a test result of 99 is going to operate the same way for the operator. The implication is that 99 and upward are indistinguishable and interchangeable is terms of that performance metric – the test has resolution that is finer than the operator’s perception of resolution and may be a level of resolution that later aspects of the radio-signal-to-human or human-to-radio-signal path has other characteristics that mask any perception.
Now here is where I get confused – this “Test Guru” identified a threshold where testing is no longer relevant, yet goes to great lengths to test radios beyond this threshold, and rates any above this threshold by the lab numbers he just told us were unimportant – lending some gravitas to what he also dismisses?
And it is confusing declaring testing beyond a usable level irrelevant than then providing a ranking of this irrelevance, it is even more confusing is how the economics work? Both in acquisition costs (the particular radio that caught my eye has a launch price of $4,000 and hopefully more than a single sample was tested, which makes a considerable layout in gear to test) and the hours of test time (hopefully the radios were operated for a period to be “burnt-in” and then subject to a series of test procedures) are large enough to question whether the altruistic service to the ham community can be supported without someone paying the bill.
Some time back my suggestion was that any manufacturer wanting their gear tested pay the lab a fee for testing plus the cost of purchasing three of their radios on the open market, with the radios being given to the manufacturer for their disposal on completion of testing. Not perfect but better than leaving everyone wondering who is picking up the bills and why?
I’m going to jump around here a bit in talking about reviews and suggest the reader at least skim the following four links:
Okay, these are about the Amazon Marketplace and how the various conflicts of interest have created what amounts to amoral world where the buyer’s trust is misplaced and misdirected because of the seller’s and sub-seller’s games.
It is interesting that Amazon Marketplace appears in print on these concerns, as some of the older on-line Bazaars like eBay have kept below the radar. Even though law enforcement speak of eBay often being a conduit for diverted, counterfeit or stolen goods, you won’t find much search for it.
Our hobby has several outlets for resale, with two offering forums and one additionally offering product ratings.
And mike they say “Surprise, Surprise” some of these outlets sell ads to manufacturers. At least one allows packs of forum trolls to run rampant under apparently the “every click is a good click” theory.
Our hobby isn’t big dollar markets like these other mega-markets, but the same principles of trust apply – a fragile trust that is put into pale light when test labs are soliciting other business, consultancies and retainers.
A trust that is further undermined when both positive and negative sock puppet reviewers skew the marketplace ratings of products.
Ditto trust damage when in un(der)moderated forums non-owners and in some cases self declared experts who have never even operated the radios they profess to have important opinions upon, are given equal standing as those with equipment and operating time investments.
So where does all of this put the average ham?
Some suggest that if you are not in the position to form a direct personal opinion about a purchase decision, that you are not ready to make that purchase.
Others point to obtaining a purchase decision based on what your friends all support – a sort of non-decision deferring to group think over individual determinism.
There are those who suggest you need to follow the best reviews, and that way you will be making the right decision.
Personally I use a personal system of evaluation, tailored to each decision set. I like the ideas in “Making Great Decisions in Business and Life” by Henderson https://www.amazon.com/Making-Great-Decisions-Business-Life/dp/0976854112 along with some decision making skills imparted by Army training.
While I listen to others, I qualify and rate what they tell me, including their expected bias AND my expected bias upon hearing their opinions.
If you have your ducks in a row, it isn’t in the least bit scary to make a less than common decision, as you will have already examined the axis of risk(s) thinking through or at least understanding you decision’s risks vs other decision choices.
If you understand that the Ratings and Reviews are at best murky and at worst untrustworthy, you can factor in the amount of impact those ratings or reviews has for you in your particular decision.