Subject:
|
Re: Why sets receive a ZERO?
|
Newsgroups:
|
lugnet.general
|
Date:
|
Tue, 19 Nov 2002 00:05:23 GMT
|
Viewed:
|
651 times
|
| |
|
|
In lugnet.general, Kerry Raymond writes:
>
> "Larry Pieniazek" <lpieniazek@mercator.com> wrote in message
> news:H5qp07.5wD@lugnet.com...
> > Maybe there should be a requirement that in order to be allowed to rate sets
> > you need to rate some sets good as well as some bad? Require that your
> > average rating across all sets you rated is at least 10 and no less than 90?
> > That is, no going and rating EVERYTHING a 0 or EVERYTHING a 100.
>
> If we have a 0-100 scale, then assuming some kind of normal distribution, we
> would expect the average rating to be about 50 with a standard deviation of
> (say) 20.
I'm not sure that an average rating of 50 would be appropriate in this
context. One of the ongoing themes of discussion in LUGNET is that LEGO set
designs deteriorate over time. The apex of LEGO set design is generally
thought to be in the past, with opinions varying from around the mid 70's
through to about the early 90's. Some people would say the first release of
Star Wars was the apex, but I think we can happily dismiss them as being
caught up in the commercial hype of the whole Star Wars thing.
Therefore, we could not peg ratings to an average of 50. The average rating
should really be based on a mathematical formula based on year of release.
If you were going to say that an average of 50 applied over time, despite
commonly accepted deterioration over time, you would have to also accepted
an average of 50 over themes. Now, the sole Insectoids fan out there might
think this is good, because his ugly (and mostly blue) sets are going to be
miraculously lifted in the ratings, but consider the legion of people who
like blue boxes with wings who would going to see the Classic Space ratings
tumble.
> Therefore, it becomes possible to moderate a member's ratings
> according. Basically you take the set of actual ratings they have contributed,
> and then calculate the actual mean and standard deviation of those ratings.
> Then you adjust each actual rating N as follows:
>
> X * N + Y
>
> where X is the relative standard deviation and Y is the relative mean. The
> effect of doing this is that the adjusted ratings should produce the desired
> mean and standard deviation. Thus *overall* set ratings will have a mean of 50
> and s/d of 20.
>
> Individual set ratings can then be computed from these adjusted ratings
> provided by each LUGnet member (just do a normal simple mean).
>
> In simple terms what this means is that people who consistently over-rate sets
> high will have their ratings scaled down, and vice versa. If people are being
> stupid (e.g. allocating 100 to all their favourite sets and 0 to every thing
> else), the standard deviation will be too large and cause the ratings to be
> scaled back to about 70 and 30 respectively.
Ahhh... but you fail to appreciate that certain people are better judges,
who would - of course - only purchase better sets. Why should those train
fans be hampered in properly rating their sets, when no hopers who bought up
big on Aquazone Hydronauts get to award an average of 50 as well?
> Before anyone says this is ridiculous, I would point out that here in
> Queensland, we operate our tertiary admissions system on this basis.
<snip commentary on Queensland tertiaray admissions system>
> Because the process is entirely mechanical, it requires only programming and no
> human intervention.
>
> Note. This system is still based on the assumption that a high rating indicates
> a set preferred by the user to a low rating. If people are reverse-rating sets
> (through error, malice, or a desire to deliberately distort the results in some
> way), then nothing can solve that problems, but the more extreme results will
> probably be scaled back to something more reasonable and the consequent impact
> on the overall rating of a set will be reduced.
I think this part could be overcome by correlating set raters. If we
established a basket of LEGO sets representing a certain range, we could
calibrate all raters against it. We do this for meat grading in Australia -
we have a system were meat graders are regularly correlated with a common
eye group to ensure that all graders are calling the same results for the
same appearance of carcases.
Raters would be able to rate so that there average rating was above 50,
whilst preserving the integrity of the whole rating system.
> So a person who submitted all 0-ratings would get scaled back to 50-ratings on
> each set (i.e. no impact on the overall means). Similarly any person who
> submitted only 100-ratings would get scaled back to 50-ratings as well. This
> means that people need to equally willing to rate the sets they like and the
> sets they dislike, as otherwise their ratings will be scaled up/down to produce
> a more "average" mean.
While this sounds plausible on the surface, it does assume that sets per
reviewer will rate an average 50 over time, which will not work as I've
outlined above.
> I've argued elsewhere on LUGnet that most recent
> releases get rated (good/bad as appropriate), but only the older *better* sets
> get rated (and hence rated favourably) as everyone forgets about the also-ran
> sets of the past and doesn't bother to rate them. IMHO this is why older sets
> tend to overrank newer sets
This is partly true, but I think nostalgia (or selective memory) comes in.
If a set comes from whatever you think the golden age is (pick any 3-5 year
period from 1975 to 1994) it's got to be worth another 10-20 points right
away, right?
Of course, I think we should move on from subjective rating and go to an
objective system which looks at measurables (number of pieces, colour
consistency, building time, presence of Timmies or Jar Jars, number of sets
sold, etc) and derives a rating which can be repeated again and again.
Cheers
Richie Dulin
|
|
Message has 1 Reply:
 | | Re: Why sets receive a ZERO?
|
| (...) Hmm, I see a lot of potential for expanding your organisation's efforts from carcases to live animals, to be specific, tertiary applicants. Given the way the parents moan about all the statistical moderation that takes place, I think a lot of (...) (22 years ago, 19-Nov-02, to lugnet.general)
|
Message is in Reply To:
| | Re: Why sets receive a ZERO?
|
| "Larry Pieniazek" <lpieniazek@mercator.com> wrote in message news:H5qp07.5wD@lugnet.com... (...) If we have a 0-100 scale, then assuming some kind of normal distribution, we would expect the average rating to be about 50 with a standard deviation (...) (22 years ago, 18-Nov-02, to lugnet.general)
|
48 Messages in This Thread:
 
- Entire Thread on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
This Message and its Replies on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
|
|
|
|
