Subject:
|
Re: Why sets receive a ZERO?
|
Newsgroups:
|
lugnet.general
|
Date:
|
Mon, 18 Nov 2002 14:49:11 GMT
|
Highlighted:
|
(details)
|
Viewed:
|
488 times
|
| |
|
|
"Larry Pieniazek" <lpieniazek@mercator.com> wrote in message
news:H5qp07.5wD@lugnet.com...
> Maybe there should be a requirement that in order to be allowed to rate sets
> you need to rate some sets good as well as some bad? Require that your
> average rating across all sets you rated is at least 10 and no less than 90?
> That is, no going and rating EVERYTHING a 0 or EVERYTHING a 100.
If we have a 0-100 scale, then assuming some kind of normal distribution, we
would expect the average rating to be about 50 with a standard deviation of
(say) 20. Therefore, it becomes possible to moderate a member's ratings
according. Basically you take the set of actual ratings they have contributed,
and then calculate the actual mean and standard deviation of those ratings.
Then you adjust each actual rating N as follows:
X * N + Y
where X is the relative standard deviation and Y is the relative mean. The
effect of doing this is that the adjusted ratings should produce the desired
mean and standard deviation. Thus *overall* set ratings will have a mean of 50
and s/d of 20.
Individual set ratings can then be computed from these adjusted ratings
provided by each LUGnet member (just do a normal simple mean).
In simple terms what this means is that people who consistently over-rate sets
high will have their ratings scaled down, and vice versa. If people are being
stupid (e.g. allocating 100 to all their favourite sets and 0 to every thing
else), the standard deviation will be too large and cause the ratings to be
scaled back to about 70 and 30 respectively.
Before anyone says this is ridiculous, I would point out that here in
Queensland, we operate our tertiary admissions system on this basis. As we do
not have external exams taken by every final year student, each school sets its
own assessment and rates students accordingly. To prevent schools from
over-stating (or under-stating) the abilities of their students, the scores
provided by the schools are adjusted to achieve the mean and standard deviation
applicable to general intelligence tests applied to the same group of students
(these tests are mandatory and standardised across the state). So, while the
students's individual results come from school-based assessment, these are
adjusted to reflect the overall class performance to the state-based
assessment. The general effect of this is that subjects that tend to attract
only the brighter students (e.g. physics) have their results up-rated and that
subjects that tend to attract less-bright students (e.g. social maths) tend to
have their overall results down-rated. This system of university admission
assessment has been operational for about 30 years now, and apart from minor
tinkering, the basic principle of adjusting each class's
mean/standard-deviation has remained unchanged as a fair basis for tertiary
admission.
Because the process is entirely mechanical, it requires only programming and no
human intervention.
Note. This system is still based on the assumption that a high rating indicates
a set preferred by the user to a low rating. If people are reverse-rating sets
(through error, malice, or a desire to deliberately distort the results in some
way), then nothing can solve that problems, but the more extreme results will
probably be scaled back to something more reasonable and the consequent impact
on the overall rating of a set will be reduced.
So a person who submitted all 0-ratings would get scaled back to 50-ratings on
each set (i.e. no impact on the overall means). Similarly any person who
submitted only 100-ratings would get scaled back to 50-ratings as well. This
means that people need to equally willing to rate the sets they like and the
sets they dislike, as otherwise their ratings will be scaled up/down to produce
a more "average" mean. I've argued elsewhere on LUGnet that most recent
releases get rated (good/bad as appropriate), but only the older *better* sets
get rated (and hence rated favourably) as everyone forgets about the also-ran
sets of the past and doesn't bother to rate them. IMHO this is why older sets
tend to overrank newer sets
Kerry
|
|
Message has 5 Replies:
 | | Re: Why sets receive a ZERO?
|
| (...) This is off-topic, but I really like the sound of your system. Here in Canada, we don't have standardised testing (like the SATs in the US) or any sort of grade balancing like this. Grade inflation is rampant. (...) I only rate sets that I (...) (22 years ago, 18-Nov-02, to lugnet.general)
| | | Re: Why sets receive a ZERO?
|
| (...) <snip well thought out but complex system> And someone had the temerity to actually agree when I called MY proposal "possibly overengineered"??? :-) I like it. I think it would work. I never know when Kerry's spoofing or not, though. :-) (22 years ago, 18-Nov-02, to lugnet.general)
| | | Re: Why sets receive a ZERO?
|
| (...) Heh-- I actually did something very similar for our company when we were sending out our customer service surveys (rated 1-5). Obviously some clients were overly thrilled with us and just gave us straight 5's. Some were mad at us and gave us (...) (22 years ago, 18-Nov-02, to lugnet.general, lugnet.admin.general)
| | | Re: Why sets receive a ZERO?
|
| In lugnet.general, Kerry Raymond writes a really neat statistical analysis thingie which I snipped: My question here is "What are you using this information for which requires such analysis?". I mean do you base your set purchases on these figures? (...) (22 years ago, 18-Nov-02, to lugnet.general)
| | | Re: Why sets receive a ZERO?
|
| (...) I'm not sure that an average rating of 50 would be appropriate in this context. One of the ongoing themes of discussion in LUGNET is that LEGO set designs deteriorate over time. The apex of LEGO set design is generally thought to be in the (...) (22 years ago, 19-Nov-02, to lugnet.general)
|
Message is in Reply To:
| | Re: Why sets receive a ZERO?
|
| (...) I would say probably not... some sets really ARE duds. (...) That's not a bad idea. It works for Figure Skating (and we know how fair and impartial THAT sport is!) Grin.... No seriously, it IS a good idea. I would say this, though.... how abou (...) (22 years ago, 17-Nov-02, to lugnet.general)
|
48 Messages in This Thread:
 
- Entire Thread on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
This Message and its Replies on One Page:
- Nested:
All | Brief | Compact | Dots
Linear:
All | Brief | Compact
|
|
|
|
