Scoring and judging discussion

[Pokemon Trainer Sarah]

Original proposal:

JUDGING

• Judges should provide feedback to all participants after each challenge and must have a clear grading rubric.
• Judges will be assigned 100 points to distribute among the entries, with the best entry receiving the biggest share of points, etc. Each entry must receive at least 1 point, but it is otherwise up to the judge to distribute.
• Judging could potentially include other awards such as "people's choice", "most improved", etc.that do not give out any points, but that members can vote on as an extra thing so more people can get recognition.
• Judges cannot participate in their section, but will receive points for submitting their judging each week in a timely manner.
• Judges should submit their 4 week plan for their section (number of challenges, what each challenge is) before PXFIRE begins.

How to use this system with a rubric:

I don't think 100 points is that hard to distribute. You can still use a rubric. The points you give out don't have to match the rubric points, but you could do it that way with a bit of maths if you wanted.

A gets 15/20 on the rubric
B gets 5/20
C gets 15/20
D gets 16/20
E gets 16/20
F gets 20/20
G gets 12/20

Max total of rubric = 7 x 20 = 140
A got 15/140 = 10.7%. 10.7% of 100 = 10.7 points awarded.
B got 5/140 = 3.5%. 3.5% of 100 = 3.5 points awarded.
Etc.

This is the current idea for how judging could work. We already had some discussion on this in the other thread. Now we have to make a decision. If you have a different/additional idea for how judging could/should work then please post it here and we can discuss it! Try and post your proposal in dot points and make it as brief/clear as possible so we can discuss it! :)

@Speed-X @Morzone
I know you guys had some alternative ideas :) Would you still like to pursue them?

[Morzone]

Mostly my ideas involved the participation stuff, I actually quite like this method.

The only thing I'm worried about is that, if we get a large number of entries, the judge won't be able to give out the amount of points they feel some entries should get. Judging the creative writing for instance, we've been told that several times there've been very close, very good, entries. If we have too many of those kind of entries, they'd all get stuck with something like "14 points" even though the quality of the work is something several points higher.

Would it be possible for judges to have a method to request the ability to give out a small amount of additional points if a time like this arises? I'd imagine they'd have to appeal to the admins, and obviously this would be allowed Very sparingly and only if the admins truly agree they deserve more points than the limit allows them to get.

[Soups]

Mostly my ideas involved the participation stuff, I actually quite like this method.

The only thing I'm worried about is that, if we get a large number of entries, the judge won't be able to give out the amount of points they feel some entries should get. Judging the creative writing for instance, we've been told that several times there've been very close, very good, entries. If we have too many of those kind of entries, they'd all get stuck with something like "14 points" even though the quality of the work is something several points higher.

Would it be possible for judges to have a method to request the ability to give out a small amount of additional points if a time like this arises? I'd imagine they'd have to appeal to the admins, and obviously this would be allowed Very sparingly and only if the admins truly agree they deserve more points than the limit allows them to get.

I'm fine with the proposed judging system; however, you bring up a valid point. If a situation like that arises, it would be best for the judge to get into contact with PXFIRE Leaders ASAP to figure out a resolution. I think 100 points is the right call though, because while you MAY get an overwhelming amount of entries, you're more likely to receive an amount that isn't too overwhelming... and I'm making that assumption based on the activity PXR sees on a daily basis. We're more of a relaxed, slower moving community, so PXFIRE activity will probably reflect that (it may actually see a slight rise).

[AD]

Ugh, math, we meet again...

One of the reasons why i was hesitant about judging. But, i am pretty sure if I approach in a manner such as the judges in the Pokemon Contests, i will be less freaked by numbers.

[Morzone]

Yeah, 100 points should definitely be enough most -if not all- of the time. I was just thinking of times like at the start of a roleplay where you have a lot of really good players, and the points limit would be, well, limiting. Like I said, it's a "just in case" measure.

Though at the same time, we might have the opposite problem if we have too few entries. Sticking with the roleplay, while the beginning always has a large number of pretty skilled layers, by the end you're likely to have dropped down to very few. The last week in WAR II I only got the bronze points was because I was the third of three posters. If the entries at that point aren't really worth 30+ points to meet the 100, should A judge be able to give out less?

[Soups]

Yeah, 100 points should definitely be enough most -if not all- of the time. I was just thinking of times like at the start of a roleplay where you have a lot of really good players, and the points limit would be, well, limiting. Like I said, it's a "just in case" measure.

Though at the same time, we might have the opposite problem if we have too few entries. Sticking with the roleplay, while the beginning always has a large number of pretty skilled layers, by the end you're likely to have dropped down to very few. The last week in WAR II I only got the bronze points was because I was the third of three posters. If the entries at that point aren't really worth 30+ points to meet the 100, should A judge be able to give out less?

I understand. I feel like this is an issue that can only be solved if we knew the exact amount of members that are going to partake in PXFIRE, which events they'll participate in, and how active they'll be... which is impossible. I think the safest way to go is to have an emergency stash of points. Maybe an extra 100? Let's say Entry A is the clear-cut winner, but you can't reward it the amount of points it's truly worth because you HAVE to split the 100 you have between the other entries (which may give other great entries less points as well). What do you do? Well, you'd be able to tap into your emergency stash of points. Of course you'd have to contact the PXFIRE leaders beforehand to get their approval. I think it's some form of a solution.

[Neo Emolga]

Yeah, 100 points should definitely be enough most -if not all- of the time. I was just thinking of times like at the start of a roleplay where you have a lot of really good players, and the points limit would be, well, limiting. Like I said, it's a "just in case" measure.

Though at the same time, we might have the opposite problem if we have too few entries. Sticking with the roleplay, while the beginning always has a large number of pretty skilled layers, by the end you're likely to have dropped down to very few. The last week in WAR II I only got the bronze points was because I was the third of three posters. If the entries at that point aren't really worth 30+ points to meet the 100, should A judge be able to give out less?

Definitely a viable concern, but the only thing is then we'd have to work in a system that bases points awarded depending on the number of players. Which IS possible, but it gets complicated.

One thing about a situation where only a few people are taking part. Sure, you'll have fewer people getting more points, but this rewards them for taking part in something where most people didn't. And the better they do, the more they'll get rewarded for it. The people that stick with it through every week will get more of an income from it, which is good.

As for the flip situation where there's a lot of people only earning small shares, this isn't a bad thing either. It encourages people to put in more effort if they can so they earn more. Or if the entry process seems easy, this will even things out a bit compared to some of the other "tougher to enter" sections.

Hence, this is kind of why I say keep it at 100.

[Morzone]

I did consider that if people were willing to stick it out when others weren't they would probably deserve more points for it anyway so...

It's fine! I like the 100 point system, and to be honest I don't think we'll have too much of a problem with having too many/too little participation. I just wanted to make sure the possibility was recognized before hand in case it came up and we got complaints about receiving so few points even if a judge went on and on about how great the entry was.

[Pokemon Trainer Sarah]

Definitely a viable concern, but the only thing is then we'd have to work in a system that bases points awarded depending on the number of players. Which IS possible, but it gets complicated.

Yeah, my issue with this is it's possible to game the system by having your team enter one really amazing entry and then everyone else submits a joke entry so now there are more participants and therefore more points awarded and their #1 entry can get more overall points.

But maybe I'm the only one who would find loopholes like that xDD

If all the events give out the same number of points, it'll encourage people to enter all the different events rather than sticking to one or two that might be more "profitable". Then we hopefully won't have too many events where not many people enter.

We could also make it that judges don't have to award the WHOLE 100 points if they feel the entries didn't deserve it. And we could also have a limit on the maximum points that can be awarded to one entry, like say 30 or 35 points or something. That way say only 1 or 2 people enter drawn art, they're not getting 50+ points just cos no one else entered.

[Caite-chan]

I know Speed had said something about how she would have her own method of scoring because of the way the GFX section would be. And that is fine but you can grade the way you want and then from there take the 100 points and distribute them between the entries.