Sneak Peek #3: New Judging Platform and Voting Algorithms coming 2019
Design and development of our new website continues to gain momentum and is on track for a big release next year. As mentioned last month, The Rookies is changing in 2019 and we are excited about what the team from Envy Labs is helping us to create.
It's easy to forget how crucial judges are to the Rookie Awards. They are the ones putting in the hard work, reviewing projects, deliberating and comparing thousands of entrants each year. With such a huge workload we needed to make sure the judging process was quick, intuitive and interactive. We also wanted to make sure it was fun so judges will continue to come back each year.
We are super excited about this one! Our new judging system will remove any potential bias. This means that judges will not be able to see any personal data like name, gender, age, location, school etc. They will only see the submitted work. It's a small thing, but it's something that we know will really help take our contests to the next level and allow us to run some pretty special contents.
This means that judges will not be able to see any personal data like name, gender, age, location, school etc.
In previous years, when a judge reviewed an entry they could see other judges scores and notes. We feel this information could potentially influence a judges score either positively or negatively. Either way it's not good. So in 2019, voting will be completely private. Judges will not have any distractions or external influences.
Interactive & Blazing Fast
Long gone are the days of clicking through dozens of pages to review one entrant. Everything happens on a single interactive page. Judges can view entries in a carousel view, stacked view or full-view. It's up to them as they move through the content adding their scores, viewing images in pixel perfect resolution and adding their own notes. They can refer to other entries they have reviewed, flag any issues they might encounter and even talk to other judges using special code names assigned to each entry (remember, no names are used).
Individual Judges Weighting
We are also developing a new algorithm to help improve consistency and accuracy from each judge. In essence it will analyse a judges individual score across the full range of their own voting data set. This means that if the judge is a high-scorer, their score gets evened out across the full range. Conversely, if the judge is a low-scorer it will be evened out too. The algorithm is pretty involved, but needless to say it’s really going to help standardise our voting system. It will also dramatically reduce the overhead of our lead judges who previously had to monitor these issues.
The algorithm is pretty involved, but needless to say it’s really going to help standardise our voting system.
Finalist Voting & Comments
There are a few key stages to our voting process. One of the more interesting stages is Finalist voting. This is after all scores have been collected and our lead judges work together to nominate entries worthy of a Finalist ranking. Judges are then asked to nominate their Top 5 entries based on their own scores. This stage was normally handled externally, but it's all integrated now with section to capture public comments about their selection and finalists that impressed them the most.
More to come
We still have lots more to share! In the coming weeks I will share news about our new contest system. We all know the Rookie Awards, and we all know our legacy of running contests back in the days of CgCoach. In 2019, we will be rolling out loads of new contests aimed at challenging artists to become the artist they know they can be. So stay tuned and get your Wacom pens ready.