You can look at how the experts are getting along during a judging task.
Select the Run Judging Session option, and then, just to make sure the judging statistics are up to date, press the Refresh Task button:
You can check how the judging is progressing for your experts by examining the information in the Judges Table:
The different columns show the following information:
- Email - This is the email address of the expert.
- Link - This is the unique link that can take you to the judging site for this expert for this task.
- Infit - A statistic related to the consistency of individual judges.
- Local - the number of judgements the expert has made.
- Mod - a measure not relevant to research uses of No More Marking that we would expect to
- Quota - the total number of judgements allocated to that expert. This information is useful if you need to remind experts to complete their judgements.
- Median Time & Time Chart - Here you can see the median time taken by a expert in their judgements on this task. Quite short median times might indicate that a expert is rushing through their judgements (it is informative to compare the median time to the infit score), and a long median time might indicate that the expert is facing some problems with the judgement process (or that they just left their computer on when they went to bed). The Time Chart shows a broader picture of the range of times taken by the expert for their decisions - uneven distributions might again indicate problems with the judging process.
- % Left Click - In the judging screen, artefacts are presented randomly on the left or right hand side of the screen. Therefore, we would generally expect that the percentage of times that an expert clicks in favour of the 'Left' artefact would be around 50%. If the % Left Click value is much higher or lower than 50%, it might indicate that the expert is not thinking carefully about their judgements and just staying with the left or right hand side. Again it is informative to compare this value with the infit statistic and the median time taken.
- Exclude - If there are real concerns about the standard of judging by a particular expert, you may wish to Exclude this expert's decisions from the final calculation of scores, so that they are more reliable. Only do this after you have tried to sort out any problems with the expert.
The Remove button (the 'bin' icon) completely removes the expert from the process, and we would normally suggest that you do not do that. Remember that in excluding or removing experts, this will have a knock-on effect on the number of judgements other experts need to make, and you may need to change the number of judgements assigned to the other experts. You can do this by adjusting the number of judgements, or selecting a expert's check box and pressing the 'Update Judge Quota' button.
You may also want to examine the different decisions made by your experts, and information on that is provided in a downloadable CSV file. To access it, click on the Decisions button on the Judges page:
