Marking > Assess: On multiple markers, annotations and moderation of scores
In the Cirrus platform you can add multiple (up to 2) Assessors or Markers per candidate that completed an assessment. This article dives into the dynamics of having multiple pairs of eyes on scoring your candidates.
- Multiple assessors and sharing annotations
- The Moderator role in Cirrus and it's setup
- Multiple assessors and no Moderator
Assessors are able make annotations when they score open questions (like essay questions). When two assessors have to mark the same candidate / question, they can both make annotations. However: they will not be able see each others annotations until they have submitted their scores. This is to ensure the highest level of impartiality while assessing.
Note: a user cannot be both an Assessor and a Moderator for the same scheduled assessment.
Moderation settings by the Marking Coordinator
When a Marking Coordinator sets up assessors for a schedule, they can choose to assign a Moderator. After a marker has assessed the candidates and submitted their scores (and comments and annotations) these will be eligible for moderation depending on the settings for moderation. For example:
- Show assessor annotations - when checked the moderator will be able to see the assessment by the assessor(s). When not checked the moderator will base his judgement purely on his own assessment. (Note: Moderator judgement trumps Assessor).
- Difference in scoring attempt - There is a difference of 10% or more in endscore of the assessment related to the maximum score possible.
- Proximity to pass-mark - The endscore of the candidate is within 2% of the pass-mark percentage.
After the marker has assessed the candidates and submitted their scores (and comments and annotations) these will be eligible for moderation depending on the settings for moderation. In these cases the moderator will need to:
- Check the answer(s) of the candidate
- Check the annotations of the assessor(s)
- Set the final score for the candidate and submit his verdict - the moderators score is final and supersedes the markers judgement.
- Following this anyone setup as a publisher by the Marking Coordinator can publish the final scores.
Please note: The moderator can check annotations made by the markers
- After the marker (assessor) has submitted the scores for a candidate
- When the marking coordinator has checked the option Moderation > Show assessor annotations for the schedule.
- If this has not been done - to increase impartiality on the part of the moderator for example - the moderator scores without seeing the exact scores and reasoning by the assessor(s).
If you are setup as Moderator for a schedule you can:
- Go to Marking > To be Assessed-tab.
- The assessment that needs moderation will be marked "Needs moderation"
- Go into the schedule and select a candidate by clicking on the name.
- Click on 'Needs Scoring' next to the first question that needs moderation.
- The 'Essay Marking' page opens - this is the same for Moderators.
- If the marking coordinator has checked the option 'Show assessor annotations' the moderator will be able to see any marker annotations. Selecting this option will allow the moderator to view annotations from the assessor(s).
- The Moderator can check annotations made by the marker(s) clicking on the right side under "Annotation" > View > the dropdown > select the marker to see the annotations of:
When assessors award a candidate / answer the same amount of point, the moderator will not come into play.
When the moderator has finished, the assessor needs to grade the exam and publish the results.
In some cases it is up to the moderator itself to choose which handed in exams to moderate. For example when you want to pick a random sample or double check the top, bottom and middle scores as awarded by your Markers. This is where bulk moderation comes in handy:
- Set the rule to moderate all scripts by setting the proximity to pass mark to 100%.
- The moderator will see all handed in exams that have been marked and submitted
- The moderator selects which ones to moderate and handles these first.
- The moderator selects the remaining candidates and clicks "Bulk Moderate"-button.
- This opens a new window (see below) showing the number candidates selected to moderate answers for. The first 10 will be shown before you have to click on 'Show all'.
- The Moderate leaves a comment (optional but recommended as this will shows the action performed!)
- Click on 'Save' > the comment will show up in the General Comment box for each candidate selected.
Tips on Bulk Moderation:
- The window will show what scores will be used in case of one or two assessors.
- If you've selected candidate that you already manually moderated a warning will show.
- When bulk moderating leaving a comment is optional but strongly recommended as this will shows the action performed!
When a scheduled assessment does not have a pre-appointed moderator, in the case of multiple assessors scoring differently, the answer will receive the average of both scores.