Marking: How to mark assessments in Cirrus
Cirrus provides a user friendly yet powerful marking workflow, which can adapt to the type of assessment and your organisation way of working. This makes Marking - also known as Scoring or Grading - assessments in Cirrus easy and intuitive.
Marking can be divided into choosing the preferred setup (workflow) and performing the actual marking:
- Setting up who can mark your assessment
- Human marking versus automatic marking
- The Simple Marking workflow: the scheduler is the marker
- The Extended Marking Workflow: a coordinator assigns one or more markers
- Vertical Marking: dividing candidates over several markers
- Horizontal Marking: dividing questions over several markers
- Setting up the Marking Coordinator
- Assess: Execution of the actual Marking by your Assessors:
So who will be marking your assessment? In Cirrus you can select to not assign any assessors and use automatic marking to skip human marking or choose a simple or extended workflow for human marking.
Automatic marking: for assessments that contain closed questions only
When your assessment only contains closed questions, also known as automatically scored items, the assessment author can choose to skip marking altogether using the "Do not assign assessors" option - All your questions are automatically scored and you do not need to add assessors:
For Assessments that need a human marker, the assessment author can chose between the simple and the extended marking workflow:
- Simple: the scheduler = the marker (the simple marking workflow).
- Extended: a coordinator assigns assessor(s) (the extended marking workflow).
If you select this when publishing the assessment, whoever schedules the assessment will also become the marker for the assessment. No moderation will be involved. This is often used in higher education for formative and certain summative assessments.
Choose wisely: If you select the Simple Marking Workflow for the assessment, you cannot change this later on to the Extended version - nor can Cirrus do this for you.
The marking coordinator is someone with the appropriate permissions to coordinate / assign markers for schedules (sessions) linked to a hierarchy level.
Marking Coordination: this is what you need to know and do
Under Marking the Coordinator tab shows marking coordinators a list of assessments. In the Cirrus platform these coordinators can add a pool of Markers to your scheduled or delivered assessments.
Depending on your choices you get to allocate who marks the candidates or questions, if a moderator needs to checks the assessment and override it and who can publish these marks.
Coordinators are assigned to certain hierarchy levels to allocate the work for schedules linked to those hierarchy levels. On the Coordinator tab they can see if a schedule needs or have assessors or moderators assigned or what stage the marking is in.
- Go to Marking > the 'Marking Coordinator Overview'-tab:
This list shows the schedules you are allowed to manage the assessors for:
- You can add/change moderators by clicking on the schedule title.
Upon mouse-over you can see how many candidates in the schedule exactly still Need Assessing / Moderation / Results that can be published.
- Use the search bar to search for schedules, filter by date or filter by hierarchy using the 'funnel'-icon (tip) - it turns black when a filter is active.
- In the 'hierarchy'-column it shows which hierarchy level the schedule was associated with. This can help you see which delivery centre was involved for example.
- In the 'status'-column you can see if assessors (and moderators when applicable) are needed and where the schedule is in the marking process:
- Assessor needed: This is the status of assessments where no assessors have been added
- Ready for assessing: This is the status an assessment gets when the assessors (and moderators) have been set by the coordinator, but the assessing has not yet started.
- Assessing in progress: This is the status of assessments where assessors have marked at least 1 candidate, but not submitted the scores
- Moderation needed: This is the status that is displayed when assessors are finished with marking, but some submissions needs to be moderated. Note that this status only shows when the moderator is the one that keeps the process from proceeding (i.e. all submissions needs to be assessed and submitted first by your markers).
- Moderation in progress: This is the status of assessments where moderators have marked at least 1 candidate, but not submitted the scores.
- Results pending: This is the status of assessments where all assessors/moderators have submitted their scores and the results are waiting to be published.
- For large sets you can filter on these statuses by clicking on the 'burger'-icon.
Please note: when candidates are added to an ongoing schedule the status for the marking coordinator will reverse to 'Assessor needed' as markers will need to be allocated to these extra candidates.
In the Cirrus platform the coordinator can choose from a pool of markers and assign them to your scheduled or delivered assessments. The coordinator decides who marks the candidates or questions and if a moderator needs to check the scoring by the marker(s). This moderator can override the scoring. The coordinator also determines who can publish these scores.
- As a coordinator you go to Marking > the Coordinator (tab) to see an overview of schedules that need your coordination.
- Click on any of the listed schedules to to start
- The 'Settings'-tab opens on the 'Manage assessors' page:
In the Cirrus platform the coordinator can choose from a pool of markers and assign them to your scheduled or delivered assessments. Depending on their choice they get to allocate who marks the candidates or questions, if a moderator needs to check the scoring by the marker(s) and if it needs to be overridden. Coordinators also determine who can publish these marks:
- Assessing type:
- One assessor - each candidate (or question) will be assessed by one marker from the pool.
- Two assessors - each candidate (or question) will be assessed by two markers from the pool. Each marker will have an equal voice in determining the mark. If selected the coordinator can choose to select:
- Sharing annotations between assessors - selecting this option will allow assessors to view each others annotations once their scores have been submitted.
- Difference in scoring (when two assessors and moderation are selected): if two assessors score a candidate very differently, the coordinator can setup a rule to have a moderator determine the final score. Simply enter a how big a difference in scoring (in % of the total score) will trigger this.
- Allocation by: choose between Candidate or Items.
- Candidate - the coordinator divides candidates to score over the selected marker(s) (also known as 'Vertical Marking'.
- Item - the coordinator divides items to score over the selected marker(s) (also known as 'Horizontal Marking'.
- Sharing annotations between assessors
- Only is assessing same - marker will be able to view scores, comment and annotations of other markers only if user is assigned as an assessor for same candidates/items
- All candidates/items - marker will have access to information for all results, even if user is not a assessor
- Anonymous candidate - this will hide the names of candidates. Only candidate IDs will be displayed to the markers. Anonymous assessor - this ensures that the assessors will be hidden from each other. Both features can be used to ensure more impartial marking.
- Use Moderation - Selecting this option enables the use of moderation for the assessment. Remember: In case of one assessor + moderator the moderators scoring will be final.
When 'Use Moderation' is selected the coordinator will be able to set the rules which control when a moderator will come into play for this schedule
- Difference in scoring attempt - when you've selected to have one candidate be marked by two different markers and their scores given differ too much, you set the %, the moderator will need to have a look and has the final verdict.
- Using the 'Proximity to pass-mark': when a candidate score is very close to the pass/not passed boundary, it is considered best practice to have a moderator check and determine the final grade. In some cases coordinators opt to set this to 100% so a moderator can check all scores or just a few and use 'Moderate multiple' to bulk accept the scores by the markers.
- Pass-mark range: this option allows to specify a specific range for results that need to be checked by moderator
- Don't forget to click 'Save and next step' to move to the 'Assessors-tab'.
Note: coordinator might not need to use all rules for moderation. In this case not necessary rules should be empty.
Read more about the dynamics of having multiple pairs of eyes on scoring your candidates in our article On multiple markers, annotations and moderation of scores.
Marking Coordination: the 'Assessors-tab' allows you to manage the assessors, moderators and publishers for your schedule
On the Assessors-tab you can add/edit the assessors and moderator involved in grading for you assessment schedule. Here you also determine who can publish the final scores:
- Simply use the 'Add' and Remove'-buttons to add users to this list
- Determine wether they are an Assessor or Moderator or Other for this schedule.
- Publish results - use the check-box for each user to determine if they are allowed to publish the final results. Hint: by selecting Other you can assign non-marking staff to publish the results of the assessment later on.
When no marking has been done you can simply change the allocation of markers to your schedule and/or to individual candidates or items. However, given the following scenarios:
- Marking Coordinator has assigned markers to assessments
- Markers have started marking and have submitted scores
- The Marking Coordinator changes the assigned markers (a warning is shown
"are you sure...")
- This will mean that submitted scores will be cancelled. So situation will be as if no one has ever done something with scoring for the candidates. The markers work is lost. This action cannot be undone.
And in the scenario were the results have been published
- The marking Coordinator has assigned markers to assessments
- Markers start marking and have submitted scores AND published results
- This will mean that the marking coordinator is no longer able to change the assigned markers. Candidates who's results have been published are highlighted on screen and the marking coordinator will get a message.
Marking Coordination: Allocation - setting who exactly will be the assessor and moderator for which candidates or which questions of a schedule
Once you've assigned a pool of assessors to mark your assessment, you can allocate who will mark which candidate/question:
- Select the Assessor/Moderator using the dropdown in the appropriate column.
- You can use 'automatic allocation' to assign assessors / moderators in bulk to all candidates or randomly (which will lead to an even distribution over the assessors available. So if you have 2 assessors each will get half the candidates to mark.
Example on marker allocation:
Example: you can add 6 markers to your planned / taken assessment session. These then can be distributed manually or automatically in pairs over the candidates. So you can set two markers per candidate, choosing from the 6 available.
Any user in Cirrus can become a marking coordinator if:
- The user has a role that allows for "Assigning Assessors"
- The user is set to be marking coordinator for the correct hierarchy level(s)
Setting a role to allow for "Assigning of Assessors"
- Go to Admin > Roles > Site (tab)
- Enable the permission "Manage assessors" for the appropriate Role(s):
Setting users as marking coordinator for the correct hierarchy level(s)
Being coordinator for a certain hierarchy level, means you will be able to coordinate the assignment of assessors (markers) for all schedules (bookings) which have that hierarchy level added as a property.
- Go to Admin > Users > a specific user (Jeremy in the example below)
- Go to the "Hierarchies" tab for this user
- Check the box for each hierarchy this user should be marking coordinator for
- Note: being marking coordinator inherits down, so you become coordinator for all sub-levels automatically.
Once you've assigned your markers and you as a marker have selected an assessment to mark, you can proceed with:
- Selecting an Assessment to mark - using the Marking Overview page
- Scoring Items (Assessing) and Submitting results
- Grading the assessment using an assessment scale based on points scored or percentage scored
- Publishing the results for the assessment.
Selecting an Assessment to mark - using the Marking overview in Cirrus
A user can see assessments that he has been assigned to mark in 'Assess-tab' overview. Simply click on any of 'Ready for Assessing' assessments to start Scoring Items (Assessing) and Submitting results. Assessments with only Multiple Choice questions skip this step and move to Grading.
- Go to Marking > Click on the 'Assess-tab':
- In this overview you will see a list of all schedules assigned to you as a marker to assess (aka mark/ grade). The Status shows the following:
- Pending - Assessments that have not been taken by and candidates and cannot be assessed.
- Ready for assessing - Assessments that have submissions by candidates that can be assessed.
- Assessing in Progress - Assessments have submissions and assessment has started.
- Assessed - Marking /Moderation was fully done and submitted. They are pending publishing and only need to be published.
- You can click the 'hamburger icon' (bottom left of the page) to filter on status. For example, you could filter the list to show only assessments that are ‘Ready for assessing’.
- To start scoring you can click on the schedule title after you read 'Scoring Items (Assessing) and Submitting results'. Tip: the mouse-over shows you details such as
- the assessment title (which can be different from the schedule title)
- Your progress: how many candidates are left to mark.
- If desired you can use the search box to narrow down the list / search for a specific schedule to mark based on the schedule title.
Moving through the Marking Workflow in Cirrus
We regularly get questions on what information shows where under Marking. Slightly oversimplifying things Marking has the following tabs (not all might be visible to you):
Read more on this in the article 'Marking: publishing results of your candidates'