Marking > dichotomous and polytomous scoring on items

Cirrus Assessment Support Team -

This article will describe different aspects of scoring rules when using Cirrus:

Marking: scoring rules when using polytomous scoring on Items

In short: Dichotomous versus Polytomous scoring

  • Dichotomous scoring - the candidate must match everything correctly in order to get a score. 
  • Polytomous scoring - gives the candidate a partial score if answer is partially correct.

The latter applies for example to auto-scored items (meaning they are 'marked automatically') like a Multiple Response question with Scoring type set to 'Polytomous'.

 

Example: Order (Order) with/without polytomous scoring:

Consider a simple example of an Order question with 4 items to put in the correct order and a maximum score of 4 to understand the basics of dichotomous versus polytomous scoring:

 Question Type: Order (4 alternatives) Dichotomous Polytomous 
no value correct 0 0
1 value is correct  0 1
2 values are correct 0 2
4 values are correct 4 4

For more complex question types like Multiple Response or Extended Match however, the amount of options a candidate can select affect the scoring model used. 

Polytomous scoring example: Multiple Response (MR)

When using polytomous scoring for MR questions in Cirrus, the author is offered two scoring models based on the number of answer options the candidate can select:

  1. The number of answers to select is equal to the correct options
  2. The number of answers to select has no limit.

2018-11-06_1640-MR-options-example.png

MR polytomous: The number of answers to select is equal to the correct options

For multiple-response questions in Cirrus using polytomous scoring - limit on answer options the candidate 'knows' only how many options will be correct. He will receive partial scoring for each option selected correctly. Consider the following example question:

Select the 2 cities with the most tourist visiting in 2018

  • Barcelona
  • Paris
  • Amsterdam
  • New York
  • Moscow
Question Type: Multiple Response MR (polytomous - limited)
Number of alternatives 5
Number of correct alternatives 2
Number of incorrect alternatives 3
Max score 2
Score per correct alternative 1

Determining the candidate's score
2 points are divided across 2 correct alternatives
In other words: points per correctly selected option 2/2 = 1. 

MR polytomous: The number of answers to select has no limit.

For multiple-response questions in Cirrus using polytomous scoring - no limit the candidate does not know the number of correct options to select. He is awarded points for

  • Selecting the correct alternatives
  • Avoiding incorrect alternatives, i.e. you are also awarded points for not selecting incorrect alternatives.

Consider the following example question:

Which cities had the most tourists in 2018?

Question Type: Multiple Response MR (polytomous - no limit)
Number of alternatives 5
Number of correct alternatives 2
Number of incorrect alternatives to avoid 3
Max score 2

This means that for determining the candidate's score 2 points are divided across five alternatives. This means each alternative is valued at 0.4 point
In other words: points per correctly selected / avoided alternative: 2/5 = 0.4.

Say your candidate selected 2 alternatives but only 1 is correct. This means the candidate selected 1 alternative correctly and avoided 2 incorrect alternatives. This means the candidate scores 3*0.4=1.2

Polytomous scoring example: Extended Match (ExtMatch)

When using polytomous scoring for ExtMatch questions in Cirrus, the author is offered two scoring models based on the number of answer options the candidate can select:

  1. The number of answers to select is equal to the correct options
  2. The number of answers to select has no limit.

2018-11-06_1640-MR-options-example.png

ExtMatch polytomous: The number of answers to select is equal to the correct options

For Extended Match questions in Cirrus using polytomous scoring - limit on answer options the candidate 'knows' only how many options will be correct. He will receive partial scoring for each option selected correctly. Consider the following example question:

Connect the options on the left with the correct numbers on the right

Even         2
Uneven   3
Prime   4
    6
    7

 

Question Type: Extended Match  (polytomous - limited) Total
Number of option 'left'   3
Number of options 'right'   5
Number of correct connections 3 (even) + 2 (uneven) + 3 (prime) 8
Max score   4
Score per correct alternative =4/8 0.5

Determining the candidate's score
4 points are divided across 8 correct alternatives
In other words: points per correctly selected connection 4/8 = 0.5. 

ExtMatch polytomous: The number of answers to select has no limit.

For multiple-response questions in Cirrus using polytomous scoring - no limit the candidate does not know the number of correct options to select. He is awarded points for

  • Selecting the correct alternatives
  • Avoiding incorrect alternatives, i.e. you are also awarded points for not selecting incorrect alternatives.

Consider the following example question:

Connect the options on the left with the correct numbers on the right

Even         2
Uneven   3
Prime   4
    6
    7

 

Question Type: Extended Match  (polytomous - no limit) Total
Number of option 'left'   3
Number of options 'right'   5
Number of correct connections 3 (even) + 2 (uneven) + 3 (prime) 8
Number of incorrect connections = 2 + 3 + 2  7
Total number of connections = 5 + 5 + 5  15
Max score   4
Score per (in)correct alternative = 4/15 0.267

Determining the candidate's score

This means that for determining the candidate's score 5 points are divided across 15 alternatives. This means each alternative is valued at 4/15.
In other words: points per correctly selected / avoided alternative: 4/15 =0.26667.

Say your candidate selected 2 alternatives but only 1 is correct. This means the candidate selected 1 alternative correctly and avoided 2 incorrect alternatives. This means the candidate scores 3*0.4=1.2

Example: Hotspot with/without polytomous scoring:

Consider a simple example of a Hotspot question with 4 items to point to. For scoring it is important to realise this gives 5 answer options: 4 hotspots and 1 "nothing clicked" option. The candidate will be scored on correctly checked options and avoided incorrect options.

When taking a maximum score of 5 points for the question this lead to the following scoring table: 

  Dichotomous Polytomous 
nothing selected 0 0
4/4 incorrect - no value correct 0 0
1/4 value is correct (3 are incorrect) 

0

1/4 value is correct (0 are incorrect)  0 2
2/4 values are correct / 2 incorrect 0 2
2/4 values are correct / 0 incorrect 0
3/4 values correct / 1 incorrect 0

3/4 values correct / 0 incorrect

4 values are correct 5 5
     

 

Marking: scoring rules when using adjust for chance score on an Assessment

For many question types candidates could opt to just simply guess the correct answer if they do not know it. This is why many organisations take this chance of guessing the answer into account during scoring and subsequently in determining a 'pass' or 'fail' or grade.

Example: When delivering for example an assessment with 80 MC4 questions (i.e. only Multiple Choice questions with four answer options) your scoring / assessment scale should take into account the chance score of your questions.

80 MC4 questions means that a candidate has a 1 in 4 chance of simply guessing the answers correctly. This basically means on average 0,25*80=20 points can be scored by simply guessing!

What does this mean for the pass mark? Let's assume the uncorrected  pass mark is set at 50%. This means you need to score 50% of 80 points = 40 points to pass. When taking chance score into account this in turn means that you would need to score (20 points + 50% of the remaining points) or (20 + (0.5*60) points) = 50 points out of 80 to pass. In other words your percentage to pass mark adjusted for chance score is 62,5%.

Example using 150 questions, pass mark at 70%:

2018-07-11_1721-Assessment-form-option-adjust-for-chance-score-example.png

Please note:

  • You set this when you've created your assessment form on the 'Options-tab'.
  • Adjusts percentage to pass and grade scale percentages using chance scores are to be used with dichotomous questions only.
  • Be sure to understand how this works before using this with our dichotomously scored questions!
  • As the screenshot shows: when using a mix of different question types the calculation become a lot more complex. Read an example.
  • Please note: If you use other than Multiple Choice questions you will use the Chance score calculations as used in Cirrus. It is imperative you understand the implications before using this.
  • If these do not match your needs you can simply create a custom Assessment Scale for your Assessment Form.
  • Feel free to contact support in case you want to know more.

Your feedback or help in creating good examples is appreciated as always! Mail us at support@cirrusassessment.com.

Comments 0

Cirrus uses Zendesk to provide our Help Center services. Check out Zendesk's Terms of Service and Privacy Policy, as they govern your use of these services.

Powered by Zendesk