COMP3100/COMP3500/COMP4500 - SE_Projects

Peer Assessment - WebPA-x Worked Example

This page provides a very simplified example of the rating algorithm used by WebPA-x to calculate indvidiaul peer assessment. The example uses a single group and rates each student against a single question. Numbers are rounded to two decimal places for readability. However, the actual algorithm is more precise and provides greater accuracy.

Step 1 - Collate the ratings submitted

	Alice	Bob	George	David	Peter
Alice	4	4	3	2	1
Bob	3	5	3	2	0
George	4	4	4	4	4
David	3	5	4	3	1
Peter	-	-	-	-	-
	14	18	14	11	6

The rows across are the ratings awarded by each student. The columns down show the ratings each student received. Ratings are totalled at the bottom. From this was can see that Bob has performed the best, while Peter has performed he worse. Also Peter did not submit a peer assessment as denoted by the dashes in his row.

Step 2 - Normalise the ratings awarded by each student

All the ratings each student awarded are added up (i.e. total each row).

Sum ratings awarded, ignoring the ratings an individual gave themselves

Alice awarded: 4 + 3 + 2 + 1 = 10 marks
Bob awarded: 3 + 3 + 2 + 0 = 8 marks
George awarded: 4 + 4 + 4 + 4 = 16 marks
David awarded: 3 + 5 + 4 + 1 = 13 marks
Peter awarded: 0 + 0 + 0 + 0 = 0 marks

Once it is known how many marks each student has awarded the fractional score for each student can be calculated. In other words, every student in the group has an identical 'cake', but they can give different sized slices to each other.

Alice awarded a total of 10 marks (so her cake was split into 10 slices).

Alice gave Bob 4 marks = 4 / 10 = 0.4
Alice gave George 3 marks = 3 / 10 = 0.3
Alice gave David 2 marks = 2 / 10 = 0.2
Alice gave Peter 1 marks = 1 / 10 = 0.1

These are the fractions awarded by Alice to each student. As a test, all the fractions can be added up and they should equal 1.

This process is repeated for every student for every question. In this example there is only one question so we only need to add up this one question.

Calculate the WebPA-x score awarded to teach student

Before each student's WebPA score is calculated, we need to know how many students were in the group, and how many of them submitted marks.

In this example the group had 5 members. However, as WebPA-x does not use self-asssessed ratings, it means 4 cakes should be split between 4 members. Unfortunately, Peter didn't submit any marks, so only 3 cakes are available!

To compensate, WebPA-x calculates a score for Peter as if he had awarded everyone the same score. In this case 1/4 or 0.25

The WebPA-x scores can now be calculated. Taking Alice as an example, we add up all the fractional scores she received. Previously we calculated the franctions Alice awarded, but the other students' fractions are produced in exactly the same way.

Bob awarded Alice 0.38 (= 3 / 8)
George awarded Alice 0.25 (= 4 / 16)
David awarded Alice 0.23 (= 3 / 13)
Peter didn't submit, so he effectively awarded 0.25

Once all the individual factors are added up the WebPA-x score is calculated. For Alice this means:

Alice's WebPA-x score = 0.38 + 0.25 + 0.23 + 0.25 = 1.11

Therefore Alice's WebPA-x peer assessed score is 1.11

If the same calculations are done for each of the other students, we find:

Bob's actual WebPA-x score = 1.28
George's actual WebPA-x score = 1.23
David's actual WebPA-x score = 0.95
Peter's actual WebPA-x score = 0.45

As a quick check, adding up all the WebPA-x scores for every student, it should equal the number of students in the group. Adding up the scores gives 5.02, which allowing for rounding errors (we're using 2 decimal places in this example) is 5, the total number of students assigned to the group.

Rating versus Evaluation

Unlike WebPA, WebPA-x has a single question that seeks an overall holistic rating of a student's peers. While the peer assessment score for the rating criterion is calculated in the same manner as shown above, rating is worth 50% of the total WebPA-x peer assessed score.

What this means, is that if Alice scored 1.2 for the rating criterion and 1.11 for the evaluation criteria, her overall WebPA-x score would be (1.11 + 1.2) / 2 = 1.155

Step 3 -- Calculating the final peer assessed mark

WebPA-x calculates a student's individual mark by taking the mark the team achieved for that assessment item and multiplying it by the WebPA-x score it has just calculated.

Thus if Alice's team was awarded 11 out of 15 for Review 1, her WebPA-x peer assessed mark would be 11 * 1.155 = 12.7.

Based on the WebPA worked example which can be found at http://www.webpa.ac.uk/?q=node/127