COMP2100
Assignment 1 Marking Guide

Background

Students were asked to submit a test plan, a test script, a number of test data files and a test report. The printout has a cover sheet, then their plan, their report, their script, their input and output data files, the results of running their script (and data) on the same version of test_formatter as they were given, and finally the result of running their script on a modified version of the program, with as few defects as possible.

Overall I want the mark on each assignment to reflect the level of achievement attained in that assignment. I hope the guide below points you in that direction. I believe that rigid marking guides have in the past tended to produce meaningless assignment marks, in particular inflating assignment marks and giving students a false sense of their level of achievement and a false expectation as to their final mark on the unit. I believe it is important that a mark of 16/20 or higher truly reflects a piece of work at High Distinction level; that a mark of 14 is work at Distinction level; 12 is a Credit and 10 is a Pass. To achieve this I am not giving a breakdown of this many marks for the report, that many for the script etc. Look at and give feedback on each of the different sections of their assignment, but then determine a single overall mark out of 20.

1. Test Plan

Consider the following points:

• Is the plan written in clear simple English? It doesn't have to be great literature, but it must be clear and unambiguous.

• How well does it cover the requirements? It's not essential that their plan cover every requirement comprehensively, but they need to show that they have read and thought about the requirements, and made a serious effort in thinking about how to test them.

  Some students may have elected to only cover some of the requirements. This is OK, but of course there should be some deduction. It should not be possible to get a High Distinction by only addressing half of the requirements.

• Are the test cases adequately described? The combination of the objective and description of each test case should be enough to give you a clear idea of what they want to do and how. It may not be quite enough for you to reproduce their test*.in and test*.out, but it should be enough for you to be confident of writing something similar.

• Is each test case traced back to the requirements?

• In places where the student believes the requirements to be ambiguous or incomplete, I have asked them to indicate this in their plan or report, and not to perform a test if the requirements make it impossible to determine the correct output. Don't let them throw all the requirements away because they're not expressed using Propositional Calculus, but do accept (and reward) reasonable instances of this. It shows that they are thinking the right way.

2. Test Script

• Does it work, or are there syntax errors, various things that cause it to crash? (Check the output.)

• Does it print a header in the correct format? Take half a mark off if they haven't managed to figure out the right options to pass to date.

  There may be other correct combinations, but one format string for "date" that works is:

  date +'%A %d %B %Y, %H:%M:%S %z'

  What I want to see is the full name of the day of the week (e.g. "Monday", not "Mon"), the day in the month, the full name of the month ("March", not "Mar"), the year (all four digits), a comma, the full time in 24-hour clock, hours:minutes:seconds, and then the time difference from GMT for this time zone (which should be "+1000" since the marker run was after the weekend -- it would have been "+1100" before the weekend).

• Will it find all test data files of the form testnn.in and run the program on them? (If it will only find test files with numbers starting at 1 and increasing in order, that's OK. They may assume that there won't be any gaps in the sequence - although it would certainly be more robust if it didn't.)

• Does it give diff output for each failed test? Is that output in the right format? Take something off if they haven't got it right.

  This year, the correct options for "diff" are:

  diff -C0 -L Expected -L Found

  If they got "-c" (with three context lines around each difference block) that's not quite right and they should lose half a mark. If they used the ordinary output format or the unified format (correct last year), take off a mark.

• Is there any risk of getting into an infinite loop (or does the output show that it actually did go into an infinite loop)?

• Does the script clean up any temporary files it creates?

3. Test Data

• Do their tests really test what the plan says they do?

• Do their output files agree with the requirements? (This is very hard to check of course. Assuming that their script works OK, a good way to check is to look at the results of running their tests on the correct version of the test_formatter. That's at the end of the printout.)

• Are the tests all there, as described in the plan?

To check everything on every test would take far too long. Check what you can and develop a general impression. I expect you to spend around 20 minutes on each assignment.

4. Test Report

• Does their report agree with the results of running their script on the original test_formatter?

• Does their report agree with their plan? (For example, if the plan says Test 7 only tests Requirement 12, and the report says that failing Test 7 is because the thing doesn't satisfy Requirement 9, then there's something wrong.) Again, check what you can, at random or if something sticks out. You can't verify everything.

• Are their comments on the failed tests written in clear, simple, concise English?

• Are their conclusions correct? How many of the defects did they find? Did they report a defect that isn't there, or isn't a defect? (If they report a defect that I haven't listed, it might be worth thinking about it and bringing it to me -- it's quite possible that I have missed something.)

• Is their summary written in clear, simple, concise English?

• Does their summary show any insight? Does it occur to them that sometimes the problem might be with the requirements rather than with the software? Some requirements might be imprecise, some might be missing, some might just be inappropriate.

5. Script output

• Does the output of their scripts, when related back to their test plan, agree with the list of defects in the programs described below?

• Is the formatting correct?

• Do they find any defects in the tweaked (corrected) test_formatter? If they do, this is something to check carefully. I've tried to remove as many defects as possible from there, but some might remain. On the other hand if they find a defect it could mean that there is a defect in their test data or their understanding of the requirements. Ask me if you're not sure.

Overall

You'll probably find yourself flipping back and forth between the plan, the data files, the script, the output and the report. It's impossible to separate them out. Some of the points above also cross those boundaries.

The criteria I believe to be most important overall for determining a mark on this assignment are:

• Clear written English communication.

• Demonstrated understanding of the requirements and devising tests that really do test them. Some understanding of black-box testing principles helps here, but it's mostly ability to think logically.

• Demonstrated ability to write a slightly more complicated shell script than those covered in the labs.

• Demonstrated ability to read instructions and online documentation. (That's what the tricky choice of diff and date options was about.)

• Traceability between requirements, test cases, data files and report.

If you absolutely insist on breaking the marks down, think of these as being worth roughly 4, 8, 4, 2 and 2 in that order. But try to think of the final mark as an indication of overall quality. If you split the marks and then end up with weak assignments getting high distinctions, then something isn't working.

---

Late Penalty

If an assignment was handed in late, this will be shown on the printout. The late penalty is simple: late assignments will be penalised four marks. Assignments more than a week late will not be accepted and you shouldn't see them.

Do not penalise assignments that are less than five minutes late. There were system problems around the deadline. If they were more than five minutes late, apply the penalty unless they have an extension. Other students struggled to get their assignments in on time and it is not fair to them if we don't stick with the policy.

The printout shows the deadline for each student. Check this, because some students were given extensions for various reasons.

Responsibilities

As a marker for this assignment, you are required to:

• Mark each assignment given to you;

• Make a photocopy of the first page of each marked assignment and return it to Richard;

• Enter the marks you awarded into the FAIS database under the name a1.

Please don't forget to photocopy the front pages.

Do not enter a zero mark for a missing assignment, just leave the field blank.

I expect you to spend about twenty minutes per student to mark this assignment. If it is much more, please come to see me about it.

---

Bugs in the original test_formatter

1. If the indentation level is changed in the middle of a line (that is, if there is already at least one word on the current line), then the left margin doesn't have the right amount of indentation.

2. It sometimes leaves an extra blank line before the first line of a paragraph that starts with a too-long word (one that is longer than the line length).

3. In justify mode it doesn't spread out the extra spaces evenly. Instead they are just put between the first few words. (So justified paragraphs have more spaces near the left margin than near the right margin.)

4. If you outdent (reduce the level of indentation) between paragraphs, the first line of the new paragraph will still have the old indentation.

5. If the indent or width commands are used in such a way that the current line becomes too long, that line is output as it is, too long.

6. In justify mode it crashes if a new paragraph starts with a word that is exactly the line length.

7. In centre mode the requirements say it should add spaces at both ends of the line to centre it between the margins, but it only adds spaces at the left. (This is an obscure one that most of them won't have seen. I didn't either until a student pointed it out.)

8. In justify mode it crashes if indents make the line length less than or equal to zero.

Bugs in the "tweaked" test_formatter

As far as I know, the "tweaked" test_formatter has only the following two defects. Of course there may be other defects in there that I don't know about.

1. The distribution of spaces in justify mode still isn't quite what I'd call "even", but it's a lot better. It may not agree with what the students put, and as this requirement was very vague, that doesn't mean the student is wrong.

2. It now adds spaces at the right-hand end of lines in centre mode. This will probably trip up almost all of them.