[Assignment 1] [Assignment 2] [Assignment 3]

COMP2100
Assignment 1: Testing

Due at 5pm on Friday 26th March 2004

Hints and FAQ | Marking guide

Your task is to test the output formatting subsystem of the software. You will be provided with an executable test program for an early version of this subsystem. This test program reads words and commands from the standard input, one per line, and uses an OUTPUT_FORMATTER object to arrange those words into lines that it then sends to the standard output.

You are to carry out the following steps:

1. Write a test plan.

2. Prepare test data (both input and output) covering the test cases described in your plan.

3. Write a Bash script to automate the process of running the test program on your test input data and comparing the output with your output data.

4. Write a test report describing the outcome of your testing and giving your evaluation of the software you have been testing.

---

1. Write a test plan

Probably your largest task is to write a detailed test plan setting out what test cases you will test and why. Your plan must conform to the template provided. As part of this you must write one or two sentences indicating the objectives and description of each test case.

You will need to read the output formatter requirements carefully. The specific requirements are numbered. The reason for this is a software engineering principle called traceability. For each test case, you must indicate which of the numbered requirements it addresses. (There may be more than one.)

You may find it helpful to read the short form of the OUTPUT_FORMATTER class and the full class text of the test program:

• class OUTPUT_FORMATTER

• class TEST_FORMATTER

We will run your tests on different versions of the output formatter with different defects, so make sure that your test cases are comprehensive. Your task is not just to find the defects in the version of the software given to you. It is to prepare a test suite that can be run on new versions of the software as they appear. Once developers have attempted to fix the defects you find, they need to be able to run your tests again to see whether their corrections worked, and to make sure that they haven't introduced any other defects into the software. This is called regression testing.

---

2. Prepare test data

You second task is to prepare test data covering all the test cases in your plan. For test case nn you need to prepare two files, an input file testnn.in and an output file testnn.out. The output file should contain what you believe to be the correct output from running the test program on the corresponding input file. Try to make each test input file as short as possible while still adequately testing the corresponding test case as described in your plan.

---

3. Write a test script

Write a bash script called run_tests which runs the test program on each of your input files and compares the output with the corresponding output file. Your script must produce output in exactly the following format (although you will certainly need more than three tests):

Formatter test run of Thursday 11 March 2004, 17:10:28 +1100
--------------------------------
Test 01: Failed

*** Expected
--- Found
***************
*** 1,3 ****
! This is a left aligned paragraph. The
! width is set to 36 and there is no
! indentation.
--- 1,3 ----
! This is a left aligned paragraph.
! The width is set to 36 and there is
! no indentation.
--------------------------------
Test 02: Passed
--------------------------------

Passed 1/2 tests = 50%

Note in particular the way the date and time are printed. In order to get full marks for this part, you must work out how to display the current date and time in exactly this format. Similarly, to get full marks you must display the differences between the expected and actual output on failed tests in exactly the format shown.

The executable test program will be located on the student system in the directory /dept/dcs/comp2100/bin/test_formatter. This directory should be on your path, so that you can run the program just by typing test_formatter. Make sure that you do this in your script. If you give the full path in your script, it will confuse the automatic marking system and cost you marks.

---

4. Write a test report

Run your tests on the executable test program supplied, and write a report summarising the results. Your report must conform to the template provided.

In the first section, give the result of each of your tests, simply as `Pass' or `Fail'. In the second section, give a more detailed description (a sentence or two) for each failed test. In the final section, give an overall evaluation of the software you have tested. This section need be no longer than a paragraph or two.

---

Submission

You are required to submit the following files:

1. plan.txt - your test plan

2. run_tests - your test script (in Bash)

3. report.txt - your test report

4. testnn.in - your test input files, one for each test mentioned in your test plan

5. testnn.out - your test output files, one for each test mentioned in your test plan

Submit your assignment from the shell by cd'ing to the directory containing your work and then typing (or cutting and pasting from your browser):

tar cvf a1.tar plan.txt report.txt run_tests test*.in test*.out
gzip a1.tar
/dept/dcs/comp2100/bin/submit_a1 a1.tar.gz

You may be prompted to enter your password.

The command `submit_a1' will run a bash script that checks whether your submission is complete before sending it to the DCS marker system.

---

Marking

Your submission will be marked out of 20 according to a standard marking guide. Some of the marking will be done automatically, by running your test script on different versions of the test program. For this reason it is important that you follow the instructions above exactly. Changing the name of your script or the output format will cost you marks.

We will make every effort to return your assignment to you in your lab class in Week 7.

---

Late submissions

Late submissions will be accepted up to one week after the deadline. They will be penalised four marks (20%).

---

Getting started

Your first task in this assignment is to spend some time reading through the requirements and thinking about how you can test whether the program satisfies them. This is what the assignment is really about; the rest is just technical detail. Before you start thinking about writing test scripts and all that stuff, you need so spend some serious time examining and thinking about the requirements. If you have any questions, ask them early. If there are any ambiguities or inconsistencies, let me know soon.

Once you are ready to start implementing your tests, I suggest that you spend a little while playing with the test program to get a feel for it. Write a couple of simple test data files and run the program on them. Examine the output. Prepare correct output files for them.

Once you have a couple of input and output files, stop and write your test script. Make it general enough that you don't have to modify it when you add new test data. Get it working early, so that you can use it to speed your work later.

Once you have your script working, it's time to start expanding and refining your set of test cases and test data. To get a high mark you will need to test each requirement thoroughly. Try to make your test data as simple as possible, while still doing what it has to. Remember (or look up in your notes from last year) the principles for black-box testing. There is no need to have Test Case 1 testing Requirement 1, Test Case 2 testing Requirement 2 etc. For some requirements you may need more than one test case, for others you may even be able to test more than one requirement with a single input file.

I think it also makes sense to start writing the report fairly early. As soon as you have a test case that the program fails on, write a short description in your final report.

---

Hints

If you have trouble writing the test script, look back at Lecture 4, Lecture 7 and Lab 1.

Look at the Info documentation for the date and diff commands. By choosing the right options for these commands, you should not need to modify their output at all. (One way to access info documentation is from the Emacs `Help' menu, in the `Manuals' submenu. Another is to type info at the command line, or to go straight to the documentation for date, type info date.) For diff, look in particular at the section on output formats.

You must keep your test plan and test data synchronised. You will have a much easier time working out whether you have missed anything if you have good objectives and descriptions for each test case. Before you submit your assignment, go through all your work carefully, checking that the cross-references are correct. If you have been working by trial and error, refining, modifying and adding to your test cases as you go, it's possible that things have gotten out of step. Check also that the references to the numbered requirements are correct.

The specifications below are for the OUTPUT_FORMATTER class, not for the test_formatter program you have been given. You must test the OUTPUT_FORMATTER as well as you can with the test program provided.

---

Output Formatter Requirements

Introduction

These are the specifications for the output formatter subsystem of the XML parser. The formatter's task is to break a stream of words into lines.

When a new output formatter is created, it must be given an output stream to write to. The main interface to the formatter then consists of the commands add_word and blank_line.

The add_word command tells the formatter to add one word to the end of the current line of output. If the word will not fit, the formatter should send the completed line to the output stream and start a new line with the word.

The blank_line command tells the formatter to send the current line to the output stream as it is, and then a blank line. This is effectively ending a paragraph.

The behaviour of the formatter can be changed by setting the width of the page (maximum number of characters on a line), the type of text alignment (left, right, centre or justified) and the indent level (for list items, quotations etc). This is done with the commands set_width, set_left_align, set_right_align, set_centre_align, set_justify, indent and outdent.

Detailed Requirements

Initialisation

When a new formatter object is created:

• the width must be set to 64,(1)

• the alignment must be set to left,(2) and

• the indent level must be set to zero.(3)

Indentation

The formatter must keep track of the current level of indentation. This may be any number greater than or equal to zero. Indentation must be applied at both the left and right margin.(4) Each level of indentation must move both margins four spaces inwards.(5) (So for example if the width is 64 and the indentation level is 2, the first character must be in column 9 and there must be no characters beyond column 56.)

All alignment must be relative to the left and right margins set by the indentation level.

The indent command must increase the level of indentation by 1 (that is, by four spaces).(6) If the indentation level is greater than zero, the outdent command must reduce the level of indentation by 1.(7) If the indentation level is zero, the outdent command must have no effect.(8)

Full lines

In all alignment modes there must be as many words on every line as possible.(9) In other words, the formatter must keep adding words to the current line until it encounters a word that will not fit.

Left alignment

In left alignment mode, every line must start at the left margin(10) and there must be exactly one space character between each pair of words.(11)

Right alignment

In right alignment mode, every line must end at the right margin(12) and there must be exactly one space character between each pair of words.(13) (Extra space characters may be added at the beginning of the line in order to satisfy these requirements.)

Centre alignment

In centre alignment mode, every line must be centred between the margins. That is, the number of spaces between the left margin and the first word must be equal to or one less than the number of spaces between the last word and the right margin.(14) There must be exactly one space character between each pair of words.(15)

Justification

In justification mode, every line (except the last line in a paragraph) must start at the left margin and end at the right margin.(16) In order to do this, the formatter must add extra spaces between the words. These extra spaces must be evenly distributed across the line.(17) The last line of a justified paragraph must be set according to the rules for left alignment.(18) If only one word will fit in a line of justified text, it should be set according to the rules for left-alignment.(19)

Long words

In all alignment modes, if a word is too long to fit between the current left and right margins, it must be set with its first character at the left margin and allowed to stick out beyond the right margin.(20) (In a more sophisticated formatter, such words would certainly be hyphenated.)

Order of operation

Each line must be formatted according to the settings in place when it is completed.(21) In other words, if the settings are changed before a line is complete, the line must be formatted according to the new settings, not the old ones.

[Assignment 1] [Assignment 2] [Assignment 3]