[ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs] [Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading] [Help]

COMP2100/2500
Lecture 19: PSP III

Summary

I will explain the updated, improved version of the process, with code review and calculations of things like defect removal rates, productivity, process yield etc.

---

There are several new elements we will add to our PSP and use for the rest of the semester.

From now on, use the Final Project Plan Summary form, the one that doesn't have any parts “greyed out”.

1. Code Review

(See Chapters 13 & 14 of Introduction to the Personal Software Process.)

Do a Code Review after the Code phase and before the Compile phase. (See Lecture 14 for the reasons why.)

• Print out your code on paper.

• Follow the Code Review Script.

• Read every single line of code.

• Fix every defect you find.

• Record every defect in your Defect Log.

• Record the time taken for your Code Review in the PSP Project Plan Summary in exactly the same way that you have been doing for all the other phases. The only point to note is that your To Date times and percentages don't include this phase, so for a while the times in the Plan column will be lower than the actual time you will need for your Code Review. This should stabilise reasonably quickly.

---

2. Personal Code Review Checklist

Follow your Code Review Checklist developed in the old Lab 7. Next to each item put an X if you don't find any defects, and the number of defects found if you do. Keep track of the To Date number and percentage of defects found by each Checklist item from now on.

I will incorporate that part of the old lab instructions here... real soon now...

---

3. Projecting Defects

(See Chapter 15 of Introduction to the Personal Software Process.)

From now on we are going to calculate the defect density in our programs. The units for defect density are Defects per thousand lines of code, abbreviated Defects/KLOC.

• Before you start projecting defect rates you need to calculate the To Date Defect Density for your PSP work so far. On your last Project Plan Summary form, find the To Date LOC figure and the To Date Total Defects. Then the Defects/KLOC is

  1000 * (Total defects) / (Total LOC)

  You will be calculating this number as part of the Postmortem phase for every program from now on.

• In the Plan phase, you will now calculate planned numbers of defects injected and removed in each phase. Starting from the To Date Defect Density and the Planned (estimated) LOC for this program, calculate the Planned Total number of defects (injected and removed) for this program.

• Still in the Plan phase, use the To Date % figures for defect injection and removal in each phase to calculate Planned numbers of defects to inject and remove in each phase. Round the calculated numbers to the nearest integer (since you can't have half a defect...).

• In the Postmortem phase, calculate the Actual and To Date defect densities.

It may seem a bit funny to be planning how many defects you will inject into a program, but that's not really the way to think about it. What you're doing is estimating, based on experience, how many defects you are likely to inject into the program. This allows you to look at the actual number of defects you find and make a judgement as to whether there are likely to be many remaining defects or not. If you know that you are likely to inject 20 defects but your Code Review only finds three, chances are that you need to go back and look again.

---

4. Defect Rates

(See Chapter 16 of Introduction to the Personal Software Process.)

This is part of something called defect “economics”. How many defects do you remove per hour in the different phases of development?

• In the Postmortem phase, calculate defect injection rates for the Design and Code phases, and defect removal rates for the Code Review, Compile and Test phases.

• Although it doesn't say so, the rates are To Date rates (not just for this program, but for all programs so far). For example, the rate of defect removal in the Test phase is given by

  60 * (To Date total defects removed in the Test phase) / (To Date total number of minutes spent in the Test phase)

---

5. Yield

(See Chapter 16 of Introduction to the Personal Software Process.)

The process yield is defined as the percentage of defects found in the Code Review phase. (This only counts defects that were there at the start of the Compile phase, not defects that were introduced during Compile or Test. The Code Review can hardly be expected to remove those.)

• For each development from now on, calculate the Plan, Actual and To Date process yield.

• In each case, the yield is

  100 * (Defects removed before Compile) / (Defects injected before Compile)

At first the Plan figures won't make much sense since you haven't been doing a Code Review until now, but the numbers will settle down quickly enough.

Of course it would be great to have 100% yield, but that's very rare. A realistic long-term goal is to achieve yield values that are consistently over 70%. Even this will take some work. You need to keep reviewing your Code Review Checklist and improving it.

It's also worth working on reducing the number of defects you introduce. Although this will make it harder to achieve a high yield, remember that a high yield value is not an end in itself, it's a means to an end: the real goal is defect-free software, produced on time and under budget.

---

6. Process Quality

(See Chapter 19 of Introduction to the Personal Software Process. The A/FR I'm describing here is the one from Section 19.8, Calculating the True Cost of Quality, not the simplified version described earlier.)

Here we focus not on the quality of the product we produce (the finished program) but on the quality of the process we use to produce it. Ultimately the quality of your software will be determined by the quality of the process you follow.

How can you know how good your process is? One way is to look at the quality of the products it produces. But there are other things you can measure, which address the process directly.

Why bother? Because as you get better at this and your product quality improves, defect removal rates necessarily fall. When there are lots of defects to be found, you will usually find most of them pretty quickly; but when there are only a very few defects to find, all the time you spend on Code Review, Compile and Test will be devoted to finding these few defects. After an initial improvement, the measures above will seem to get worse again.

As you spend longer and longer looking for the last few defects, there has to be a point when you stop, when it's no longer worth the extra time spent. No matter how much you might hate this, you have only limited time to spend on getting your programs right, and out there in the business world time is money. So you need to measure the cost of quality, the amount of time you spend achieving low defect rates.

There are three parts to the cost of quality: prevention, appraisal and failure costs. We will only look at appraisal and failure costs here. Failure cost is the amount of time you spend fixing defects. This is the sum of all the Fix Time entries in your Defect Log. Appraisal cost is the time you spend looking for defects. This is all the time you spend in Code Review, Compile and Test when you weren't fixing defects.

We could measure both of these and record them on the Project Plan Summary, but it turns out that it's not those times (or their percentage of the total development time) that's most important, but their ratio. Research shows that the Appraisal to Failure Ratio, or A/FR is a crucial measure of quality. High A/FR correlates very strongly with low rates of defects found in the Test phase and with low time spent in the Test phase. We should be aiming for an A/FR of 3.0 or greater.

• In the Plan phase, fill in the Plan A/FR by copying the To Date A/FR from the previous program. For the first program with the new process, calculate the Plan A/FR by following the instructions below for the To Date A/FR working off the previous program.

• In the Postmortem phase work out the Actual A/FR as follows. First calculate the failure cost of quality F by adding up all the Fix Times for all defects found in this program. (You'll need to pay attention to getting fairly accurate fix times as you work, or the ratio will fluctuate wildly.) Then calculate the appraisal cost of quality A as the sum of the times spent in Code Review, Compile and Test, minus F. Then the A/F ratio is A/F.

• In the Postmortem phase also work out the To Date A/FR using the same method as for the Actual A/FR, but using the total time spent for all programs so far. If you want to record these values routinely, I recommend that you write down To Date values for the failure time in your notebook somewhere so that you don't have to keep going back over all your defect logs every time.

This section describes an advanced quality measure, so I'm making it optional for you to calculate in your homework exercises. It's probably only useful once you have had some experience with the PSP, code reviews, continuous improvement of your code review check list and process yield. I recommend that you calculate at least the Actual A/FR, because it's not much extra trouble and could be a useful pointer for you.

---

Notes

All the quality measures we're talking about here are concerned with defects: how many there are, how fast and how effectively we find them, and so on. But remember that there's a lot more to quality than this. For now, we're focussing on defects, as an important first step in achieving quality. You can't have quality software that's full of defects, but just having no defects is no guarantee of quality.

So what else is involved? For a start, it's possible to write a defect-free program that does the wrong thing. Making sure that you understand the requirements is the task of analysis, which we have hardly touched, but which you will look at in detail in COMP3110.

Secondly, there's the question of design. As you become experienced with the PSP, you will probably find that you make fewer and fewer coding errors, and that design defects become more significant. So how can you improve your designs? You will study some of that (although predominantly the higher-level aspects) in COMP2110. Some of COMP3110 is concerned with detailed design.

For those of you continuing to fourth year software engineering, quality will become your major focus. In particular you will take a whole course on process improvement.

[ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs] [Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading] [Help]