New Page 1

Quality & Testing

Topics from SEPA 8

Software quality is a complete approach, not just something you do after the code is written. Read quote on page 193.

SW quality = compliant to spec + absence of bugs + on time and budget + benefit to user. (page 196)

Errors are easiest to fix at the beginning of the software development cycle, and progressively more expensive the later they are fixed.

This is the central, most important, idea about creating high-quality software.

Look at chart on page 198 and discuss.

Read quote on page 198.

Example: Poor performance of a data comms software related to poor design -- basically impossible to fix.

Example: Logical error in functional specification (user allowed to do contradictory operations). Can be fixed later, but may involve changing LOTS of code.

Defect amplification

Explain what it is.

Examine pictures on page 204, 205.

Technical reviews

What are they?

Can be used for functional spec, or design, or coding.

How often are they really used?

I find review most useful for technical design. Catching errors here saves A LOT of time later, so reviews are worth the time and effort.

Read and explain guidelines on pages 207, 208.

Statistical measures of software quality... I will not discuss them, but would make a good extra credit or presentation topic.

Some useful quality measures

MTTF -- mean time to failure. Average time between software failures.

MTTR -- mean time to repair. How long does it take to fix the problem and get the system back up.

MTBF -- Sum of the above. Average time from the start of one failure to the start of the next.

Availability -- Percentage of time that system is up and running.

Draw example on board of one system, its up and downs, showing each item above.

Of course, the definition of "failure" is murky. Give some examples.

ISO 9000 approach to software quality. We will not discuss it but this is a good extra credit / presentation topic. I have some material you can borrow on this.

Topics from SEPA 17

White box testing... Explain what it is. Exercises all (or most) of the code paths.

Black box testing. Explain what it is. Tests that a given set of inputs produces the correct outputs.

Basis-path testing (17.4). This is a good EC / presentation topic. Read this, plus find other material and teach the class about it.

Boundary (or edge) testing.

What is it. Testing the extremes of input or operating conditions.

Examples -- Most negative number, biggest number, zero. Really big file, zero length file. Really fast input, really slow input, no input.

Why is boundary testing important? (Most bugs happen at boundary conditions.)

Why do you think most bugs happen at the edges? The logic is trickiest here and programmers are often thinking about the usual cases.

TIP -- Test each routine for four inputs: lowest value, highest value, two values in the middle. If these work, the routine is most likely correct. Further testing is unlikely to be productive.

It sometimes requires creativity to think of (or create) edge conditions. It is worth the time to do this though.

Usability testing

What is it? Not focused on bugs, per se, but testing ease of use. Can be a big deal for consumer products that are UI based.

Could be a good EC or presentation topic. I may be able to arrange a visit to the Lotus usability lab.

Topics from SEPA 18

Validation and Verification

Validation -- Checking that the behavior is what the customer wants.

Verification -- Checking that the behavior is correct.

How to remember this? Validation comes first (customer says what they want) and A comes before E.

Examples: Validate = a report has the right rows and columns of info as specified in the functional spec. Verify = numbers in a spreadsheet add correctly.

There is certainly overlap between these two.

How do we know when we are done testing?...

All known and probable major bugs are gone.

Count of minor bugs is acceptable and decreasing.

Performance and usability are good.

Look at graph on page 483.

What does it mean. (The longer we test, the fewer bugs we find.)

Is this graph reality? (Sometimes.)

Sometimes, real testing is a saw tooth graph. (Draw it.) This kind of debugging can be very frustrating for new engineers and managers. You think everything is going great, at the bottom of the first dip. Then after a couple spikes, you don't know what is happening.

Stubs and drivers.

Driver -- Simple code that creates input for a function you are testing. Give some examples.

Stub -- Simple function that can be called by a higher-level routine you are testing. Give some examples.

Is it always easy to write drivers and stubs? No, they can be hard to create for complex situations.

Usually it is worth the effort to write these, especially for complex systems.

Example -- For programs that take input from a file, a small data file can act as a driver. You can construct a sample input file by hand, then use it as the input case. Can someone think of an example? (A program that reads daily NYSE stock prices and prints the names of companies that have lost 20% value in one day. Create a small, simple input file with the proper structure, rather than testing on the giant, real input file.)

Unit testing. Explain what it is.

Integration testing.

Testing the way two or more routines work after you put them together.

Also applies to testing sets of subsystems that have previously been integrated.

Big bang integration testing -- Integrating everything at once, then starting to test. Usually a bad idea. There are so many errors, you don't know where to start. Some errors hide each other. The program won't even run often.

Incremental integration testing -- A better approach. Integrating a few modules, then testing, then integrating a few more, testing some more, etc.

Top-down integration testing.

Putting the highest-level functions together first, then adding lower-level functions.

Does it use a lot of driver, or stubs. (Stubs.) Why?

Bottom-up integration testing.

Putting the lowest level functions together first, then adding higher levels.

Does it use a lot of driver, or stubs. (Drivers.) Why?

Regression testing.

Re-testing a feature that used to work before a recent software change.

Can a bug re-appear after you have fixed it? (Yes!) How?

Alpha and beta testing.

Alpha -- Complete system testing by the development group or their associated test team.

Beta -- System testing by real users outside of the development team.

Why do you need beta testing, if the program has been checked by a whole staff of smart testing people? (Real users do things differently, that engineers and testers never thought of.)