Software Quality Assurance Plan

1.0 Purpose

This document describes the plan by which the CPE 309 project will produce a quality product.

2.0 Referenced Documents

Project Plan

3.0 Management

3.1 Organization

The team will designate an individual "quality assurance leader," named in section 3.3. The course may also have a QA "guru" who coordinates QA activities between teams.

3.2 Tasks

QA tasks during CPE 309 shall include

  • documentation,
  • informal, internal QA reviews
  • software inspections
  • daily build
  • defect reporting and tracking
  • requirements traceability
  • verification (including testing),
  • metrics collection and analysis

These tasks are detailed in this document.

3.3 Responsibilities

Each team member is responsible for the quality of his or her work. Each person should be sure their work conforms to the QA criteria provided on the course web page.

It is the quality assurance leader's responsibility to see to it that the tasks in 3.2 are done, and to ensure that the prescriptions in this document are followed, including scheduling the inspections specified and the collection of metrics data. The QA leader is ultimately responsible for the quality of the deliverables.

The Quality Assurance leader for team TWAIN is Brian Mark.

4.0 Documentation

All the documentation required for this project is described in the project plan (SPMP).

The GNOME style guide is a very good reference for writing tips for user documentation. You might require your analyst to read this document. Of particular use to QA people is the section on quality reviews.

5.0 Standards, practices, conventions and metrics

5.1 Purpose

This section describes the standards, practices, conventions and metrics to be used for the CPE 309 project. These are intended not only to ensure quality for CPE 309 project, but also to obtain quantitative metric data on the SQA process itself. This data is to be used to help establish the key practices of a CMM level four organization.

5.2 Standards

  1. The IEEE standards, with appropriate modifications, are to be used for documentation. In most cases teams should follow the templates Dr. Dalbey has prepared for major documents.
  1. Quality criteria and checklist are to be used for specification and design documents.
  1. UML notation is recommended for Analysis and Design documents.
  2. Detailed Designs must follow the class pseudocode standard.
  3. Java source code must adhere to the class coding standard.
    Download the Checkstyle extension from their website and check your code using the the 308 config file.
  4. E-mail message "subject lines" must follow the guidelines in the project plan.
  5. Directory names in the team account must begin with a capital letter.

5.3 Practices

  1. The document which describes the quality attributes of the finished software must exhibit is the non-functional requirements section of the SRS. The QA manager should work with the Analyst to ensure this document is well written and meets the customer's needs. Also assist the Test Manager in writing tests for these requirements.
  2. From a technical perspective, the success of the software depends on the detailed designs (pseudocode). It's worth putting extra effort into making these correct and very detailed. Internal review is required.
  3. Because delaying quality is expensive, engineers are strongly encouraged to apply quality precepts while they work, rather than as an afterthought. "If you don't have time to do it right in the first place, when are you going to find time to fix it after it breaks?"
  4. QA is an important function in CPE 309 as part of the student course grade depends on the quality of the project deliverables. A cooperative, team-oriented attitude is crucial.
  5. Even though the QA person is ultimately responsible for quality, the QA person is NOT expected to edit and rework every artifact that s/he receives. Each individual must produce quality work. The QA job is to assure that the product has achieved the required level of quality before it is delivered.

5.4 Conventions

  1. HTML (or Wiki) is the preferred format for documents that are to be shared in computer readable form.
  2. Guidelines for all written work are given in the syllabus.
  3. Source code printouts must follow the guidelines given in the syllabus.
  4. The web pages on the team web site must be viewable in Internet Explorer and Firefox.

5.5 Metrics

5.5.1 Individual Metrics

Individuals will track the number of defects found by the inspectors of their code during formal software inspections.

(Recommended) Programmers will plan and track their own unit development work using the Personal Software Process (PSP). Consult the PSP script for details on gathering size, cost, and quality data. During Stage 1, only size and cost measures will be required. During Stage 2, quality measures will also be required.

5.5.2 Team Metrics

The QA person is responsible for compiling and reporting the following metrics, but each person is responsible for gathering their own data.

  • Productivity Metrics (Size and Cost)

Actual Lines of Code. Use the LOC procedure. Categorize by individual and team total. Keep a separate total for the lines of testing code.

Actual Hours of effort (Summary of Individual hours, categorized by project phase, as gathered from timecards. Also team total by phase.) For example:

Name Stage 1 Stage 2 Total
Sue 55.25 47.75 113.5
Tom 63.50 54.25 128.75
Bill 50.25 56.25 115.5
Team Total 179.00 158.25 357.75
  • Defect Metrics

Number of issues (software inspections). Inspections will be conducted using a formal process and any issues found will be recorded.

Number of compile defects. That is, when a defect causes integration failure ("broken build"). This can be during the official daily build, or unofficially by any developer who compiles the code from the team repository. Any time after a module is checked in to the repository, if the system won't compile it counts as a compile defect. These defects are not entered into the defect tracking system, so they must be counted separately. All developers must gather this data.

Number of test defects. Count of defects found during integration and system testing (not unit testing). Just as above, once a module is checked in to the team repository, if it causes a defect, it gets counted as a test defect. Each of these defects is entered into the team defect reporting system, so this measure can be obtained from the reporting system itself.

Defects per KLOC. Compute as: total test defects * 1000 / Total LOC.

Total Defect Repair time. Repair times are logged for each defect in the defect reporting system. The total must be computed by manually summing the all the individual times.

  • Software Design Quality Metrics

It is recommended that each team use some objective means to determine the quality of their design. The QA person may compute the following metrics to assess the quality of design and code. These can be collected using the Together CASE tool available in the lab. The QA person is responsible for understanding the meaning and how to interpret each of these metrics. These metrics should be collected every other week and a briefing given to the team.

Lines of Code, Comment Ratio, Number of Classes, Lack of Cohesion of Methods, Cyclomatic Complexity, Response for Class, Weighted Methods Per Class, Coupling between Objects, Coupling Factor, Fan Out, Depth of Inheritance Hierarchy.

  • Change Metrics

Number of Formal Change Proposals Submitted.

Number of Formal Change Proposals Accepted.

Number of Uncontrolled changes to SRS. Save a copy of the original baselined SRS. Perform an automated "diff" against the final SRS. Count each item appearing in the diff for which there isn't a change request.

Number of Uncontrolled changes to High Level Design (javadocs). Save a copy of the original baselined javadocs. Perform an automated "diff" against the final javadocs. Count each item appearing in the diff for which there isn't a change request.

5.6 Goals

[Note to the student: The quality targets used here are based on historical data obtained from previous student teams on similar projects. Based on previous team data, these targets are aggressive but not unrealistic.]

CPE 309 quality goals for delivered products are as follows.

  • Software Inspections: no more than one severe and three minor issues per inspection.
  • Compile: no more than one broken build for the entire quarter.
  • Integration and System Test: no more than fifty defects per KLOC (thousand lines of code).
  • Acceptance Test: no defects during acceptance testing.

The actual metric data from this project (all the metrics in 5.5.2) are to be reported in the final project submission.

6.0 Demos and Reviews

6.1 Purpose

The purpose of demos and reviews is to provide a means of focusing the attention of engineers on quality of the application as it develops.

  • Internal Reviews are a peer review by the QA leader or another team member of individual work products. Here is a recommended QA submittal form.
  • Software Inspections are peer reviews of source code before unit testing.
  • Class Demos are presentations of completed work products or deliverables in front of the entire class.
  • Customer Demos are a formal presentation for approval of the customer.

6.2 Minimum Requirements

6.2.1 Internal Reviews

All new detailed designs in Stage 1 and 2 will undergo internal review.

6.2.2 Software Inspections

All new source code modules for Stage 1 and 2 require a formal code inspection.

6.2.3 Post Mortem Review

The team will conduct a post-mortem review at the end of CPE 309 to gather the insights and lessons learned during the course.

7.0 Test Plan

A software project test plan describes the objectives, scope, approach, activities, and procedures your team will use for software verification. It explains who is responsible for each kind of testing, how it is to be carried out, and how it will be documented and tracked.

Here is the Test Plan to be followed for CPE 309, based on the IEEE standard.

8.0 Defect reporting, correction, and tracking

Defects in the software found during integration and system test are to be tracked using Trac. When you find a defect, you will create a ticket in Defect Tracking.

Refer to pg 129-130. Defects are corrected using this Defect Repair Procedure. Make sure all developers follow this procedure to the letter!

9.0 Tools, Techniques and Methodologies

SQA techniques include the application of standards, requirements tracing, FTR's and software verification (testing). The SQA tools consist of form templates and checklists.

  1. Form Templates are provided on the course web site.
  2. QA Criteria Checklists for each major deliverable are provided in the deliverables list.

9.1 LOC Counting Procedure

As a measure of product size we will use logical lines of code. Do not include unit test code as part of the product size measure, only code in the product itself. Our procedure requires following the class coding standard and a tool to count lines of code. We do not include comment lines in our line count. Here are two suggested tools for you to consider:

9.2 Requirements Traceability

See McConnell page 151. Each requirement in the specification is traced to each module(s) in design and code that implements it. Table format works fine for this.

9.3 Unit Development Procedure

This procedure describes every step that a developer undertakes in producing a single module (unit). The Unit Development Procedure (aka "PSP script") for CPE 309 is a more elaborate version of Table 14-1 pg 203. (Note that McConnell? calls this an "integration" procedure, but we will call it a Unit Development Procedure).

This script has been painstakingly prepared for you. Read the directions completely, carefully, and be sure you understand every word. There is also a FAQ. If there is anything in these directions about which you have the slightest question, ask for clarification! Completing the PSP forms is recommended, but not required. You are expected to complete the PSP forms accurately and precisely.

Be sure you understand this procedure and follow it religiously. If the procedure doesn't work for you, seek the instructor's permission to modify it. Note this procedure only applies to new code, not to maintenance tasks.

9.3.1 PSP Submission Procedure

The PSP forms for a unit you develop are to be submitted to the QA manager during the first class session after the unit has been submitted to the build. The QA manager gathers all the PSP forms and may publicize which have been completed (but do not publicize individual data). The QA manager submits all the PSP forms to the instructor with the Release. (Completing these forms is an individual choice as described in the previous section).

9.4 Internal Review

Internal reviews are the best way for a team to ensure quality. The QA person must write the procedure for internal review and place it here. At a minimum it requires review of the work product by one other team member and documentation using a form such as this review form. The QA leader may create a scoring mechanism, checklists, or other artifacts that seem helpful for the team. The reviews may be staff from a different team. The QA leader may choose to do all the reviews himself (or herself). Be sure to work with the team manager to allocate time for review meetings.

9.5 Software Code Inspections

All new modules for Stage 2 require a code inspection.

Read these guidelines to Software Inspections for Software Engineering Student Teams.

Here is the required CPE 309 Software Inspection Process.

9.5.1 Software Inspection Report

A summary report of results from the software inspections is to be posted on the team web page under the link "Software Inspection Report"

9.6 Design By Contract

Design by Contract is a module specification technique that intends to improve design reliability and reduce development costs. After completing the homework activities and participating in the class discussion about DBC, each team must decide which specification style they choose to adopt: Design by Contract or Error Checking. For the second stage of development, all modules must be specified in the style the team has chosen. It is the QA manager's job to enforce conformance to this style.

On (decision date) our team decided to adopt the following module specification style (check one):

___ Design by Contract
___ Error Checking

10.0 Integration Procedure

Integration is a crucial technical area that can make or break a project. The implementation manager is responsible for having a detailed integration plan and the QA manager is responsible for enforcing conformance to the integration procedure. Don't let people cheat on the integration procedure! Our integration procedure will include a team "daily build" which helps avoid the "big bang" syndrome.

10.1 Integration Procedure

Our integration procedure is based on an Incremental Integration strategy (see notes). It assumes the design is complete and the interfaces have been compiled. The implementation manager create the Implementation Plan which lists the order in which modules are written and integrated. Developers then write their individual units using the Unit Developer Procedure. No partial implementations allowed. (See "Done is Done" pg 203).

10.2 Build Procedure

There must be a regular ("daily") build of the software product. The build must start at the beginning of the third week of class. The build must occur at least 4 times a week (suggested SMWF).

The implementation manager is responsible for writing the specific technical instructions for how to run the build on your development platform. Include the schedule for when the build will be run and procedures for dealing with broken builds..

Here is a ten-step summary of the steps in the build process. Here is an example build procedure from a previous student project. Here is a REAL build procedure for Mozilla.

10.3 Daily Build Report

The results from running the daily build are to be posted on the team web site. Refer to Ch 14. You may want to create a list of chronological links to a separate page for each build report. A build report contains

  • Date and Time the build was run.
  • Optional Build Number.
  • Status: Compiled / Broken.
  • Which modules were included in the build (Could be CVS log info for most recent version).
  • Computer output from build script.
  • A hyperlink to the build itself for testers and developers to download and run.

Here are example build reports from a previous student project.

11.0 Software Release

Release Criteria

These criteria dictate when the software can be released. Pay attention to these; the software can't be released until these criteria are met (regardless of the planned schedule). Discussed in Chapter 16. Here are the CPE 309 release criteria. (negotiable until end of week 3). Release Checklist This is a list of all tasks that must be completed before the product can be delivered to the customer. Discussed in Pfleeger Chapter 10 and McConnell? Chapter 16. See McConnell?'s template. It is the responsibility of the QA mgr to create the release checklist and to supervise related tasks.

For example, one release issue is how to package the software and test artifacts.

  • Here is a simple packages demo: PkgDemo.zip
  • Here is a more sophisticated example: LOC.ZIP

Software Deployment Plan A Deployment Plan describes how you intend to get the software transformed from source code in your team repository to executing bits on the user's machine. This Implementation Manager is the logical person to create the plan with assistance from the QA manager. Here are issues and considerations in writing your plan. Place a link to the plan either here or on the team home page.

12.0 Records collection, maintenance & retention

The SQA records collected and archived shall include ...

  • Individual Status Reports
  • Individual PSP forms (if collected)
  • Internal Review Forms
  • Software Inspection Reports
  • The Current Defect List
  • Release Checklist for each Release (and associated documentation)

13.0 Training

The only SQA training planned is class lectures. If additional training seems necessary, describe it here.