NOTE: Plan not complete. Updates in progress…
Software Quality Assurance Plan
1. Purpose
This document describes the plan by which Luna Software will produce a quality product.
2. Referenced Documents
3. Management
3.1 Organization
Luna Software will designate an idividual "quality assurance leader," named in section 3.3.
3.2 Tasks
QA tasks shall include:
- documentation
- informal, internal QA reviews
- software inspections
- daily build
- defect reporting and tracking
- requirements traceability
- verification
- metrics collection and analysis
3.3 Responsibilities
Each team member is responsible for the quality of his or her work. Each person should be sure their work conforms to the QA criteria provided on the course web page.
It is the quality assurance leader's responsibility to see to it that the tasks in 3.2 are done, and to ensure that the prescriptions in this document are followed, including scheduling the inspections specified and the collection of metrics data. The QA leader is ultimately responsible for the quality of the deliverables.
The Quality Assurance leader for Luna Software is Ian Stewart.
4. Documentation
All the documentation required for this project is described in the project plan (SPMP).
5. Standards, Practices, Conventions and Metrics
5.1 Purpose
This section describes the standards, practices, conventions and metrics to be used for the CPE 309 project. These are intended not only to ensure quality for CPE 309 project, but also to obtain quantitative metric data on the SQA process itself. This data is to be used to help establish the key practices of a CMM level four organization.
5.2 Standards
The IEEE standards, with appropriate modifications, are to be used for documentation. In most cases teams should follow the templates Dr. Dalbey has prepared for major documents.
Quality criteria and checklist are to be used for specification and design documents.
SRS quality criteria
Design quality checklist
Detailed Design quality criteria
UML notation is recommended for Analysis and Design documents.
Detailed Designs must follow the class pseudocode standard.
Java source code must adhere to the class coding standard.
E-mail message "subject lines" must follow the guidelines in the project plan.
5.3 Practices
- The document which describes the quality attributes the finished software must exhibit is the non-functional requirements section of the SRS. The QA manager should work with the Analyst to ensure this document is well written and meets the customer's needs. Also assist the Test Manager in writing tests for these requirements.
- From a technical perspective, the success of the software depends on the detailed designs (pseudocode). It's worth putting extra effort into making these correct and very detailed. Internal review is required.
- Because delaying quality is expensive, engineers are strongly encouraged to apply quality precepts while they work, rather than as an afterthought. "If you don't have time to do it right in the first place, when are you going to find time to fix it after it breaks?"
- QA is an important function in CPE 309 as part of the student course grade depends on the quality of the project deliverables. A cooperative, team-oriented attitude is crucial.
- Even though the QA person is ultimately responsible for quality, the QA person is NOT expected to edit and rework every artifact that s/he receives. Each individual must produce quality work. The QA job is to assure that the product has achieved the required level of quality before it is delivered.
5.4 Conventions
- HTML (or Wiki) is the preferred format for documents that are to be shared in computer readable form.
- Guidelines for all written work are given in the syllabus.
- Source code printouts must follow the guidelines given in the syllabus.
- The web pages on the team web site must be viewable in Internet Explorer and Firefox.
5.5 Metrics
5.5.1 Individual Metrics
Individuals will track the number of defects found by the inspectors of their code during formal software inspections.
5.5.2 Team Metrics
The QA person is responsible for compiling and reporting the following metrics, but each person is responsible for gathering their own data.
- Productivity Metrics (Size and Cost)
Actual Lines of Code. Use the LOC procedure (Section 9.1). Categorize by individual and team total. Keep a separate total for the lines of testing code.
Actual Hours of effort (Summary of Individual hours, categorized by project phase, as gathered from timecards. Also team total by phase.)
- Defect Metrics
Number of issues (software inspections). Inspections will be conducted using a formal process and any issues found will be recorded.
Number of compile defects. That is, when a defect causes integration failure ("broken build"). This can be during the official daily build, or unofficially by any developer who compiles the code from the team repository. Any time after a module is checked in to the repository, if the system won't compile it counts as a compile defect. These defects are not entered into the defect tracking system, so they must be counted separately. All developers must gather this data.
Number of test defects. Count of defects found during integration and system testing (not unit testing). Just as above, once a module is checked in to the team repository, if it causes a defect, it gets counted as a test defect. Each of these defects is entered into the team defect reporting system, so this measure can be obtained from the reporting system itself.
Defects per KLOC. Compute as: total test defects * 1000 / Total LOC.
Total Defect Repair time. Repair times are logged for each defect in the defect reporting system. The total must be computed by manually summing the all the individual times.
- Change Metrics
Number of Formal Change Proposals Submitted.
Number of Formal Change Proposals Accepted.
Number of Uncontrolled changes to SRS. Save a copy of the original baselined SRS. Perform an automated "diff" against the final SRS. Count each item appearing in the diff for which there isn't a change request.
Number of Uncontrolled changes to High Level Design (javadocs). Save a copy of the original baselined javadocs. Perform an automated "diff" against the final javadocs. Count each item appearing in the diff for which there isn't a change request.
5.6 Goals and Reporting
CPE 309 quality goals for delivered products are as follows.
- Software Inspections: no more than one severe and three minor issues per inspection.
- Compile: no more than one broken build for the entire quarter.
- Integration and System Test: no more than fifty defects per KLOC (thousand lines of code).
- Acceptance Test: no defects during acceptance testing.
The actual metric data gathered from this project (all the metrics in 5.5.2) are to be reported at each release.
6. Demos and Reviews
6.1 Purpose
The purpose of demos and reviews is to provide a means of focusing the attention of engineers on quality of the application as it develops.
- Internal Reviews
- a peer review by the QA leader or another team member of individual work products.
- Software Inspections
- peer reviews of source code before unit testing
- Class Demos
- presentations of completed work products or deliverables in front of the entire class
- Customer Demos
- a formal presentation for approval of the customer
6.2 Minimum Requirements
6.2.1 Internal Reviews
All new detailed designs in Stage 1 and 2 will undergo internal review (Section 9.4).
6.2.2 Software Inspections
All new source code modules for Stage 1 and 2 require a formal code inspection (Section 9.5).
6.2.3 Post Mortem Review
The team will conduct a post-mortem review at the end of CPE 309 to gather the insights and lessons learned during the course.
7. Test Plan
A software project test plan describes the objectives, scope, approach, activities, and procedures your team will use for software verification. It explains who is responsible for each kind of testing, how it is to be carried out, and how it will be documented and tracked.
Here is the Test Plan to be followed for CPE 309, based on the IEEE standard.
8. Defect Reporting, Correction and Tracking
Defects in the software found during integration and system test (NOT unit test) are tracked using the tick system provided with Trac. According to the Integration Procedure (10.1) only unit tested code can be submitted to the repository.
Defects are corrected using this Defect Repair Procedure.
9. Tools, Techniques and Methodologies
SQA techniques include the application of standards, requirements tracing, FTR's and software verification (testing). The SQA tools consist of form templates and checklists.
- Form Templates are provided on the course web site.
- QA Criteria Checklists for each major deliverable are provided in the deliverables list.
9.1 LOC Counting Procedure
As a measure of product size we will use logical lines of code. Do not include unit test code as part of the product size measure, only code in the product itself. Our procedure requires following the class coding standard and a tool to count lines of code.
9.2 Requirements Traceability
See McConnell page 151. Each requirement in the specification is traced to each module(s) in design and code that implements it. Table format works fine for this.
9.3 Unit Development Procedure
This procedure describes every step that a developer undertakes in producing a single module (unit). The Unit Development Procedure (aka "PSP script") for CPE 309 is a more elaborate version of Table 14-1 pg 203. (Note that McConnell calls this an "integration" procedure, but we will call it a Unit Development Procedure).
9.4 Internal Review
All products/deliverables will be reviewed by the QA leader. Any item which fails to meet Quality standards is required to be repaired by the original author. Once the owner revises the item he/she is required to submit the item to the QA leader for re approval.
9.5 Software Code Inspections
All new modules for Stage 2 require a code inspection.
Read these guidelines to Software Inspections for Software Engineering Student Teams.
Here is the required CPE 309 Software Inspection Process.
9.5.1 Software Inspection Report
A summary report of results from the software inspections is to be posted on the team web page under the link "Software Inspection Report"
10. Integration Procedure
10.1 Integration Procedure
Our integration procedure is based on an Incremental Integration strategy (see notes). It assumes the design is complete and the interfaces have been compiled. The implementation manager create the Implementation Plan which lists the order in which modules are written and integrated. Developers then write their individual units using the Unit Developer Procedure. No partial implementations allowed.
10.2 Daily Build Procedure
11. Software Release
Release Criteria
These criteria dictate when the software can be released. The software can't be released until these criteria are met (regardless of the planned schedule). Discussed in Chapter 16. Here are the CPE 309 release criteria.
Release Checklist
This is a list of all tasks that must be completed before the product can be delivered to the customer. Discussed in Pfleeger Chapter 10 and McConnell Chapter 16. See McConnell's template. It is the responsibility of the QA mgr to create the release checklist and to supervise related tasks.
Software Deployment Plan
TODO: Software Deployment Plan
12. Records Collection, Maintenance and Retention
The SQA records collected and archived shall include:
- Internal Review Forms
- Software Inspection Reports
- The Current Defect List
- Release Checklist for each Release (and associated documentation)
