See also the complete listing of my publications and the web site for my "Software Quality Engineering" book.
Abstract: In this paper, we examine external failures and internal faults traceable to web software and source contents. We develop related defect and quality measurements based on different perspectives of customers, users, information or service hosts, maintainers, developers, integrators, and managers. These measurements can help web information and service providers with their quality assessment and improvement activities to meet the quality expectations of their customers and users. The different usages of our measurement framework by different stakeholders of web sites and web applications are also outlined and discussed. The data sources include existing web server logs and statistics reports, defect repositories from web application development and maintenance activities, and source files. We applied our approach to four diverse websites: one educational website, one open source software project website, one online catalog showroom for a small company, and one e-Commerce website for a large company. The results demonstrated the viability and effectiveness of our approach.
Abstract: In this chapter, we maximize component diversity as a means to achieve the goal of system dependability. Component diversity is examined from four different perspectives: 1) environmental perspective that emphasizes a component.s strengths and weaknesses under diverse operational environments, 2) target perspective that examines different dependability attributes, such as reliability, safety, security, fault tolerance, and resiliency, for a component, 3) internal perspective that focuses on internal characteristics that can be linked logically or empirically to external dependability attributes, and 4) value-based perspective that focuses on a stakeholder's value assessment of different dependability attributes. Base on this examination, we develop an evaluation framework that quantifies component diversity into a matrix, and use a mathematical optimization technique called data envelopment analysis (DEA) to select the optimal set of components to ensure system dependability. Illustrative examples are included to demonstrate the viability of our approach.
Abstract: Feedback loops for software quality assurance (QA) and improvement analyze measurement data from QA activities and other sources using various models to assess product quality for decision making and project management and to improve product quality via appropriate follow-up actions. We characterize these feedback loops into categories of feedback mechanisms and individualized feedback paths by examining their data input, analyses performed and models used, analysis results and modeling output, and corresponding followup activities. A good understanding of these feedback categories can help software quality professionals find ways to better incorporate feedback into their QA activities and related processes for assuring and improving their software quality.
Abstract: In this paper, we adapt an existing defect classification and analysis framework, orthogonal defect classification (ODC), to analyze web errors and identify problematic areas for focused reliability improvement. Based on information extracted from existing web server logs, web errors are classified according to their response code, file type, referrer type, agent type, and observation time. We also introduce an analysis procedure to identify high-risk/high-leverage sub-classes of problems and consolidate analysis results to recommend appropriate followup actions. Results applying our approach to the www.seas.smu.edu and www.kde.org web sites are included to demonstrate its applicability and effectiveness.
Abstract: Software Reliability Engineering provides a way to quantify the likelihood of software failures in software intensive systems during test and field operations. One simple but practical technique is introduced and discussed with application to fighter aircraft avionics software. A case study illustrates the technique and compares predicted to observed software field failures. During software development, closing on a target software reliability objective provides assurance that the software is achieving an acceptable operational performance mark, one that can be predicted, observed, and measured.
Abstract: In this chapter, we characterize problems for web applications, examine existing testing techniques that are potentially applicable to the web environment, and introduce a strategy for web testing aimed at improving web software reliability by reducing web problems closely identified with web source contents and navigations. Using information about web accesses and related failures extracted from existing web server logs, we build testing models that focus on the high-usage, high-leverage subsets of web pages for effective problem detection and reliability improvement. Related data are also used to evaluate web site operational reliability in providing the requested pages as well as the potential for reliability growth under effective testing. Case studies applying this approach to the web sites www.seas.smu.edu and www.kde.org are included to demonstrate its viability and effectiveness. We also outline extensions to our approach to address testing, defect analysis, and reliability improvement issues for the constantly evolving web as a whole by analyzing the dynamic web contents and other information sources not covered in our current case studies.
Abstract: Identifying the change-prone modules can enable software developers to take focused preventive actions that can reduce maintenance costs and improve quality. A positive correlation between change-proneness and some structural measures that can be obtained from source code, such as size, coupling, cohesion, and inheritance measures, has been observed by some researchers. However, the modules with the highest measurements were not found to be the most troublesome modules by some of our colleagues in the industry. To obtain additional evidence, we identified and compared the high-change and high-measurement modules in two large-scale open-source products, Mozilla and OpenOffice, and we characterized the relationship between them. We extracted a large number of structural measures from Mozilla and OpenOffice source code, and obtained the change count from the source code control repositories of these projects. Contrary to the common intuition, we found that the top-change and top-measurement modules were different. In addition, we observed that the top-change and high-change modules did not have the highest but a fairly high place in the measurement rankings. These results were based on formal hypothesis testing, and they were in accordance with our previous results obtained from six large-scale and closed-source industrial products. Such accumulated findings from both open and closed-source products should provide practitioners with additional guidance in identifying the change-prone modules.
Abstract: In this paper, we characterize usage and problems for web applications, evaluate their reliability, and examine the potential for reliability improvement. Based on the characteristics of web applications and the overall web environment, we classify web problems and focus on the subset of source content problems. Using information about web accesses, we derive various measurements that can characterize web site workload at different levels of granularity and from different perspectives. These workload measurements, together with failure information extracted from recorded errors, are used to evaluate the operational reliability for source contents at a given web site and the potential for reliability improvement. We applied this approach to the web sites www.seas.smu.edu and www.kde.org. The results demonstrated the viability and effectiveness of our approach.
Abstract: We conducted an on-line survey about the defect handling approaches employed in medium and large open source projects. The quantitative results and related findings based on 119 responses from 52 such projects are reported in this article.
Abstract: We analyzed a large set of complexity metrics and defect data collected from six large-scale software products, two from IBM and four from Nortel Networks, to compare and characterize the similarities and differences between the high defect and high complexity modules. We observed that the most complex modules often have an acceptable quality and high defect modules are not typically the most complex ones. This observation was statistically validated through hypothesis testing. Our analyses also indicated that the clusters of modules with the highest defects are usually those whose complexity rankings are slightly below the most complex ones. These results should help us better understand the complexity behavior of high defect modules and guide future software development and research efforts.
Abstract: This paper presents a new approach to software reliability modeling by grouping data into clusters of homogeneous failure intensities. This series of data clusters associated with different time segments can be directly used as a piecewise linear model for reliability assessment and problem identification, which can produce meaningful results early in the testing process. The dual model fits traditional software reliability growth models (SRGMs) to these grouped data to provide long-term reliability assessments and predictions. These models were evaluated in the testing of two large software systems from IBM. Comparing to existing SRGMs fitted to raw data, our models are generally more stable over time and produce more consistent and accurate reliability assessments and predictions.
(An earlier version of this paper also appeared as:)
C. Kallepalli and J. Tian,
"Usage Measurement for Statistical Web Testing and Reliability Analysis",
7th International Software Metrics Symposium,
pp.148-158,
London, England, April, 2001.
Abstract: Statistical testing and reliability analysis can be used effectively to assure quality for web applications. To support this strategy, we extract web usage and failure information from existing web logs. The usage information is used to build models for statistical web testing. The related failure information is used to measure the reliability of web applications and the potential effectiveness of statistical web testing. We applied this approach to analyze some actual web logs. The results demonstrated the viability and effectiveness of our approach.
(An earlier version of this paper also appeared as:)
J. Tian, A. Nguyen, C. Allen, and R. Appan,
"Identifying and Characterizing Problem Prone Modules in
Telecommunication Software Systems",
10th International Conf. on Software Quality,
pp.3-15,
New Orleans, LA, Oct. 2000.
Abstract: This paper compares several risk identification techniques and applies tree-based defect models to analyze project measurement and defect data for several large telecommunication software systems from Nortel Networks. The modeling results are used to identify problem-prone modules and provide constructive information to guide quality improvement activities aimed at these modules. The results are also compared with previous results using similar analysis techniques for both legacy and new software systems to help us generalize our results. Based on these results, we recommend an integrated approach so that problematic areas can be identified and characterized for focused defect removal and quality improvement.
(An earlier version of this paper also appeared as:)
J. Tian,
"Early Measurement and Improvement of Software Quality",
22nd Annual International Software and Applications Conference,
pp.196-201,
Vienna, Austria,
Aug., 1998.
Abstract: Software reliability is one of the most important aspects of software quality. However, most of the existing techniques focus on the system testing and other late stages of development life-cycle. Therefore, they are not effective for early problem identification and reliability improvement. This paper describes our recent work in establishing predictive linkage between software reliability and other entities that we can measure and control early in the development cycle, and using such predictive relations to drive continuous reliability improvement.
(An earlier version of this paper also appeared as:)
S. Yih and J. Tian
"Prescriptive Specification Checking for Hazard Prevention
in Computer controlled Safety-Critical Systems",
D. Gritzalis, editor,
Reliability, Quality and Safety of Software-Intensive Systems,
pp.109-120, Chapman & Hall, 1997.
Abstract: This paper examines the characteristics of computer-controlled safety-critical systems (CCSCS), and analyzes the common causes for hazard in such systems using a formal framework called two frame model (TFM). Based on this analysis, a set of prescriptive specifications is derived to guard the consistency between the computer controller and the physical system to be controlled. A taxonomy and a derivation procedure for such prescriptive specifications are developed to ensure comprehensive coverage of potential hazardous situations for effective safety improvement in CCSCS. Some implementation and automation issues, as well as comparison with existing approaches, are also discussed. The feasibility and effectiveness of this approach is demonstrated by a comprehensive case study.
Abstract: This paper examines two existing approaches to software reliability analysis, time domain reliability growth modeling and input domain reliability analysis, and presents a new approach that combines some of their individual strengths. An analysis method called tree-based modeling is used to build models based on the combined measurement data. This new approach can be used to assess the reliability of software systems, to track reliability change over time, and to identify problematic subparts characterized by certain input states or time periods. The results can also be used to guide various remedial actions aimed at reliability improvement. This approach has been demonstrated to be applicable and effective in the testing of several large commercial software systems developed in the IBM Software Solutions Toronto Laboratory.
Abstract: A formal model of program complexity developed earlier by the authors is used to derive evaluation criteria for program complexity measures in order to determine appropriate measures for a particular application domain. A set of rules for determining feasible measures for a particular application domain are given and an evaluation model for choosing among alternative feasible measures is presented. This model is used to select measures for the empirically guided software development environment of Selby and Porter, and shown to be effective.