Performance Measurement in Product Development Research : A Literature Review

This paper reviews product development research with focus on performance indicators. The following are the four primary indicators of product development performance: (1) degree of success, (2) survival, (3) product competitiveness, and (4) development productivity and development lead-time. From 1960 to 1980, when the field of product development research was established, most studies adopted indicator (1). In contrast, after 1990, indicator (4) became mainstream. The trigger for this transition was the comparative study of international product development projects in the automobile industry by Harvard University’s Kim Clark and Takahiro Fujimoto (Clark & Fujimoto, 1991). They proposed the groundbreaking empirical research method of statistically analyzing organizational structure, process, and strategies that influence the three product development performance indicators: development productivity, development lead-time, and total product quality.


Measurement of Product Development Performance
The common indicators for measuring product development performance are as follows: (i) Degree of success: This includes the two-dimensional "success/failure" method and "success levels" as a form of overall evaluation.Major measuring methods are mixes of objective performance indicators, the respondent's subjective judgment (e.g., Rickert scale), and the researcher's subjective judgment.A prevalent statistical method for this indicator is factor analysis.Until the mid-1980s, much of empirical studies of product development adopted this performance indicator and measuring method.Qualitative single-case studies and multiple-case studies are also included in this category.
(ii) Survival: This sets the survival or survival rate of a business or company as the indicator for success.The indicator is an effective means for measuring performance industries that are young and in a nascent stage and those with rapid technological changes and in markets characterized by frequent entries and exits.Studies by Abernathy, Utterback, and Christensen use these performance indicators to clarify the impact of technology on the survival rate and/or the survival numbers of companies (Abernathy, 1978;Abernathy & Utterback, 1978;Christensen, 1997;Christensen, Suarez, & Utterback, 1996;Utterback, 1994).
(iii) Product competitiveness: This indicator comprises performance, cost, and quality of the developed product.As for product performance, following Lancaster's (1966) analytical model, several researchers have performed the evolutionary path analysis and equilibrium analysis with product performance frontiers (Foster, 1986;Sahal, 1981;Shintaku, 2005).This approach is effective for high-tech products that can demonstrate their performance by means of a few objective indicators and show rapid performance improvements.Concerning product costs, the relationship between learning (or experience) curves and dynamic changes in cost has been analyzed.During and after the 1980s, however, business administrative studies re-interpreted learning curves as no longer being functions of the amount of experience gained (e.g., cumulative production volumes), but that technological and organizational decision making are capable of influencing these curves (Hayes & Clark, 1985;Hayes & Wheelwright, 1984;Takahashi, 2013aTakahashi, , 2013b)).Regarding product quality, since the end of the 1980s, multiple indicators including subjective judgment are being used for products that are difficult to evaluate with objective performance indicator1 (Clark & Fujimoto, 1991;Fujimoto, 1989).

Empirical Research of Product Development from 1960s to 1980s
Since empirical research of innovation management began in earnest in the 1960s, in approximately the first 20 years, the analysis of general factors that divided the success and failure of innovation projects flourished in the U.S. and U.K.Many research topics were based on aerospace, high-tech industrial goods, and basic research fields, and some research included international perspectives.In addition, in many cases innovation performance was measured using a qualitative indicator of success/failure.Some research adopted development lead-time or productivity, which directly evaluates the competitiveness of product development activities.
The following background formed the above-mentioned characteristic of this period of innovation studies: First, in the U.S., the post-Cold War situation provided the backdrop for the thriving research and development related to national interests such as military, aviation, and space exploration, and effective management for these fields became an important topic.Also, there seemed to be a ready abundance of research funds for these field.Reflecting on the situation, researchers focused on the success or failure, rather than the cost or productivity of the product development project.
Second, the intensification of international competition occurring in durable consumer goods, electronics, and semiconductors was not yet a pressing problem between the 1960s and 1970s.Therefore, the perspective of "product development capability as a source of international competitiveness" was yet to become an issue of research concern.As a result, investigated firms and industries were mostly limited to domestic ones.(Clark, 1989).In other words, there was a strong tendency to view technology development in the same light as product development.
In any case, most research conducted from the 1960s to the 1980s adopted the degree of success as a performance indicator of innovation and product development.As the specifics of this research are reported elsewhere (Clark & Fujimoto, 1991), they will not be repeated here.The following are some characteristics of the Harvard Study that relate to product development research.However, a difference was seen in the adoption levels among the Japanese companies, especially with regard to the "heavyweight product manager (HWPM)" system.
Second was the international scope of this study.The empirical studies of technology and innovation management, which until then had focused on the U.S. and the U.K., did not have much of an international perspective.Some exceptional research that did were just case studies or dealt with themes other than competitiveness.It is probably not an exaggeration that, with the exception of research based on existing statistics of international comparisons of research and development expenditures, the Harvard Study was the first statistical empirical analysis of product development that focused on international competitiveness.
Third, as a natural consequence of the above approaches, the Harvard Study expended much energy in the measurement and correction of product development performance.Specifically, it spent much time in ensuring the measurement of accurate performance data, such as eliminating measurement errors as much as possible regarding the three attributes of development lead-time, development productivity, and total product quality, as well as performing data corrections in relation to differences in the product or project content.
In the field of technology and innovation management, past research had carefully measured innovation performance to a certain degree; however, only few earnest performance measurements relating to product development had been conducted until then.
Fourth, the priority given to the performance measurements caused the necessity to limit the scope of the study, resulting in limitation in the Harvard Study.For example, to conduct meaningful development performance comparisons, the research needed to be constrained to a single industry and single product of passenger vehicles.Thus, the application of this research to other industries was naturally limited.In addition, individual product development project was chosen as an unit of analysis for an accurate comparative analysis.Although most contemporary companies simultaneously manage multiple projects (Nobeoka & Cusumano, 1997), the issue of such coordination between projects was omitted from the analysis.Moreover, statistical comparative analysis must be static and cross-sectional, and thus, the analysis of the dynamic aspect of product development was limited.
Finally, the empirical research could not use specific theoretical framework and became eclectic as a result of stressing performance measurement.The results of the Harvard Study was shaped through field surveys for each company and questionnaire surveys administered four times.Although it effectively took the appearance of hypothesis testing, it is more accurate to call it the results of trial and error.To this effect, we can say that the Harvard Study, in a positive manner of speaking, possessed a flexible research style that was not trapped within traditional frameworks.In a negative manner of speaking, it had an eclectic style with no clear relationship with any theoretical framework.
Incidentally, within the Harvard Study, it was confirmed that information processing, problem solving, specialization and integration, and organizational contingencies, which are all archaic concepts of a theoretical framework, could be effectively used as empirical analysis tools if suitably modified.There are ebbs and flows in the popularity of theoretical researches, and if the applicability for the empirical analysis is not carefully considered, the research can be labeled as being "behind the times."Refocusing on those traditional theoretical tools may be an unintended contribution of eclectic empirical analysis.
Nonetheless, major contributions of the Harvard Study conducted by Clark and Fujimoto (1991) were the precise measurement of product development performance and the clarification of the causes of performance disparities based on those measurements.This type of empirical research rapidly increased in the beginning of the 1990s, and formed that period's product development research trend.While much of this research has commonalities with the conclusions of Clark and Fujimoto (1991), at the same time, one can see the development toward a direction that overcomes the limitations of Clark and Fujimoto's study: single industry, individual projects, static analysis, for example. 5

(
iv) Development productivity and development lead-time: These measure the performance of development project activities.Until the first half of the 1980s, some research used the goal achievement rate of development lead-time as the previously mentioned degree of success indicator; however, not much research has directly evaluated lead-times and/or man-hours. 2Empirical research using these performance indicators began in earnest only after the onset of the 1990s.On the basis of the above overview, this paper continues to review existing research related to the theme "What exactly is high performance product development?"From the perspective of performance indicators, we divide existing research of product development into two broad periods, with the latter half of the 1980s as the dividing line.Simply put, until the latter half of the 1980s, research using the degree of success as the performance indicator was mainstream, but from around 1990, research that employed development productivity and/or development lead-time as the performance indicator began to flourish.
Third, many western companies at the time had not yet recognized diversification and the changing uncertainty of markets and technology as important management topics.Thus, they were not enthusiastic about analyzing expediting (shortening the development lead-time) and optimization (increasing development productivity) as measures to address these issues.Fourth, since most research focused on high-tech industrial goods and research laboratories, little research regarded product development management as a research topic independent from component technology management and basic research management

3 3.
Contributions and Limitations of "Harvard Study" In the 1990s, empirical research exploring effective product development processes and organizational structure and management entered a new phase.Based on the general formula "product development performance = f (organizational pattern, situation)," we can say that this change mainly occurred due to a development of the measurement for product development 3 See Kuwashima (2012) for more details.performance, which is a dependent variable.Simply put, new research came to fore that analyzed product development performance more explicitly in relation to company competitiveness.One of the triggers for this was the comparative study of international product development projects in the automobile industry by Clark and Fujimoto of Harvard University (hereinafter the "Harvard Study") (Kuwashima, 2012).The survey and data collection in this study began in 1985, and reflected nearly 30 projects from about 20 automobile companies in Japan, the U.S., and Europe.This study was originally due in part to the prediction of Abernathy, who began his competitiveness analysis of the automobile industry in the 1970s, and stated "product development is key to new industrial competition for the future."In addition, based on Clark's expertise in measuring productivity, this research began somewhat removed from the theoretical tradition of conventional technology management 4 and as a part of international competitiveness analysis.

First
, the Harvard Study analyzed the following causal chain focusing on the influence of competitiveness: Organizational capabilities of product development (organizational structure, routine, and process) → product development performance → overall competitiveness.Specifically, the three attributes of development productivity (development man-hours), development lead-time, and total product quality were first defined as indicators of product development performance that contribute to competitiveness.Next the variables of organizational structure, routine, process, and strategy that were likely to cause development performance disparities between companies were extracted, and the statistical correlation between these and development performance was analyzed.From a development performance perspective, the results confirmed that, in the 1980s, the Japanese companies had a relative competitive advantage in the two indicators of development productivity and development lead-time.Regarding the total product quality, however, Japanese companies did not have a competitive advantage.Analysis from the perspective of organizational capabilities showed that Japanese companies have the following common characteristics: participation of parts companies in the development, utilization of manufacturing capabilities for product development, overlapping problem solving in product development and production preparation, and compact development teams.