1.1 Design decisions, design defects and missed requirements
The design of complex systems involves activities such as sizing subsystems and components, selecting configurations, and determining possible design solutions and implementations. Throughout, the process requires decisions (Hazelrigg 1998) that determine what needs to be analyzed, considered and determined. We call these design decisions. Design decisions define requirements, create concepts, form specifications from requirements, shape the final configuration and physical form, and define production and testing (Saad et al.
2013) and lifecycle/operating principles. A design research question of interest is the relative importance of design decisions made early versus late in the design process (Ulrich & Scott 1998; Ullman 2015).
One way to study relative decision importance of is to consider the relative impact of incorrect decisions made. An incorrect decision is observable in terms of the corrective actions required to fix the problem. We define a design defect as a poor design decision made creating an attribute not desired in the design, which must either be fixed (decision undone) or accepted as a permanent non-compliance denigrating the produced design. Design defects come in form at different levels of abstraction, from high level missed requirement (e.g., insufficient flying distance range), to engineering sizing deficiencies (e.g., insufficient battery size) or unspecified design features (e.g., insufficient fillet radius) and inherent requirements that are not explicitly stated.
An engineering design process can be described in a requirements compliance framework; in this view, the process is a sequence of defining technical requirements and in turn developing and engineering concepts to meet the requirements, and then finally verifying and validating the design meets the requirements (VDI 1993; Clausing 1994; Otto & Kristin 2001). In this context, design decisions, when not met, can be tracked for impact based on traditional effort tracking against requirements within program management tools.
Therefore, we define missed requirements as design defects on managed requirements. Missed requirements are the important design defects with such impact that they require program management tracking during development. We propose to examine the importance of design decisions by studying managed requirements in industrial development programs and compare their relative timing and impact of missed requirements. Note that these requirements based approach includes the large early design decisions such as the decision on the concept selected. If the concept selected decision was not corrected and must be re-decided, it was due to an inadequacy in missed information needed early at the point when the concept was selected.
For more complex systems, this includes defining a network of requirements and managing their multiple interactions (Clarkson & Eckert 2010; Eckert, Isaksson & Earl 2012; Ladyman, Lambert & Wiesner 2013). Requirements are statements that describe targeted functionality, e.g., minimum range, minimum payload, etc. (Otto & Kristin 2001). Constraints on the other hand, are the physical limits based upon the chosen design implementation, e.g., maximum propeller tip speed when using propellers, or stress limits of chosen materials (Otto & Kristin 2001). When developing new systems, it can be unclear how these multiple requirements and constraints impact one another. These interactions may not be uncovered until late in development during testing (Clarkson & Eckert 2010), when it is realized the requirements and constraints are inconsistent. The consequence can be iterations to resolve these inconsistencies and longer development time (Keller et al.
2005; Clarkson & Eckert 2010).
The aerospace and defense industry in particular, has a history of late deliveries, cost overruns and missed requirements (Obaid et al.
2005; Mark et al.
2006; Bernard, Eric & Archag 2015). On average, the typical procurement process for a new defensive system takes nine (9) years with an average delay of 21 months (Charette 2008). As such, there is constant schedule pressure to deliver the product (system) on time and within cost. However, this time pressure is not necessarily the root cause of late deliveries; Chong et al. (2011), showed that time pressure can have a positive or a negative impact and is highly dependent on team level dynamics.
On the other hand, complex systems can have many nested interactions and hierarchical relations between requirements (Crawley et al.
2004). These interactions and relations can introduce development time-lagged dependencies amongst requirements, and early concept phase decisions are typically made assuming other requirements can be met. The outcome of these assumptions will not be known until later in the program. This uncertainty can result in missed requirements, leading to in-service performance deficiencies (Keller et al.
2005). Late deliveries, peculiar behaviors in certain operating conditions, and outright failures in performance are some of the explicit symptoms of missed requirements (Bernard et al.
While it is known that complex programs exhibit cost and schedule overruns, the relationship amongst inconsistent requirements and when they are defined is less established. Requirements are often noted to have been inconsistent, but it is less reported whether these are inconsistencies between early-defined system-level requirements, inconsistencies between later-defined component requirements, or inconsistencies between early-defined system requirements and later-defined requirements, as an evolving system. These statements highlight the importance of this work as it provides clarification of how much more important early design decisions are by providing justifications for added investments in early design processes, methods and tools.
1.2 Related research
A well-established 80–20 rule has been noted where product design accounts for 80% of the manufacturing cost, and has been studied extensively (Ulrich & Scott 1998; Otto & Kristin 2001). The eventual cost committal accounts for manufacturing cost including material, labor, sourced components and other production overheads. This also includes the testing and certification costs. It generally does not include life cycle costs such as maintenance and repair. The 80–20 rule is also supported by Ullman (2015) who studied copier machines and argued that as high as 75% manufacturing cost savings is committed during the conceptual design phase, and further that design freedom decreases as the knowledge of the design problem increases. To a lesser extent, industrial standards such as S-3000L (Aerospace Industries Association 2009) use a similar premise as the basis to calculate opportunity for life cycle cost savings, although not explicitly stated.
Specific case studies demonstrating the 80–20 rule are less reported. In a product archaeology research study of coffee makers, Ulrich & Scott (1993, 1998) confirmed the 80–20 ratio as the design to manufacturing cost contributors, and found it could be made lower by adopting design for manufacturing and assembly methods (Geoffrey 1994). Standards on the cost impact of design for manufacture also provide useful guidelines to reduce developmental cycles and discuss the importance of early design decisions (Aerospace Industries Association 2009). In a study of space systems development costs, NASA reports defects not found until operations carry 50 times more cost than defects found in the early concept phase (NASA 2016). In a meta-study of multiple defense systems, the RAND corporation found the cost of changing a requirement escalate over time in a program (Obaid et al.
2005; Mark et al.
2006; Bernard et al.
2015) highlights that early decisions cost more. Software development process research also reports a similar phenomenon as those in electro-mechanical systems. The National Institute of Standards observed that the longer the time of the discovery of a bug from its creation, the greater the cost impact (Tassey 2002). Boehm reports bugs found in operation carry 10 times more cost than defects found early in the coding phase (Boehm & Papaccio 1988; Boehm & Basili 2001). Hamada (1996) reported that with work done at Ricoh, the cost of fixing a design defect was $35 in design compared to $690,000 in field service. All these studies support that design defects discovered late cost more to fix, due to the costs of changing fixed assets such as tooling and repairing multiple units in operation.
Almefelt, Berglund & Nilsson (2006) found use of requirements management tools can significantly reduce requirements’ errors. The likelihood of requirements compliance can be improved by using formalized processes like the VDI 2211 process or V-model for development to manage requirement changes (VDI 1993). The United States Department of Defense (US DoD) implemented use of Engineering Change Requests (ECRs) and configuration management protocols (Department of Defense 2001) to ensure the changes to design after Critical Design Review (CDR) do not affect the intended function of engineered defense systems. Capers & Lipton (1993) reported that the choice of integration and testing processes can greatly impact the performance of a system. This was illustrated in their reported Hubble Space Telescope case study, where failure to adequately test requirements caused expensive repairs.
While such requirements management systems help reduce design errors and help in their mitigation, understanding the root causes of missed requirements and their impact is less understood. De Weck, Olivier & Eckert (2007) studied the impact of various uncertainties on requirements management. Uncertainties can range from being stochastic in nature, such as fabrication or field-use variations, and those that are epistemic in nature, such as the unknown behavior response in the use of immature technologies. They note the lack of knowledge on the contextual background of the design, corporation, usage, market and politico-socio elements can lead to requirements being missed.
Overall, there are multitude interpretations and discussions of the 80–20 rule in the literature. This work seeks to clarify where in the early design stage that 80 percent cost committal is valid, from concept, preliminary, detailed or the verification design phases. Specifically, in terms of monetary resources, time or developmental work activities, we seek to understand the cost impact of decision from the various development process phases.
Table 1. Examples of complex systems design defects
1.3 Examples of design decisions and consequences in the literature
The impact of design decisions has had a number of negative consequences, including performance failures, cost overruns and late delivery problems. As background to the industrial case study work presented here, we first also studied the recent literature for examples of particular failures due to design defects in complex systems. A sample of 10 case studies of design defects publicized in the literature are shown in Table 1. While these examples may have had multiple problems and incurred very high program costs, here we isolated the cost impact of an individual defect and studied that individual design defect’s cost and delay impact alone. We examined the cost and schedule overrun and scrutinized for the published issue and self-stated primary root cause. For example, A380 the wiring design defect was discovered late in aircraft final assembly. This led to issues such as liquidity damage from delivery delays (2 years), recertification after compatibility was established and other production rework cost. This single design defect resulted in 18.5% of the total program cost. Table 1 lists examples, issue and causes, and attributes whether the root cause is before (early design) or after (late design) design freeze. Figures 1 and 2 summarize the impact on cost and schedule overruns of the one design defect alone. These publicized design defects all have the same characteristic of not being detected until late. Further, from Figures 1 and 2, it is also observable that the earlier the design defect was attributed, the higher the cost of rework.
Figure 1. Cost impact of missed requirements.
Figure 2. Schedule impact of missed requirements.
In this sample, all of the requirements changes made occurred late in their programs. Beyond the higher cost impact of changing a requirement late in a program, these results further show the cost of changing a requirement based on when the requirement was defined. From the figures, we can see that there is a clear indication that early/phase missed requirements have a distinctively higher cost than late/phase missed requirements.
The publicized design defect examples reviewed in Table 1 included analysis on their missed requirements. Analyzing these findings, in Table 2 we summarize the list of missed-requirement root causes, as self-reported within the works. The root causes include inadequate trade-off analysis of requirements, inadequate reviews, budget and time pressure, and immature technology or industrial base. Yet, the relative occurrence and impact of these root causes is less reported.
Given this, we next present a study of the root causes of missed requirements in the development process and their impact on development cost through rework activities. This is a longitudinal study over several years at the industrial development at a large aerospace firm responsible for developing and delivering UAVs.
Table 2. Self-identified root causes of missed requirements
An empirical perspective was applied to quantify and study why requirements are missed in the development of complex systems. While the results shown in Table 1 remain anecdotal evidence from publicized cases, it also creates a basis of hypotheses for root causes. We seek to understand if missed requirements occur more often or not in the early phase of design, as separated before and after concept freeze.
Hypothesis 1: Design decisions made earlier in development carry a higher probability of being incorrect than decisions made later in development.
Given that a missed requirement occurs, the cost impact can be different between early and late phase requirement misses. We seek to understand if early phase decisions cost more or less to fix than late phase decisions.
Hypothesis 2: There is a distinctively higher cost to rectify design requirements originating early in development than later.
From the literature, Table 2 lists several possible root causes of missed requirements. A third research question concerns the root cause of missed requirements.
Hypothesis 3: The most frequent root cause of missed requirements in complex systems development is the difficulty in trading off requirements.
The three research hypotheses will be tested by studying the available project documentation in the corporate setting, including the requirements documentation found in design reports and gate reviews documents, their time history, the Engineering Change Request (ECR) logs, the Failure Reporting System (FRACAS), and finally with Kaizen event workshops with the engineers discussing the problems and tracing their causes. The starting point of the analysis is an individual design defect as either an ECR, a FRACAS event, or a changed requirement in the gate reviews. Each event’s relevant requirement was traced through the requirements documentation for the associated root cause inconsistency, and when those requirements were set. The cost impact can be determined through the chain of activities required to rework the design.
Beyond this, interviews were also conducted with the stakeholders to enhance understanding of the origin of the missed requirements and mitigation strategies. Finally, we will provide our recommendations and conclusions for theoretical contributions and design practice.
3 Findings: Occurrence and impact of decisions
3.1 Activities and design defects
In step 1, we listed the design defects from the program data sources (Figure 4). Overall, out of 211 total system-level requirements, a total of 58 system-level design defect missed requirements occurred from the two major product development programs. Note there could be multiple misses over time for a particular requirement; for example, the endurance requirement changed at each gate review and after the final flight test.
Any design defect found is quantified with an engineering target specification for the rework activities to meet, so that the design would be made compliant to the desired specifications. For example, a particular avionics module may have had an over-heating problem when the operating temperature rose to
in test where the required operating limit was 50 degrees Celsius. The ECR proposed relocating the air scoop location to increase airflow to the cooling system such that the avionics module maintained a temperature range between 40 and 50 degrees Celsius. The root cause was traced to inadequate heat load analysis from the modeling and simulation design activity. By tracking these design rework measures, we could understand which subsystem was associated with the design defect.
In total, 264 ECRs, FRACAS, gate review and design reports were reviewed. The breakdown of the 58 design defects by subsystem is shown in Figure 6 and the breakdown by data source is shown in Table 3. Most of the design defects are attributed to the novel structure and avionics, which was expected as most of the design activities are concentrated on these systems. The electrical and mechanical systems were traditional designs available from vendors. The breakout in Figure 6 also indicates most of the design defects were on the in-house activities, consistent with that found elsewhere (Obaid et al.
2005). The engineering novelty was concentrated in these structure and avionics systems developed in-house. 66% of the design defects were from ECR and FRACAS data sources, which is an indication that most design defects were discovered late in the development process.
Figure 6. Breakdown of design defects detected by subsystem.
Figure 7. Breakdown of design defects by activity where detected.
Table 3. Data source breakdown
Next, the breakout by activity is shown in Figure 7. We found that eight of the design defects were detected before concept freeze, seven before design freeze, 19 after design freeze during testing, and 24 arose only once in field service. Of the 19 design defects detected during testing, 13 of them were not identified until final flight testing began. This result indicates that at least 13 of the 211 total requirements, required system-level testing activities to determine consistency. Overall, 6% of the total system requirements could not be determined prior to final flight testing in the development process used at this company.
Next in step 2 (Figure 4), we trace each design defect to its root cause activity. Each design defect was followed regressively back through the design process Gantt chart, and included analysis of the design review documentation and design reports associated with each activity. From this information, we are able to isolate the onset activity for each design defect. This is plotted as the dark bars in Figure 8 for the 58 total design defects.
Figure 8. Breakdown of design defects root cause activity.
To ensure that the design defects are adequately categorized to its root program segment and subsystem, an inter-rater agreement measure was carried out. The design defect root cause analysis was repeated twice using two engineers on the projects, for all 58 design defects. The Cohen’s Kappa was 0.81 for the root cause activity identification (Figure 8) and 0.90 for the root cause subsystem identification (Figure 6). Both results show a good agreement for the categorization of root cause program.
Studying Figures 7 and 8, conclusions can be further drawn. At 23 design defects, the dominant root cause of design defects is attributed to the requirements analysis, which occurs in the early concept phase BCF program segment. This is the point in the program when requirement targets for the design are defined, and requires estimation of subsystem capabilities and their mutual impact. This result follows and confirms others who have noted the difficulty and impact of requirements definition (Almefelt et al.
2006). The BCF segment had more than half of the design defects as root cause (32 total), and the remainder were in the ACF and ADF design program segments (25 and 1 total). No root cause design defects arose in the testing or operation phases, as expected.
Figure 9. Breakdown of design defect root causes by subsystem.
Figure 10. Occurrence rate of root cause of design defects by program segment.
In addition to identifying the root cause activity, the root cause subsystem was also identified. This is shown in Figure 9. Notice the airframe had the most root cause design defects, which was due to its novel construction which was a major objective of the design concept. Conventional electrical and mechanical systems were used which the design team had extensive previous experience.
The count of activities and number of design defects is shown in Table 4, and the occurrence rate graphed in Figure 10. Notice the likelihood of a mistaken decision culminating in a missed requirement is much higher for early phase design decisions. The likelihood a decision made in the early BCF segment was computed as 59%, a very high rate, significantly different from the latter segments. This result holds for this company using their particular development process. Note the 95% confidence intervals were rather large given the small sample sizes (Table 4), but the concept phase versus later phases result remains significant.
Figure 11. High rework activities for defects originating BCF.
Table 4. Design defect occurrence rates
3.2 Impact analysis
Following the next step 3 outlined in Figure 5, we next analyzed for the rework activities needed to fix each design defect. This allows for a comparison of impact. When doing this rework assessment of each design defect, a second point of comparison was computed. For each design defect, not only was the actual rework determined, but we also computed the difference in rework activities needed if the defect had been discovered immediately after it was detected.
These results are shown in Figure 11 for the concept segment BCF design defects (32 total) and in Figure 12 for the program segment ACF defects (25 total). One comparison to consider is the difference in count of rework activities needed between the concept phase and design phase defects. That is the comparison of rework activity counts (graph vertical axis height) between Figures 11 and 12. As can be seen, concept phase defects require more rework. A second comparison to consider is the difference in actual rework required and the theoretical rework that would have been required if the defect were immediately detected. In each graph, this is the difference in white and dark vertical bars for each activity.
Figure 12. Lower rework activities for defects originating ACF.
An observation of Figures 11 and 12 shows that it is costlier in terms of rework activities to resolve design defects originating from the BCF program segment, as more rework activities have to be executed. This is shown in Figure 13, which considers the increased rework needed for concept phase design defects that are discovered later. Also, the rework activity would always be substantially less if it were to be discovered immediately at the point of occurrence. This is all consistent with observations of impact of defects determined late versus early reported elsewhere (Tassey 2002). We here build on those works in our analysis of root cause occurrence.
Figure 13. Average rework multiplier for when a BCF design defect is detected.
The average rate of increase in percentage of rework activities between the BCF and ACF program segments is as illustrated in Figure 13. It is observed that a statistically significant average of 13.0, or 13 times more activities were required for this company to fix a concept phase design defect not detected until after system-level testing and in-service. This statement is now quantified for this data set. A similar analysis shows a multiplier of 15.1 for the ADF program segment, and 2.9 for the ACF program segment.
This compares with earlier from Boehm & Papaccio (1988), Boehm & Basili (2001), who found a multiplier of 50 for the BCF segment, and with NASA who found a multiplier of 29-1000X for spacecraft systems (NASA 2016). Our results differ from previous in that the last column of BCF defects detected very late while in the APGA segment (in-field operation) had a lower multiplier than defects found during testing (Figure 13). This is because in our case, such defects were not actually fixed; instead usually the defect was interrogated and requirements revised and the issue accepted as a non-compliance and operations modified. Therefore, the actual rework activities simply involved issuing the non-compliance and so were less.
Table 5. Contribution of work and potential rework attributed to each program segment
Next, we apply Eq. (2) and compute the contributions of each segment to expected cost. As a percent, this is in the last column of Table 5, as graphed in Figure 14. The results show that, for this company using their development process, 86% of cost was determined by design before design freeze. Within that, 34% had been determined before concept freeze. These results were statistically significant at 95% confidence (Figure 14).
Figure 14. Cost contributions of each program segment.
3.3 Confirming results through interviews
After this analysis, interview sessions were conducted to confirm the results and ensure that the interviewee’s responses were complete, thorough, and meaningful. Six question categories were interactively asked in a recursive 5-why’s approach (Ohno 1988) as shown in Table 6.
Table 6. Question categories to understand design defects
The responses in the interviews were then holistically coded (Saldana 2015), and two (2) coders were used to conduct an inter-rater agreement measure. The resulting Cohen’s Kappa was 0.80, indicating good agreement. Typical responses are shown in Table 7; where the responses provided for better understanding of the root cause activity and subsystem.
Table 7. Typical interview responses
These interview statements were then categorized into eight interview categories and the number of interview responses against these categories was summed across the interviews, as shown in the second column of Table 8.
Table 8. Interview response agreement with design defect analysis
To relate the interview responses to the previous activity quantification, the 58 design defects (Figure 7) were also grouped into eight categories. This is shown in the third column of Table 8. The relative percentage of design defects and interview statements is shown in Table 8, and a Chi squared test indicates the two agree with p-value
. This agreement indicates the earlier quantified analysis is supported by the engineers’ perceptions.
4 Discussion: Impact of early phase decisions
4.1 Reviewing the evidence gathered
We have shown from ECR, FRACAS and gate reviews that there is indeed a distinctively higher number of corrective activities required to rectify missed design requirements originating from before a design concept is frozen. H1 and H2 are therefore supported.
For H3, our data has shown that 25 of the 58 (most frequent) missed requirements are attributed to concept phase requirements analysis. The interviews with engineers also reaffirmed that requirement changes could be better managed with better understanding of requirements trade-off and definition during conceptual design. Hence, H3 is supported.
This is a longitudinal study at one large company, limited to the tools, methods, experience and processes used to develop new products. The high impact root cause of requirements management may not hold true at other firms should they have a means to establish requirements targets early. However, other studies echo our results here (Almefelt et al.
2006). Therefore, if the company were to invest in improved conceptual design stage methods and tools, perhaps the error rate could be reduced. There is an incentive to do so, since 86% of the rework cost committed is determined before design freeze. Nonetheless, this study is typical of an aerospace firm today.
The second limitation is our findings with relation to the 80–20 rule. Based on the evidence, this validation can only hold for new product development in a mid-sized aerospace-level system complexity. This may or may not extend to very large-scale systems and may not extend to small-scale individual projects. Further investigations are needed.
Another limitation is the consideration of the means to mitigate design defects and their effectiveness. In the literature, we find at least two distinct approaches used to address concept phase design defects. One path is to build prototypes and a second is to analyze concepts using modeling and simulation. Both provide improved estimates of expected as-delivered system-level requirement outcomes at different configurations and sizes.
The first approach to mitigate design defects is through the use of physical prototypes and empirical testing of many concepts early in the conceptual design stage (Camburn et al.
2013; Camburn et al. 2014; Dunlap et al.
2014; Camburn et al.
). They provide the designers with information on whether the conceptual design is sound. This applies the design philosophy ‘Fail fast, fail often’ (Ullman 2015) and the concept of a ‘minimum viable product’ (Ries 2011) to provide rapid assurance of system-level requirements. A difficulty with prototyping can be the build time and cost of highly complex technology, e.g., jet engines, high performance aircraft, long range strategic submarines, etc.
A second approach to mitigating design defects also often used is model-based analysis of preliminary concepts (Agte, Shougarian & De 2012). Models can be used to provide rapid assessment and trade-off analysis of different requirements. This applies the philosophies ‘Emphasize the key set of issues’ (Almefelt et al.
2006) and ‘get it right the first time’ (Thomke 1998) to provide rapid assurance of system-level requirements. A difficultly with modeling can be the effort needed to build high fidelity models and also model error (Kenway & Martins 2014; Huang et al.
Finally, for the analysis approach used, another limiting factor is that the data originates from the design defect documentation and the case is built around correction activities on these design defects. This does not address whether some requirements are close to the threshold of failing. What is detectable is the rework activities required as a result of design defects.
Overall, this work showed the impact and occurrence rates of design defects. Our results would suggest that by assessing requirement change impact early, a design team could develop a forward perspective and plan strategies to mitigate or correct their impact. This indicates research is needed to establish formalized systems that can incorporate and manage not just design requirements but also their margins and mitigation strategies early and throughout development. This should be easily complemented with current product development processes to aid in the enhancement of design practitioners. These views are also supported by the work of Almefelt & Andersson (2004), Bergsjö, Almefelt & Malmqvist (2010), Bertoni, Bertoni & Isaksson (2016).
Our work showed that design defects occur equally before and after the conceptual design freeze. However, the design activities required to correct design defects arising before concept freeze are 13 times higher than after. This then translates into design decisions carrying 86% of total work plus potential rework costs. As such, the anecdotal 80–20 rule of design development cost is validated, that the 80 percent cost committal stems from before the design freezes for complex systems. We also found that the majority of the design defects originating from before concept freeze is attributed to requirement analysis. Consequently, we found that difficulty ensuring consistency and trading off requirements was the highest occurring root cause of design defects.