2.1. Prototyping in new product development

In literature, prototypes are typically defined at an abstract level; as preliminary representations of a product. For example, as “an approximation of the product along one or more dimensions of interest” (Ulrich & Eppinger Reference Ulrich and Eppinger2016) or the very broad definition of Houde & Hill (Reference Houde and Hill1997) – “any representation of a design idea regardless of medium”. Such high-level definitions allow space for the wide range of prototypes and prototyping activities that exist in design industry, in which, prototyping methods and purposes are often in essence bespoke, in themselves, a task for the designer to select and implement as part of the product design process.

Others extend these artefact-driven definitions to also encompass the importance of the role of prototyping. Lauff, Kotys-Schwartz & Rentschler (Reference Lauff, Kotys-Schwartz and Rentschler2018) state that:

As such they identify that a prototype is more than the sum of its parts, and plays a broad and critical role in the learning and understanding processes of designers and across design teams. It is therefore critical for researchers to understand how prototyping methods influence the learning they afford it generates such that better tools and methods may be developed and selected by designers (Menold et al. Reference Menold, Jablokow and Simpson2017; Lauff et al. Reference Lauff, Menold and Wood2019).

This breadth of definition and criticality outlines a key challenge for industry – the selection and application of prototyping methods has a substantial impact on the success, quality and value of both the design process and design output (Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017; Rao et al. Reference Rao, Kim, Kwon, Agogino and Goucher-Lambert2020); for example, through the cost of methods employed (Christie et al. Reference Christie, Jensen, Buckley, Menefee, Ziegler, Wood and Crawford2012) and suitability of methods for the intended learning (Pei, Campbell & Evans Reference Pei, Campbell and Evans2011). However, prototyping can itself often be ad hoc and lacking direction (Petrakis et al. Reference Petrakis, Wodehouse and Hird2021a, Reference Petrakis, Wodehouse and Hird2021b), with both selection of methods and implementation left solely as a task for the designer indicating an absence of a clear prototyping strategy. This apparent lack of strategy risks inappropriate or sub-optimal selection, leading to extended processes or yielding products that do not reach their full potential. With novice designers, prototypes can be under-utilised during the front end of design (Deininger et al. Reference Deininger, Daly, Sienko, Lee and Kaufmann2019). It is subsequently suggested that prototyping practice could be improved by ensuring the right prototyping method is used at the right time (Goudswaard et al. Reference Goudswaard, Gopsill, Harvey, Snider, Bell and Hicks2021a).

To this end, researchers have developed many classifications of prototyping that have focused on elements including:

• • prototyping methods and physical characteristics of prototypes (Hannah, Michaelraj & Summers Reference Hannah, Michaelraj and Summers2009; Ulrich & Eppinger Reference Ulrich and Eppinger2016);

• • the embodiment of prototypes (Camburn et al. Reference Camburn, Dunlap, Kuhr, Viswanathan, Linsey, Jensen, Crawford, Otto and Wood2013);

• • prototype types (Houde & Hill Reference Houde and Hill1997);

• • prototype functionality (Buchenau & Suri Reference Buchenau and Suri2000);

• • prototype fidelity (McCurdy et al. Reference McCurdy, Connors, Pyrzak, Kanefsky and Vera2006; Pei et al. Reference Pei, Campbell and Evans2011);

• • purpose of prototyping activities (Otto & Wood Reference Otto and Wood2001; Ulrich & Eppinger Reference Ulrich and Eppinger2016; Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017);

• • prototype scope (McCurdy et al. Reference McCurdy, Connors, Pyrzak, Kanefsky and Vera2006) and

• • how prototyping enables progress in design processes (Lauff et al. Reference Lauff, Kotys-Schwartz and Rentschler2018).

Through these perspectives, researchers aim to understand how prototyping is used and may be better used to support the design process, with learnings then passed forwards to designers by way of better prototyping tools, methods and selection approaches.

2.2. Prototyping as iterative knowledge generation

Within a design process, and as highlighted by Lauff et al. (Reference Lauff, Kotys-Schwartz and Rentschler2018), a key role of prototyping lies in supporting the learning of the designer(s) implementing the prototyping activity. This view – of iterative knowledge generation lying central to prototyping activities – aligns closely with how we understand the wider design process itself. For example, Maher & Poon (Reference Maher and Poon1996) and Smulders, Reyman & Dorst (Reference Smulders, Reyman and Dorst2009) describe design as an iterative co-evolution of the problem space and the solution space, where activity within one increases knowledge of the appropriate state of the other with iteration until both align. Similarly, the Function, Behaviour, Structure (FBS) model (Gero & Kannengiesser Reference Gero and Kannengiesser2004; Gero Reference Gero1990) highlights the iteration between a design’s expected behaviour and actual behaviour with the learning generated from their comparison leading to alignment and the final product, while C-K theory (Hatchuel & Weil Reference Hatchuel and Weil2009) describes design as an iteration between an exploratory conceptual space and an explicit knowledge space where conceptual effort iteratively increases knowledge.

Through these models and definitions, it is clear that design, and hence prototyping as an activity within it, can be considered as a knowledge generation process – a view echoed by many (see Ulrich & Eppinger Reference Ulrich and Eppinger2016; Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017; Lauff et al. Reference Lauff, Menold and Wood2019; Goudswaard et al. Reference Goudswaard, Snider, Gopsill, Jones, Harvey and Hicks2021b). Increasing knowledge of the current design representation and comparison against expectation or some pre-defined ideal supports the next stage of iteration, with type and quality of knowledge generated then impacting the success of the design process.

Considering the breadth of prototyping tools and methods required and employed to meet the variant needs of different design projects, it then stands that the prototyping activity employed may directly impact the form of knowledge that is generated through the process. As such it is important to understand how the prototyping method employed may affect the knowledge that is subsequently generated, and whether it forms the appropriate set given the needs of the activity.

This argument underpins this work – direct investigation of the relationships between prototyping activity and knowledge generated, with an eventual view to influence said knowledge for the benefit of the wider design process itself. This is achieved by mapping of prototyping activity to KDs generated across eight comparable design projects, then exploring consistency both between cases, and between methods employed. In so doing, the article sets grounding for better method selection with respect to the knowledge that is most appropriate to the needs of the designer and design process.

2.3. Knowledge dimensions in prototyping

Where the development of a product or system requires knowledge to be generated across a range of dimensions (Ulrich & Eppinger Reference Ulrich and Eppinger2016), prior works have investigated the application of frameworks to observe the knowledge contributions from prototyping against such a range of dimensions. For instance, in the study of architecture, Schon & Wiggins (Reference Schon and Wiggins1992) and Schon (Reference Schon1968) outline a set of knowledge domains to which the designer registers knowledge; describing the process as a reflective conversation with the materials of a design’s solution, with the designer attributing information beyond the medium itself to a range of different domains.

Schon & Wiggins (Reference Schon and Wiggins1992) give the example of a designer working in the medium of drawing, ‘the designer sees what is “there” in some representation of a site, draws in relation to it, and sees what has been drawn, thereby informing further designing’. The designer not only registers visual information, but identifies and gives meaning to patterns beyond the medium itself, registering, and further evaluating this knowledge against the different dimensions required for the overall understanding of a concept. Further, research has shown that the medium used in an activity affects the form of and learning of the activity itself (Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017; Kent et al. Reference Kent, Snider, Gopsill and Hicks2021). This raises the question of the relationships between activity, medium and knowledge generated, and specifically to what degree the medium benefits, constrains, or otherwise influences knowledge generation such that it affects the progress of the design process itself.

In the context of NPD, Real et al. (Reference Real, Snider, Goudswaard and Hicks2021) recontextualise Schon’s domains to classify and characterise the design knowledge afforded by different prototype fabrication and evaluation methods (Real et al. Reference Real, Snider, Goudswaard and Hicks2023). The KDs proposed by Real et al. (Reference Real, Snider, Goudswaard and Hicks2021) delineate design knowledge into a set of 10 dimensions against which knowledge is required for the creation of a new product in NPD (Table 1). These dimensions are not claimed to be an exhaustive set, instead representing those proposed by Schon translated into the NPD domain. Where this work examines their feasibility as a research tool to generate insight, specific KDs of interest should be investigated or developed on a case-by-case basis in future work.

Table 1. Knowledge dimensions that adapted from Schon & Wiggins (Reference Schon and Wiggins1992)

By considering these different KDs, researchers can better understand the complex and multidisciplinary nature of prototyping, as well as the challenges and opportunities involved in developing effective prototypes for different purposes and contexts.

Real et al. (Reference Real, Snider, Goudswaard and Hicks2021, Reference Real, Snider, Goudswaard and Hicks2023) highlight a number of relevant findings in their study of dimensions, predominantly, in the distinctions between the physical and digital fabrication and evaluation methods commonly employed in the product development process. The authors in this work identify strong relationships between the methods selection and the KD(s) that they contribute towards.

The study, based on a participant questionnaire, in which raters review and score a broad and representative selection of prototyping media by their perceived contributions to each of the dimensions, posits the hypothesis that different media or methods will provide varying degrees of learning against different dimensions. Thus, by observing dimensions and understanding the methods, prototyping activities can be better supported with prescribed strategies to streamline knowledge generation in the necessary dimensions.

2.4. Summary and research hypotheses

Prototyping can be understood as a process in which a designer utilises any number of methods to form preliminary representations of their design, from which they generate knowledge about it as a product, iterate and refine until reaching a satisfactory final design.

The variant properties of different media (Lim, Stolterman & Tenenberg Reference Lim, Stolterman and Tenenberg2008; Pei et al. Reference Pei, Campbell and Evans2011; Christie et al. Reference Christie, Jensen, Buckley, Menefee, Ziegler, Wood and Crawford2012; Ulrich & Eppinger Reference Ulrich and Eppinger2016; Kent et al. Reference Kent, Snider, Gopsill and Hicks2021) and previous preliminary work (Goudswaard et al. Reference Goudswaard, Snider, Gopsill, Jones, Harvey and Hicks2021c; Real et al. Reference Real, Snider, Goudswaard and Hicks2021) imply that different media are differently suited to generating knowledge against different dimensions. However, the specific suitability and relationships between such media and knowledge generated is not currently understood. There is, hence, a need to investigate the knowledge generated during prototyping in order to enable improved method selection. Further, there is a need to understand the degree by which knowledge generation may be expected to be consistent or variant dependent on designer, scenario, or common prototyping properties such as fidelity and process stage.

Additionally, there is a need for a methodology that can be used to elicit and analyse the appearance of KDs during prototyping activities in product development. As stated in the article’s contributions, the methodology proposed in this article constitutes a first-of-a-kind approach to enable this.

To indicate the utility of KDs as a method to measure and support design activity, this work probes an initial situated investigation of KDs in a real-world design context by considering the what, when and how of prototype contributions towards each KD. To aid this intention, four hypotheses will be tested, chosen to investigate where across the breadth of prototyping activities KDs may provide insight. These frame the study of knowledge generation through dimensions in three ways: i) that of determining the knowledge profiles generated by designers within a set design context; ii) that of studying relationships between prototype media and characteristics of application and knowledge generated and iii) that of studying relationships between process characteristics and knowledge generated.

This article will investigate the relationships between prototyping media and methods, project stages and KDs through comparison of eight different teams completing a 16-week design project with identical brief. Through this, it will explore similarity in appearance of KDs, relationships between KDs and media and process stages, and potential value both of different prototyping activities, and of KDs as a research tool to understand design. To explore similarities in appearances of KDs, the knowledge profiles of different teams will be examined. A knowledge profile is the prevalence of knowledge generated through the duration of a project categorised against each of the different KDs.

Given a consistent design task with similar designer experience, design process and availability of prototyping methods, it may be expected that the knowledge generated will also show consistency. This would indicate that knowledge generated is not random or individualistic, and instead closely related to the design scenario meaning that knowledge profiles would be dependent upon the design task, team composition and prototyping methods used during a project. This leads to H1, exploring what knowledge is generated for each KD, and consistency in knowledge profiles across teams for the same design brief:

Next, this work explores the relationships between prototyping and generation of knowledge. This intends to investigate whether there is indeed a determinance on knowledge generation derived from the prototyping method used. While many specific properties could be investigated, this work focuses on the fundamental distinction between prototypes in the physical domain (i.e., made of atoms) and the digital domain (i.e., made of bits), and the specific distinction between prototypes of varying representative fidelity (i.e., the degree to which they present a realistic representation of the complete design solution (see McCurdy et al. Reference McCurdy, Connors, Pyrzak, Kanefsky and Vera2006; Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017). H2 and H3 are therefore as follows:

Finally, this work investigates the relationship between knowledge generation and process stage via prototypes. With a general shift in such characteristics as prototyping purpose (Camburn et al. Reference Camburn, Viswanathan, Linsey, Anderson, Jensen, Crawford, Otto and Wood2017; Menold et al. Reference Menold, Jablokow and Simpson2017), fidelity (Ulrich Reference Ulrich2003), type (Houde & Hill Reference Houde and Hill1997) and more as the design process progresses from idea to solution, it may be expected that the form of knowledge generated also evolves. Hence, H4 investigates whether a change in knowledge generation itself may be detected as the design process progresses, stating:

Through these four hypotheses, this work continues to investigate the generation of knowledge within prototyping, and its relationship to prototyping media, characteristics of implementation and process stage.

3.1. Study setup: team and task structure

Design teams comprised of students completing Proyecto Intermedio (Intermediate Project – Uniandes unit code IMEC2700) – a project-based learning course on design prototyping at the Universidad de los Andes (Uniandes), Colombia. The course follows a conceive, design, implement and operate (CDIO) scheme and ran from late January to June 2021. Students were required to complete the unit as a credit-bearing part of their course. The course is the second design project in the Mechanical Engineering syllabus at Uniandes and is expected to prepare students for their graduate theses in the final semesters of the course. Students can be considered to be novice designers with some prior design experience acquired from the previous taught design course.

Group demographics are presented in Table 2. Groups initially had eight or nine students, with a single seven-member team are due to students dropping the unit part way through the course. All students were majoring in Mechanical Engineering and if students were also pursuing a minor degree, this was recorded in the minor column. The average experience across all groups was 5.77 semesters. All groups with the exception of groups three and six have means within one semester of this global average. These outlying teams are due to single participants being in semesters 18 and 19, respectively, which can be due to a change in major, study breaks and/or part time study. Group composition was determined in accordance with Uniandes practice and sought to enable equal groups but without marginalising under-represented groups. For this reason, six groups had two female students, and two had none. Uniandes is based in Bogotá, and due to pandemic conditions during the study, some students were working from their home cities and towns. This ratio of students in versus out of Bogotá is also presented in Table 2. Due to consistency of major across participants, similarities in experience in terms of completed semesters of undergraduate study and that all students met requisite requirements to partake in the course, teams are considered comparable for this study however, any potential impacts of team composition will be considered in Section 5.

Table 2. Team demographics

While differing significantly in experience, the design activities undertaken by novice designers have been observed to generally encompass those of design practitioners (Cash, Hicks & Culley Reference Cash, Hicks and Culley2013). The population selected for this study is therefore considered to be appropriate for testing the hypotheses of the study set out in the previous section.

Ethics procedures were followed in accordance with those from Uniandes. All students gave informed consent to participate in the study. Whilst undertaking of the course was mandatory, participation in the study was optional and students were given the option to opt out of the study and continue as normal with the course. All students on the course chose to take part in the study.

The design task asked the students to design, prototype, and test a system to safely transport vaccines to isolated locations while maintaining cold temperatures and hence vaccine viability. The environment given was that of Colombia, comprising a mix of mountainous, rainforest, and otherwise isolated locations requiring a range of airborne, waterborne, or manually carried transportation.

The task comprised four stage gates: 1) concept design (CD); 2) design selection (DS); 3) system-level design (SD) and 4) detail design (DD). Each stage gate occurred over a 4-week period. At the end of the DS stage, gate teams were expected to generate a low/mid fidelity digital prototype. At the end of SD stage, gate students undertook a performance test of a design representation of their choice. By the end of the project after the DD stage gate, students were expected to design, make and evaluate a physical prototype. To meet the goals, set out for each stage students were expected to prototype their designs according to their own preference of methods. The project structure is shown in Table 3. As part of course formalities, students presented their progress and outputs at three stage gates and in final presentations for feedback, although these reviews did not form part of the analysed dataset.

Table 3. Design project structure

As all groups possessed similar experience and process knowledge, followed the same framework, and performed the same design task, this study does not claim that specific results are fully generalisable and care must be taken when applying these findings to alternative design scenarios or team structures. Rather, it intends to demonstrate and evaluate the capability of the approach to generate research-level knowledge that provides insight into design through study of the relationships between prototyping domains and KDs in a specific, comprehensive case, with sufficient internal repetition to allow within-task comparison between design teams and hence establish consistency.

3.2. Data capture: design logs

As part of the course, students were encouraged to complete design logs charting their progress and documenting prototypes made during the project. Completion of the log was not mandatory and did not have weight on course outcome. Students were encouraged to use them in order to keep track of their prototypes and design process, facilitate both intra-team and external communication and to encourage on-going documentation for stage-gate and/or final presentations. The outputs of these formed the primary dataset for analysis.

The design logs comprised a spreadsheet completed by each team with one row per prototype, asking students to input a range of information describing their prototyping activity, its method, purpose and learning. Information requested in the design log is detailed in Table 4. The information captured in the design log was based upon the information included for different prototype capture tools (Nelson, Berlin & Menold Reference Nelson, Berlin and Menold2019; Erichsen et al. Reference Erichsen, Sjöman, Steinert and Welo2020) as well as additional information required to address the study’s research questions.

Table 4. Information requested in the design log

Format for the design logs was iterated through a previous design project where they were first applied, following which students discussed categories and utility with researchers. This involved ensuring that meaning of categories were well understood by students and that the design logs were able to support and improve students’ design processes rather than making them more difficult.

Prior to completing the design logs, the participants were briefed on the meaning and intention of each data category, with opportunity for questions. At each stage gate, the design logs were reviewed by the course director to ensure data entry was in line with intention. Between stage gates, design logs were observed by a course monitor who had direct insight into team progress and behaviour. This course monitor was able to input on how the ‘as-done’ design work was reflected in the design logs. An example of a design log entry is given in Figure 2.

Figure 2. Extract of a team’s design log.

Prototype method was selected from a list produced and utilised in previous work (Real et al. Reference Real, Snider, Goudswaard and Hicks2021) that was abridged for the purpose of this study (see Table 5). Prototype domain was selected from physical (made of atoms), digital (made of 0s and 1s) and sketch. Sketches, including all pen-/pencil- and paper-based representations, were considered as their own category as physical and digital sketches have been shown to be analogous (Ranscombe & Bissett-Johnson Reference Ranscombe and Bissett-Johnson2017). Prototype fidelity (the level of visual representation of the design; McCurdy et al. Reference McCurdy, Connors, Pyrzak, Kanefsky and Vera2006; Pei et al. Reference Pei, Campbell and Evans2011) was rated on a three-point Likert scale labelled as low, medium and high. All other entries were free text. To ensure understanding of the above terms, students were provided with a video resource made by the research team that defined each category the students needed to assign their prototypes into. Students also had the opportunity to ask questions to the study team in the event that terms were not fully understood.

Table 5. Prototyping methods and their associated domain

Design logs were captured from each of the 8 teams, with between 4 and 22 prototypes listed by each (median 11). Design logs were completed by each team member on creation of a prototype and thus served as a boundary object between team members for communication of design status. Teams were able to amend their prototype entries to align the ‘as-documented’ to ‘as-done’ work from the project making the design log a living document.

3.3. Data coding

Entries were collated for all teams and coded for appearance of KDs by the three raters. These were two lecturers in Design and Manufacture and one PhD candidate with 13, 17 and 11 years’ design experience, respectively. The KDs considered were the same as those presented in Section 2.3 that are consolidated from previous work (Real et al. Reference Real, Snider, Goudswaard and Hicks2021, Reference Real, Snider, Goudswaard and Hicks2023) and shown in Table 1.

Appearance of KDs was not considered to be mutually exclusive, that is, a prototype may contribute to multiple KDs. Each entry was coded for each KD as ‘2’ (substantial contribution to the KD), ‘1’ (some appearance of the KD), or ‘0’ (no appearance of the KD). An example of coded data is shown in Figure 3.

Figure 3. Example of design log coding. Empty cells indicate a 0 – no appearance of this KD.

To ensure validity and consistency in coding, all data were coded according to a three part inter-coder reliability process (see Klenke Reference Klenke2008; Blessing and Chakrabarti 2009; Krippendorff Reference Krippendorff2018). Before rating, each rater was familiarised with the KDs, study process and coding process to build shared understanding. Following coding best practice (Krippendorff Reference Krippendorff2018), during this process, the definitions for each KD were discussed to ensure consistency between raters and alignment between KD definitions and their intention.

• • (1): Each rater was presented with the same log comprising 13 entries, and coded it in isolation. Rating was assessed for Fleiss’ kappa (Fleiss Reference Fleiss1971) and showed moderate agreement ( $ \kappa $  = 0.557, p  $ < $  0.0005). Following, each conflict was discussed by all raters until agreement was reached. Conflicts were defined either as disagreement in presence of a KD (a score of ‘0’ vs ‘1’ or ‘2’) and in scale of contribution (a score of ‘1’ vs ‘2’).

• • (2): Each rater was assigned two-thirds of the total dataset, ensuring that every log was coded by two raters. Each coded their sample in its entirety, in isolation.

• • (3): All double-coded samples were compared in a single sitting attended by all raters. During this meeting, the duplicated samples were compared to identify conflicts and discussed by all raters, with the rater who did not initially code each set acting as adjudicator. In this process, every code was assessed for the entire dataset, thereby reaching three-way consensus between raters on all data points.

This process produced a single agreed set of coded data for all design logs to then be processed in subsequent analysis.

3.4. Analysis

Coded data were compared via means of a range of statistical tests. To compare variance between groups one-way ANOVA was used in all cases as underlying data were found to meet assumptions of normality. One-way ANOVA is a parametric test and as such has greater statistical power than non-parametric alternatives such as the Kruskal–Wallis test. To augment these comparisons, Tukey multiple comparison tests were carried out to identify between which groups statistical differences could be observed. Where this was used, correction for multiple comparison was made by using statistical hypothesis testing. Descriptive statistics, one-way ANOVA, Tukey multiple comparison tests and normality tests were all carried out using Graphpad Prism 9 software.

Chi-squared tests were carried out to determine statistical differences in count data considering the occurrences of KDs. These were carried out using Social Science Statistics (Stangroom Reference Stangroom2022) with P and chi-square values corroborated with Graphpad Prism 9.

Further information about the applications of each statistical tests including how assumptions were met are outlined along with each result.