Keywords

1 Introduction

Specifying the needs and requirements is the starting point for a procurement project, which is especially challenging for complex and innovative procurements [1, 2]. Even though the agile paradigm with emphasis on incremental requirements management has led to substantial savings and risk reduction in software engineering, the procurement process has been “(o)ne of the most difficult areas to renew.” In contrast, requirements must be fixed before the tendering for procurement [1]. Therefore, previous literature on requirements engineering (RE) in procurement calls for dialogues with the vendors to clarify the requirements before tendering [3].

From the procurement management perspective, the literature on RE in public procurement also advocates a pre-tendering dialogue, especially for innovative solutions. Recent studies encourage the public buyers to utilize early-phase market dialogue [4, 5]. Specification of needs and requirements is one of the main challenges in the public procurement of innovation, and suppliers are reluctant to provide innovative solutions with overly rigid requirement specifications [6]. More critically, the procurement entities need to articulate their demands and transform them into requirements, and those requirements must also be matched with supply possibilities within time and budget limits [7]. However, the pre-tender dialogue has attracted very little academic attention [4, 8] and even less for RE during this dialogue. We aim to understand critical factors of the dialogue phase that clarify and focus needs and requirements. Consequently, our research question is: How can we clarify and focus needs and requirements for a new solution in the market dialogue phase?

As our case study, we use the dialogue phase before the call for tender in the Autoscore project at St. Olavs Hospital, one of Norway’s most prominent hospitals. The objective of the Autoscore project is to procure an innovative solution for contactless measurement of vital signs to simplify the activities in the emergency room (ER).

2 Background

2.1 Innovative Partnership and Market Dialogue

In 2016, the EU introduced innovation partnerships to simplify the innovative development of the EU through public procurement. The public authorities in the EU spend around 14% of GDP (approximately €2 trillion per year) on procurement – where software-intensive solutions are a significant part. As stated in the EU directive [9], “public authorities should make the best strategic use of public procurement to spur innovation.” The directive “allows contracting authorities to establish a long-term innovation partnership for the development and subsequent purchase of a new, innovative product, service or works provided that such innovative product or service or innovative works can be delivered to agreed performance levels and costs, without the need for a separate procurement procedure for the purchase.”

Unlike traditional public procurement, which is strictly regulated by national and international regulations with a strong focus on transparency, fairness, and competition, innovation partnerships allow more interaction between the public purchasers and vendors [5]. Negotiation between the public and private parties during the procurement process enables the public buyers to procure complex contracts with innovative solutions tailored to buyers’ specific needs [10].

As exact requirements of the innovative solutions are usually not known by the public buyers, market dialogue is encouraged at the pre-tender stage [11]. This is not unique for the innovation partnership but is recommended for all innovative procurements. The market dialogue is a two-way interaction between suppliers and the public buyers to map needs and improve the requirements specifications prior to a tendering phase, including early market consultation and technical dialogue [8, 12]. This pre-tender market dialogue encourages the purchasers “to write more realistic and ‘inspiring,’ innovation-driven specifications” [5]. Moreover, market dialogue is an excellent way to interact with the suppliers and inform the market about forthcoming needs [12]. By conducting market dialogues with potential suppliers, the procurers can avoid risks, such as by emphasizing price rather than quality, formulating overly rigid specifications, and specifying without sufficient competencies and knowledge of the innovation solution [12].

2.2 Procurement and Requirements Engineering

The search for literature on experiences with innovation partnerships in RE retrieved no relevant articles. This is not surprising because the EU introduced innovation partnerships in 2016, and only a few empirical studies investigate this new procedure. However, we have found relevant articles on RE in procurement and innovation and outline some key points. Messina and Rogers [1] describe two obstacles to innovation in the procurement process in software engineering. The first obstacle concerns public procurement rules: “(b)ureaucracies, rigidly structured organizations, and formal administrative processes do not like innovation. They kill it.” The second obstacle is inadequate commitment: “Leaders and top managers play one of the most relevant roles in introducing innovation by expressing willingness to accept the associated risk and to support and reward innovative ideas and approaches.” Similar obstacles are also discussed in [13] by Moe et al. They address the dilemma that public sectors should follow strict procurement regulations while at the same time specifying complex requirements to procure information systems. It is challenging to clarify requirements before talking to vendors [2].

The following literature mentions some proposals to improve the requirements of procurement. Hiisilä et al. [3] describe an iterative process for improving requirements during the procurement phase of acquiring software for a Finnish pension insurance company. As a result of interviews, they present prioritized lessons learned, where this lesson was one of the most important: “Requirements should be discussed with the supplier and refined during the procurement phase.” They also explained that “requirements should be improved based on the solutions available on the market. New requirements may also emerge after demonstrations or analyses of the bids.”

Moe et al. [14] describe the dilemma when a public entity procures an information system. Procuring an information system requires lengthy dialogues with the vendors to clarify the specification, whereas strict public procurement regulation restricts such dialogues. This article discussed that even though the newly introduced innovation partnership procedure allows more interaction between the public entities and the vendors, this dilemma still exists because the public entities are under strict regulations to limit interaction. Similarly, Moe and Newman show the importance of dialogue meetings with potential vendors to shape the requirement before the tendering phase [15].

3 The Autoscore Case

The Autoscore project started Medio 2020 to simplify the collection of vital signs from patients in the ER at St. Olavs university hospital, as well as other locations where monitoring is essential. Today, monitoring is a time-consuming task for health care workers, connecting sensors and cables to patients, which have little freedom of movement. Health care workers also have a significant cognitive load from working with multiple data sources in a hectic work environment. Consequently, the idea for this innovation partnership was shaped after talks with one vendor with visionary technology for contactless monitoring of vital signs.

The vision for the Autoscore project is to create a solution for contactless measurement of vital signs with no cables and preferably no sensors attached to the patient and where information is well integrated into the health care workers’ information systems. The vital signs are related to the National Early Warning Score (NEWS), including respiration rate, oxygen saturation, blood pressure, pulse rate, level of consciousness and awareness, and temperature. Because no medically approved contactless solutions exist, this innovation partnership project was initiated with financial support from Innovation Norway, a state-owned company that stimulates entrepreneurship in Norway.

As described in Fig. 1, Autoscore has five phases: 1) Mapping of needs, user involvement, and planning, 2) Market dialogue with suppliers and experts, 3) Tender competition, 4) Innovation partnership (where the technology is developed jointly), and 5) Distribution and procurement.

Fig. 1.
figure 1

Case-study timeline.

Full size image

The focus of our study is phase 2 – the market dialogue. The natural endpoint of this phase is when the needs and requirements in the call for tender are specified.

4 Method

4.1 Research Method

We carried out this research in the form of a case study, collecting information and data about and from the Autoscore case in the market dialogue phase of the innovation partnership process. As a research methodology, the case study can be characterized by at least five aspects commonly mentioned in the literature, e.g., Wohlin, 2021 [16]. Our study relates to these five aspects as follows:

  • Being an empirical inquiry or investigation. We collected data about the process, covering the entire timespan, including data prior to the process, e.g., plans and data such as observations throughout the process.

  • Studying a contemporary phenomenon: The case was investigated as it unfolded, e.g., collecting data by observing essential meetings. Collected information was used to identify new observation points, e.g., new meetings. We also collected evolving versions of vital documentation to track developments.

  • Within its real-life context: Data were collected within the same context as the process we studied. All meetings were online video meetings due to Corona restrictions.

  • Using multiple sources: We collected both documentation and observations.

  • The boundary between phenomenon and context is unclear: the focus of our study was to understand the dialogue process, but contextual information was essential to understand the workings of the process, although vague where the boundary was.

4.2 Data Points

Our data points were documentation, observations, and a continuous dialogue with the project manager:

  • Documentation was retrieved from the project’s public website and shared by the project manager.

    • (2020 Q4) “Measurement of Vital Signs in the Emergency Department an Overview of Needs:” a comprehensive slide-set presenting the vision, high-level needs, and relevant context. This document resulted from a needs-mapping process prior to the dialogue phase and may be perceived as initial needs and requirements. This document was openly shared and presented at a dialogue conference in March 2021Footnote 1.

    • (2021 Q1) “Q&As from the Dialogue conference:” an openly shared document collecting all questions from potential vendors at the dialogue conference and the immediate responses from the Autoscore project group. This document ensures insight and transparency for potential vendors that did not participate in the conference.

    • (2021 Q3) The tender documentation: the formal documentation defining requirements that are issued in the request for tenders. A preliminary version was released openly in July for commenting and in the final version in September. We collected different versions throughout the process.

  • Observations of the procurement team’s sessions were made by either one or two researchers (first & second author). Notes were taken, as well as screenshots of relevant presentations. Notes transcribed the progress and dialogue of the meetings, including the researcher’s evaluation of potentially interesting aspects.

    • (2021 Q2) Nineteen one-to-one vendor meetings (five one-hour meetings for three days and four one-hour meetings the final day).

    • (2021 Q2–Q3) Seven project group meetings.

    • (2021 Q2–Q3) Two steering group meetings.

  • Continuous feedback and correspondence between the researchers (first, second, and third author) and the Autoscore project manager (fourth author).

  • The three researchers observed but did not influence the procurement process, except that they advised the project manager to discuss the relevant risks with the steering group after making the decision on the requirements.

4.3 Data Analysis

The observation notes and screenshots from the one-to-one, project group, and steering group meetings were coded in 80 nodes in NVivo. To ensure good coverage, codes were not defined up-front but identified by reading the text in multiple passes. Typically, we coded statements, feedback, or discussions related to themes such as requirements or needs, how the team evaluated vendors and reflections of the dialogue process itself. An example of a single code that was applied to several data points is ‘Critical view on the project’s ambition’ – where vendors express doubt or critique (e.g., that the level of ambition is too high). The collected documents were not coded but used to verify notes and provide context for the analysis. After grouping these nodes, three main themes emerged. Sub-themes and representative statements are explored below:

  • Maturing the vision and the requirements:

    • Trust or doubts both from the vendors and among the procurement team in the overall contactless vision, e.g., one vendor stated, “It will take at least five years until contactless sensors are mature… In the meantime, … use available technology.” One project team member stated, “If we shall cover all the requirements (for six NEWS parameters), we need both contactless and wireless sensors.”

    • Sharpening the requirements, e.g., less than six NEWS parameters.

    • Risks, e.g., the accuracy of contactless measurements, where one vendor stated that they would not be reliable.

    • “Non-functional” requirements, e.g., integration with other hospital IT systems, certification of sensors, and the need for medical competency by vendors.

  • Reflections on negative and positive aspects of the procurement process:

    • Negative evaluation from researchers: sometimes, the project meetings were unstructured because key personnel had to focus on their medical work.

    • Positive evaluation from the project owner to the steering group 28 May 2021: “I am pleased with the project team … Those who participate have ownership of the project … Get the ambassadors involved from the start, and it will be easier to get the system up and running.” Question from the head of the steering group and hospital assistant CEO in the same steering group meeting: “How does the rest of the world compare to this (contactless vision)?… This has great potential.” Answer from project manager “A lot happens on the wireless, but we focus on contactless.” Replay from assistant CEO: “We do not need to implement what others are doing.”

  • Assessments of the vendors, e.g., the hospital has good or bad experiences with a given vendor earlier and the willingness of vendors to share future plans (trust).

5 Findings

Based on the data analysis described in Sect. 4.3, this section details how the needs and requirements are managed in the market dialogue phase of an innovation partnership. Market dialogues with different types of vendors are critical to sharpening the requirement, to identify, reduce, and accept the potential risks, and to build trust between the procurer and vendors. A cross-functional procurement team with a safe and inclusive team atmosphere is also fundamental.

5.1 Focusing Requirements Through Vendor Dialogue Meetings

After the dialogue conference, vendors shared input notes that shaped the one-to-one dialogue meetings. Nineteen vendors requested a one-to-one meeting with the project group. During these meetings, the procurement team and the vendor presented themselves, and the vendor presented content from their input notes or suggested solutions for solving the needs. The procurement team had prepared points for dialogue which was shared with the vendor in advance: (1) Necessary competencies, including existing competencies, the need for external competencies or desire for matchmaking with other vendors, and what competencies the vendor needed from the hospital or the procurement team. (2) Technology, including what needs to be developed and what existing technology or infrastructure was thought to be used (or was thought to be lacking). (3) Modularity with other instruments, sensors, and systems. (4) Experience with the development of medical technology. Furthermore, the procurement team requested information on challenges for developing the solution and the solution itself, and they also wanted the vendor to present their future ambitions.

The 19 one-hour dialogue meetings were an efficient learning arena for the project team. By discussing the vision and getting feedback on the opportunities and limitations of existing and projected technologies, the procurement team focused on the kernel of the innovation. In other words, the requirements were focused on the part of the vision that is realizable within time and budget limits. This meant that the ambitious target of contactless monitoring of all six vital signs was reduced to contactless monitoring of the two most critical vital signs—respiration and pulse. Below, we present a variation in how the vendors approached the dialogue meetings in terms of their degree of alignment with the vision:

Vendors Aligned with the Vision:

These partners reassured the procurement team of the viability of the bold vision on contactless measurements. However, it became clear that asking for six vital signs as a minimum requirement would not return any tenders because current technology could not reliably measure them all. Accordingly, they focused on two vital signs – respiration rate and pulse rate. Moreover, subsequent dialogues with hospital IT platform representatives revealed that it could cover several aspects of integration and presentation. Finally, the requirement for certification as medical equipment is a complex and time-consuming process with a low probability of reaching full certification within the project period. Hence, a clear strategy for certification became the requirement.

This sharpening of requirements was a significant strategic move to maximize the chances of achieving the core part of the vision, which they learned was very ambitious in this 10 MNOK innovation partnership (approximately 1 million €). The procurement team realized that patient identification was an onerous requirement. This becomes critical with the movement of patients and medical personnel, e.g., patients walking between a bed and a chair or when a nurse bends over a patient in a bed.

Vendors with More Conservative Solutions:

These solutions require physical intervention with the patient or restrict their movement. Some of these vendors expressed criticism and argued that the vision was overly ambitious, advising the team to lower the level of ambition to fit their more mature but also more conservative solutions. However, this made it even more apparent to the team that the role of the innovation process was to identify the limits of the technology and not play safe with known solutions requiring tedious cleaning or maintenance tasks.

The interdisciplinary procurement team concluded that the more conservative solutions would not sufficiently increase value compared to today’s solutions and would not lead to a contactless future anytime soon. Accordingly, because some of these vendors allowed themselves to understand “contactless” as “wireless,” the procurement team clearly defined the term “contactless” in the call for tender as not requiring contact with the patient and as monitoring continuously while moving between furniture.

Vendors with Complementary Offerings:

These vendors could complement more innovative vendors so that they together could deliver a more “complete” solution, e.g., by (1) supporting more vital signs with more conservative technology or (2) by offering integration with the hospital IT infrastructure, or (3) with a meaningful presentation of measurements. The initial signal from the project was that either the project or the vendors could propose two or more matching vendors. However, in line with the sharpening of the vision, the procurement team decided on skipping matchmaking altogether.

To summarize, we see that the team made informed decisions in sharpening the vision, the needs, and the requirements as an effect of the dialogues with a large and varied group of vendors, representing an overview of existing technologies, near future innovation opportunities, and limitations of both technologies and project resources.

5.2 Risk Management

The procurement team gradually clarified, understood, accepted, and reduced the project’s potential risk by jointly focusing on the requirements while formulating the final needs in the call for the tender. The original vision of the solution requiring all six vital signs would involve high risks in terms of potential loss of time and money by not being able to realize any solution in time. Moreover, in a hospital context, the primary risk is patient harm. Hence, the risks involved in demanding a radical technology were not perceived as high by the clinicians in the procurement team (i.e., the project owner) or the steering group because the solution will not immediately replace existing monitoring technology. On the contrary, not being able to explore the possibilities of contactless technology was perceived as a risk by the procurement team because the potential added value would then not be identified. The support from Innovation Norway mitigates the financial risk, and it may be possible to apply for more funding later. Lastly, the innovation partnership procurement instrument aims to co-create innovative solutions instead of a more traditional procurement.

5.3 A Cross-Functional Procurement Team

The cross-functional team was created from the beginning of this innovation procurement project because the project owners recognized that involving ambassadors early was essential for successful change processes. We observed the procurement team through all 19 dialogue meetings, including short recap discussions and several internal meetings. The members of this cross-functional team had complementary competencies and resources relevant to the vision of contactless measurements of vital signs:

  • Top-level management: the steering group, led by the hospital assistant CEO, supported the team with the commitment to the sharpened vision and hence supported its risk-taking in focusing on contactless and not wireless technology.

  • Medium-level management: Being also the assistant head of the ER, the project owner gave the team members flexibility with their regular work obligations to attend dialogue meetings. The head nurse also contributed to increased commitment among the nurses by being a regular procurement team member.

  • End-user level: represented by ER nurses and a patient representative.

  • A medical technology and information security expert with deep knowledge of the technological state of the art served as a semantic broker for technology to the rest of the team.

  • A procurement expert who understood the innovation partnership instrument.

  • Project manager with expertise in innovation and knowledge management (and the fourth author of this paper).

The team atmosphere was safe and inclusive, indicating psychological safety. Experts with natural authority invited other roles with less authority, e.g., the senior medical doctor actively seeking the viewpoints from the patient representative. The team environment enabled discussions on conflicting views without creating conflicts, and varying viewpoints were discussed openly in the team. We observed several cases where, e.g., the medical experts expressed enthusiasm about technologies presented in the dialogue meetings, but the medical expert was able to correct and balance the view based on technical experience and expertise. Thus, the members showed high mutual trust by sharing their knowledge and acknowledging each other’s views and knowledge related to different types of requirements. The members had a stronger voice for requirements associated with their competencies, where health care workers were more active in discussions on using the suggested technology and had the last word on focusing such requirements. This was not only with the other members’ blessing but by direct encouragement. Therefore, the team was successfully united in the sharpening of the vision.

The bold vision gave a robust commitment to the team. Key members even stated that they would not have participated in the procurement with a less ambitious objective. The team members were all highly engaged throughout the dialogue meetings and able to find time in busy schedules. However, we observed a shift in the engagement when the writing of the call for tender started, and it became harder to involve the team members and verify parts of the documentation. The project manager was the critical author but highly reliant on input and verification of the content by the rest of the team. Hence the writing process became cumbersome but eventually resulted in a complete call for tender. In several minor meetings, the project manager struggled to engage team members with clinical commitments. However, the procurement team often expressed confidence in the shared vision based on the thorough dialogue and inclusive dialogue meetings and argued that they trust the project manager to realize this in written format. In this process, the project manager needed to know “who knows what” to get the correct input and verification during finalization. The procurement expert was also a fundamental part of this process with knowledge about the procurement documents and process, ensuring compliance with the formal procedure and contents.

5.4 Trust from Vendors

Trust between procurer and vendor is vital as an innovation partnership will be an intense R&D cooperation for at least 18 months if both parties agree to continue the collaboration. During the market dialogue phase, the hospital knows that an innovation partnership requires time from critical resources to educate the vendors and to provide access to facilities and users during the development phase. Moreover, because the contract only ensures a purchase option, the vendor can refuse to sell the developed solution. Correspondingly, vendors may have to invest their resources to mature the result, e.g., from a proof of concept to a ready product. The willingness of both the hospital and vendors to commit time and resources shows that their level of trust surpasses the threshold of perceived risk, and they are willing to engage in the relationship [17]. Meanwhile, sensitive information on limitations and plans for future technology development shared by vendors also indicated a high level of trust. To ensure that the market dialogue is an effective and safe process to interact and exchange information, an essential prerequisite for this trust was that the procurement team and the researchers had signed a non-disclosure agreement before the dialogue meetings and reassured the vendors about this before every dialogue meeting.

6 Discussion

Our overall objective was to understand how we can clarify and focus needs and requirements for a new solution in the market dialogue phase. We will discuss how this was achieved below.

6.1 Clarifying and Focusing Needs and Requirements

It was valuable with broad inclusion of vendors spanning from those that propose relatively mature technologies to those willing to discuss innovation and development opportunities. Conservative vendors were eager to discuss technology limitations with contactless sensors, while innovative vendors discussed their opportunities. Suppliers were willing to share ideas and knowledge, including limitations and development challenges. This is in line with Uyarra et al. [6], who found that overly rigid requirements hinder innovative solutions.

An inclusive dialogue phase took time for valuable human resources. However, it has been a vital learning process enabling the team to refine and focus on the vision and clarify and justify requirements. The initial vision was ambitious but open to create interest, and the final sharpened vision was innovative and realistic with acceptable risk, e.g., requiring two vital signs instead of the initial six, with the remaining four signs optional. Other vital requirements were adjusted. For example, the team learned that the user interface would be developed through the planned improvement of existing infrastructure systems – leaving more room to focus on the core part of the vision. Learning points came from the dialogue with a great variety of vendors; some large with experience in medical equipment certification that smaller vendors do not have, some small vendors with specific technologies, and existing hospital’s IT system vendors. This sharpening of the needs and requirements is required because the innovation partnership instrument requires the initial specifications of a purchasable solution.

6.2 Understanding of the Innovation Partnership Instrument Among Vendors

The understanding of the innovation partnership instrument varied amongst the vendors. Vendors are accustomed to traditional procurement processes where requirements are specified in detail, and the relationship between the customer and vendor is transactional. In an innovation partnership, the relationship is based on collaboration and co-creation during an innovation process but with a higher risk of not fulfilling the vision [5]. We observed that several vendors took a traditional approach of presenting existing products with less focus on their potential role as an innovation partner. This puzzled us as we expected vendors to more actively describe their potential partner role and abilities to collaborate, e.g., experience in co-creation processes and user involvement. The reason may be that this process is new, unknown, and complex and that the dialogue does not follow established patterns from traditional procurement [1].

Some vendors discussed the innovation potential and their role more as a future development partner than a traditional supplier. Although with a few cases, we suspect there is a sweet spot for good vendors: those with a high level of technological know-how but without being locked by existing products. We believe the intention of the innovation partnership instrument should be clarified and that vendors should be challenged to describe their role as development partners explicitly in the dialogue meetings.

6.3 Vendor Matching

Before the dialogue meetings, the initial ambition was to have an open approach to the potential matching of vendors; both vendors and the project team could propose matching to cover the width and complexity of the requirements. Throughout the process, it became clear that the best strategy was to focus on the core of the vision and reduce the need for complementary competencies, covering initial broader requirements like integration, user interface, and certification. Hence, with learning-based justification, they were looking for one vendor with a focused innovation process for contactless measurement of vital signs.

6.4 A Well-Aligned Cross-Functional Procurement Team

We see that a well-aligned cross-functional procurement team has been a critical factor with the following characteristics:

  • Cross-functionality: The cross-functional team represented all stakeholders affected by the innovation process and the envisioned outcome and those with knowledge of innovative procurement processes. This helped to adjust each members’ impression from the meetings. In particular, the medical technology expert could inform the doctors and nurses about the realism and technical limitations of a solution that at first glance looked promising.

  • Resources: The team members had sufficient resources for participation in addition to a hectic schedule. The associate chief of the ER (i.e., the project owner) considered the project and the dialogue process to be of great importance and allowed his medical staff to prioritize the project (at the cost of their regular tasks).

  • Motivation: The team had a considerable task with a potentially significant impact on the ER and its patients. This ambitious task caused motivation and enthusiasm, compared to the traditional procurement of more conservative technology.

  • Formal procurement competency: In the dialogue and team meetings, the team was supported by a procurement expert, ensuring compliance with formal procurement routines. This role was crucial for increasing the team members’ understanding of what requirements were allowed and what were not (and when), which was a repeated topic during project team meetings and dialogue meetings; and when writing the call for tender. This expertise created safety and order for the project manager and the team.

  • Trustful team dynamics: Although the team members had varying formal power at the hospital and the ER, the associate chief of the ER and the project manager deliberately sought the viewpoints from all members during the dialogue meetings, especially in the team-internal evaluation at the end of each dialogue meetings.

  • Commitment from top-level management: The steering group supported the strategic decision to focus on contactless measurement of vital signs.

6.5 Summary of Discussion

In sum, the procurement team has succeeded in using the dialogue phase as intended:

  • Limitations and opportunities of the technology have been explored by seeing available technology and getting insight into some vendors’ plans and strategies, including R&D.

  • The new knowledge of vendors and possible solutions have enabled the team to adjust requirements with confidence (a balance between realism and risk).

  • New insight enabled the team to identify the critical risk factor in this case, the risk of not exploring the core of the vision instead of the risk of not getting a fully functional system. The innovations’ improvement potential to healthcare workers is too promising not to pursue fully.

From this analysis, we see two main components that have affected the process of prioritizing the core requirement of the vision and adjusting additional requirements: first, a well-functioning learning process, and second, a sufficient level of trust in the team, internally, within the hospital, and towards vendors. This is illustrated in Fig. 2.

Fig. 2.
figure 2

The requirements funnel

Full size image

7 Threats to Validity

Being a study of a single case, our findings have explorative and explanatory value [18]. We provide insight into a new phenomenon with no previous research available (the dialogue phase in innovation partnerships) by openly collecting and analyzing nearly all available data within the study period. We indicate potential explanations of influencing factors through a thematic analysis (summarized in Fig. 2). However, the study is subject to a set of threats to its validity [18]:

  • Construct validity (whether we studied the right phenomenon – the dialogue phase): We have sought to ensure construct validity by collecting data through observations of the phenomena itself (dialogue meetings, project group, and steering group meetings), as well as input and output documentation (the vision document, and the resulting request for tender specification). Other potentially influencing sources have been avoided.

  • Internal validity (whether we have understood casual effects correctly): There is obviously a threat to internal validity with only a single case and no reference cases in the literature. We have, however, sought compensation in building support for claims, combining different aspects in the analysis by openly coding the material (using Nvivo), e.g., several factors explaining the team.

  • External validity (whether our findings are valid to other cases): This needs to be investigated in other studies, but we have set a restricted scope for the study – the dialogue phase, which is part of a well-defined process, meaning that our findings should have relevance to other cases that are restricted by the same type of process.

  • Reliability (whether other researchers would reach the same conclusions): This is hard to evaluate, but we have provided a rich insight into findings and how we interpret the findings (discussion). This should enable the reader to assess the reliability of our interpretations.

The project manager of the Autoscore project is the fourth author of this paper, which may create bias and influence validity. This is compensated for by defining that member’s role to only cover correction of facts about the innovation partnership process and not the design and implementation of the study.

8 Conclusions

Our research question was: “How can we clarify and focus needs and requirements for a new solution in the market dialogue phase?” The case study shows that the procurement instrument, an innovation partnership, has worked according to its intention. Dialogues with a broader group of suppliers contributed to learning, addressing both the envisioned innovation and the innovation process. New knowledge of opportunities and limitations has helped the procurement team focus their vision and requirements with confidence. Although a costly process, our analysis shows that building an excellent cross-functional team with a high level of trust is a valuable investment.

We believe insights into the Autoscore case are of value both to practice and research, especially as an innovation partnership is a new instrument with sparse empirical experience. This case study has implications for practice and further research:

  • Implications for practice: (1) A wide range of vendors is valuable for clarifying and focusing initial needs and requirements. Therefore, the initial vision and the initial description of needs and requirements should be so broad that it attracts interest from a variety of vendors. (2) The dialogue phase of the innovation partnership process can benefit from making the vendors’ potential role as an innovation partner more explicit, both in the initial description of needs and requirements and as a discussion point in the dialogue meetings. (3) Reducing the team’s competency and capacity, e.g., to save time for critical stakeholders, pose a significant risk.

  • Implications for research (further work): (1) The dialogue phase in this case study may be streamlined based on more experience, finding a sweet spot between the quality of the dialogue phase and its cost. This is important since market dialogues are recommended for all public procurements [12]. (2) Apart from observing a procurement team and their interactions with vendors, it would be interesting also to observe vendors and their internal prioritizations and evaluations prior to and after the dialogue meetings. This would give a more complete picture.

Looking into the near future for the Autoscore project, we may ask: how should the innovation partnership phase build on the promising results from the dialogue phase? Based on our experience with innovative software engineering processes, we believe that an agile approach would be fruitful. Iterations and increments simplify development and synchronous clarification of functional and commonly under-focused quality requirements, e.g., as addressed by Brataas et al. [19].