Int. Journal of Business Science and Applied Management, Volume 9, Issue 2, 2014

Using Delphi technique to build consensus in practice

Lefkothea Giannarou

Hellenic Open University, School of Science and Technology

Sahtouri 11, 26222, Patra

Telephone: + 30 2610 367 566

Email: giannarou@eap.gr

Efthimios Zervas

Hellenic Open University, School of Science and Technology

Sahtouri 11, 26222, Patra, Greece

Telephone: + 30 2610 367 566

Email: zervas@eap.gr

Abstract

This paper focuses on the use of Delphi technique in building consensus in practice. More specifically,

it reviews some fuzzy issues regarding the expert’s panel selection and the questionnaire design, while

it provides two case examples for the consensus measurement. Hence, examining some controversies,

it makes obvious that the purpose of the study and the homogeneity of the sample are crucial factors

when designing the Delphi procedure. However, what still remains unclear is the approach in

measuring consensus, which varies from study to study. In this case, the present paper recommends a

complementary use of three measures to assess consensus, since each one separately could not be

thought of as a good proxy of it. These measures are: (i) the interquartile range, (ii) the standard

deviation and (iii) the 51% percentage of respondents lying in the ‘highly important’ or ‘strongly

agreeing’ category.

Keywords: Delphi method, methodological problems, consensus building, application, research

guidelines

Int. Journal of Business Science and Applied Management / Business-and-Management.org

1 INTRODUCTION

Delphi technique was firstly introduced by Rand Corporation in 1950 and evolved as a

‘consensus’ tool in 1970. It was based on the assumption that ‘group judgments’ are more reliable than

individual’s and has applications on various sectors, such as public health, public transportation,

education etc. (Dalkey, 1969; Kittell-Limerick, 2005: 55). This technique is preferred as a problem

solving or policy making tool when the knowledge about a problem or a phenomenon is incomplete

and is used with the aim of obtaining the most reliable group opinion (Adler & Ziglio, 1996; Kittell-

Limerick, 2005: 53; Kreitner & Kinecki, 1992). Thus, Delphi is used in forecasting tasks when there is

no appropriate or available information and is based on the assumption that “N+1 heads are better than

one” (Hill, 1982; Nerantzidis, 2012; Rowe & Wright, 2001).

Delphi has been criticized for ‘apparent consensus’ (Rowe & Wright, 1999: 363). However, it is

supported that consensus is not forced but elicited (Shields, Silcock, Donegan, & Bell, 1987), with the

results being conducted and recorded through a focused conversation, without the disadvantages of the

interpersonal conflict (Agwe & Sharif, 2007; Dalkey & Helmer, 1963; Landeta, 2006).

Even if this method measures the consensus, there is no common practice regarding the statistical

analysis of the results, with this approach varying from study to study (Landeta, 2006). In addition,

critics about its lack of accuracy and reliability check are also existent (see Landeta, 2006).

Undoubtedly, the aim of this paper is to provide practical assistance to management or business

researchers in designing and applying the Delphi technique. For this reason, the purpose of the case

examples presented is to clarify the way of reaching consensus among experts. Hereafter, the main

characteristics of Delphi technique along with the questionnaire design, the expert’s panel selection and

the consensus measurement are described, whilst 32 prior empirical studies in the field of management

and business are presented in order to record a trend on these issues. Finally, two case examples are

provided for an in depth understanding.

2 DELPHI PROCEDURE

2.1 Background

Delphi technique is considerably desirable to reach consensus on a field where a lack of

agreement or incomplete knowledge is evident. Its application is primarily based on anonymity, giving

the opportunity to participants to express their opinions freely, eliminating any possible personal

conflict (Christie & Barela, 2005; Dalkey, 1969; Linstone & Turoff, 1975; Skulmoski, Hartman, &

Krahn, 2007). Respectively, it is characterized for (i) iteration, which allows participants to reconsider

and refine their opinion, (ii) controlled feedback, which provides them with information about the

group’s perspectives in order to clarify or change their views and (iii) statistical response, to represent

the group’s views quantitatively (Dalkey, 1969; Landeta, 2006; Rowe & Wright, 1999; Shields et al.,

1987; Skulmoski et al., 2007).

However, two of the most fundamental issues in Delphi application are related with the

questionnaire design and the expert’s panel selection. The former is referred to the Likert scale choice

and the number of rounds, while the latter to the panel size, its main characteristics and the response

rate.

2.2 Questionnaire design

Of the first priorities when conducting such a research, is to decide upon the questionnaire

structure and the appropriate rounds. On the one hand, the Likert scale choice depends on the study’s

purpose. This means that when the researcher wants to identify between three situations, a 3-Likert

scale is used, while when he/she attempts to assess the degree of agreement, he/she usually chooses a

10-point one (Christie & Barela, 2005). On the other hand, the Delphi rounds are not an easy task as

they are usually related with the group size. This means that, although Delphi is a repeated process of

‘feedback’ until consensus is reached, in most cases – when the sample is small – no more than one

round may be needed (Mullen, 2003). However, a minimum of two rounds is required in order to allow

feedback and ‘revision of responses’ (Butterworth & Bishop, 1995; Christie & Barela, 2005; Gallagher,

Branshaw, & Nattress, 1996; Mullen, 2003). Respectively, there are also cases where three rounds are

usually recommended (for large samples, >30) (Christie & Barela, 2005; Dalkey, Rourke, Lewis, &

Snyder, 1972; Helmer, 1967; Linstone & Turoff, 1975). Nevertheless, the scope of the study, for

example when the goal is to understand the ‘nuances’, and the sample homogeneity may accept a

smaller number; i.e. less than 3 rounds (Skulmoski et al., 2007). Undoubtedly, it is up to the researcher

Lefkothea Giannarou and Efthimios Zervas

to choose his/her study rounds, while, according to Landeta (2006: 479), he/she may prefer to sacrifice

rounds in order to “guarantee panel participation and continuity”.

2.3 Experts’ panel

When constructing the experts’ panel it is important to consider that their experience (‘expertise’)

or knowledge (‘knowledgeability’) determines the reliability and validity of the results (Adler & Ziglio,

1996; Kittell-Limerick, 2005: 53; Rowe & Wright, 1999). Hence, the experts should satisfy four

requirements: (i) to acquire knowledge and experience through investigation, (ii) to be willing to

participate, (iii) to have sufficient time (to participate) and (iv) to possess effective communication

skills (Adler & Ziglio, 1996; Skulmoski et al., 2007). In any case, ‘knowledgeable persons’ could be

identified either through literature search or recommendations from institutions and other experts,

demanding techniques of purposive and snowball sampling (for more information, see Bryman & Bell,

2011: 192-193; Saunders, Lewis, & Thorhill, 2009: 237-240).

In addition, two more important factors, when conducting Delphi technique, are the panel size and

the response rate. In both cases, there are not strict rules. It is referred that the group size is highly

related to the purpose of the investigation (Cantrill, Sibbald, & Buetow, 1996; Mullen, 2003) and the

response rate may be ranging between the different disciplines, according to the participants’ research

interest (Mason & Alamdari, 2007). However, it is evident that the group error reduces and the decision

quality is reinforced as the sample increases (Skulmoski et al., 2007); Although the sample ranges from

7 to 30 (Armstrong, 1985; Cavalli-Sforza & Ortolano, 1984; Dalkey, 2003; Mullen, 2003; Phillips,

2000; Turoff, 1970), the ‘drop-out’ rate is higher in large groups (Reid, 1988). In any case, it is

believed that a sample size of 20 tending to retain the members (Mullen, 2003). Undoubtedly, what

determines the panel’s size selection is the homogeneity, since in this case a sample of between 10 to

15 people can yield sufficient results (Skulmoski et al., 2007) and assure validity (Listone & Turoff,

1975).

3 MEASURING CONSENSUS

Although the principal aim of Delphi technique is to reach consensus among the participants, still

a common practice to measure it does not exist. Hence, there are studies that measure agreement

through frequency distributions and others using the standard deviation or the interquartile range. In the

first case, the percentage of responding to any given category is defined, which according to McKenna

(1989) is determined to 51%, while there are cases where a specified distance from the mean is also

considered. For example, Christie & Barela (2005: 112) propose that at least 75% of participants’

responses should “fall between two points above and below the mean on a 10-point scale”. As for the

studies using standard deviation or interquartile range to assess consensus, the former should be less

than 1.5 (Christie & Barela, 2005) and the latter less than 2.5 (Kittell-Limerick, 2005) or 1 (Raskin,

1994; Rayens & Hahn, 2000: 311).

However, each analysis should also include the calculation of mean and median, since these are

used to describe the middle and most typical response, depicting the central tendency (Binning,

Cochran, & Donateli, 1972; Kittell-Limerich, 2005), as well as the coefficient of variation (i.e. the

division of the standard deviation with the mean), denoting the observations’ homogeneity, and the

mode, representing the most frequently occurred value (Gupta & Waymire, 2008: 104; Saunders et al.,

2009: 444-448).

4 PRIOR EMPIRICAL STUDIES USING DELPHI TECHNIQUE

In this part, a number of studies, using Delphi technique, between years 1975 to 2013 in the

scientific fields of management and business were chosen. These studies are summarized in table 1

focusing on the way they used Delphi and providing implications for the most controversial issues of

the panel size, the Likert scale, the measure of consensus and the Delphi rounds.

More specifically, in the first two columns the authors (in chronological order) and the country of

research are referred. From the total 32 studies analyzed here, 11 were conducted in Europe, 9 in USA,

4 in Canada, 3 in Asia, 2 in Africa and 1 in Australia; while 2 were cross-national.

The third column depicts the participants in every study, showing that the majority uses a number

up to 30 experts, namely 18 out of 32 studies. In these 18 studies of Delphi 10 used the opinion of less

than 20 experts. However, there are studies using more than 30 experts, with the number ranging

between 30 and 50 participants in 5 studies and between 50 and 100 in 4 more. Also, there are 5 studies

which used an even greater number of participants, i.e. >100.

Focusing on the Likert point scale (fourth column of table 1), it is obvious that 10-point and 5-

point scales are the most common, since these are used by the 29 out of 32 studies (14 studies using a

Int. Journal of Business Science and Applied Management / Business-and-Management.org

10-point scale and 15 studies a 5-point one). Nevertheless, the most important in the Likert point scale

selection is the aim of the study. What can be extracted by the use of the Likert point scale, is that a 10-

point one is used when the level of importance is investigated, since from the 14 studies which used the

10-point scale, 11 measured the importance while from the 15 studies that used the 5-point scale, only

3 did so. On the other hand, when the level of agreement is investigated, or in case of increase/decrease

measurement, a 5-point scale is most common. This could be inferred by the fact that 5 out 15 studies

used a 5-point scale to investigate the level of agreement and 3 the level of decrease/increase, while

only 1 out of 14 studies which used a 10-point scale measured the level of agreement.

The fifth column shows the measure of consensus with the majority of studies (12 out of 32) using

the standard deviation. An also common measure of consensus is the interquartile range which in many

cases is used supplementarily with standard deviation, or with median, or with a specific percentage of

the participant responding to a given category, as for example the percentage of experts responding to

the ‘strongly agreeing’ category, or the percentage of experts responding to the ‘highest priority’

category etc. However, there are also cases using only the percentage of the participant responding to a

given category as an exclusive measure of consensus, others using the coefficient variation and others

implementing the Kendall’s coefficient W. Also, there are studies combining the standard deviation

with the coefficient variation, or the standard deviation with the mean, or even more the interquartile

range with the standard deviation and the median, or the interquartile range with the median and the

percentage of the participant responding to a given category.

Finally, focusing on the number of rounds implemented for reaching consensus, the last column

shows that the majority needed 2 or 3 rounds. From the 32 studies presented in table 1, 17 reached

consensus after two rounds, 11 after three rounds, 2 after four rounds, 1 after five rounds and an

additional one used a combination of two panels, reaching consensus in the 2

and the 4

round

respectively.

Lefkothea Giannarou and Efthimios Zervas

Table 1: Prior empirical studies of Delphi

Authors

Research scope

Country

Participants

Likert-scale

Measure of

consensus

Delphi

Rounds

Lamb (1975)

This study appraises 12 research projects in the field of

electricity utilization by using Delphi combined with

benefit/cost rankings

Canada

160

10-point

(zero/negligible value

to extremely valuable

research program)

2 rounds

Ley & Anderson

(1975)

The Delphi technique was used to forecast the urban

development of Nanaimo, British Columbia along a range of

physical, social and political dimensions.

Canada

5-point

2 rounds

Kaynak &

Macaulay (1983)

Gather data concerning the factors that will influence the

future growth of tourism

Europe

(Scotia)

round: 111/150

round: 44/60

5-point (significant

decrease to

significant increase)

2 rounds

Nelms & Porter

(1985)

This study estimates the maximum possible impact that

technology could have on clerical productivity as well as the

actual expected impact.

USA

(Atlanta,

Georgia)

n/d3

SD, IR,

median

2 rounds

Fish & Piercy

(1987)

This study used Delphi to examine the similarities and

differences in the theory and practice of structural and strategic

family therapy

USA

7-point for agreement

IR, median

3 rounds

Green, Hunter &

Moore (1990)

Assessment of the environmental impacts stemming from

tourist projects.

Europe (UK)

Preliminary stage:

Round: 31

Round: 21

n/d

SD and CV

2 rounds & a

preliminary

stage

Niederman,

Brancheau &

Wetherbe (1991)

The study uses Delphi to determine the most critical issues in

Information Systems (IS) management. For this reason the

importance of 25 issues was investigated.

USA

round: 114/241

round: 126/241

round: 104/175

10-point (least

important to most

important)

3 rounds

Kaynak, Bloom

& Leibold (1994)

This study uses Delphi to analyze the future of tourism in

South Africa by investigating factors which will influence the

future growth of the tourism industry

South Africa

round: 50/100

round: 37/50

5-point (significant

increase to significant

decrease) and 10-

point (non important

to critically

important)

2 rounds

Dekleva &

Zupančič (1996)

Evaluating the importance of 26 IS management issues

Europe

(Slovenia)

Round: 105/330

Round: 163/330

Round: 129/186

round: 148/186

10-point (from

unimportant to most

important)

4 rounds

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Greninger et al.

(2000)

Delphi was used to determine retirement planning guidelines:

to ascertain retirement planning considerations and guidelines,

to determine if a consensus of opinion existed or could be

established and to determine what differences in opinions

might exist.

USA

188

5-point (definitely do

not agree to strongly

agree)

% of experts

responding

categories:

agree,

uncertain

and disagree.

3 rounds

Hayne & Pollard

(2000)

Assessing the importance of 23 issues in Information Systems

(IS) management.

Canada

157

10-point (least

important to most

important)

2 rounds

Miller (2001)

The study used Delphi technique in order to develop indicators

to measure the movement of the tourism product at a

company/resort level towards a position of greater or lesser

sustainability. More specifically, the author ascertained the

opinion of experts on indicators presented to measure

movement towards sustainable tourism.

Europe

Round: 54/74

Round: 37

5-point (strongly

disagree to strongly

agree)

2 rounds

Keil, Tiwana &

Bush (2002)

The study explores the issue of IT project risk from the user

perspective and compares it with risk perceptions of project

managers.

USA

10-point of

importance

Kendall’s

coefficient

concordance

(W)

2 rounds

Hackett, Masson

& Phillips (2006)

The study explores levels of consensus among practitioners

about good practice in relation to youth who are sexually

abusive.

Europe (UK

& Ireland)

10-point (strongly

disagree to strongly

agree)

5-point (no relevance

to highly relevant)

IR, median,

% of

strongly

agreeing

statement (8-

10 and 4-5)

3 rounds

Kaynak &

Marandu (2006)

The study explores the most probable scenario for the tourism

industry in Botswana by the year 2020. For this experts

commended on the extent of changes in societal values and

ranked the expected impact these changes would have on the

industry.

Africa

(Botswana)

round: 104

round: 68

5-point (significant

decrease to

significant increase)

and 10-point (no

impact at all to very

high impact)

2 rounds

Ku Fan & Cheng

(2006)

The study uses Delphi technique in order to identify the needs

for continuing professional development for life insurance

sales representatives and to examine the competencies needed

by those sales representatives.

Asia

(Taiwan)

5-point (strongly

disagree to strongly

agree)

3 rounds

Saizarbitoria

(2006)

The scope of this study was to analyze the influence on

companies’ performance of the two most important models for

Quality Management practice, using Delphi technique.

Europe

(Spain)

11-point

IR, median

2 rounds

Lefkothea Giannarou and Efthimios Zervas

Mason &

Alamdari (2007)

The paper used Delphi to forecast the structure of air transport

in EU in 2015 in respect of network carriers, low cost airlines

and passenger behavior. For this reason the experts were

required to agree or disagree with 27 statements.

26/61

5-point

A 75% of

agreement as

a “broad

consensus”

threshold.

2 rounds

Chang et al.

(2008)

Delphi was used to assess the importance or ERP life cycle

activities

Asia

(Taiwan)

1st round: 27/40

2nd round: 24

10-point for

importance

2 rounds

Czinkota &

Ronkainen

(2008)

The scope of the study was to identify international business

dimensions subject to change in the next 10 years and

highlight the corporate and policy responses to these changes

Africa

Asia

Europe

America

10-point (very low

impact to very high

impact)

n.d.

3 rounds

Nakatsu &

Iacovou (2009)

They investigated the importance of 25 risk factors of

outsourced software development from a client perspective in

domestic and offshore settings

USA

1st round: 29/32

2nd round: 26/32

3rd round: 27/32

10-point (unimportant

to very important)

3 rounds

Lee & King

(2009)

The study proposes a guiding framework for the future

development of hot springs tourism in Taiwan, drawing upon

factors influencing the competitiveness of the sector.

Asia

(Taiwan)

1st round: 31/36

2nd round: 28/31

3rd round: 26/28

5-point for

importance

IR<1 & 80%

responded to

categories

“highest

priority”

(mean score

above 4.5)

and

“important

elements”

(means score

between 4

and 4.49)

3 rounds

Asonitis &

Kostagiolas

(2010)

Delphi technique was employed to highlight the most

important library services for the central Greek public

libraries.

Europe

(Greece)

1st round: 11/12

2nd round: 9/12

10-point for

importance

2 rounds

Geist (2010)

Evaluating the importance of organizational goals

and a follow-up survey asking questions about the ease of use,

the merit or value and enjoyment

USA

Paper-pencil delphi:

Round 0: 14/30

Round 1: 16/30

Round 2: 12/30

Round 3: 13/30

Real-time Delphi:

Round 0: 10/30

Round 1: 11/30

7-point (not important

to very important

5-point (strongly

disagree to strongly

agree)

SD, IR

4 rounds & 2

rounds

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Hussein (2010)

The study examines Corporate Social Responsibility (CSR)

theorists’ criteria from the corporate executive’s perspective.

For this reason it uses Delphi technique in order to identify

specific criteria recognized and used by organizational

executives vital to evaluating CSR.

USA

1st round: 26/35

2nd round: 27/35

3rd round: 25/35

5-point for

importance

IR<1.2

3 rounds

Culley (2011)

This study uses Delphi to evaluate the efficacy of using online

computer, Internet and e-mail applications.

USA

7-point (not useful to

very useful)

IR, ≥70%

agreement

2 rounds

Giannarakis,

Litinas &

Theotokas (2011)

The paper identifies both general and sector-specific indicators

in order to measure the Corporate Social Responsibility (CSR)

performance in Telecommunication sector

Europe

(Greece)

8/17

n/d

3 rounds

Post, Rannikmäe

& Holbrook

(2011)

The study aims to create a theoretical tool for determining

competencies and knowledge in science education (which a

school leaver should have in order to be successful in the

workforce and/or as a citizen in society).

Europe

(Estonia)

1st round: 38

2nd round: 85

5-point (non

important to very

important)

Mean

(divided into

two

categories:

over 4 and

under 3)

2 rounds

Hadaya, Cassivi

& Chalabi (2012)

The purpose of the study is to identify the most important IT

project management resources and capabilities.

Canada

1st round: 30/34

2nd round: 30/30

3rd round: 28/30

4th round: 24/28

5th round: 19/24

10-point for

importance

Kendall’s W

5 rounds

Hefferan &

Wardner (2012)

It uses Delphi to demonstrate how demand drivers and

accommodation priorities for emerging knowledge-intensive

firms are understood and how corporate property and asset

managers can respond to them.

Australia

5-point (low priority

to very high priority)

n/d

4 rounds

Goula (2013)

This study uses Delphi technique to explore ways of public

transition from bureaucracy to a participation-culture model of

human resources.

Europe

(Greece)

10/12

5-point (strongly

disagree to strongly

agree)

IR, SD

2 rounds

Jones, Day &

Quadri-Felitti

(2013)

This study uses Delphi to determine what defines “boutique”

and “lifestyle” hotels.

Europe

USA

Asia

1st round: 20

2nd round: 24

3rd round: 25

10-point (least

important to most

important)

3 rounds

1. IR: interquartile range

2. SD: standard deviation

3. n/d: not defined

4. CV: coefficient variation

Lefkothea Giannarou and Efthimios Zervas

5 CASE EXAMPLES

5.1 Design

The presented case examples focus on one of the most controversial issues in Delphi technique

application, namely the consensus measurement. This issue triggered our effort to provide complete

guidelines to conduct Delphi as a means of eliciting experts’ opinion. Based on our experience, the way

of reaching consensus, is presented, using two case examples to illustrate how the various measures of

consensus could be applied in practice. These examples are used to indicate our basic conclusions on

consensus measurement in a practical way.

In the following two case examples, the way of eliciting the experts’ opinion is demonstrated,

regarding the importance of 10 variables and their agreement upon 8 statements respectively; which are

two of the most common uses of Delphi (see for example Geist, 2010; Hadaya, Cassivi, & Chalabi,

2012; Hayne & Pollard, 2000; Ku Fan & Cheng, 2006; Miller, 2001; Nakatsu & Iacovou, 2009). For

this reason, a well-structured questionnaire is formulated (see appendix 1); using a 10-Likert scale for

assessing the importance of a variable (1

case) and a 5-Likert one for the measurement of agreement

case). The data that are used to illustrate these case examples are taken in part from one of the

authors PhD thesis. However, since the aim of this study is to provide guidance to any researcher in any

scientific field, the names of the variables and the statements are not referred. Nevertheless, the

selected data are used to describe the problems that may arise in the consensus measurement and are

described thereafter.

5.2. 1

Case

In the first case, the consensus measurement when the scope of a Delphi study is to assess the

importance of a variable is demonstrated. Such examples are the Hayne and Pollard’s (2000) study,

where the importance of 23 issues in Information Systems (IS) management was evaluated, or the

Nakatsu and Iacovou’s (2009) one where the importance of 25 risk factors of outsourced software

development from a client perspective in domestic and offshore settings was investigated.

To illustrate this case, in a Likert scale of 0-10 (respectively for non- and high- importance)

(Asonitis & Kostagiolas, 2010; Ishikawa et al., 1993; Mullen, 2003; Nerantzidis, 2013), the opinion of

12 experts is shown in table 2.

To assess consensus, three measures are used combinatory:

(i) The 51% responding to the category ‘highly important’, which is between values 8 and

10 on a 10-Likert scale (Hackett, Masson & Phillips, 2006),

(ii) the interquartile range below 2.5 (Kittell-Limerick, 2005) and

(iii) the standard deviation below 1.5 (Christie & Barela, 2005).

Each of the above three measures has been separately proposed for consensus measurement.

However, there are cases where the interquartile range may be lower than 2.5 and/or the standard

deviation lower than 1.5, but only a low percentage of experts (less than 51%) evaluate the variable as

‘highly important’ (between values 8 and 10). Respectively, it is also possible that although at least

51% of the experts evaluate a variable as ‘highly important’, its interquartile range may be higher than

2.5 or/and its standard deviation higher than 1.5. These cases are presented in table 2 in variables 4,7

and 8.

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Table 2: Delphi results regarding the importance of the variables

Expert

median

Q=Q3-Q1

mode

average

8-10%

standard

deviation

Delphi round

Variable 1

9.3

1.25

8.75

83.33

1.06

0.12

Variable 2

8.8

0.25

8.75

100.00

0.45

0.05

Variable 3

1.00

8.67

91.67

0.89

0.10

Variable 4

6.5

2.50

7.42

66.67

1.78

0.24

Variable 5

8.8

1.25

9.17

91.67

1.27

0.14

Variable 6

5.8

3.25

6.75

41.67

2.22

0.33

Variable 7

7.5

2.00

7.17

50.00

1.40

0.20

Variable 8

5.8

2.25

6.75

41.67

1.54

0.23

Variable 9

2.00

8.50

83.33

1.73

0.20

Variable 10

7.5

2.00

7.08

50.00

1.83

0.26

Delphi round

Variable 1

8.3

0.75

8.90

100.00

0.74

0.08

Variable 2

1.00

8.60

100.00

0.52

0.06

Variable 3

1.00

8.70

100.00

0.67

0.08

Variable 4

1.00

7.40

60.00

1.43

0.19

Variable 5

1.00

9.50

100.00

0.71

0.07

Variable 6

7.8

1.75

6.60

30.00

1.71

0.26

Variable 7

8.8

1.75

7.70

60.00

1.25

0.16

Variable 8

6.5

7.8

1.75

6.60

30.00

1.17

0.18

Variable 9

9.5

1.00

9.20

90.00

1.03

0.11

Variable 10

1.00

7.60

60.00

1.07

0.14

Lefkothea Giannarou and Efthimios Zervas

More specifically, although the 66.67% of respondents evaluate the ‘variable 4’ as ‘highly

important’ (i.e. value this variable between 8 and 10 in the Likert scale), its interquartile range is 2.5

and its standard deviation over 1.5. Thus, how can we infer that this variable reaches consensus?

Respectively, ‘variable 7’ has an interquartile range 2 and standard deviation 1.40, but only a 50% of

respondents consider the variable as ‘highly important’ (its average value is 7.17). Similarly, ‘variable

8’ also has an unsatisfactory average value of 6.75 and an even lower percentage of respondents

evaluate it as ‘highly important’ (41.67%), although its interquartile range is 2.25.

All things considered, in this example, only 4 variables could be thought of as reaching consensus

(variables 1, 2, 3, 5) from the 1

Delphi round and a 2

round of feedback is considered necessary in

order to conclude for the most important variables.

For this reason, a questionnaire of a controlled feedback of the group’s perspective should be

designed, for the second Delphi round, so that the respondents can clarify or change their views. For

this reason, the interquartile range of each variable should be identified (the shadow area in appendix 2)

and the respondents should change or state their answer when this is out of this range.

In case where fewer respondents than in the first round participate, the response rate must be

calculated. In this case example, we consider the answers of 10 out of 12 experts participating in the

second round; a response rate of 83.33%.

As it is apparent, the second round has improved the agreement among the experts. This means

that, apart from variables 1, 2, 3 and 5, consensus is also reached for the importance of variables 4, 7, 9

and 10 (see table 3). More specifically, all these variables satisfy the criteria of an interquartile range

below 2.5, a standard deviation below 1.5 and a percentage of experts over 51% evaluating them as

‘highly important’ (between values 8-10). Hence, in this example, where the importance of 10 variables

was investigated and diverse views existed (lack or agreement), the Delphi technique provided us with

a reliable way to conclude to the most significant ones; namely these where agreement was reached

among the experts.

Table 3: Variables’ consensus

% 8-10

1st Round

2nd

Round

1st

Round

2nd

Round

1st

Round

2nd

Round

Variable 1

83.33

100.00

1.25

0.75

1.06

0.74

Variable 2

100.00

0.25

1.00

0.45

0.52

Variable 3

91.67

100.00

1.00

0.89

0.67

Variable 4

66.67

60.00

2.50

1.00

1.78

1.43

Variable 5

91.67

100.00

1.25

1.00

1.27

0.71

Variable 6

41.67

30.00

3.25

1.75

2.22

1.71

Variable 7

50.00

60.00

2.00

1.75

1.40

1.25

Variable 8

41.67

30.00

2.25

1.75

1.54

1.17

Variable 9

83.33

90.00

2.00

1.00

1.73

1.03

Variable 10

50.00

60.00

2.00

1.00

1.83

1.07

5.3 2

Case

In this second case, an example of eliciting consensus upon the agreement of experts in 8

statements is provided, using a 5-Likert scale, with value 1 denoting strongly disagreeing and value 5

strongly agreeing (Hackett et al., 2006; Verhagen et al., 1998). This use of Delphi is presented, for

instance, in Miller’s (2001) study to ascertain the opinion of experts on indicators considered to

measure the movement towards sustainable tourism. For this reason, he asked the experts whether they

agree or not that an indicator is understandable or is measured on an ongoing basis etc. In these

statements, experts were asked to provide their opinion choosing a value from 1 (strongly disagree) to 5

(strongly agree).

In such a case, the consensus is proposed to be assessed using three measures combinatory:

(i) The 51% of experts responding to the category ‘strongly agreeing’ (which according to Hackett et

al., 2006, is between values 4 and 5 on a 5-Likert scale),

(ii) the interquartile range below 1 (Raskin, 1994; Rayens & Hahn, 2000: 311) and

(iii) the standard deviation below 1.5 (Christie & Barela, 2005)

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Table 4: Delphi results regarding the agreement of the statements

Expert

median

Q=Q3-Q1

mode

average

8-10%

standard

deviation

Delphi round

Statement 1

4.8

0.25

4.75

100.00

0.45

0.10

Statement 2

1.00

4.58

100.00

0.51

0.11

Statement 3

3.8

1.25

4.00

75.00

0.95

0.24

Statement 4

4.8

0.25

4.67

91.67

0.65

0.14

Statement 5

3.00

3.33

41.67

1.56

0.47

Statement 6

4.3

1.25

3.92

66.67

0.79

0.20

Statement 7

2.8

1.25

3.17

41.67

0.83

0.26

Statement 8

1.00

3.67

58.33

0.89

0.24

Delphi round

Statement 1

4.3

0.75

4.70

100.00

0.48

0.10

Statement 2

1.00

4.60

100.00

0.52

0.11

Statement 3

4.8

0.75

4.10

80.00

0.74

0.18

Statement 4

0.00

4.90

100.00

0.32

0.06

Statement 5

3.5

3.00

3.50

50.00

1.43

0.41

Statement 6

1.00

4.50

90.00

0.71

0.16

Statement 7

3.5

1.00

3.50

50.00

0.53

0.15

Statement 8

4.8

0.75

4.10

80.00

0.74

0.18

To prove the need of this combinatory use, the answers of 12 experts for the 1

Delphi round and 10

experts for the 2

one are provided (table 4).

As it is obvious, in the first Delphi round, there may be statements with standard deviation below 1.5

and/or a 51% or experts responding to the category ‘strongly agreeing’ (i.e. between values 4 and 5), while their

interquartile range may be above 1 (statements 3 and 6). Respectively, there may be a case where the percentage

of experts’ responses lying into the ‘strongly agreeing’ category is below 51%, even if the standard deviation

and/or the interquartile range are below 1.5 and 1 respectively (statement 7).

The question of how can one assure that these statements are reaching consensus among the experts still

exists. Thus, combining the above three measures, in our example, only 4 statements could be thought of as

overall consensus and a second round of enhancing agreement is required (see appendix 2).

In the second round of changing or stating the opinion (using the interquartile range as guidance), the level

of agreement of two more statements was improved. That’s was because the combination of the three measures

of consensus, namely the 51% of experts responding to the ‘strongly agreeing’ category, the interquartile range

below 1.5 and the standard deviation below 1, were denoting overall consensus among six statements.

Obviously, consensus was reached in addition to statements 3 and 6, where their interquartile range value was

improved to 0.75 and 1 respectively.

Finally, table 5 denotes the difference between these measures from round to round for each statement.

Undoubtedly, the combinatory use of these three measures ensured, once more, the way of reaching consensus

in Delphi technique and provided a reliable manner to conclude on the expert’s overall agreement upon the eight

statements assumed.

Table 5: Statement’s consensus

% 4-5

1st Round

2nd

Round

1st

Round

2nd

Round

1st

Round

2nd

Round

Statement 1

100.00

0.25

0.75

0.45

0.48

Statement 2

100.00

1.00

0.51

0.52

Statement 3

75.00

80.00

1.25

0.75

0.95

0.74

Statement 4

91.67

100.00

0.25

0.00

0.65

0.32

Statement 5

41.67

50.00

3.00

1.56

1.43

Statement 6

66.67

90.00

1.25

1.00

0.79

0.71

Statement 7

41.67

50.00

1.25

1.00

0.83

0.53

Statement 8

58.33

80.00

1.00

0.75

0.89

0.74

6 CONCLUSION

The Delphi technique is a qualitative tool, which is used to elicit expert’s opinion, without the cost of ‘face-

to-face’ interaction, when information about the existing problem is restricted. Although time consuming, it is

quite simple in application and allows interaction. However, its implementation on different sectors has also

yielded issues of fuzziness regarding the expert’s panel selection (size and characteristics), the consensus

measurement and the number of rounds, as well as the response rate and the questionnaire design.

This paper clarifies the above issues both theoretically and practically, to assist any researcher in

management or business field to conduct Delphi technique. In particular, through literature review this study

shows that the purpose of each study defines the questionnaire design, and more specifically the Likert scale

choice, while the homogeneity of the sample determines the panel size and therefore the Delphi rounds;

demanding, in any case, a response rate above 70%. However, since there is a great variation among the studies

using Delphi, regarding the Likert scale, the number of participants, the number of rounds and the measures of

consensus, 32 prior empirical studies are analyzed to show the major trends.

On the other hand, using two examples, the way of reaching consensus was demonstrated in practice,

leading to the need of using more than one statistical measures in order to assess the consensus. Hence, this

study shows that there are cases where the interquartile range or/and the standard deviation may be within the

accepted limit but the average value may be low and hence the experts may do not assess the importance of a

variable as high (between values 8-10 in a 10-Likert scale) or may not ‘strongly agree’ with a statement

(between values 4-5 in a 5-Likert scale). For this reason, these three measures should be considered at the same

time, so that consensus can be ensured.

All things considered, Delphi is a quite useful tool in decision making process in the scientific field of

management or business, when a lack of agreement or incomplete knowledge is evident. It is useful in case

study analyses, because of its limitation of non generalizability of the results, and provides a great advantage for

Int. Journal of Business Science and Applied Management / Business-and-Management.org

the researcher who does not need a representative sample to implement this method. Its diffusion and

contribution in any scientific field could be the aim of a longitudinal study. This means that, selecting the

applications of Delphi from the very first years, such a study could highlight the scientific field with the greatest

contribution and practical implementation. Undoubtedly, this is not the only implication for future studies, since

an open case is the great time that this method demands in order to reach consensus. This issue may also be

central in the near future, where technology could provide a clear assistance on its implementation. Hence, what

was an obstacle in 1970s, could now be confronted through on-line applications, providing friendlier

environment and quicker responses with real time interactions between the experts.

REFERENCES

Agwe, J. N., & Sharif, N. M. (2007). Prioritizing the relative dominance of drivers for intellectual

entrepreneuring through the tertiary knowledge industry. Journal of Technology Management &

Innovation, 2(4), 20-43.

Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi method and its application to social policy

and public health. London: Kingsley Publishers.

Armstrong, J. S. (1985). Long Range Forecasting: From crystal ball to computer (2nd ed.). New York: Wiley.

Asonitis, S., & Kostagiolas, P. A. (2009). An analytic hierarchy approach for intellectual capital. Evidence for

the Greek central public libraries. Library Management, 31(3), 145-161.

Butterworth, T., & Bishop, V. (1995). Identifying the characteristics of optimum practice: Findings from a

survey of practice experts in nursing, midwifery and health visiting. Journal of Advanced Nursing, 22(1),

24-32.

Binning, D., Cochran, S., & Donatelli, B. (1972). Delphi panel to explore post-secondary educational needs in

the state of New Hampshire. Manchester, NH: Decision Research.

Bryman, A., & Bell, E. (2011). Business research methods (3rd ed.). New York: Oxford University Press, Inc.

Cantrill, J. A., Sibbald, B., & Buetow, S. (1996). The Delphi and nominal group technique in health services

research. International Journal of Pharmacy Practice, 4(2), 67-74.

Cavalli-Sforza, V., & Ortolano, L. (1984). Delphi forecasts of land-use – Transportation interactions. Journal of

Transportation Engineering, 110(3), 324-339.

Chang, S., Yen, D. C., Huang, S., & Hung, P. (2008). An ERP system life cycle-wide management and support

framework for small- and medium-sized companies. Communications of the Association for Information

Systems, 22(1), 275-294.

Christie, C. A., & Barela, E. (2005). The Delphi technique as a method for increasing inclusion in the evaluation

process. The Canadian Journal of Program Evaluation, 20(1), 105-122.

Culley, J. M. (2011). Use of a computer-mediated Delphi process to validate a mass casualty conceptual model.

CIN: Computers, Informatics, Nursing, 29(5), 272-279.

Czinkota, M. R., & Ronkainen, I. A. (2008). Trends and indications in international business: Topics for future

research. Management International Review, 49(2), 249-266.

Dalkey, N. (1969). The Delphi method: An experimental study of group opinion. Santa Monica, CA: Rand

Corporation.

Dalkey, N. C. (2003). The Delphi methodology. [pdf] Retrieved April 10, 2013 from http://

www.optimizationgroup.com/wp-content/uploads/2013/02/IdeaGeneration -Turning-Questions-Into-

Answers.pdf

Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts.

Management Science, 9(3), 458-467.

Dalkey, N., Rourke, D. L., Lewis, R., & Snyder, D. (1972). Studies in the quality of life: Delphi and decision-

making. Lexington, MA: Lexington Books.

Dekleva, S., & Zupančič, J. (1996). Key issues in information systems management: a Delphi study in Slovenia.

Information & Management, 31(1), 1-11.

Fish, L. S., & Piercy, F. P. (1987). The theory and practice of structural and strategic family therapies: A Delphi

study. Journal of Marital and Family Therapy, 13(2), 113-125.

Gallagher, M., Branshaw, C., & Nattress, H. (1996). Policy priorities in diabetes care: a Delphi study. Quality in

Health Care, 5(1), 3-8.

Geist, M. R. (2010). Using the Delphi method to engage stakeholders: A comparison of two studies. Evaluation

and Program Planning, 33(2), 147-154.

Malin Olander Roese and Sverker Sikström

Giannarakis, G., Litinas, N., & Theotokas, I. (2011). A Delphi study to identify corporate social responsibility

indicators: The case of Greek telecommunication sector. Journal of Sustainable Development, 4(2), 16-32.

Goula, A. (2013). The organizational culture of public hospital: Factors of transition from the bureaucratic to a

new model of management (in Greek). PhD. Panteion University of Political and Social Sciences.

Green, H., Hunter, C., & Moore, B. (1990). Application of the Delphi technique in tourism. Annals of Tourism

Research, 17(2), 270-279.

Greninger, S. A., Hampton, V. L., Kitt, K. A., & Jacquet, S. (2000). Retirement planning guidelines: a Delphi

study of financial planners and educators. Financial Services Review, 9(3), 231-245.

Gupta, V. K., & Waymire, E. C. (2008). Spatial variability and scale invariance in hydrologic regionalization. In

S. Garrison (Ed.), Scale dependence and scale invariance in hydrology (pp. 88-135). New York:

Cambridge University Press.

Hackett, S., Masson, H., & Phillips, S. (2006). Exploring consensus in practice with youth who are sexually

abusive: Findings from a Delphi study of practitioner views in the United Kingdom and the Republic of

Ireland. Child Maltreatment, 11(2), 146-156.

Hadaya, P., Cassivi, L., & Chalabi, C. (2012). IT project management resources and capabilities: a Delphi study.

International Journal of Managing Projects in Business, 5(2), 216-229.

Hayne, S. C., & Pollard, C. E. (2000). A comparative analysis of critical issues facing Canadian information

systems personnel: a national and global perspective. Information & Management, 38(2), 73-86.

Hefferan, M. J., & Wardner, P. (2012). New firms … old assets: Using Delphi technique to identify

accommodation priorities for knowledge-intensive firms. Journal of Corporate Real Estate, 14(4), 202-

213.

Helmer, O. (1967). Analysis of the future: The Delphi method. Santa Monica, CA: RAND Corporation.

Hill, G. W. (1982). Group versus individual performance: Are N+1 heads better than one? Psychological

Bulletin, 91(3), 517-539.

Hussein, M. M. (2010). Corporate social responsibility: finding the middle ground. Social Responsibility

Journal, 6(3), 420-432.

Ishikawa, A., Amagasa, M., Shiga, T., Tomizawa, G., Tatsuta, R., & Mieno, H. (1993). The max-min Delphi

method and fuzzy Delphi method via fuzzy integration. Fuzzy Sets and Systems, 55(3), 241-253.

Jones, D. L., Day, J., & Quadri-Felitti, D. (2013). Emerging definitions of boutique and lifestyle hotels: A

Delphi study. Journal of Travel & Tourism Marketing, 30(7), 715-731.

Kaynak, E., Bloom, J., & Leibold, M. (1994). Using the Delphi technique to predict future tourism potential.

Marketing Intelligence & Planning, 12(7), 18-29.

Kaynak, E., & Macaulay, J. A. (1983). The Delphi technique in the measurement of tourism market potential:

The case of Nova Scotia. Tourism Management, 5(2), 87-101.

Kaynak, E., & Marandu, E. E. (2006). Tourism market potential analysis in Botswana: A Delphi study. Journal

of Travel Research, 45(2), 227-237.

Keil, M., Tiwana, A., & Bush, A. (2002). Reconciling user and project manager perceptions of IT project risk: a

Delphi study. Information Systems Journal, 12(2), 103-119.

Kittell-Limerick, P. (2005). Perceived barriers to completion of the academic doctorate: A Delphi study. Texas:

Texas A&M University-Commerce.

Kreitner, R., & Kinecki, A. (1992). Organizational behavior (2nd ed.). Homewood, IL: Irwin.

Ku Fan, C., & Cheng, C. (2006). A study to identify the training needs of life insurance sales representatives in

Taiwan using the Delphi approach. International Journal of Training and Development, 10(3), 212-226.

Lamb, P. (1975). Research appraisal in the electricity industry – An interesting application of the Delphi

method. Industrial Marketing Management, 4(2-3), 89-98.

Landeta, J. (2006). Current validity of the Delphi method in social sciences. Technological Forecasting &

Social Change, 73(5), 467-482.

Lee, C., & King, B. (2009). A determination of destination competitiveness for Taiwan’s hot springs tourism

sector using the Delphi technique. Journal of Vacation Marketing, 15(3), 243-257.

Ley, D. F., & Anderson, G. (1975). The Delphi technique in urban forecasting. Regional Studies, 9(3), 243-249.

Listone, H. A., & Turoff, M. (1975). Introduction to the Delphi method: Techniques and applications. In H.A.

Linstone & M. Turoff (Eds.), The Delphi method: techniques and applications (pp. 3-12). Reading, MA:

Addison-Wesley Publishing Company, pp. 3-12.

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Mason, K. J., & Alamdari, F. (2007). EU network carriers, low cost carriers and consumer behavior: A Delphi

study of future trends. Journal of Air Transport Management, 13(5), 299-310.

McKenna, H. P. (1989). The selection by ward managers of an appropriate nursing model for long-stay

psychiatric patient care. Journal of Advanced Nursing, 14(9), 762-775.

Miller, G. (2001). The development of indicators for sustainable tourism: results of a Delphi survey of tourism

researchers. Tourism Management, 22(4), 351-362.

Mullen, P. M. (2003). Delphi: Myths and reality. Journal of Health Organization and Management’, 17(1), 7-

52.

Nakatsu, R. T., & Iacovou, C. L. (2009). A comparative study of important risk factors involved in offshore and

domestic outsourcing of software development projects: A two-panel Delphi study. Information &

Management, 46(1), 57-68.

Nelms, K. R., & Porter, A. L. (1985). EFTE: An interactive Delphi method. Technological forecasting and

social change, 28(1), 43-61.

Nerantzidis, M. (2012). Delphic Hierarchy Process (DHP): A methodology for the resolution of the problems of

the evaluation of Corporate Governance Quality. [pdf] Retrieved April 4, 2013 from

http://ssrn.com/abstract=2056926

Nerantzidis, M. (2013). Corporate governance in the Greek private sector, in state-owned enterprises and

organizations and in public administration: Comparisons, results and proposals for improvement (in

Greek). PhD. Panteion University of Social and Political Sciences.

Niederman, F., Brancheau, J. C., & Wetherbe, J. C. (1991). Information Systems management issues for the

1990s. MIS Quarterly, 15(4), 475-500.

Phillips, R. (2000). New applications for the Delphi technique. Annual “San Diego” Pfeiffer and Company, 2,

191-196.

Post, A., Rannikmäe, M., & Holbrook, J. (2011). Stakeholder views on attributes of scientific literacy important

for future citizens and employees – a Delphi study. Science Education International, 22(3), 202-217.

Raskin, M. S. (1994). The Delphi study in field instruction revisited: Expert consensus on issues and research

priorities. Journal of Social Work Education, 30(1), 75-89.

Rayens, M. K., & Hahn, E. J. (2000). Building consensus using the policy Delphi method. Policy, Politics &

Nursing Practice, 1(4), 308-315.

Reid, N. (1988). The Delphi technique: Its contribution to the evaluation of professional practice. In R. Ellis

(Ed.), Professional competence and quality assurance in the caring professions (pp. 230-262). London:

Chapman & Hall.

Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International

Journal of Forecasting, 15(4), 353-375.

Rowe, G., & Wright, G. (2001). Experts opinions in forecasting: The role of the Delphi technique. In J.

Armstrong (Ed.), Principles of forecasting (pp. 125-144). Boston: Kluwer Academic.

Saizarbitoria, I. H. (2006). How quality management models influence company results-Conclusions of an

empirical study based on the Delphi method. Total Quality Management & Business Excellence, 17(6),

775-794.

Saunders, M., Lewis, P., & Thornhill, A. (2009). Research methods for business students (5th ed.). Harlow,

Essex: Pearson Education Limited.

Shields, T. J., Silcock, G. W. H., Donegan, H. A., & Bell Y. A. (1987). Methodological problems associated

with the use of the Delphi technique. Fire Technology, 23(3), 175-185.

Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of

Information Technology Education, 6, 1-21.

Turoff, M. (1970). The design of a policy Delphi. Technological Forecasting and Social Change, 2(2), 149-171.

Verhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., & Knipschild, P.

G. (1998). The Delphi list: A criteria for quality assessment of randomized clinical trials for conducting

systematic review developed by Delphi consensus. Journal of Clinical Epidemiology, 51(12), 1235-1241.

Malin Olander Roese and Sverker Sikström

APPENDIX 1

1st Delphi round: Questionnaire sample

Clarifications

In the following questionnaire your may state your opinion regarding the level of each variable’s

importance, compared to the others, by choosing a value among 0 to 10. More specifically, you may choose the

zero (0) value when the variable is considered unimportant and, while value ten (10), when it is considered as

highly important. Respectively, you should express your opinion on 8 statements by choosing a value among 1

to 5. You may choose value one (1) when you highly disagree, while value five (5) when you highly agree.

Arithmetic scale

Not Important → Highly important

Answer

VARIABLES

Variable 1

Variable 2

Variable 3

Variable 4

Variable 5

Variable 6

Variable 7

Variable 8

Variable 9

Variable 10

Highly Disagree → Highly agree

Answer

STATEMENTS

Statement 1

Statement 2

Statement 3

Statement 4

Statement 5

Statement 6

Statement 7

Statement 8

Additional information – observations:

APPENDIX 2

Delphi round: Questionnaire sample

Clarifications

In the following questionnaire you are to restate your opinion regarding the contribution level of each of

the 10 variables, compared to the others, by choosing a value between 0 and 10 and your disagreement or

agreement upon the 8 statements, by choosing a value between 1 and 5. In addition, the shadowed cells depict

the range of the 50% of the first Delphi round responses as follows: the lower values imply lower importance for

this specific variable or low levels of agreement, while the higher values, higher importance or higher levels of

agreement.

In the two next tables, you are to restate your opinion, either by maintaining or changing your previous

choice (your answer in 1

Delphi round). In the case where the chosen value is outside the shadowed range, you

should justify your choice providing a short explaining text.

Int. Journal of Business Science and Applied Management / Business-and-Management.org

Arithmetic scale

Answer

Stating your answer

Not important → Highly important

VARIABLES

Variable 1

Variable 2

Variable 3

Variable 4

Variable 5

Variable 6

Variable 7

Variable 8

Variable 9

Variable 10

Arithmetic scale

Answer

Stating your answer

Highly disagree → Highly agree

STATEMENTS

Statement 1

Statement 2

Statement 3

Statement 4

Statement 5

Statement 6

Statement 7

Statement 8