Delphi Survey

Online Foresight Guide

The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts in the relevant field(s).

Overall description

The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts. It therefore yields both qualitative and quantitative results and draws on exploratory, predictive even normative elements. A single Delphi methodology exists but the applications are diverse. In the most common form, the opinions sought concern the particular developments that are likely to take place. Such Delphis have been widely used in Technology Foresight studies and other exercises. But there are many other types of Delphi exercise possible. They may focus on different topics – on social developments, for example. Instead of trying to forecast the time scales of particular developments, Delphis can be constructed to help identify and prioritise policy goals, for example.

There is agreement that Delphi is an expert survey in two or more 'rounds' in which, in the second and later rounds of the survey the results of the previous round are given as feedback (Cuhls 1998). Therefore, the experts answer from the second round on under the influence of their colleagues' opinions, and this is what differentiates Delphi from ordinary opinion surveys. The idea is that the respondents can learn from the views of others, without being unduly influenced by the people who talk loudest at meetings, or who have most prestige, etc. Ideally, significant dissenters from a developing consensus would be required to explain their reasons for their views, and this would serve as useful intelligence for others. Unfortunately, to carry this out is time consuming, and is quite often missed out from Delphi studies.

Thus, the Delphi method is a 'relatively strongly structured group communication process, in subjects, on which naturally unsure and incomplete knowledge is available, are judged upon by experts', write Häder and Häder (1995, p. 12). Giving feedback and the anonymity of the Delphi survey are important characteristics. Wechsler describes a 'Standard-Delphi-Method' in the following way: 'It is a survey which is steered by a monitor group, comprises several rounds of a group of experts, who are anonymous to each other and for whose subjective-intuitive prognoses a consensus is aimed at. After each survey round, a standard feedback about the statistical group judgement calculated from median and quartiles of single prognoses is given and if possible, the arguments and counter arguments of the extreme answers are fed back...' (Wechsler 1978, pp. 23f.).

Characteristics of Delphi are therefore specified as (see e.g. Häder and Häder 1995):

See also the definition of a Delphi Survey in the key terms section.

When is this method appropriate?

Usually, the goal (and the result) of a Delphi study is to organise a debate, to collect and synthesise opinions and to achieve a degree of convergence. It is a valuable tool for communication and for exchanging opinions on a topic, making experts' tacit knowledge of the future more explicit. It is also useful for longer-term assessments where extrapolations make no sense. It can help to gather the opinions of a larger group of experts and in fields where there is not a lot of evidence about the developments and where experts often do not dare to explain their real opinion. It is conducted anonymously in order not to let anyone lose face in the event of a change of opinion. The methodology is designed to avoid domination by particular individuals.

Normally, the number of respondents is small, Delphis are not intended to produce statistically significant results. In fact, the outcomes provided by a panel do not predict the response of a larger population or even a different Delphi panel. The outcomes represent the view of a particular group of experts. But in recent surveys, there has been an attempt to achieve a kind of representative sample from a particular community.

The Delphi method is mainly used when long-term issues (up to 30 years) have to be assessed. It is a useful means of predicting and assessing emerging developments where there is no empirical database, where external factors are likely to have a determining effect and where social arguments may dominate economic or technical considerations. As it implies identifying statements (topics) that are relevant for the future, it reduces the tacit and complex knowledge to a single statement and makes it possible to judge. On the other hand, in more complex issues, when the themes cannot be reduced that much or when thinking about and discussing alternatives are the major target, Delphi is not the method of choice. It is also suitable if there is the (political) will to involve a large number of people in processes (Eto 2003).

The Delphi method seems to have managed to withstand the pressures of the changing challenges of the past 50 years. It has been able serve different perceptions of forecasting or Foresight and was probably understood by the users as being an appropriate way to tackle technical perspectives, organisational perspectives, and also personal perspectives. The individual could express a distinctly different opinion as compared to the group perspective and this to a differing degree between the technical details under scrutiny. As multiple perspectives are recommended for decision-making (Linstone/Mitroff 1994; Linstone 1998), the Delphi technique seems to have an appeal in quite diverse situations at the the longer-range end of the scale. As controlled scientific experiments have shown that Delphi estimates are no better than those of other consensus-oriented methods (Woudenberg 1991), it must be the communicative force of Delphi approaches that facilitates the switching between different perspectives. Indeed, Delphi is no longer used to create consensus but to test if there is already consensus about the shape of things to come. What users like in particular are the sets of data about the future that are gathered. Writing down future topics seems to have an immense psychological effect because it transfers implicit and tacit knowledge to more visible, explicit, and therefore transferable knowledge.

Therefore, one can conclude that Delphi and other surveys are tools to bring together the opinions or judgements of a large number of persons. These kinds of surveys are mainly useful in processes where the exchange of opinions and the communication effect is important but which are mainly result-oriented. Especially when data sets are needed for priority-setting, they deliver the essential basics.

Delphi is mainly used in science, technology or education fields, where topics can be stated concisely. For more complex issues, Delphi is not the method of choice.

Who is this approach appropriate for?

The major users are companies, particularly strategy departments. In the case of national Delphi studies, the target audience is usually defined as anyone interested in information about the future so, along with companies, the major users tend to be research institutions, ministries, journalists, teachers, students and pupils. This formalised and traceable method has credibility with policy-makers.

Who is typically involved?

Typically involved (i.e. constituting the panel of respondents) are experts from business, government, research, associations and other persons competent in the field of subject, but usually the term expert is used in a relatively broad sense.

Step-by-step guide

    Selection of the subject to forecast

The subject should be one where there is a lack of hard data on future trends. In some cases, one thematic field is enough, in many cases the aim is to get an overview so that more fields are decided on and handled in a flexible way. There is always the possibility to add, remove or re-name fields.

Definition of the procedure

The whole procedure has to be fixed in advance. Will panel meetings be set-up or will the teams work on-line? Is the questionnaire an electronic or a paper one? This means that logistics (whether setting up the website or typing in results from the paper versions) have to be organised. Will there be follow-up workshops, interviews, presentations? If so, these also have to be organised and prepared. Printing of brochures, leaflets, questionnaire, reports also have to be considered.

When designing the questionnaire, it is important to consider from the beginning how to give feedback to the participants during the second round. The usual way is to provide percentages or graphics from the accumulated data in a similar way as in the first round questionnaire. But that often gives the impression of a very 'full' picture and too much information has to be shown on one page. The new electronic media provide many more possibilities. There is much room for creativity.

Formulation of the statements

This is a time-consuming process. It has to be made clear where the statements come from. The easiest way is desk research and to take them from literature and surveys that are available. But the more creative way is to set up working groups who have the task to structure the field and formulate topics. It is often necessary to filter the topics twice or even three times because often, the experts in working groups add topics instead of reducing the number. The last step is formulation of the details.

Formulation of the questions

The questions derive from the objectives of the Foresight exercise in general and have to be adapted to them. They should be clearly defined, possible to answer, and match the statements made. The statements have to be formulated in a way that the criteria or questions can be judged on the basis of them.

Often questions are related to the date on which an event or development occurs. This is important in order to understand what time horizon you are talking about. The time of realisation is normally asked in five year steps because single years would be so exact that nobody would be able to estimate. A common time horizon in Delphi studies is 30 years ahead (e.g. from now to 2035), but it is also helpful to ask for a later time (after 2035) or "never". The analysis is often done in percentiles (lower quartile, median, upper quartile) in order to show the breadth of the opinions. But simple graphics or percentages can also be used, especially if there is the hypotheses that "statistical camels" occur (there are two opposing groups of participants, one part judges an early time, added normally by high importance, and the other with late time horizons and low importance, representing different lobbies, or different schools of thought).

Other questions may be related to the possible constraints (economical, technological, social, political) to the occurrence of event or development.

The presentation of the data should be thought of in advance and depends on the "clients" or users.

Other criteria necessary for the assessment of the validity of sample and answers like the self-estimation of the "expertise" of the participants may be included.

Selection of the panel of experts

Care is needed in recruiting the panel and the criteria for selection should be set out (problem of bias, which is present in every panel group). Before an expert agrees to take part in a Delphi inquiry, he/she should understand the purpose of the inquiry and should be aware that his/ her expertise should be made available in different rounds of the inquiry. The Delphi method has an iterative nature. If the exercise is to maintain its credibility the tendency for panel members to drop out after the first round should be minimised.

Administration of the questionnaire

The same person should administer and manage the questionnaire and communicate results to the panel members.

Analysis of responses

The analysis of responses consist of statistical analysis of the data, so as to quantify qualitative issues, but also qualitative assessment.

The results are presented in a statistical way. A common approach is that for each question the median (i.e. the central tendency) and inter-quartile ranges (i.e. the middle half of the range outside which lie the upper and lower 25%, or quarters, of the range) are calculated. This information is the basis of the second round of the inquiry and it is sent to the panel members, who are asked to review their estimates in the light of the group opinion. Members who maintain an estimate outside the interquartile ranges are asked to provide a brief justification for their opinion. A new median and interquartile ranges can be calculated and either used as the final forecast or circulated again for further refinement. The questionnaire can circulate until convergence of opinions is reached, but a Delphi inquiry should not have more than four rounds.

Organisation of the Delphi-Process

Figure showing the Organisation of a Delphi survey

Resources needed (time, budget, labour force, skills)

Delphi studies are time-consuming and labour intensive and require expert preparation. However, the total cost will depend very much on the number of participants, the length of the questionnaire, and the general approach. Electronic surveys are much less costly than paper surveys, so Delphi surveys can a range from 20,000 to 1.5 million euros. If printed questionnaires are used, there needs to be sufficient manpower to print, type or scan the results.

Time is essential as the preparation for the data base, the topic creation and the questionnaire design take time. A single Delphi round can easily require three weeks; a three round Delphi questionnaire requires at least three to four months including preparation and analysis of outcomes. In-between the rounds, the experts involved need to have enough time to answer (twice), so that at least half a year should be calculated for (preferably more).

The organisers need management skills, neutrality and to be open to creativity.

Outputs

Typical outputs are reports with tables, lists and figures. The data can be presented in a variety of ways. Even comics (Japanese manga) are used for presentation purposes.

Most commonly, Delphi produces a prediction of a date of occurrence of a sequence of events and assessments of the topic with different criteria. However, other types of judgement may be elicited – the importance of goals, the drivers of change, and practically anything else that might be of interest to study.

Pros and cons

Advantages:

Disadvantages:

Possible variations of the approach

There are implementations of Delphi that are explicitly designed to identify different clusters of opinion, rather than zones of consensus.

"Goals Delphi" methods aim to make it compatible with a normative approach.

There are a number of variations, e.g. making changes to the questionnaire between rounds.

In practice, some so-called Delphis do not make much, if any, use of iterations of the questionnaire. Quasi-Delphi surveys only ask for the topics in the first round and let them be judged upon in the second (only once). Others organise a one-round survey and then discuss the results in workshops. In fact, these are surveys, misappropriating the name "Delphi".

Possible complementary methods

The Delphi method implies identifying statements (topics) that are relevant for the future. Therefore, creativity methods (e.g. brainstorming, 6-3-5 or others), scenarios or key technology can be used in the preparatory phase to define these statements. Data from desktop studies: literature research, patent analysis or bibliometrics can be added. In the analytical phase, different modelling or statistical methods (calculation, rankings, correlations) or the re-building of scenarios as well as pseudo-roadmaps can be used. For comments or additional explanations, qualitative analyses are necessary. A SWOT analysis can be based on the results.

"Is it for me?" - Checklist

Examples

There is a description of a Delphi survey in one of the example cases:

References


ForTransRIS Final Seminar - March 18th 2009, Brussels