The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts in the relevant field(s).
The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts. It therefore yields both qualitative and quantitative results and draws on exploratory, predictive even normative elements. A single Delphi methodology exists but the applications are diverse. In the most common form, the opinions sought concern the particular developments that are likely to take place. Such Delphis have been widely used in Technology Foresight studies and other exercises. But there are many other types of Delphi exercise possible. They may focus on different topics – on social developments, for example. Instead of trying to forecast the time scales of particular developments, Delphis can be constructed to help identify and prioritise policy goals, for example.
There is agreement that Delphi is an expert survey in two or more 'rounds' in which, in the second and later rounds of the survey the results of the previous round are given as feedback (Cuhls 1998). Therefore, the experts answer from the second round on under the influence of their colleagues' opinions, and this is what differentiates Delphi from ordinary opinion surveys. The idea is that the respondents can learn from the views of others, without being unduly influenced by the people who talk loudest at meetings, or who have most prestige, etc. Ideally, significant dissenters from a developing consensus would be required to explain their reasons for their views, and this would serve as useful intelligence for others. Unfortunately, to carry this out is time consuming, and is quite often missed out from Delphi studies.
Thus, the Delphi method is a 'relatively strongly structured group communication process, in subjects, on which naturally unsure and incomplete knowledge is available, are judged upon by experts', write Häder and Häder (1995, p. 12). Giving feedback and the anonymity of the Delphi survey are important characteristics. Wechsler describes a 'Standard-Delphi-Method' in the following way: 'It is a survey which is steered by a monitor group, comprises several rounds of a group of experts, who are anonymous to each other and for whose subjective-intuitive prognoses a consensus is aimed at. After each survey round, a standard feedback about the statistical group judgement calculated from median and quartiles of single prognoses is given and if possible, the arguments and counter arguments of the extreme answers are fed back...' (Wechsler 1978, pp. 23f.).
Characteristics of Delphi are therefore specified as (see e.g. Häder and Häder 1995):
- Delphi studies always tackle issues formulated in statements about which uncertain and incomplete knowledge exists. Otherwise there are more efficient methods for decision-making.
- Delphi involves making judgments in the face of uncertainty. The people involved in Delphi studies only give estimates.
- The experts involved need to be selected on the basis of their knowledge and experience so that they are able to give a competent assessment. They have the opportunity to gather new information during the successive rounds of the process.
- The Delphi method stresses the psychological processes involved in communication rather than mathematical models (Pill 1971 p. 64; Dalkey 1968 and 1969; Dalkey, Brown, Cochran 1969; Dalkey, Helmer 1963; Krüger 1975).
- Delphi tries to make use of self-fulfilling and self-destroying prophecies in the sense of shaping or even 'creating' the future.
See also the definition of a Delphi Survey in the key terms section.
Usually, the goal (and the result) of a Delphi study is to organise a debate, to collect and synthesise opinions and to achieve a degree of convergence. It is a valuable tool for communication and for exchanging opinions on a topic, making experts' tacit knowledge of the future more explicit. It is also useful for longer-term assessments where extrapolations make no sense. It can help to gather the opinions of a larger group of experts and in fields where there is not a lot of evidence about the developments and where experts often do not dare to explain their real opinion. It is conducted anonymously in order not to let anyone lose face in the event of a change of opinion. The methodology is designed to avoid domination by particular individuals.
Normally, the number of respondents is small, Delphis are not intended to produce statistically significant results. In fact, the outcomes provided by a panel do not predict the response of a larger population or even a different Delphi panel. The outcomes represent the view of a particular group of experts. But in recent surveys, there has been an attempt to achieve a kind of representative sample from a particular community.
The Delphi method is mainly used when long-term issues (up to 30 years) have to be assessed. It is a useful means of predicting and assessing emerging developments where there is no empirical database, where external factors are likely to have a determining effect and where social arguments may dominate economic or technical considerations. As it implies identifying statements (topics) that are relevant for the future, it reduces the tacit and complex knowledge to a single statement and makes it possible to judge. On the other hand, in more complex issues, when the themes cannot be reduced that much or when thinking about and discussing alternatives are the major target, Delphi is not the method of choice. It is also suitable if there is the (political) will to involve a large number of people in processes (Eto 2003).
The Delphi method seems to have managed to withstand the pressures of the changing challenges of the past 50 years. It has been able serve different perceptions of forecasting or Foresight and was probably understood by the users as being an appropriate way to tackle technical perspectives, organisational perspectives, and also personal perspectives. The individual could express a distinctly different opinion as compared to the group perspective and this to a differing degree between the technical details under scrutiny. As multiple perspectives are recommended for decision-making (Linstone/Mitroff 1994; Linstone 1998), the Delphi technique seems to have an appeal in quite diverse situations at the the longer-range end of the scale. As controlled scientific experiments have shown that Delphi estimates are no better than those of other consensus-oriented methods (Woudenberg 1991), it must be the communicative force of Delphi approaches that facilitates the switching between different perspectives. Indeed, Delphi is no longer used to create consensus but to test if there is already consensus about the shape of things to come. What users like in particular are the sets of data about the future that are gathered. Writing down future topics seems to have an immense psychological effect because it transfers implicit and tacit knowledge to more visible, explicit, and therefore transferable knowledge.
Therefore, one can conclude that Delphi and other surveys are tools to bring together the opinions or judgements of a large number of persons. These kinds of surveys are mainly useful in processes where the exchange of opinions and the communication effect is important but which are mainly result-oriented. Especially when data sets are needed for priority-setting, they deliver the essential basics.
Delphi is mainly used in science, technology or education fields, where topics can be stated concisely. For more complex issues, Delphi is not the method of choice.
The major users are companies, particularly strategy departments. In the case of national Delphi studies, the target audience is usually defined as anyone interested in information about the future so, along with companies, the major users tend to be research institutions, ministries, journalists, teachers, students and pupils. This formalised and traceable method has credibility with policy-makers.
Typically involved (i.e. constituting the panel of respondents) are experts from business, government, research, associations and other persons competent in the field of subject, but usually the term expert is used in a relatively broad sense.
Selection of the subject to forecast
The subject should be one where there is a lack of hard data on future trends. In some cases, one thematic field is enough, in many cases the aim is to get an overview so that more fields are decided on and handled in a flexible way. There is always the possibility to add, remove or re-name fields.
Definition of the procedure
The whole procedure has to be fixed in advance. Will panel meetings be set-up or will the teams work on-line? Is the questionnaire an electronic or a paper one? This means that logistics (whether setting up the website or typing in results from the paper versions) have to be organised. Will there be follow-up workshops, interviews, presentations? If so, these also have to be organised and prepared. Printing of brochures, leaflets, questionnaire, reports also have to be considered.
When designing the questionnaire, it is important to consider from the beginning how to give feedback to the participants during the second round. The usual way is to provide percentages or graphics from the accumulated data in a similar way as in the first round questionnaire. But that often gives the impression of a very 'full' picture and too much information has to be shown on one page. The new electronic media provide many more possibilities. There is much room for creativity.
Formulation of the statements
This is a time-consuming process. It has to be made clear where the statements come from. The easiest way is desk research and to take them from literature and surveys that are available. But the more creative way is to set up working groups who have the task to structure the field and formulate topics. It is often necessary to filter the topics twice or even three times because often, the experts in working groups add topics instead of reducing the number. The last step is formulation of the details.
Formulation of the questions
The questions derive from the objectives of the Foresight exercise in general and have to be adapted to them. They should be clearly defined, possible to answer, and match the statements made. The statements have to be formulated in a way that the criteria or questions can be judged on the basis of them.
Often questions are related to the date on which an event or development occurs. This is important in order to understand what time horizon you are talking about. The time of realisation is normally asked in five year steps because single years would be so exact that nobody would be able to estimate. A common time horizon in Delphi studies is 30 years ahead (e.g. from now to 2035), but it is also helpful to ask for a later time (after 2035) or "never". The analysis is often done in percentiles (lower quartile, median, upper quartile) in order to show the breadth of the opinions. But simple graphics or percentages can also be used, especially if there is the hypotheses that "statistical camels" occur (there are two opposing groups of participants, one part judges an early time, added normally by high importance, and the other with late time horizons and low importance, representing different lobbies, or different schools of thought).
Other questions may be related to the possible constraints (economical, technological, social, political) to the occurrence of event or development.
The presentation of the data should be thought of in advance and depends on the "clients" or users.
Other criteria necessary for the assessment of the validity of sample and answers like the self-estimation of the "expertise" of the participants may be included.
Selection of the panel of experts
Care is needed in recruiting the panel and the criteria for selection should be set out (problem of bias, which is present in every panel group). Before an expert agrees to take part in a Delphi inquiry, he/she should understand the purpose of the inquiry and should be aware that his/ her expertise should be made available in different rounds of the inquiry. The Delphi method has an iterative nature. If the exercise is to maintain its credibility the tendency for panel members to drop out after the first round should be minimised.
Administration of the questionnaire
The same person should administer and manage the questionnaire and communicate results to the panel members.
Analysis of responses
The analysis of responses consist of statistical analysis of the data, so as to quantify qualitative issues, but also qualitative assessment.
The results are presented in a statistical way. A common approach is that for each question the median (i.e. the central tendency) and inter-quartile ranges (i.e. the middle half of the range outside which lie the upper and lower 25%, or quarters, of the range) are calculated. This information is the basis of the second round of the inquiry and it is sent to the panel members, who are asked to review their estimates in the light of the group opinion. Members who maintain an estimate outside the interquartile ranges are asked to provide a brief justification for their opinion. A new median and interquartile ranges can be calculated and either used as the final forecast or circulated again for further refinement. The questionnaire can circulate until convergence of opinions is reached, but a Delphi inquiry should not have more than four rounds.Organisation of the Delphi-Process
Delphi studies are time-consuming and labour intensive and require expert preparation. However, the total cost will depend very much on the number of participants, the length of the questionnaire, and the general approach. Electronic surveys are much less costly than paper surveys, so Delphi surveys can a range from 20,000 to 1.5 million euros. If printed questionnaires are used, there needs to be sufficient manpower to print, type or scan the results.
Time is essential as the preparation for the data base, the topic creation and the questionnaire design take time. A single Delphi round can easily require three weeks; a three round Delphi questionnaire requires at least three to four months including preparation and analysis of outcomes. In-between the rounds, the experts involved need to have enough time to answer (twice), so that at least half a year should be calculated for (preferably more).
The organisers need management skills, neutrality and to be open to creativity.
Typical outputs are reports with tables, lists and figures. The data can be presented in a variety of ways. Even comics (Japanese manga) are used for presentation purposes.
Most commonly, Delphi produces a prediction of a date of occurrence of a sequence of events and assessments of the topic with different criteria. However, other types of judgement may be elicited – the importance of goals, the drivers of change, and practically anything else that might be of interest to study.
- The formalisation of the methodology, the amount of data, the number of experts involved and the fact that diverging opinions are partially hidden behind the main converging one, make it popular and credible approach for policy makers.
- As with other well-formalised methods, it forces people to think about the future.
- It gives participants the opportunity to think in more depth and gather further information between the rounds (psychological effect).
- It highlights clearly whether there is consensus on an issue or not.
- There is a psychological effect and a communication effect in being forced to express ideas in a clear and concise way.
- The judgements allows for analyses, rankings and priority-settings.
- The output is in a form which is operational for many actors including policy makers.
- Even oriented towards action, Delphi surveys allow for longer-term thinking.
- Delphi studies are difficult to perform well. They are fairly time-consuming and labour intensive and require (external) expert preparation. They are therefore expensive.
- The consensus obtained in the second round is often artificial.
- However, there is a danger of regarding results as facts.
- Single opinions that might be of special value are also pooled and normally ignored. Only the accumulated results are published to preserve anonymity. It is difficult to find out reasons for dissenting answers later on, as this anonymity has to be respected.
- A poorly designed Delphi will provoke antagonism and elicit poor quality information. It may fuel criticisms of the overall Foresight activity with which it is associated. Therefore, a great deal of attention must be given to the choice of participants, the questionnaire must be meticulously prepared and thoroughly tested to avoid ambiguity.
- Care has to be taken over group effects. As in all panels or expert groups, the opinions will reflect the set of participants involved: a narrow set of criteria for these may lead to unrepresentative views or miss out important sources of knowledge.
- Some participants drop out during the process (especially after the first round). In addition, although further qualitative assessment of Delphi inquiry may produce useful information, this step is often not carried out due to lack of time.
- It is often difficult to convince people to answer a questionnaire twice or more and incentives may be needed (e.g. that the experts receive the results). The dropout-rate increases after the second or third round, so most current studies are limited to preparation and two rounds.
- A Delphi survey is actually always a mix of methods because a topic generation procedure is needed.
- It is not applicable in all fields or cases, because the statements have to be formulated relatively quickly. Even when it is applicable, this short formulation reduces the complexity.
There are implementations of Delphi that are explicitly designed to identify different clusters of opinion, rather than zones of consensus.
"Goals Delphi" methods aim to make it compatible with a normative approach.
There are a number of variations, e.g. making changes to the questionnaire between rounds.
In practice, some so-called Delphis do not make much, if any, use of iterations of the questionnaire. Quasi-Delphi surveys only ask for the topics in the first round and let them be judged upon in the second (only once). Others organise a one-round survey and then discuss the results in workshops. In fact, these are surveys, misappropriating the name "Delphi".
The Delphi method implies identifying statements (topics) that are relevant for the future. Therefore, creativity methods (e.g. brainstorming, 6-3-5 or others), scenarios or key technology can be used in the preparatory phase to define these statements. Data from desktop studies: literature research, patent analysis or bibliometrics can be added. In the analytical phase, different modelling or statistical methods (calculation, rankings, correlations) or the re-building of scenarios as well as pseudo-roadmaps can be used. For comments or additional explanations, qualitative analyses are necessary. A SWOT analysis can be based on the results.
- How broad should the study be?
- How many fields should I ask for (and which should they be)?
- How will it be organised? Who manages the process?
- Do I have the necessary skills and resources?
- What results can be expected?
- What are the questions being asked?
- How is the questionnaire designed?
- Who will be invited to participate (active or non-active)?
- What kind of analysis is needed for the objectives of the survey and has to be facilitated by the data gathered?
- How do you intend to implement the results?
- Will there be follow-up activities (public relations, publications, workshops, presentations, conferences etc.)?
There is a description of a Delphi survey in one of the example cases:
- Delphi method and its relevance in management
- Blind, K.; Cuhls, K. and Grupp, H.: Personal attitudes in the assessment of the future of science and technology: A factor analysis approach, in: Technological Forecasting & Social Change 68 (2001), pp. 131-149.
- Cuhls, K. (2005): Delphi surveys, Teaching material for UNIDO Foresight Seminars.
- Cuhls, K. (2000): Opening up Foresight Processes, in: Économies et Sociétés, Série Dynamique technologique et organisation, no. 5, pp. 21-40.
- Cuhls, K. (1998): Technikvorausschau in Japan, Heidelberg: Physica (Technik, Wirtschaft und Politik 29).
- Cuhls, K.; Blind, K. und Grupp, H. (2002): Innovations for our Future. Delphi '98: New Foresight on Science and Technology. Technology, Innovation and Policy, Series of the Fraunhofer Institute for Systems and Innovation Research ISI no. 13. Physica Heidelberg, pp.15ff.
- Cuhls, K.; Blind, K. and Grupp, H. (eds., 1998): Delphi '98 Umfrage. Zukunft nachgefragt. Studie zur globalen Entwicklung von Wissenschaft und Technik, Karlsruhe.
- Cuhls, K.; Breiner, S. and Grupp, H. (1995): Delphi-Bericht 1995 zur Entwicklung von Wissenschaft und Technik - Mini-Delphi -, Karlsruhe 1995 (later print as BMBF brochure, Bonn 1996).
- Cuhls, K. and Kuwahara, T. (1994): Outlook for Japanese and German Future Technology. Comparing Technology Forecast Surveys, Heidelberg : Physica (Technology, Innovation, and Policy 1).
- Dalkey, N. C.: Analyses From a Group Opinion Study, in: Futures, vol. 2 (1969) no. 12, p. 541-551
- Dalkey, N. C. (1968): Predicting the Future, Santa Monica .
- Dalkey, N. C. (1969): The Delphi Method: An Experimental Study of Group Opinion, prepared for United States Air Force Project Rand, Santa Monica .
- Dalkey, N. C.; Brown, B. and Cochran, S. (1969): The Delphi Method, III: Use Of Self Ratings To Improve Group Estimates, Santa Monica.
- Dalkey, N. C. and Helmer, O.: An Experimental Application Of The Delphi-Method To The Use Of Experts, Journal of the Institute of Management Sciences, in: Management Science, 9. Jg. (1963) S. 458-467.
- Eto, H. (2003): The suitability of technology forecasting/ foresight methods for decision systems and strategy. A Japanese view, in: Technological Forecasting and Social Change, no. 70 (2003) pp. 231-249.
- Gordon, T. J. und Helmer, O. (1964): Report on a Long-Range Forecasting Study, Rand Corporation, Santa Monica / California .
- Häder, M. and Häder, S. (1995): Delphi und Kognitionspsychologie: Ein Zugang zur theoretischen Fundierung der Delphi-Methode, in: ZUMA-Nachrichten, vol. 37, 19. November 1995, p. 12.
- Kaplan, A., Skogstad, A. L. and Girshick, M. A. (1950): The Prediction of Social and Technological Events, in: The Public Opinion Quarterly, XIV, pp. 93-110.
- Krüger, U. M. (1975): Die Antizipation und Verbreitung von Innovationen. Entwicklung und Anwendung eines kommunikations-strategischen Konzeptes unter besonderer Berücksichtigung der Delphi-Technik, Köln.
- Linstone, H.A. (1999): Decision Making for Technology Executives. Using Multiple Perspectives to Improve Performance. Artech House: Boston / London .
- Linstone, H. A. (1998): Multiple Perspectives Revisited, IAMOT Conference, Orlando .
- Linstone, H. A. with Mitroff, I. I. (1994): The Challenge of the 21st Century: Managing Technology and Ourselves in a Shrinking World, Albany : State University of New York Press.
- Linstone, H. A. and Turoff, M. (eds., 1975): The Delphi Method - Techniques and Applications, Reading: Addison-Wesley.
- National Institute of Science and Technology Policy (NISTEP, 1997): The Sixth Technoloy Forecast Survey - Future Technology in Japan - (NISTEP Report, English Translation), Tokyo .
- Nedeva, M.; Georghiou, L.; Loveridge, D. and Cameron, H. (1996): The use of co-nomination to identify expert participants for Technology Foresight, R&D Management, vol. 26, no. 2, pp. 155-168
- Pill, J. (1971): The Delphi Method: Substance, Context, A Critique and an Annotated Bibliography, in: Socio-Economic Planning Science, vol. 5 (1971) p. 64.
- Wechsler, W. (1978): Delphi-Methode, Gestaltung und Potential für betriebliche Prognoseprozesse, Schriftenreihe Wirtschaftswissenschaftliche Forschung und Entwicklung, München.
- Woudenberg, F. (1991): An Evaluation of Delphi , in: Technological Forecasting and Social Change, vol. 40, pp. 131 – 150.