- 2.1 Context
- 2.2 Political support
- 2.3 Defining the focus
- 2.3.1 Lessons learned - focus
- 2.4 Setting the objectives
- 2.5 Defining the users
- 2.6 Defining the outcomes
- 2.6.1 Tangible outcomes
- 2.6.2 Intangible Outcomes
- 2.7 Defining the scope
- 2.7.1 Selection of the topics/ perspective adopted
- 2.7.2 Lessons learned regarding scoping
- 2.8 Time horizon
- 2.9 Time frame
- 2.10 Cost/ Funding of the Manufacturing Visions project
- 2.11 Skills and competencies
- 2.12 Team/Coordinator
- 2.13 Implementation plan
- 2.14 Methodology
- 2.15 The main challenges of the design phase – Sum Up
The ManVis project was launched in response to the following factors:
The results from previous Foresight activities and an empirical survey indicated that manufacturing in Europe needs to strengthen its innovative capacity to get into a more proactive position in the face of the increasing pace of product innovation. The need was therefore felt for policy action at both European and national level to support such a transformation.
The increasing debate on relocation of manufacturing to locations outside of Europe and the resulting fear of loss of European competitiveness, employment and quality of life.
The activities of the European Commission (DG RTD Industrial technologies) in support of manufacturing (Manufacturing Action Plan – MATAP) needed to be underpinned by visions of future states of manufacturing and possible paths towards these visions.
As the 7th framework programme for research funding was being developed, priorities for research funding had to be revised and – if necessary redefined – within the whole range of topics that are subject to research funding. ManVis was to support this process in the area of industrial technologies and specifically production processes. The objective was to make sure that the priorities were supporting European manufacturing to meet not only current but also future challenges.
Many Foresight studies already existed but mainly on national level, no vision for Europe as a whole.
The ManVis project took place in the context of various initiatives from the European Commission aiming to address the challenges posed by the future of manufacturing on a European level. Thus, it was explicitly targeting the problem from the perspective of EU policy making. The NMP unit of DG research of the European Commission which is responsible for research funding in the realm of manufacturing had long been aiming at basing their funding decisions on a coherent vision of sustainable and competitive manufacturing in Europe. Therefore, it had financed the 'FutMan' project which was a fully fledged Foresight exercise within the 5th framework programme. The FutMan project, which was finished in 2003, was exploring the 'Future of Manufacturing in Europe 2015-2020' with a focus on 'The challenge for Sustainability'. Within the FutMan project scenarios on future manufacturing patterns had been developed together with a number of detailed case studies on selected sectors and crosscutting themes. At the end of the FutMan project DG Research initiated the Manufuture initiative which aimed to bring together various actors concerned with the future of Manufacturing in Europe, especially industry representatives, to support their strategic decisions and to build up a technology platform on manufacturing. A high level expert group was established and elaborated a vision document as well as a strategic research agenda.
Another actor aiming in the same direction was the EUREKA Factory initiative that is financing concerted national research projects on manufacturing. To generate anticipatory intelligence on manufacturing related issues, EUREKA factory financed a project that was collecting, structuring and analysing currently available studies on the Future of manufacturing (Informan).
The ManVis project consortium was partly formed by research institutes that had been taking part in at least one of these activities. Therefore, there the team was aware of this part of the political landscape and was able to build on the results and experience. The NMP unit, as the client, ensured a close cooperation with the Manufuture initiative, so e.g. they asked the ManVis coordinator to participate in various meetings of the Manufuture group. Some of the ManVis results were taken up in the work of the Manufuture high level group and in turn the ManVis team reacted on the Manufuture documents in their various reports. The ManVis team supported one of the Manufuture conferences with substantial input from the Delphi survey results. Nevertheless, due to time constraints and also different perspectives of the two groups a real exchange in terms of content and results between ManVis and Manufuture was not achieved.
While the ManVis project was being implemented a new emphasis on manufacturing emerged within the European Commissions industrial policy (DG Enterprise and Industry). A Communication was launched outlining a 'Policy framework to strengthen EU manufacturing' (COM(2005) 474 final) comprising several measures to support manufacturing competitiveness in Europe. This highly relevant EU level policy initiative was in no way connected with the ManVis project. Although this did not have direct negative impact on the project it can be concluded that an opportunity to support an ongoing policy making was missed. This could partly be attributed to an incomplete analysis of the political landscape. On the other hand it is due to typical features of the policy making process such as lack of coordination between different actors and fast changing priorities as most Foresight projects have experienced.
Political support for the ManVis project was secured on various levels. From the beginning there was strong support from the client (Directorate Research, NMP) and this positive attitude remained the same even as the head of unit, who had initiated the project, left. However, the principal support did not lead to a high level of engagement in the process. Although provided several opportunities for the client to give input to the project (e.g. to suggest topics for the Delphi survey) this was not done. Support from other DGs was not provided but had not been actively sought. On the national level several of the partners in the various countries approached government agencies for support of the project. In some countries high level participation of policy makers to the workshops could be achieved in others, ministries provided co-funding to enable additional analysis of the ManVis results highlighting national aspects. The final conference was strongly supported by the Slovenian Ministry of Economy and the Slovenian Ministry of Higher Education, Science and Technology.
The ManVis project focussed on the issue of how to underpin the competitiveness of the European manufacturing industry. Accordingly, it was primarily an issue driven Foresight. The geographical territory that was considered was Europe but global aspects were taken into consideration to determine the pathways for Europe. The issue to be addressed by the project as well as the European level of the analysis were defined by the need of the client (see objectives).
It might have been worthwhile to define the focus of the project in closer interaction with the client to improve the impact and usefulness of the outcomes. (see also lessons learned scope).
Providing support to European policy makers on all levels for informed decision making on policy actions and research funding towards sustainable and competitive manufacturing in Europe by describing visions of future manufacturing as well as various elements forming pathways towards these visions.
Support European manufacturing industries in facing the long-term challenges of changing socio-economic framework.
Initiating a discourse on manufacturing issues across boarders of nations and disciplines to enhance communication and knowledge flow between actors of manufacturing innovation in Europe thus enhancing the capability of European manufacturing to proactively meet future challenges by self sustaining innovation initiatives.
The prime user of the ManVis results was supposed to be the client and sponsor (DG RTD of the European Commission NMP unit). Other DGs were not actively targeted but it was expected that they would be informed of the outcomes through the Manufuture initiative. However, national governments in the various countries involved were seen as potential users of the results. Also it was reckoned that other stakeholders such as industry and associations would make use of the results. There was no special interface for this target group (originally it had been planned to develop an 'SME manual' on how to use results from manufacturing Foresight for companies strategic planning but this was skipped in the course of the negotiation in the early phase).
Statements on the future of manufacturing
The core project team developed 100 draft Statements on the future of manufacturing to be used within the Delphi survey. These statements were discussed within workshops that were held in all the 22 countries participating in the project. On the base of the outcome of these workshops the statements were revised and finalised by the core team. The final list of statements can in itself be considered as a tangible outcome of the ManVis process as it was the result of a widespread debate among stakeholders all-over Europe.
The statements were relating to the following topics:
Overall manufacturing sector (55 statements):
- Manufacturing Technology
- Strategy, Organisation and management
- Product features and concepts
- Logistics supply chain
- working conditions
Sector specific statements on the following sectors:
- Machinery (Nace 29)
- Electrical / electronical / optical equipment (Nace 30-33)
- Fabricated metal products (Nace 28)
- Transport equipment (motor vehicles and others) (Nace 34, 35)
- Rubber and plastics (Nace 25)
- "traditional products" combining aspects from the sectors Textile, Textile Products, Leather and Footwear (Nace 17-19), Furniture (Nace 36), non-metallic minerals (ceramics) (Nace 26)
Each county was allowed to add 5 'national specific' statements that would only be answered by experts in their country.
Database from Delphi survey
The online Delphi survey was answered by 2993 experts from all-over Europe. However, not every expert answered all statements. The median number of answers per statement was 1983.
The answers were stored in a database which allowed for an in depth analysis of the results such as comparison between countries, groups of experts (age, type of organisation etc.). The core team members had access to this database to compile the various reports. The national correspondents received the datasets for their respective countries to be able to analyse them in more depth.
When the project was launched a number of formal deliverables in the form of reports had been agreed upon with the client and sponsor of the project (DG-RTD Industrial Technologies, European Commission). These deliverables were all duly elaborated (although some of them with a time delay):
- Documentation describing the results of 22 workshops in 22 European countries, where selected manufacturing experts had been discussing the statements for the Delphi survey proposed by the core team. The documentation gave details of the various arguments, suggestions for new statements as well as the outcome of a 'voting procedure' on the statements to be selected carried out at the workshops. The documentations were elaborated by the national correspondent from each country who had also organised and moderated the workshops and selected the participants using a list of criteria prepared by the core team.
- Intermediary report giving recommendations for the midterm review of the 6 th Framework program for EU R&D funding. Due to the urgent need for input to the midterm review and the planning of the next framework program, a report was produced shortly after the finishing of the 1 st round of the Delphi survey giving preliminary results and first 'messages' for policy in support of European manufacturing.
- Documentation of 'Global context workshop'. This workshop was held to highlight the influence of emerging changes in the global context of manufacturing focussing especially on developments in China, Korea, Japan an U.S. The participants were manufacturing experts from these countries who discussed their views on future developments of manufacturing industries worldwide. This workshop was also documented but the results were then integrated in the Delphi report.
- Documentation of 'Demand perspective workshop'. Within the ManVis project one separate strand of activities was focussing on the influence of possible changes in the demand side on the future of manufacturing. The demand aspects were analysed in two clusters that is
- changes in demands towards products and services
- changes in values and lifestyles leading
The emerging changes were supposed to have effects on manufacturing working environment, management concepts, modes of knowledge generation, interaction of manufacturing industry with customers and users as well as on product features. An interactive workshop was held with participants from various stakeholder organisations concerned with these aspects such as trade unions, environmental NGOs, consumer organisations, and students’ representatives. The participants collected emerging issues in these fields and discussed their perspectives on future developments and its impact on manufacturing. The results of this workshop were documented in depth. The documentation was integrated in the ' Delphi report' (see below).
- Scenario report: The results from the demand perspective strand were further developed within a scenario building approach. Existing scenarios on the future of manufacturing were advanced by integrating the demand side aspects identified in the first workshop. This was supported by a scenario workshop with experts from the respective fields and complemented by a literature analysis. The results were documented in a 'Scenario report'.
- ManVis Delphi Report. This report described the result of the ManVis Delphi Survey in great detail (with the actual data as an annex of the report). The results were discussed within various thematic clusters such as manufacturing and sustainability and manufacturing innovation capacity. The results of the demand strand and on the analysis of the global context were also integrated.
- Policy Report. This Report was summarising the results from the various strands of activities. In addition, conclusions were drawn for policy strategies towards sustainable and competitive manufacturing in Europe.
Naturally, an assessment of intangible outcomes is not as straightforward as the listing of tangible products. Moreover, it needs some time to judge the durability of intangible outcomes such as new networks or changed attitudes among a community of actors. Nevertheless it is possible to give some indications starting from the targeted intangible outcomes as mentioned in the ManVis proposal.
Discourse across established lines of thinking
The project consortium was formed by 30 partners from all-over Europe with widely differing disciplinary and professional background. These diverse partners had to work closely together to implement the project. This initiated a mutual learning process followed by a new type of discourse between perspectives that had not been linked before. This was especially valid for the core team consisting of seven European organisations concerned with widely differing aspects of the future of manufacturing employing different methodology and tools to investigate it. As most of the deliverables were elaborated jointly among the core team and many tasks especially the development of the Delphi questionnaire were fulfilled together within various meetings an intensive exchange of perspectives took place. Although this process of mutual learning also involved a number of conflicts the understanding of different perspectives on the future of manufacturing was certainly enhanced.
A further extension of this dialogue was triggered through the various workshops which again brought together experts from different realms of manufacturing that had to interact to contribute to the projects deliverable (e.g. propose the Delphi statements or advance the manufacturing scenarios). It is most likely, that all these people will spread the cross-disciplinary discourse on the future of manufacturing will into their various contexts such as companies, associations and policy agencies all-over Europe.
Finally, the Delphi exercise served to carry the debate about the future of manufacturing to an even larger audience of manufacturing experts. Even though experts were free to answer only a subset of the statements (which many chose to do), they were at least confronted with the whole set of problems addressed by the survey. Thereby it can be assumed that the survey served to make people with various professional backgrounds aware of perspectives from other fields.
Within the Manufuture conference where a number of workgroups were fed with ManVis results as well as in the ManVis final conference this discourse was further disseminated.
Through the many workshops and gatherings that took place through the ManVis project new linkages between various stakeholders in the realm of manufacturing were established. The networking initiated by ManVis ranged from the local scale (i.e. meeting of managers from two companies within a national workshop) up to the European and even global level (e.g. representative from overseas companies discussing ManVis results within manufuture conference). The network embraced actors from industry, research and policy. The networking benefited from a mutual reinforcement with the Manufuture initiative.
The perspective to be adopted was holistic as this was perceived to be the only way of deriving meaningful knowledge related to the issue. This was reflected in the selection of topics which was in no way constrained to specific policy fields or fields of expertise.
To determine the topics to be covered within the project, the ManVis project team tried to find out which topics were needed to give a satisfying input to the issue at stake that is: how to secure lasting competitiveness for the European manufacturing sector. The following considerations were made:
- In the face of globalisation of markets and production to understand the situation of Europe and how it can be enhanced one needs to look at the global picture. Accordingly it was decided to establish a separate strand of activities looking at global developments in manufacturing (specifically, Japan, China, India and U.S.)
- Based on results of previous projects and studies it was reckoned that one of the major drivers for change of manufacturing in the future would be changes in the demand side of manufacturing. Accordingly, another separate strand in ManVis was set up to deal with demand related changes in manufacturing.
- As a tool to derive expert opinion on important future issues in manufacturing the Delphi survey method was applied. However, it had to be decided, which kind of topics should be addressed in this survey. Although the client was most interested in information about emerging technologies the ManVis team decided to adopt a holistic view of manufacturing as the team thought that technological innovation needs to be embedded in a wider framework embracing organisational innovation as well as workplace innovation. Thus the survey addressed the following topics:
- Manufacturing Technology
- Strategy, Organisation and management
- Product features and concepts
- Logistics supply chain
- Working conditions
The same kind of decision had to be taken when defining the aspects that should be asked in the questionnaire. The team reasoned that to allow for the generation of a vision of sustainable and competitive manufacturing in Europe and also to asses the existing visions within European manufacturing experts it was not enough to ask only about the likelihood or time horizon of the statements. In addition a need was perceived to find out which were the effects and framework conditions attached to these statements to be able to give reasonable recommendations. For this reason the questionnaire covered a range of aspects from quality of life to economic viability. On the other hand there was an urgent need to limit the number of statements and aspects to be asked to limit the time required to answer the survey and thereby raise the reply rate. The choice of topics was therefore guided by a compromise between depth of analysis and practicality.
Another scoping decision was dealing with the sectors of industry to be cobered by the project. The following sectors were selected to be dealt with in more detail:
- Machinery (Nace 29)
- Electrical / electronical / optical equipment (Nace 30-33)
- Fabricated metal products (Nace 28)
- Transport equipment (motor vehicles and others) (Nace 34, 35)
- Rubber and plastics (Nace 25)
- "traditional products" combining aspects from the sectors Textile, Textile Products, Leather and Footwear (Nace 17-19), Furniture (Nace 36), non-metallic minerals (ceramics) (Nace 26)
The sectors were decided by the core team in agreement with the client.
Although the holistic approach is certainly justified when looking at manufacturing from a research point of view, it turned out that it was not easy to draw conclusions from the results that were meaningful for the technology funding programme of the client. Especially the 'soft issues' (such as organisational aspects and working conditions) that were addressed by the survey remained largely untapped for the recommendations to the client. It turned out to be very difficult to integrate the various outcomes from different perspectives and translate them into useful messages for policy making. Probably the holistic perspective should rather have been targeted through integration of aspects than through separation and add on. One could argue that as policy is not acting in a holistic way (that is different departments addressing different policy realms) it is difficult for Foresight to convey messages from a holistic perspective. On the other hand this way the project was able to pass a strong message to policy about the need to align measures from various realms to enable a transition in European manufacturing. However, this message was partly lost as only one of the actors (i.e. the client received it).
Probably it would have been useful to think more carefully before launching the project how, by whom and for which purpose the results will or could be used and to select the topics based on this assessment instead of mainly relying on the research perspective. It should be kept in mind that a broad range of topics does not automatically add up to a holistic perspective and that it is quite difficult to integrate these topics afterwards into a holistic view. From the ManVis project it can be concluded that the integration should come earlier and be more emphasised than the diversification. It can also be concluded, that a holistic approach concering various aspects might lead to an overloading of a project. In this case the holistic approach considered
- 22 nations
- innovations in technology, organisation, working conditions, logistics and supply chain
- 6 sectors (including many NACE classifications)
- researchers from different disciplines and countries
- technology driven and demands side driven innovations
- perspective from inside Europe and from outside Europe.
To reduce complexity and to ensure an integrated holistic result for at least some of these aspects it seems to be necessary to accept a non-holistic approach for some dimensions.
ManVis did not adopt a particular time horizon but most of the developments discussed were ranging from 2010 to 2020
The ManVis project was roughly carried out within two years (2004 and 2005).
The project was funded with 1.57 million Euros by the European Commission (DG RTD G2). Additional funding was provided to some partner through national sources.
The bulk of costs were providing for the work of the project consortium (172 person month). The costs of the workshops were covered through other resources (e.g. partners used their own facilities or were supported by universities etc.). Travel expenses for the experts were not paid except for the demand side stakeholder workshops.
The project was proposed by the coordinator within a competitive call for projects within the 6 th framework programme for research funding of the European Commission. It was evaluated positively and selected for funding. During negotiations between coordinator and client the budget originally foreseen was reduced and some activities cut accordingly.
The project demanded a wide range of skills and competencies. Not least of all, the coordination of the large and extremely diverse consortium (30 partners from 22 countries) required considerable coordination efforts which took almost a full person most of the time.
In-depth knowledge of manufacturing issues in the various fields addressed by the project was necessary to set up the various activities ( Delphi survey generation, scenario workshop etc.) in an adequate way. The facilitation of the workshops demanded skilled facilitators able to moderate the exchange of people from very different professional and institutional backgrounds. Beside that, knowledge on Foresight methodology was crucial to carry out the project.
The online Delphi survey was demanding highly skilled programmers but also design skills to ensure user friendliness.
The coordinator of the ManVis project was Fraunhofer-ISI, a German contract research institute with a strong background in innovation studies and Foresight. The group that was carrying out the project is specialised in analysis of manufacturing innovation and had been involved in both preceding projects (FutMan and Informan).
The demand strand was lead by the Foresight group of JRC-IPTS who specialises in Foresight methodology and also has experience with future oriented analysis of manufacturing through the FutMan project which it was coordinating.
The overseas strand was lead by the Cambridge Manufacturing Institute having in depth knowledge of manufacturing as well as well established connections to manufacturing related research and futures analysis in Asia and United States.
As the ManVis project was carried out as a regular project funded within FP6 it was following the respective administrational requirements. Therefore the implementation plan was elaborated before the start of the project in form of a technical annex of the contract between coordinator, consortium and the client. Accordingly, it contained all the elements required by the contract such as detailed time table, list of deliverables, description of work packages, allocation of resources to tasks and partners.
The main method used in the ManVis project was the Delphi method. In the 'demand strand' a scenario type analysis was carried out. In the following paragraphs the main features of the Delphi survey will be outlined.
ManVis Delphi Survey statement generation
The core element of a Delphi survey is a set of statements describing possible future states of the area under investigation. In ManVis the survey comprised a list of 100 statements on a wide range of manufacturing topics. These statements were generated in the following way:
A draft list of statements was prepared by the ManVis core team within a special meeting. It was developed mainly on the base of results from previous studies on the future of manufacturing. The draft list was sent to all 22 national partners. Each partner translated the statements (if thought necessary) and organised a workshop with around 20 national manufacturing experts. These experts were selected using the same criteria as for the respondents of the survey. The coordinator elaborated and distributed clearly defined tasks for the workshops.
The draft statements were to be assessed with respect to clarity. Furthermore each workshop was supposed to rank the statements as to their relevance for European manufacturing and suitability to be included into the survey. For the latter some criteria such as timeframe of realisation had been proposed by the coordinator.
On top of this, experts were invited to propose additional statements on issues that they thought relevant but the statements did not address.
The core team collected all the input of the workshops and decided on the final list of statements on the base of the workshop results.
Lessons learned – Statement generation
It was crucial to have a workshop in each country, primarily because of spreading the debate was one of the aims of the projects that was very much fostered through the workshops. However the work done in the workshops was a necessary precondition to be able to derive meaningful results from the survey later on. In almost every workshop there was a long discussion on the meaning of the statement and how this could best be 'translated' into the respective understanding of the country. This was not only a matter of language but even more a matter of different understanding of various topics within different national and cultural contexts. The different perspectives within the countries alone were a result that gave interesting insights into the fragmented nature of the European manufacturing community (so e.g. one statement was ruled out in one country as being too 'unrealistic/futuristic' while in another workshop it was dismissed with the argument that this was 'long realised' and therefore not worth to be dealt with in a future oriented study).
Nevertheless there were some doubts afterwards whether the setting of the workshop was ideal. Due to the strict instructions given by the coordinator the discussion in the workshops focused almost exclusively on the assessment of the list of statements prepared by the core team. On the one hand this enabled a focussed discussion and ensured comparable results from the workshops. On the other hand it turned out that almost all workshops were completely occupied with understanding and interpreting the statements as well as suggesting clarifications or correcting the translation. While this was useful for the carrying out of the survey it somewhat restricted the openness and creativity. Almost no workshop came up with new suggestions for topics to be dealt with in the statements. Probably it would have been better to start with a more open brainstorming session to make better use of the wealth of knowledge available in the workshops.
Almost all workshops were conducted only by the national partners, which on one hand facilitated the discussion of the experts by e.g. eliminating language problems, but on the other hand interesting commends, remarks and the common understanding of the statements could only partly be transmitted to the core team by a written report of the national partner.
Another problem that was encountered during the generation of the statements was the lack of interaction with the client. Although the team tried to engage the client into the discussion on the statements this was not taken up. Certainly it would have been much easier to draw relevant conclusions for the client later on if there had been a clear common understanding of the statements and even more if at least some of the statements would have directly related to topics the NMP unit had an immediate concern. Ideally a discussion on the statements alone would have been a support to the decision making process of the client. Probably, it should have been stressed by the team early on that the cooperation of the client was part of the process. Maybe a workshop, similar to the national ones, should have been organised on European level integrating the EU actors.
ManVis Delphi survey – the design of the questionnaire
For each statement the following aspects were asked in the survey (cf. figure below):
- Degree of expertise (Low to high)
- Time of Realisation (<2010 …>2020, Never, Do not know)
- Expected effects of realisation for Europe compared to today (none, increase decrease, do not know):
- Environmental Quality
- Living and working conditions
- Regional Differences
- Two main barriers in Europe blocking the realisation of the statement
- Technical feasibility
- Social acceptability
- EU legislation/standards
- economic viability
- lack of R&D funding
Importance to European manufacturing Industry (4 grades from low to high)
Highest Level R&D (EU, Japan, US, Other, Do not know)
Current position of your country in comparison to Europe (4 grades from top to lagging)
This design is based on previous large scale Delphi studies with some modifications. It was the result of a long discussion of the core team and an attempt to balance the conflicting aims of reducing the effort required to answer the survey while getting as much relevant information out of the results as possible. (See scope)
Carrying out the online survey
The ManVis project carried out the Delphy survey using an online questionnaire. The questionnaire for each Delphi statement was programmed by a company specialised in web design (see picture below). The aspects to be asked in the questionnaire were decided upon by the core team based on previous large scale Delphi exercises (German Delphi, Japanese Delphi, Spanish Delphi, Turkish Delphi) but some modifications were made (see scope). The aspects asked were supposed to be self explanatory but a further explanation was available via the Help (?) Function.
The questionnaire was first developed in English language. It was then translated into 15 languages (some partners carried out the survey in English as they saw no language barrier within their expert community). The translation was done by each national partner online directly within the questionnaire via a tool provided by the web design company.
Selection of experts to answer the survey
The core team established criteria for the selection of experts to participate in the survey. Each national partner was asked to identify a minimum number of experts according to these criteria. The number was corresponding to the number of people employed in the manufacturing sector in the respective country. The criteria were the following:
Type of Organisation
- University or research institute (40%),
- Manufacturing company (40%),
- Others such as consultants, associations, policy actors (20%)
For companies: minimum share of 20% of SME (<250 employees)
Type of expertise (to be specified during data selection only if available)
- Technology (50%)
- Organisation/ Management, Working Conditions (50%)
- Users and Developers of Technologies/Solutions to be included (companies R&D personnel as well as production managers).
- Different levels within the companies to be represented (upper management, operative level, ...)
- Sector structure of the country to be represented
- As many female experts as possible to be included
Approaching the experts
The experts to answer the survey were selected by each national partner (see selection of participants). A list of experts was forwarded to the programmers who created a personal link for each nominated expert to his/her questionnaire and compiled a standardised database containing the data of all experts to be approached in the survey. Each partner had access to their section of the database to allow them to monitor the communication with the experts, and to send e-mails to them containing the personal link to the questionnaire.
In addition a self-registration function was established, where experts could register themselves for the survey via a website. This was done for two purposes:
- to approach experts with unknown e-mail address via a letter indicating the link
- to allow experts to forward a link to the self registration page to other colleagues (this was demanded by a number of partners and participants)
At a fixed date every expert who was registered in the database received an e-mail by the national ManVis partner in her/his country containing his/her personal link and asking them to answer the survey within a certain period of time (six weeks). This e-mail had been previously prepared by each partner in the respective language. Some partners were using a special tool developed by the programming company to address the experts personally (title and name) others were using a general address. After a certain time the experts received one reminder and after another period a second one. After an agreed period of time the survey and the results database were closed (with some allowance for latecomers). Each national partner was responsible to reach a certain minimum number of replies using whatever means they felt appropriate. Most of the partners choose to call some experts personally to ask them to respond to the survey other partners sent reminding letters by post.
As a motivation to participate, a lottery was announced among the participants where experts who had answered the survey could win a weekend in Slovenia (in connection with the ManVis final conference). In addition the participants of the survey were given access to a restricted area in the ManVis website, where the reports could be downloaded and other information related to the topic was posted.
Answering the survey
Before starting the actual Delphi-survey, the respondents were asked to give some information as to their expertise as well as personal details (age, gender, position, type of organisation). In the next stage they were asked to give their opinion about four visionary statements corresponding to four different basic scenarios on the future of manufacturing. Then they were asked to select two manufacturing sectors they would like to answer in addition to the general part. The questionnaire was automatically assembled according to that choice so each expert had around 100 statements to answer. Then every expert, independent of his/her expertise or choice of sectors, was asked to answer the general section of the questionnaire (that is all four thematic fields). The rationale behind this was, that in order to assess the visions within European expert communities it would be important to know how experts were judging developments in fields outside their immediate concern (e.g. expectations of technical experts on organisational changes and the other way round). However, to be able to access the assessment of specialists in each field for each statement the expert was first asked to estimate his/her degree of expertise (see picture of questionnaire).
The programming of the questionnaire allowed the respondent to view a list of all statements and to have an overview of the answers. The questions could be answered one by one, but it was also possible to jump to a particular section. While answering the survey a bar-lever showed the progress within the whole questionnaire. It was also possible to save the questionnaire to continue answering later on. Before finishing an overview was given as to what answers had been done and which ones were missing. When an expert finished the survey the results were transferred to an SPSS database. Here the data was stored anonymously so it was not possible to link answers to names later on.
For experts preferring to answer the survey on paper it was possible to request a pdf version that could be printed out and answered manually. However, as this meant sending a more than 100 pages document by post and then re-enter the results manually into the database, the team tried to restrict this as much as possible. However, some experts actually asked for a paper version.
2 nd round of Delphi survey
As foreseen within the Delphi methodology the ManVis survey was carried out in two rounds. However, the second round was done differently than in previous applications of the Delphi method. First of all a limited number of statements (20 out of 100) was selected for the second round to avoid frightening off the experts with another long questionnaire. For the second round the core team selected the statements that had produced the most controversial results among various expert groups (e.g. different countries) or else were contradictory to other statements’ results. The second round questionnaire contained small texts discussing these contradictions. This proceeding was seen as a major methodological innovation. In fact the results between the two rounds differed much more than usual within Delphi surveys. As usual in a Delphi survey the reply rate was considerably lower in the second round but still sufficient to draw relevant conclusions.
Lessons learned – Delphi survey
- Online survey
In many respects the experience in ManVis verified that an electronic Delphi survey has many advantages over a paper questionnaire. It is cheaper and quicker to send the questionnaire, and the answers are available as soon as they are released by the expert. As the results are already in an electronic format, they can quickly be processed and analysed. Furthermore, additional features can be provided to the expert while filling in the survey such as statistics on the answers given etc. Finally, it is easier to elaborate the questionnaire with a team of researchers from various countries (e.g. the translation could be done via a joint web workspace). However, there are also drawbacks. In some countries there is a high reluctance to answer e-mails from unknown sources. This problem was partly avoided in ManVis, as the e-mails were sent by the national partners who were in most cases known to the experts. Nevertheless, in some countries the e-mails containing the link were erased by spam filters, this was especially a problem in the Netherlands where there seems to be a particular feeling of 'information overload'. Some countries reported that in some companies it is difficult for experts to work online, which kept them from answering the survey.
A number of experts complained about the time required to answer the questionnaire (ca. 2 hours). Many of them abandoned answering the survey after only one statement most likely because they were annoyed by the complexity and length of the survey. It became clear that the e-mail addressing the experts for the first time should have contained a clear indication on how long the answering would roughly require. In any case it was obvious that the questionnaire was too long. Probably the number of statements and even more the number of aspects asked per statement should have been severely reduced. This is even more valid as it turned out later, when analysing the results, that many aspects and differentiations could not be analysed in a meaningful way for the client, due to lack of time but also because of too much detail. The main lesson learned was that one should start with a clear idea on what kind of conclusion one would like to draw from the results and starting from that, work out the questionnaire. Although this was attempted within ManVis, it was probably not carried out rigorously enough. Another conclusion was that there should have been more extensive pre-testing of the questionnaire which – due to time constraint and high pressure to start the survey - had been done mainly internally and in a sporadic manner. A feedback of external experts on the questionnaire would certainly have improved the questionnaire substantially and also facilitated a better assessment of the results later on.
The same is true for the content of the statements. Some of the outcomes could not be interpreted in a way relevant for future oriented strategic decisions for R&D and other policies as it was not clear what to make of it. When analysing the results the team was often unsure whether different assessments among experts resulted from different opinions or different understanding of the statement especially when comparing expert groups from different countries. This type of problem could have been avoided, if more time and resources would have been spent on the preparation phase. Also, the problem is also part of the intercultural approach. Different understanding of a statement can only be avoided to a certain extent, since the reduced form a short statement instead of a longer text leads to diverse understandings even among one cultural group.
Also when analysing the statements the team concluded that it would have been useful to allow for a normative dimension in the questionnaire. As it was, the experts had no possibility to say whether they would like or dislike a statement to come true. Many experts (especially in Sweden) remarked that they were missing this element of personal assessment. But also for the analysis this caused problems as it was often suspected that normative aspects were hidden in the answers so e.g. when experts were stating that some developments will not take place when indeed they meant to say that they hoped that these would never happen.
The survey was answered by almost 3000 experts. With a few exceptions all countries reached the minimum number of answers that was thought necessary to form a valid subset. However, as it was to be expected, not every expert answered all the statements. The median number of answers was 1289 (see below). To be able to make assessments on 'European manufacturing experts' opinion it was decided to apply a weighting on the results depending on the number of people employed in the manufacturing sector. Otherwise it would have meant that countries that were able to motivate more people to answer the survey would have influenced the survey’s result disproportionably.
Lessons learned – data analysis
The survey produced an enormously huge amount of data. To be able to analyse this data required a lot of work and support from specialists in statistical analysis. Accordingly, it took some time to perform all steps of data preparation before the team could for the first time discuss so much as a diagram showing a ranking of the statements in terms of importance for manufacturing. Especially in the face of high pressure from the client to get results quickly, the task was extremely demanding. The effort needed in this phase had been severely underestimated when planning the project. This is especially problematic as the quality of all future analysis of the survey depends on the quality of the work done in this phase. A very important lesson to be learned from ManVis is therefore, to foresee in advance sufficient capacities for preparing the data from the survey and to warn the client well in advance that closure of the survey does not mean immediate access to impressive and meaningful diagrams.
Information gathering to provide an input for the Foresight process was crucial for the project in various stages and for various purposes:
Delphi survey statement generation:
The initial information needed to compile the draft set of statements was taken from previous studies on the future of manufacturing (FutMan and InforMan see Landscape) as well as relevant future oriented studies containing information on manufacturing issues (e.g. Japanese Delphi study). In addition, expertise from the core team partners was used to refine the statements especially the sector specific ones. There was no attempt to derive fresh ideas e.g. through interviews with experts in the field, mainly because the previous studies had been recently carried out using extensive interviews throughout Europe but also due to time pressure in the preparation phase.
Information to underpin decisions on the Set up of project
Some decisions such as number of answers to be achieved per country, weighting factors to be applied in the analysis) needed to be based on quantitative information on the manufacturing sector within the countries involved. To achieve this, an in depth search in the relevant databases (CIS, OECD etc.) was carried out.
Global perspective strand
The analysis within the global perspective strand was based on personal interviews with actors in various countries outside Europe (especially Asia) as well as an analysis of Foresight studies in these countries.
Demand perspective strand
The influence of developments within the demand side of manufacturing was one of the aspects that had been neglected by previous studies. This was why ManVis made a particular effort to integrate this aspect in its analysis. However, this meant that the analysis could only partly start from results of the pre-ceding studies. Information gathering for the demand perspective workshop and the scenario workshop involved an analysis of relevant results from socio-economic research (literature review).
- client integration
- dealing with time constraint
- achieving holistic perspective
- start from imagining required outcomes when developing the methodology
- screening all aspects of the landscape
Running an exercise
Continuous Adaptation of the process
In the course of the manufacturing visions project the process had to be adapted many times. Partly this was due to a delay that was accumulated from the beginning when the contract preparation took longer than expected. Other changes occurred due to new demands from the clients that had again been caused by political events. Finally some of the ideas outlined in the implementation plan proved to be difficult to be put into practice so the proceeding had to be changed. Some examples that might be useful:
- From the beginning the team felt the need to try a new methodological approach to avoid some of the drawbacks of known Foresight tools, especially the Delphi survey. Therefore, it was planned to combine the Delphi approach with a scenario type of analysis. The idea was to assign the Delphi statements to basic scenarios (that were available from a previous project) right from the start. However, this turned out to be difficult in practice. The scenarios were not on a level of detail that allowed for derivation of suitable statements for a Delphi survey. Therefore even though some statements were indeed linked to the scenarios this line of analysis was not further explored.
- While the project was carried out the client initiated a parallel activity to support the strengthening of the European manufacturing research area. Under the heading of Manufuture several expert groups were set up and a series of was conferences held to arrive at a strategic research agenda and set up a Technology Platform. Naturally, the client was interested to form synergies between this initiative and ManVis. At the same time the ManVis team was interested to ‘be in phase with the policy makers needs' to achieve a maximum of impact. This resulted in the need to interact with the manufuture initiative which called for the rescheduling and reshaping of some of the activities. One example was the need to release preliminary results from the Delphi survey earlier than foreseen to feed them into some of the Manufauture conferences. Also the coordinator was asked to participate in meetings of the experts groups and core team members were moderating sessions in the conferences.
- During the projects implementation phase the process of setting R&D priorities on European level was changing. So e.g. the concept of technology platforms was gaining more prominence. In terms of content the focus on 'knowledge intensity' was increasing. The mid-term evaluation of the 6 th framework programme that was supposed to be a major use for the Manvis results was missed due to the delay.
- Finally almost all the partners of the consortium experienced a fluctuation in personnel resources so the team was changing continuously and along with it some orientations of the project.
see methods: Delphi survey
The participants of the demand strand workshops were selected following a careful screening of relevant stakeholder organisations dealing with the main areas that had been identified as demand aspects (demands on products, demands on working and living conditions). While the first workshop was involving activists from the various organisations, the participants in the second one were mainly researchers dealing with the topics that the first workshop had put on the agenda.
Involving participants - Lesson learned (Demand strand)
It was far from easy to establish a group of people from the targeted stakeholder organisations to participate in the project. The reasons were manifold. First of all, to many organisations it was not immediately obvious why they should engage in this kind of process. Organisations like consumer associations as well as environmental and social NGOs are focussing on influencing policy decision making in their respective areas. Accordingly, they are used to participate within negotiations on new legislation or standardisation. However, they are not very much familiar with other policy fields such as R&D policy (with the exception of the trade unions that have strongly developed links to this policy area) and often do not see an immediate reason for engaging into this policy field. Therefore, in many cases it was necessary for the team to convince the participants that this exercise was relevant for them. Another problem was the work overload that many of the activists in such organisations are facing. Many of them have an extremely tight agenda so it is not easy to find a date that suits a relevant number of people. Finally few of the organisations are able to pay for the participation of their members in such exercises. For this reason, the stakeholder workshops were the only ones within the ManVis project were the participants travelling expenses were covered by the projects funding.
The main challenges of running the exercise – Summing up
- Managing Timing
- Maintaining flexibility to adapt the process
- Coping with shifting priorities in the policy arena