Introduction
In the last decade especially, the issue of monitoring and evaluation (M&E) of counter-terrorism activities—both those related to dealing with the consequences of terrorist activities, that is detection, prevention of attacks, and possible prosecution of their perpetrators (CT), and prevention of extremist activities leading to violence (PCVE)—has become the subject of a great deal of analysis and a number of lively academic discussions as well as the interest of practitioners (coming from both governmental and non-governmental institutions) (Neumann, 2016; Schmid, 2021; Vidino and Brandon 2012). However, most authors have focussed their attention on the methodological problems in an attempt to find the best possible (if not perfect) “evaluation toolset” in order to be employed universally by the practitioners (Horgan, 2021; Marret et al., 2017; Ris and Ernstofer, 2017). In our research we intended to check how widespread are real M&E efforts in three selected neighbouring European states: Belgium, France, and the Netherlands. All these states were selected for the research because of their longstanding terrorist threat, vast experience in combatting terrorism, and relative cultural similarity and geographical proximity. We were interested for the period of 2004–2023, because the first substantial terrorist murder took place in Amsterdam on 4 November 2004.
Romaniuk and Chowdhury Fink (2012, p. 9) observed the following more than a decade ago:
In the field of counter-terrorism generally, the measurement of outcomes and impacts is inherently difficult. Despite the massive investment of resources in this field, especially in the post-9/11 period, few states and multilateral organizations have elaborated robust and succinct methodologies to evaluate their counter-terrorism measures per se.
In the last decade, however, it is clear that the issue of evaluating counter-terrorism measures, both those related to dealing with the consequences of terrorist activity, that is detecting and preventing attacks and possible prosecution of their perpetrators, and those associated with preventing violent extremist activities (PCVE) (van Leeuwen, 2023), has become the subject of extensive analysis and lively academic discussion as well as the interest of practitioners from both governmental and non-governmental institutions (Argomaniz and Bures, 2016; Bauman et al., 2018; Baykal et al., 2021; Holdaway and Simpson 2018; Holmer et al., 2023). Today, we can even observe a surge of interest in this issue among civilian specialists and experts—the number of publications is growing (Baruch et al., 2018; Gielen, 2019; Malet, 2021; Nehlsen et al., 2020; Paul and Virgili, 2019; Sinai et al., 2019) and new research and development centres are being established1 focusing their attention on improving methodological tools that can be used for M&E of ongoing activities in this important area of state activity (Cherney et al., 2020; de Graaf, 2011; Ellis et al., 2011; Horgan and Braddock, 2010; McBride, 2022).
Yet, the issue has rarely received a wider attention in both public and journalistic debates. This lack of public awareness creates a certain dissonance with a broader interest in the subject in professional circles even in the states that are seriously and constantly threatened by extremist activity and its destructive consequences in the form of terrorist acts. Somewhere between this lack of public interest and the increased interest of professionals in the M&E lies the attention of politicians and policymakers as well as government officials involved in counter-terrorism activities. Because state security activities, including those of a anti-terrorist nature, that is CT and PCVE, are overwhelmingly the prerogative of state institutions—they are initiated, designed, planned, implemented, executed, controlled, and financed by these institutions—the question of their evaluation is overwhelmingly dependent on the attitude of political elites and state administration towards the process. Even if counter-terrorism activities are implemented in partnership with non-governmental organisation (NGOs), they are mainly financed with public funds available to the governments of a nation-state.
The issue of terrorist threat itself tends to arouse serious public emotions, especially when terrorists succeed in executing a terrorist attack or when information about the containment of a spectacular attack reaches the public. However, the general population tends not to care much about the concrete measures taken by their governments to ensure their security, and whether the resources allocated for this purpose are justified, as long as they appear to be effective and have the intended result. It is accepted that a state threatened by terrorist activity operates effectively when there are no terrorist attacks; however, in the event of an attack, ad hoc analyses usually suggest that mistakes have been made. In such cases, extraordinary commissions are established, by either the executive branch or parliaments, which attempt to assess ex post where and by whom these mistakes were made, why they happened, and what should and could be fixed (e.g. Benbassa and Troendle, 2017; Feneche and Pietrasanta, 2016).
Sometimes there are efforts to specify what changes in the existing “anti-terrorist system” are needed to make it work more effectively. This motive is often evident in the actions of the countries we analysed, namely Belgium, France, and the Netherlands. The authorities of the nation-states often justify the alleged need to introduce new anti-terrorist instruments that infringe, if not violate, other civil rights and freedoms (the right to privacy, the right to anonymity, the right to know what actions the authorities are taking against them) in order to increase the effectiveness of the state in ensuring security of the population, precisely based on the alleged inadequacies of the existing solutions in the area of anti-terrorism. Rarely, if ever, there are calls for the implementation of new counter-terrorism measures and actions supported by a thorough, informed, and factual analysis of the effectiveness and necessity of those used so far, as well as projections of how the introduction of new countermeasures can increase this effectiveness. Public debates on the issue of terrorist threat and counter-terrorism measures are often held in a moment of shock and awe caused by a successful terrorist attack and are thus marked by the stigma of public emotions aroused by terrorists, including, above all, an emotion as powerful as fear. Politicians confronted with a spectre of an escalation of terrorist threat are compelled in such a situation to calm inflamed emotions, and one of the methods is the assurance that hard and necessary decisions are taken to bolster public security. Rhetorically, these declarations are also intended to increase confidence in the actions taken by state authorities but are rarely supported by evidence-based arguments. Since terrorism (and to some extent anti-terrorism) is a “policy implemented by other means,” that is violence, when analysing the problem of evaluating the actions taken by state services in combating terrorism, political factors must be taken into account, as they determine the nature of the actions taken by the state in response to a terrorist threat.
Methodology
We conducted our research using qualitative content analysis of non-classified documents and in-depth expert interviews. Special attention was afforded to official reports from various executive and parliamentary commissions that evaluated actions and policies in response to terrorist crises. We reviewed and cross-checked available literature in the field to confront our impressions with the scientific work already exercised (Hecker, 2021; Lahnait, 2021; Leeuw, 2009; Schuurman and Bakker, 2016; Seze, 2019; Veldhuis, 2015; Veldhuis et al., 2011). In-depth interviews were conducted with twenty-two respondents from expert, academic, and government sectors as well as counter-terrorism services in the three studied countries. Purposive sampling was used to select the sample based on the respondents’ role in the analysis of deradicalisation activities in the three countries studied. We used a snowball sampling method—initially we approached the persons we knew personally or had recommendation to contact and asked them for further contacts. Some of the people we contacted refused to take part in the study, others requested anonymity, but the majority of respondents did not object. The study adhered to ethical guidelines ensuring transparency and deference to all contributors. Informed consent was obtained, and the participants were made aware of their right to withdraw from the study at any time.
The sample included individuals with the following characteristics:
Age range: 28–78 years.
Gender distribution: twenty-one men and one woman.
Occupation: two government officials, ten academic workers, five think-tank experts and journalists, and five military, police, and special services members.
Overall, there are more men than women employed in the sectors dealing with anti-terrorist activities; this gender-imbalanced sample was a limitation of our study. Certainly, there remains the problem of representativeness of the sample because our respondents form only a fraction of individuals involved in CT/PCVE activities in respective states. This means that we for sure have not achieved a full knowledge of our research area; however, this was not our goal. We treat our study as an initial step in understanding vital issues and the invitation to go further in the direction we were pursuing. We do not claim that we were able to find a clear, flawless cause and effect “explanation” of problems under study (we consider it a very difficult task to research a complex phenomenon such as CT/PCVE activities in the states involved in anti-terrorism), but we definitely think that we were able to discover a repeating motives which might be a part of possible non – causal explanation. The majority of our respondents were not active participants of governmental actions in the area of CT/PCVE but had an extensive experience in such areas or access to people involved and they all had remained critical of some activities they had performed. Hence, we consider that their analytical skills prevailed in their assessments, providing us with “objective” statements rather than “subjective opinions driven by their personal prejudice.”
All interviews, averaging 45–75 minutes, were conducted in person. Open-ended questions were designed to encourage participants to elaborate on all relevant themes. Interviews were recorded and transcribed for in-depth analysis. The interviews were transcribed by using Braun and Clarke’s (2013) transcription notation system. Owing to the sensitivity of the topics discussed and the need to protect confidentiality of the participants, no transcription software and additional help was used. The diary comprised four parts: interview process, transcribing process, coding, and creating themes, including an overview of merged codes and themes with reasonings.
Owing to the lack of suitable theoretical frameworks to address the focus of this study at the beginning of the research process, a combination of grounded theory (Glaser and Strauss, 1967) and reflexive thematic analysis (RTA) (Braun and Clarke, 2021) was made and transcriptions of the interviews were coded twice to adjust material. During the creation of themes, which was done inductively, and the writing of topic summaries, the codes and subthemes were constantly revisited, merged, and renamed. Adopting the widely accepted approach of Braun and Clarke (2021), coding was part of a six-phase analytical process. The coding procedure itself was broken down into two main steps within a larger thematic analysis framework: familiarisation and generating initial codes, then searching for themes and reviewing them and finally defining and naming the same. In line with sociological tradition, the analysis centred on key excerpts that clearly expressed participants’ perspectives and contributed significantly to research questions.
Why is evaluation important?
Looking for an answer to this question, we opted for a desktop research method and structured interviews, in which we asked experts from the Netherlands, Belgium, and France about the issues related to evaluation. All experts agreed that systematic evaluation should take place because it brings significant benefits.2
Benefits derived from evaluation
Despite the positive example of the use of evaluation in the case of the Netherlands, most projects, strategies, and programmes in other countries do not undergo systematic evaluation, making it difficult to understand which measures used to counter extremism are producing positive results. This is important because critics very often argue that counter-terrorism measures, especially in the area of PCVE, are replicated, and no empirical evidence suggests whether they are actually effective (van Leeuwen, 2023, p. 2). CT/PCVE needs a methodological rigorous framework that allows a comparative evaluation of the goals to be achieved with those actually achieved (van Leeuwen, 2023, p. 5).
Listing the main advantages of evaluation, it is often pointed out that evaluation allows learning from mistakes and avoiding those in continued activities, which consequently makes CT and PCVE policies more effective.3 Interviewee 2 said that evaluation allows learning from the past and avoid mistakes from earlier CT practices. The interviewee added that it must be proved through evaluations that old CT measures are no longer adequate and that new measures are necessary. If this is not proved, no new measures should be adopted. He warns that states have a (dangerous) tendency to constantly adopt new CT legislation/measures, even if the need for such laws/measures is not established. That is a dangerous approach, which only leads to an unnecessary expansion of CT apparatus.4 An evidence-based approach to policy making is the only way forward, concludes interviewee 2.5 It also makes it possible to apply the lessons learned from other projects and programmes, thus avoiding the repetition of the same mistakes.6
Systematic evaluation also helps to improve individual police counter-terrorism operations, resulting in learning from mistakes and avoiding them in subsequent operations. This does not have to be limited to just one country but should apply to international cooperation in the field of police counter-terrorism operations. However, it is important that this process should not be used to point out those “guilty” of possible mistakes, but to draw constructive lessons for the future.7
Interviewee 4 points to another important aspect of evaluation, namely, its role in safeguarding human rights, freedom of expression, and the right to privacy, thus ensuring the proper distribution of power and the non-abuse of granted authority by certain institutions.8 It should not be forgotten that CT/PCVE policies are funded by taxpayers’ money, who have the right to demand transparency about how the money is spent and whether it makes sense. Evaluation makes it possible to measure time, money, and other resources invested in given projects and programmes and compare them with the results achieved. This helps to build trust in public institutions, especially those responsible for a critical area such as CT/PCVE (Organization for Economic Cooperation and Development [OECD], 2020).
Evaluation is a part of good public management, enabling policy improvements and project implementations. In addition, the process shows how the existing resources can be better utilised to both achieve desired goals and change strategies. Moreover, it allows for strengthening the capacity of organisations, individual teams, and individuals (Dawson et al., 2014).
The experts we conversed shared their opinions on the conditions that must exist for an evaluation to be carried out. An organisation’s internal culture that promotes the freedom to share ideas, including the most problematic and difficult ones, is crucial.9 In addition, as interviewee 2 observes, it is important to have
a true willingness from politicians and policy makers to keep an open mind about how to best engage in the CT field. This also means being able not to go ahead with a certain measure if the necessity for it is not proven, despite what the public may demand from those politicians and policy makers.10
Evaluation is particularly valuable if it is planned before the start of a CT/PCVE project or programme. First, it allows for improvements to be made even before an initiative is launched, and second, it is not used as a scapegoat to find fault with possible negligence, which might be the case when an ad hoc evaluation is ordered after occurrence of a negative incident.
Internal or external evaluation?
Evaluation should be carried out by external bodies that control the work of police and security apparatus.11 Here, however, no complete agreement was established among experts. Interviewee 3 argues that evaluation should be done internally, because the people involved in the process are more willing to discuss openly and to admit mistakes than when the process is done by an external body. Here he points to the method of evaluation in the Belgian police, where it was carried out by the leader of the operation, assisted by his deputy. Most evaluation procedures in the Belgian police are of this nature. External evaluators are involved only in the largest cases.12 A similar opinion was expressed by interviewee 1, in whose opinion continuous evaluation should be carried out by an internal committee, and only in exceptional circumstances, such activities should be implemented by an external entity.13 According to interviewee 5, it all depends on the nature of the subject of evaluation, but internal evaluation is always desirable, although larger programmes may also require external evaluation. This also helps to guarantee the needed neutrality and objectivity of evaluation.14
Methodology of evaluation
A widely debated issue is the use of appropriate methodologies in evaluation. Experts interviewed indicate that methodologies should depend on what is actually being subjected to the evaluation procedure. If, for example, it concerns international humanitarian law, then the evaluation framework should be international. In other cases, it should be a project- and programme-specific approach that takes into account the conditions of the project and programme.15 Interviewee 1 believes that the basis should be a universal framework that makes it possible to compare the evaluations carried out, supplemented—if necessary—with specific methods.16 We can talk of a partial fulfilment of this premise in case of the European Union (EU), where there are detailed guidelines and indicators of evaluations that must be applied to the assessment of all funded activities (European Commission, 2008). However, they are flexible enough to be modified depending on the project and programmes.17
Problems with evaluation
Despite the undoubted advantages of evaluation, however, the process has potential risks and can harm the effective implementation of CT/PCVE activities. Interviewee 6 mentioned the assessment of the Utrecht terrorist attack18 as an example of the misuse of evaluation. In this case, the evaluation was done after the incident had occurred, which severely hampered, or even in some cases paralysed, institution’s work. Many feared that the evaluation was a means of holding them accountable, despite the fact that in the event of a terrorist attack the procedures worked very well.19 The spectre of the post factum evaluation, as a consequence of the attack carried out, can temporarily have a “freezing effect” until the final presentation of its results. On the other hand, the evaluation made by the French parliament with suggestions to make changes in the French anti-terrorism system (Fenech and Pietrasanta, 2016) was disavowed in this reformist capacity by the French executive, and most of the systemic recommendations were ignored by it.20 Thus, evaluation cannot be seen as a panacea for the weaknesses of counter-terrorism efforts, because, ultimately, its usefulness and impact depend on the (un)willingness of the state bureaucracy to implement the changes proposed by the evaluators.
Evaluation can also be influenced by a country’s political structure. Our research in Belgium showed that the federal structure of the state interfered not only with the conduct of CT and PCVE activities but also with their evaluation. The diverse administrative structure, the complex division of responsibilities, and, in the case of Belgium, ethnic and linguistic diversity were serious obstacles to evaluation.21
Political conditions are also often not conducive for evaluation because of the fact that politicians typically expect unambiguous results, often presented in the form of quantitative data, which evaluation rarely produces, as data most often is qualitative rather than quantitative. This approach can also discourage funding for evaluation activities, which unfortunately makes them difficult or sometimes even impossible to undertake. That is why it is important that an evaluation always has a specifically defined objective and a clearly identified scope.22 A well-prepared evaluation requires a suitable budget and time to conduct necessary research and provide appropriate recommendations.23 Unfortunately, evaluation is regularly reduced to a box-ticking task, and this is not the essence of the process of improving counter-terrorism operations.
Another important issue is restricting access to information and data because of confidentiality or reluctance to share with people outside the organisation, which can warp evaluation results.24 Sometimes this problem arises because organisations involved in CT/PCVE activities are unwilling to cooperate or disclose information.25 In case of accessing and disclosing data, it is crucial to remember that information should never be disclosed to an adversary, which poses an additional challenge in evaluation process.26 Such a problem is particularly observed in deradicalisation programmes, because it is very difficult to assess whether a person put through deradicalisation has definitively renounced violence. Even if recidivism does not occur during first months or even years, there is no definitive assurance that it won’t happen. As many experts have noted, it is difficult for someone to “read their minds” and determine that irreversible changes in attitudes and behaviour have occurred.27 The existing databases are far from complete and lack key information about potential radicals and terrorists, which can distort evaluation results.28
Evaluation of counter-terrorism activities in its current form has its limitations, which must be noted by institutions and those planning to use it. That is why it is important to look for new research methods and tools to employ and test new methodologies, and, above all, to have a greater share of empirical research.29
Why evaluation is not widespread (except the Netherlands)?
In an interesting policy brief by Gielen and van Leeuwen (2023, p. 2) of the Dutch think-tank, International Centre for Counter-Terrorism (ICCT), the authors distinguish the following four major “presumptions” that are said to make evaluation of counter-terrorism efforts difficult, if not, according to its critics, downright impossible: (i) “Prevention and non-events cannot be measured and evaluated,” (ii) “concepts are too abstract to make it measurable,” (iii) “M&E of PCVE interventions is too politically sensitive,” and (iv) “We don’t have the time.” Despite these difficulties, as the authors note:
In the complex arena of Preventing/Countering Violent Extremism (PCVE), Monitoring and Evaluation (M&E emerges as a linchpin for success. While its role in determining impact, steering evidence-informed policy decisions, and guiding efficient resource distribution is undeniable, a web of assumptions often obstructs its full integration and utilization. (Gielen and van Leeuwen, 2023, p. 2)
Gielen and van Leeuwen (2023, p. 2) point out the difficulties of implementing M&E programmes for counter-terrorism activities by assuming that these activities are the basis for their success.
Our survey, conducted with twenty-two respondents (n = 22) in Belgium, France, and the Netherlands overwhelmingly supports the thesis that M&E of counter-terrorism activities should take place as a systemic part of counter-terrorism activities. It also points to somewhat different reasons both for why M&E should take place and for the fact that, with the exception of the Netherlands, it is done to a rather limited extent and certainly not systematically and methodically. In this section, we focus on identifying the reasons most frequently cited by our respondents as causing reluctance to use M&E-like procedures in counter-terrorism operations.
Based on the findings of the survey, we would like to expand on the third “presumption” proposed in the policy paper of the Dutch researchers, namely, “M&E of PCVE interventions is too politically sensitive,” and to identify three additional important reasons for the aforementioned reluctance (in case it exists) to conduct systematic M&E of counter-terrorism activities: (1) lack of a broader “culture of evaluation” in the organisation of nation-state conducting counter-terrorism activities as well as long-term planning and review of the effects of implemented policies; (2) lack of knowledge and skills in the use of M&E in both individual institutions involved in these activities and in the state counter-terrorism system, combined with the lack of development of “national” evaluation programmes; (3) belief in the supposed uniqueness of the problem of radicalism and terrorism in nation-states, making any comparative research impossible.
Political sensitivity of evaluation
Regarding the third point of the ICCT report, that is, the political sensitivity of the M&E of counter-terrorism efforts, it can be mentioned that the reluctance of politicians and policy-makers to check effectiveness of the tool used and its real impact on the external environment may have a number of diverse local causes. According to our interviewees from three European countries threatened by terrorism, the reluctance of politicians to introduce planned, systematic, and permanent M&E of counter-terrorism activities has particularist reasons. The fear that evaluation results may reveal the negative side of counter-terrorism, such as violation of human rights and civil liberties, is an important, but not the only, reason for the unwillingness of political class to use it.
Politicians may also be concerned about whether the evaluation is able to “prove” the presumed need of a proposed new remedy in CT or PCVE operations they are pushing. If a particular remedy is presented by decision-makers as absolutely necessary to improve public safety, while detailed research and analysis show its negligible impact on reducing terrorist threat, then this may in effect compromise the authority of those promoting the implemented solution. Policy planners’ fear of evaluation results does not necessarily stem from the fear of media and public criticism, because, as we have noted above, the detailed methods used in counter-terrorism efforts rarely generate wide and long-lasting media and public interest unless they involve issues closely linked to human rights. One reason for policy planners’ concern may also be a reluctance to seek alternative solutions to the measures already in place but deemed ineffective, which may be the result of evaluation. State bureaucracies, which are, after all, the security apparatuses of nation-states, are reluctant to act beyond their familiar routines and are hesitant to innovate and test measures already in place. Constant critical analysis of one’s own conduct requires open minds to evaluative action, and this is not a dominant characteristic of state bureaucracy personnel. It requires their leadership to be resilient to possible substantive criticism from experts, which is much more difficult to counter than criticism from other politicians.
Lack of culture of evaluation
This problem is closely linked to the presence, or lack thereof, of the so-called “culture of evaluation” in the state organisation. We understand by this term “the broad acceptance and systemic application of evaluation procedures and methods by the organization involved to assess objectively its performance in respective areas.” If esprit de corps lacks the need for applying such methods and they are used scarcely or not at all, we can talk about the non-existence of the “culture of evaluation” in the organisation. Our research shows that the widespread use of M&E in counter-terrorism activities in the Kingdom of the Netherlands is connected to the wide acceptance of such activities in other areas of the country’s operations and their almost routine use. Consistently, although in a negative way, this relates to the lack of such a culture in the state bureaucracies of Belgium and, above all, France, where systematic evaluation of officials’ actions in other areas of its operations is extremely limited. However, it should be noted that even in countries with limited applicability of general M&E activities, they are carried out systematically in some sectors.
In France, for example, systematic and routine evaluation of overseas counter-terrorism operations conducted by units of the French army and the Gendarmerie Nationale (GIGN) is carried out under a procedure known as Retex (Retour de Experience). Owing to the nature of its activities (long-term planning for future military operations of different types and in different theatres), the French Ministry of Défense works with a multi-year perspective, which requires it to analyse past operations and anticipate the nature of future ones, with a strong evaluation and forecasting component. While the details of these activities are, of course, classified, our civilian as well as military interlocutors assured us that significant importance is attached to evaluation activities, as they allow lessons to be learned for future. Interviewee 17 aptly pointed out the fact that the state (unlike the EU) finances activities on an annual (fiscal) basis, rather than a multi-year one, makes it difficult to take a longer-term view of actions.30 Most of our French respondents stressed that anticipating future risks and their prevention is not within the pragmatics of operation of most state institutions. The same is true about M&E. However, even in ministries that operate mainly on a “here and now” basis, such as the French Ministry of the Interior, systematic evaluation of counter-terrorism operations carried out by special police units is sometimes conducted. Here, evaluation is a part of the organisational culture of these relatively small and extremely specific counter-terrorism units within the French National Police (GIGN, Recherche, Assistance, Intervention, Dissuasion [RAID], Groupes d’Intervention de la Police Nationale [GIPN], and Brigade de Recherche et d’Intervention [BRI]), which is understandable, because in the case of counter-terrorism units, the stakes in their game are human lives—including the terrorists, possible hostages, the casual public, and the operators of these units. There the evaluation is carried out to eliminate mistakes that lead to losses of lives and equipment. Although the details of this process are also classified, our interviewed experts confirmed the very fact that evaluations are conducted. Another important factor is a strong bond between members of such units and identification with the goal of this specific organisation, which is to save human lives. In Belgium, too, counter-terrorism units accomplish analogous activities.31
The same is true of the penitentiary systems of the aforementioned countries. The magnitude of the problem they have to solve, that is the ever-growing presence of extremists and terrorists in detention centres and prisons, increases the threat of radicalisation of inmates by incarcerated extremists, which, as it were, necessitates verifying both positive and negative effects of the rehabilitation measures used in penitentiary institutions. In this case, the effect is clearly operationalised, and it is the lack of recidivism of the crime of terrorism among those leaving prisons. The need to evaluate the rehabilitation measures resulted in experimental measures and then in the systematic implementation of M&E processes in penitentiary systems dealing with radicalised inmates.32 Thus, we observe that even in countries where the “culture of evaluation” is not rooted strongly, it is possible if practice influences the formation of a necessary and required belief. However, this does not automatically translate into the prevalence and wider acceptance of such activities in other areas of the state activity involved in counter-terrorism.
Lack of sufficient resources
The second factor often cited by our interviewed experts as influencing the reluctance to implement M&E in counter-terrorism activities was the lack of sufficient numbers of people with necessary knowledge and experience in this area in their respective countries who could provide expert background to state institutions (which is also related to the lack of a “culture of evaluation”).33 Particularly in the case of holistic assessment, finding competent experts is a challenge, given the complexity and interdependency of the CT/PCVE system.34 There is a lack of a broader focus on M&E results in a dearth of research centres devoted to the subject, leaving far too few places where appropriate cadres can be trained to enrich the institutions that conduct M&E. It is assumed that the greater the scale of the state’s counter-terrorism activities, the greater the need to employ educated and experienced specialists in this field. If the state does not treat M&E of counter-terrorism activities as a priority and does not see the need to fund centres that provide expertise in this area, there is a greater likelihood that even limited attempts at evaluation may be deemed unsuccessful (and therefore unnecessary) because of lack of professionalism. Without appropriate and specific methodologies implemented by experts, the process becomes meaningless to produce expected results. A simple comparison of the number of research centres and specialists developing M&E methodologies, the number of experts participating in professional discussions on these topics, and the number of publications on the subject in the countries analysed could be an interesting indicator of each country’s ability to conduct professional M&E activities.
Uniqueness of the problem and the lack of transferability
The third factor identified by the experts we interviewed is the belief about the supposed uniqueness of the problem among those involved in combating and countering political extremism, and therefore the lack of transferability of experiences from other countries or organisations to their own institutional environment. Indeed, the problem of radicalisation and socialisation of individuals involved in political violence is a highly complex, multidimensional, and complicated problem, very much dependent on the specific social environment in which it occurs. However, the belief in the supposed uniqueness of one’s own experiences and the lack of transferability from other organisations builds an unwillingness to learn not only from own mistakes but also that of others. Aversion to, or even ignorance of, “universal” evaluation methods, coupled with a lack of interest in developing one’s own “national” counter-terrorism M&E programmes that take into account these local specificities leads to the abandonment of experimental efforts. Only an empirical determination of whether a given methodology fits the specific conditions or not makes it possible to develop such M&E procedures that can achieve expected goals. However, this requires their systematic growth, and therefore the creation of an infrastructure capable of meeting the demands.
Interestingly, the vast majority of our experts interviewed stated that evaluation of counter-terrorism activities, including both CT and PCVE, should definitely take place, because through it there can be a process of improving and increasing the effectiveness of national programmes to combat political extremism that leads to terrorist violence. Thus, it is assumed that there is considerable unanimity in the expert community regarding a positive assessment of the potential impact of this activity on improving the security of communities attacked by radicals. Accordingly, experts recognise that evaluation makes sense and can bring positive results. At the same time, many experts interviewed were critical of the experience to date in this regard and pointed to what they considered insufficient efforts by key decision-makers to make M&E activity a priority. The existing and already visible potential for M&E in this area can be put to good use, provided that the issue of M&E of counter-terrorism activities gains prominence in the eyes of those responsible for shaping state policy in this important area.
Does the Netherlands have the silver bullet? Evaluation in practice
A proud exception on the map of the EU member states where the evaluation of anti-terrorism activities is carried out systemically is the Netherlands. While in Belgium and France, M&E activities are relatively rare, ad hoc, and usually post factum (they take place after some “spectacular terrorist act disaster”), in the Netherlands these activities are conducted on a routine basis, as justified by our focus on the experience of this particular state. This makes the presentation of this particular case study relevant, because it gives a real example of positive approach towards “the culture of evaluation” combined with a substantial number of experts possessing appropriate skills in professional M&E.
Suyver Committee and the evaluation beginning in the Netherlands
The country was hit by political violence at the beginning of the 21st century, when a leading Dutch politician, an openly gay man and a favourite to take over the prime minister’s position, Pim Fortuyn (BBC, 2014), was assassinated in 2002, followed 2 years later by film director Theo Van Gogh (Castle, 2005). The murders shocked the tolerant and open-minded Dutch society and led to groundbreaking changes in the CT/PCVE system led by the passage of “The Crimes of Terrorism Act” in 2004 to combat terrorism more efficiently (Committee of Experts on Terrorism [CODEXTER], 2008). Action in the Netherlands did not stop there but continued. In 2007, MP Alexander Pechtold submitted a parliamentary inquiry into the institutionalisation of the Netherlands’ anti-terrorism policy. His proposal initiated the formation of a committee of anti-terrorism policy experts chaired by Dr. J.J. Suyver. Its work resulted in a series of recommendations adopted by the Dutch parliament in 2009, one of which was a recommendation on the need to conduct evaluations of counter-terrorism policies from a practical perspective. As a result of the efforts of the Suyver Committee, work began on the first national counter-terrorism strategy. From the very beginning, it was agreed that it should undergo an evaluation procedure every 5 years to assess the effectiveness of the measures introduced and their proportionality to the threat (Parlementaire Monitor, 2009). Of course, the introduction of evaluation was not without problems and challenges. The problem was finding the right methodology or defining correct research questions.35
Evaluation of the fist CT strategy
The results of the evaluation of the first CT strategy were used in the second strategy document of this type, which was published in 2016 as planned. They led to changes in the naming of the main points of the strategy from the term “pillars” to “areas of interventions” and other minor changes (National Coordinator for Counterterrorism and Security, 2020). The evaluation also helped to define trends that negatively affected the authorities’ intervention capabilities and recommended a comprehensive approach to counter-terrorism and PCVE.36 The most important aspect, however, was that the evaluation of the strategy document was not just an academic exercise but had a real impact on the shape of all strategy documents.
The evaluation of the “National Counterterrorism Strategy 2016–2020” was even more methodologically rigorous, with detailed main and side research questions, and a division of the work into three phases. Pawson and Tilley’s realistic evaluation model and the (I)CMO model were also used. During the evaluation, external experts were consulted, sharing the knowledge gained during the evaluation process (Scientific Research and Data Center, 2021). The evaluation showed its strengths as well as its limitations; hence, as the work progressed, areas were selected where it could be applied on the widest scale. Changes that were made after the evaluation of the first version of the strategy were also assessed positively (Scientific Research and Data Center, 2021, pp. 4–6). The latest anti-terrorism strategy published in 2022 continued the trend started with the publication of the first version of the evaluation document. At the time of its publication, it was explained that the changes made were precisely the result of the evaluation. The provisions of evaluation were confirmed in the strategy while the document would undergo this procedure (National Coordinator for Counterterrorism and Security, 2022). Thus, the “counterterrorism strategy” in the Netherlands is one of the few documents in the entire EU that is subject to continuous evaluation, planned in advance using the methodologies created with the support of external experts. Additionally, this evaluation is not just an intellectual exercise but actually serves to improve elements of subsequent strategies.
Programmes and projects evaluated in the Netherlands
Evaluation in the Netherlands is not just a matter of strategy; practically every major programme or project at both national and local scales undergoes this procedure. Here it is worth pointing to the evaluation of the following document: “Changing needs—need for change: integrating responses to extremist threats. An evaluation of the Dutch Ministry of Foreign Affairs’ counter-terrorism policy and operations.” This evaluation, as in the case of the strategy, was characterised by a specific methodology, a research question referring to the results achieved by the Dutch Ministry of Foreign Affairs (2021) in the area of CT and PCVE during 2015–2021. The evaluation concluded with a number of recommendations that should be implemented to improve operations of the Dutch Ministry of Foreign Affairs. Other evaluation activities addressed the links between terrorism and the state of emergency following the 2015 attacks in France.37 It was not only CT-related activities that were evaluated but also PCVE initiatives. In 2014, the National Coordinator for Security and Counterterrorism (NCTV) and the Ministry of Social Affairs and Employment introduced “The Netherlands Comprehensive Action Program to Combat Jihadis,” in which thirty-eight measures were divided into five groups. This document was subjected to evaluation by the Inspectorate of Security and Justice, and its main task was to check how organisations implement these measures (Ministry of Security and Justice, 2017). It is noteworthy that, as in the case of the previously indicated evaluations, the purpose of this one too was not to evaluate the effects of the measures introduced, because this type of evaluation was considered very difficult, if not impossible to carry out. The PCVE evaluation in 2013–2014 included an evaluation of the reintegration programme for offenders on probation or parole suspected of being involved in terrorism or extremism. The main task of this evaluation was to examine whether the reintegration programme for extremists was a success (Schuurman and Bakker, 2015). This is an example of the evaluation of PCVE programmes; many such projects undergo this process at local levels.
Evaluation in the Netherlands is not only about strategies, projects, and programmes but also includes informal meetings of experts who evaluate and expand solutions at conferences and workshops. This whole ecosystem of formal and informal evolution is a strong point of the Dutch CT/PCVE system and one of the factors that has played an important role in preventing major terrorist incidents since the two killings in the early 2000s.
Conclusions
Currently, the M&E of counter-terrorism activities, both CT and PCVE, is in an ambivalent state. On the one hand, among Belgian, French, and Dutch experts, there is a definite prevalence of the belief that M&E is necessary and should be a permanent element of the counter-terrorism measures undertaken, while on the other hand, as our survey shows, M&E is prevalent only in the Netherlands. The development of evaluation methodologies is of interest for external specialists and experts associated with think tanks and academic centres but is rarely developed strictly in government institutions. The result is an insufficient number of competent specialists to take on the task of evaluating ongoing and implemented counter-terrorism activities in these three countries. Governments also invest far too few financial resources not only in the development but also in the implementation of evaluation programmes. While systematically implemented M&E activities cannot be seen as a panacea for the weaknesses and ills plaguing counter-terrorism programmes, they are undoubtedly capable of providing a wealth of objectified data for in-depth analyses to verify the validity and effectiveness of their application. The sources and causes of reluctance to implement M&E counter-terrorism activities that we have identified, and the benefits as well as the problematic issues arising from their application, are surmountable through activities that promote the solutions already available in this area, and those that help to develop and expand necessary knowledge. This goal should be served by further systematic scientific studies that diagnose the current state of affairs in this area and point out new perspectives for the development of this discipline of terrorism studies, which is developing dynamically today.
Recommendations
Evaluation should be introduced at the design and development stage of CT/PCVE programmes and strategies, as it translates into effective policies and helps to make these policies more transparent.
Further work is needed on evaluation methodology, which is currently based on rather simple methods that do not always guarantee the desired success.
There is a need for a solid education of politicians, experts, and citizens about what evaluation is, what objectives it is supposed to achieve, and what benefits it brings.
It is important to strengthen cooperation between decision-making groups, practitioners, and academia, and to translate theoretical models of evaluation into practical action.
It is essential to share experiences of evaluation at the international level, including within the EU (Glazzard and Reed, 2018). In view of the obvious differences in counterterrorism strategies, which are due to the nature of the target group, the cultural and economic context, etc., it is worth considering whether—at least at a basic level—it is possible to construct a framework for the process of evaluation of such policies which could be used for subsequent objective comparison of results.
Related to the above, it could be useful, for example, to develop clear “success criteria” for the programmes being evaluated.
Encouraging project stakeholders to submit their projects to evaluations carried out by independent experts, and to subsequently disseminate the results of these evaluations to stakeholders, including international stakeholders (NGOs and think tanks). This is done to isolate methods that have proved successful and to discard or modify unsuccessful ones. This can prevent the replication of errors in the programme designing of such projects.
Stakeholders, as those directly affected by the implementation of the programme, should be able to prioritise evaluation by defining the importance of different parameters to be evaluated. The evaluation should be useful to stakeholders (i.e. utilisation-focused evaluation) and should answer questions about the operation and effectiveness of the programmes that are particularly relevant to them.
Involving stakeholders in decision-making processes usually increases their involvement in evaluation: they are potentially to understand the purpose of evaluation, cooperate in its implementation, and are more prospective to leverage its results.
