In the last decade especially, the issue of monitoring and evaluation (M&E) of counter-terrorism activities—both those related to dealing with the consequences of terrorist activities, that is detection, prevention of attacks, and possible prosecution of their perpetrators (CT), and prevention of extremist activities leading to violence (PCVE)—has become the subject of a great deal of analysis and a number of lively academic discussions as well as the interest of practitioners (coming from both governmental and non-governmental institutions) (Neumann, 2016; Schmid, 2021; Vidino and Brandon 2012). However, most authors have focussed their attention on the methodological problems in an attempt to find the best possible (if not perfect) “evaluation toolset” in order to be employed universally by the practitioners (Horgan, 2021; Marret et al., 2017; Ris and Ernstofer, 2017). In our research we intended to check how widespread are real M&E efforts in three selected neighbouring European states: Belgium, France, and the Netherlands. All these states were selected for the research because of their longstanding terrorist threat, vast experience in combatting terrorism, and relative cultural similarity and geographical proximity. We were interested for the period of 2004–2023, because the first substantial terrorist murder took place in Amsterdam on 4 November 2004.

Romaniuk and Chowdhury Fink (2012, p. 9) observed the following more than a decade ago:

In the last decade, however, it is clear that the issue of evaluating counter-terrorism measures, both those related to dealing with the consequences of terrorist activity, that is detecting and preventing attacks and possible prosecution of their perpetrators, and those associated with preventing violent extremist activities (PCVE) (van Leeuwen, 2023), has become the subject of extensive analysis and lively academic discussion as well as the interest of practitioners from both governmental and non-governmental institutions (Argomaniz and Bures, 2016; Bauman et al., 2018; Baykal et al., 2021; Holdaway and Simpson 2018; Holmer et al., 2023). Today, we can even observe a surge of interest in this issue among civilian specialists and experts—the number of publications is growing (Baruch et al., 2018; Gielen, 2019; Malet, 2021; Nehlsen et al., 2020; Paul and Virgili, 2019; Sinai et al., 2019) and new research and development centres are being established¹ focusing their attention on improving methodological tools that can be used for M&E of ongoing activities in this important area of state activity (Cherney et al., 2020; de Graaf, 2011; Ellis et al., 2011; Horgan and Braddock, 2010; McBride, 2022).

Yet, the issue has rarely received a wider attention in both public and journalistic debates. This lack of public awareness creates a certain dissonance with a broader interest in the subject in professional circles even in the states that are seriously and constantly threatened by extremist activity and its destructive consequences in the form of terrorist acts. Somewhere between this lack of public interest and the increased interest of professionals in the M&E lies the attention of politicians and policymakers as well as government officials involved in counter-terrorism activities. Because state security activities, including those of a anti-terrorist nature, that is CT and PCVE, are overwhelmingly the prerogative of state institutions—they are initiated, designed, planned, implemented, executed, controlled, and financed by these institutions—the question of their evaluation is overwhelmingly dependent on the attitude of political elites and state administration towards the process. Even if counter-terrorism activities are implemented in partnership with non-governmental organisation (NGOs), they are mainly financed with public funds available to the governments of a nation-state.

The issue of terrorist threat itself tends to arouse serious public emotions, especially when terrorists succeed in executing a terrorist attack or when information about the containment of a spectacular attack reaches the public. However, the general population tends not to care much about the concrete measures taken by their governments to ensure their security, and whether the resources allocated for this purpose are justified, as long as they appear to be effective and have the intended result. It is accepted that a state threatened by terrorist activity operates effectively when there are no terrorist attacks; however, in the event of an attack, ad hoc analyses usually suggest that mistakes have been made. In such cases, extraordinary commissions are established, by either the executive branch or parliaments, which attempt to assess ex post where and by whom these mistakes were made, why they happened, and what should and could be fixed (e.g. Benbassa and Troendle, 2017; Feneche and Pietrasanta, 2016).

Sometimes there are efforts to specify what changes in the existing “anti-terrorist system” are needed to make it work more effectively. This motive is often evident in the actions of the countries we analysed, namely Belgium, France, and the Netherlands. The authorities of the nation-states often justify the alleged need to introduce new anti-terrorist instruments that infringe, if not violate, other civil rights and freedoms (the right to privacy, the right to anonymity, the right to know what actions the authorities are taking against them) in order to increase the effectiveness of the state in ensuring security of the population, precisely based on the alleged inadequacies of the existing solutions in the area of anti-terrorism. Rarely, if ever, there are calls for the implementation of new counter-terrorism measures and actions supported by a thorough, informed, and factual analysis of the effectiveness and necessity of those used so far, as well as projections of how the introduction of new countermeasures can increase this effectiveness. Public debates on the issue of terrorist threat and counter-terrorism measures are often held in a moment of shock and awe caused by a successful terrorist attack and are thus marked by the stigma of public emotions aroused by terrorists, including, above all, an emotion as powerful as fear. Politicians confronted with a spectre of an escalation of terrorist threat are compelled in such a situation to calm inflamed emotions, and one of the methods is the assurance that hard and necessary decisions are taken to bolster public security. Rhetorically, these declarations are also intended to increase confidence in the actions taken by state authorities but are rarely supported by evidence-based arguments. Since terrorism (and to some extent anti-terrorism) is a “policy implemented by other means,” that is violence, when analysing the problem of evaluating the actions taken by state services in combating terrorism, political factors must be taken into account, as they determine the nature of the actions taken by the state in response to a terrorist threat.

We conducted our research using qualitative content analysis of non-classified documents and in-depth expert interviews. Special attention was afforded to official reports from various executive and parliamentary commissions that evaluated actions and policies in response to terrorist crises. We reviewed and cross-checked available literature in the field to confront our impressions with the scientific work already exercised (Hecker, 2021; Lahnait, 2021; Leeuw, 2009; Schuurman and Bakker, 2016; Seze, 2019; Veldhuis, 2015; Veldhuis et al., 2011). In-depth interviews were conducted with twenty-two respondents from expert, academic, and government sectors as well as counter-terrorism services in the three studied countries. Purposive sampling was used to select the sample based on the respondents’ role in the analysis of deradicalisation activities in the three countries studied. We used a snowball sampling method—initially we approached the persons we knew personally or had recommendation to contact and asked them for further contacts. Some of the people we contacted refused to take part in the study, others requested anonymity, but the majority of respondents did not object. The study adhered to ethical guidelines ensuring transparency and deference to all contributors. Informed consent was obtained, and the participants were made aware of their right to withdraw from the study at any time.

The sample included individuals with the following characteristics:

Age range: 28–78 years.

Overall, there are more men than women employed in the sectors dealing with anti-terrorist activities; this gender-imbalanced sample was a limitation of our study. Certainly, there remains the problem of representativeness of the sample because our respondents form only a fraction of individuals involved in CT/PCVE activities in respective states. This means that we for sure have not achieved a full knowledge of our research area; however, this was not our goal. We treat our study as an initial step in understanding vital issues and the invitation to go further in the direction we were pursuing. We do not claim that we were able to find a clear, flawless cause and effect “explanation” of problems under study (we consider it a very difficult task to research a complex phenomenon such as CT/PCVE activities in the states involved in anti-terrorism), but we definitely think that we were able to discover a repeating motives which might be a part of possible non – causal explanation. The majority of our respondents were not active participants of governmental actions in the area of CT/PCVE but had an extensive experience in such areas or access to people involved and they all had remained critical of some activities they had performed. Hence, we consider that their analytical skills prevailed in their assessments, providing us with “objective” statements rather than “subjective opinions driven by their personal prejudice.”

All interviews, averaging 45–75 minutes, were conducted in person. Open-ended questions were designed to encourage participants to elaborate on all relevant themes. Interviews were recorded and transcribed for in-depth analysis. The interviews were transcribed by using Braun and Clarke’s (2013) transcription notation system. Owing to the sensitivity of the topics discussed and the need to protect confidentiality of the participants, no transcription software and additional help was used. The diary comprised four parts: interview process, transcribing process, coding, and creating themes, including an overview of merged codes and themes with reasonings.

Owing to the lack of suitable theoretical frameworks to address the focus of this study at the beginning of the research process, a combination of grounded theory (Glaser and Strauss, 1967) and reflexive thematic analysis (RTA) (Braun and Clarke, 2021) was made and transcriptions of the interviews were coded twice to adjust material. During the creation of themes, which was done inductively, and the writing of topic summaries, the codes and subthemes were constantly revisited, merged, and renamed. Adopting the widely accepted approach of Braun and Clarke (2021), coding was part of a six-phase analytical process. The coding procedure itself was broken down into two main steps within a larger thematic analysis framework: familiarisation and generating initial codes, then searching for themes and reviewing them and finally defining and naming the same. In line with sociological tradition, the analysis centred on key excerpts that clearly expressed participants’ perspectives and contributed significantly to research questions.

A widely debated issue is the use of appropriate methodologies in evaluation. Experts interviewed indicate that methodologies should depend on what is actually being subjected to the evaluation procedure. If, for example, it concerns international humanitarian law, then the evaluation framework should be international. In other cases, it should be a project- and programme-specific approach that takes into account the conditions of the project and programme.¹⁵ Interviewee 1 believes that the basis should be a universal framework that makes it possible to compare the evaluations carried out, supplemented—if necessary—with specific methods.¹⁶ We can talk of a partial fulfilment of this premise in case of the European Union (EU), where there are detailed guidelines and indicators of evaluations that must be applied to the assessment of all funded activities (European Commission, 2008). However, they are flexible enough to be modified depending on the project and programmes.¹⁷

Despite the undoubted advantages of evaluation, however, the process has potential risks and can harm the effective implementation of CT/PCVE activities. Interviewee 6 mentioned the assessment of the Utrecht terrorist attack¹⁸ as an example of the misuse of evaluation. In this case, the evaluation was done after the incident had occurred, which severely hampered, or even in some cases paralysed, institution’s work. Many feared that the evaluation was a means of holding them accountable, despite the fact that in the event of a terrorist attack the procedures worked very well.¹⁹ The spectre of the post factum evaluation, as a consequence of the attack carried out, can temporarily have a “freezing effect” until the final presentation of its results. On the other hand, the evaluation made by the French parliament with suggestions to make changes in the French anti-terrorism system (Fenech and Pietrasanta, 2016) was disavowed in this reformist capacity by the French executive, and most of the systemic recommendations were ignored by it.²⁰ Thus, evaluation cannot be seen as a panacea for the weaknesses of counter-terrorism efforts, because, ultimately, its usefulness and impact depend on the (un)willingness of the state bureaucracy to implement the changes proposed by the evaluators.

Evaluation can also be influenced by a country’s political structure. Our research in Belgium showed that the federal structure of the state interfered not only with the conduct of CT and PCVE activities but also with their evaluation. The diverse administrative structure, the complex division of responsibilities, and, in the case of Belgium, ethnic and linguistic diversity were serious obstacles to evaluation.²¹

Political conditions are also often not conducive for evaluation because of the fact that politicians typically expect unambiguous results, often presented in the form of quantitative data, which evaluation rarely produces, as data most often is qualitative rather than quantitative. This approach can also discourage funding for evaluation activities, which unfortunately makes them difficult or sometimes even impossible to undertake. That is why it is important that an evaluation always has a specifically defined objective and a clearly identified scope.²² A well-prepared evaluation requires a suitable budget and time to conduct necessary research and provide appropriate recommendations.²³ Unfortunately, evaluation is regularly reduced to a box-ticking task, and this is not the essence of the process of improving counter-terrorism operations.

Another important issue is restricting access to information and data because of confidentiality or reluctance to share with people outside the organisation, which can warp evaluation results.²⁴ Sometimes this problem arises because organisations involved in CT/PCVE activities are unwilling to cooperate or disclose information.²⁵ In case of accessing and disclosing data, it is crucial to remember that information should never be disclosed to an adversary, which poses an additional challenge in evaluation process.²⁶ Such a problem is particularly observed in deradicalisation programmes, because it is very difficult to assess whether a person put through deradicalisation has definitively renounced violence. Even if recidivism does not occur during first months or even years, there is no definitive assurance that it won’t happen. As many experts have noted, it is difficult for someone to “read their minds” and determine that irreversible changes in attitudes and behaviour have occurred.²⁷ The existing databases are far from complete and lack key information about potential radicals and terrorists, which can distort evaluation results.²⁸

Evaluation of counter-terrorism activities in its current form has its limitations, which must be noted by institutions and those planning to use it. That is why it is important to look for new research methods and tools to employ and test new methodologies, and, above all, to have a greater share of empirical research.²⁹

In an interesting policy brief by Gielen and van Leeuwen (2023, p. 2) of the Dutch think-tank, International Centre for Counter-Terrorism (ICCT), the authors distinguish the following four major “presumptions” that are said to make evaluation of counter-terrorism efforts difficult, if not, according to its critics, downright impossible: (i) “Prevention and non-events cannot be measured and evaluated,” (ii) “concepts are too abstract to make it measurable,” (iii) “M&E of PCVE interventions is too politically sensitive,” and (iv) “We don’t have the time.” Despite these difficulties, as the authors note:

Gielen and van Leeuwen (2023, p. 2) point out the difficulties of implementing M&E programmes for counter-terrorism activities by assuming that these activities are the basis for their success.

Our survey, conducted with twenty-two respondents (n = 22) in Belgium, France, and the Netherlands overwhelmingly supports the thesis that M&E of counter-terrorism activities should take place as a systemic part of counter-terrorism activities. It also points to somewhat different reasons both for why M&E should take place and for the fact that, with the exception of the Netherlands, it is done to a rather limited extent and certainly not systematically and methodically. In this section, we focus on identifying the reasons most frequently cited by our respondents as causing reluctance to use M&E-like procedures in counter-terrorism operations.

Based on the findings of the survey, we would like to expand on the third “presumption” proposed in the policy paper of the Dutch researchers, namely, “M&E of PCVE interventions is too politically sensitive,” and to identify three additional important reasons for the aforementioned reluctance (in case it exists) to conduct systematic M&E of counter-terrorism activities: (1) lack of a broader “culture of evaluation” in the organisation of nation-state conducting counter-terrorism activities as well as long-term planning and review of the effects of implemented policies; (2) lack of knowledge and skills in the use of M&E in both individual institutions involved in these activities and in the state counter-terrorism system, combined with the lack of development of “national” evaluation programmes; (3) belief in the supposed uniqueness of the problem of radicalism and terrorism in nation-states, making any comparative research impossible.

A proud exception on the map of the EU member states where the evaluation of anti-terrorism activities is carried out systemically is the Netherlands. While in Belgium and France, M&E activities are relatively rare, ad hoc, and usually post factum (they take place after some “spectacular terrorist act disaster”), in the Netherlands these activities are conducted on a routine basis, as justified by our focus on the experience of this particular state. This makes the presentation of this particular case study relevant, because it gives a real example of positive approach towards “the culture of evaluation” combined with a substantial number of experts possessing appropriate skills in professional M&E.

The country was hit by political violence at the beginning of the 21st century, when a leading Dutch politician, an openly gay man and a favourite to take over the prime minister’s position, Pim Fortuyn (BBC, 2014), was assassinated in 2002, followed 2 years later by film director Theo Van Gogh (Castle, 2005). The murders shocked the tolerant and open-minded Dutch society and led to groundbreaking changes in the CT/PCVE system led by the passage of “The Crimes of Terrorism Act” in 2004 to combat terrorism more efficiently (Committee of Experts on Terrorism [CODEXTER], 2008). Action in the Netherlands did not stop there but continued. In 2007, MP Alexander Pechtold submitted a parliamentary inquiry into the institutionalisation of the Netherlands’ anti-terrorism policy. His proposal initiated the formation of a committee of anti-terrorism policy experts chaired by Dr. J.J. Suyver. Its work resulted in a series of recommendations adopted by the Dutch parliament in 2009, one of which was a recommendation on the need to conduct evaluations of counter-terrorism policies from a practical perspective. As a result of the efforts of the Suyver Committee, work began on the first national counter-terrorism strategy. From the very beginning, it was agreed that it should undergo an evaluation procedure every 5 years to assess the effectiveness of the measures introduced and their proportionality to the threat (Parlementaire Monitor, 2009). Of course, the introduction of evaluation was not without problems and challenges. The problem was finding the right methodology or defining correct research questions.³⁵

The results of the evaluation of the first CT strategy were used in the second strategy document of this type, which was published in 2016 as planned. They led to changes in the naming of the main points of the strategy from the term “pillars” to “areas of interventions” and other minor changes (National Coordinator for Counterterrorism and Security, 2020). The evaluation also helped to define trends that negatively affected the authorities’ intervention capabilities and recommended a comprehensive approach to counter-terrorism and PCVE.³⁶ The most important aspect, however, was that the evaluation of the strategy document was not just an academic exercise but had a real impact on the shape of all strategy documents.

The evaluation of the “National Counterterrorism Strategy 2016–2020” was even more methodologically rigorous, with detailed main and side research questions, and a division of the work into three phases. Pawson and Tilley’s realistic evaluation model and the (I)CMO model were also used. During the evaluation, external experts were consulted, sharing the knowledge gained during the evaluation process (Scientific Research and Data Center, 2021). The evaluation showed its strengths as well as its limitations; hence, as the work progressed, areas were selected where it could be applied on the widest scale. Changes that were made after the evaluation of the first version of the strategy were also assessed positively (Scientific Research and Data Center, 2021, pp. 4–6). The latest anti-terrorism strategy published in 2022 continued the trend started with the publication of the first version of the evaluation document. At the time of its publication, it was explained that the changes made were precisely the result of the evaluation. The provisions of evaluation were confirmed in the strategy while the document would undergo this procedure (National Coordinator for Counterterrorism and Security, 2022). Thus, the “counterterrorism strategy” in the Netherlands is one of the few documents in the entire EU that is subject to continuous evaluation, planned in advance using the methodologies created with the support of external experts. Additionally, this evaluation is not just an intellectual exercise but actually serves to improve elements of subsequent strategies.

Evaluation in the Netherlands is not just a matter of strategy; practically every major programme or project at both national and local scales undergoes this procedure. Here it is worth pointing to the evaluation of the following document: “Changing needs—need for change: integrating responses to extremist threats. An evaluation of the Dutch Ministry of Foreign Affairs’ counter-terrorism policy and operations.” This evaluation, as in the case of the strategy, was characterised by a specific methodology, a research question referring to the results achieved by the Dutch Ministry of Foreign Affairs (2021) in the area of CT and PCVE during 2015–2021. The evaluation concluded with a number of recommendations that should be implemented to improve operations of the Dutch Ministry of Foreign Affairs. Other evaluation activities addressed the links between terrorism and the state of emergency following the 2015 attacks in France.³⁷ It was not only CT-related activities that were evaluated but also PCVE initiatives. In 2014, the National Coordinator for Security and Counterterrorism (NCTV) and the Ministry of Social Affairs and Employment introduced “The Netherlands Comprehensive Action Program to Combat Jihadis,” in which thirty-eight measures were divided into five groups. This document was subjected to evaluation by the Inspectorate of Security and Justice, and its main task was to check how organisations implement these measures (Ministry of Security and Justice, 2017). It is noteworthy that, as in the case of the previously indicated evaluations, the purpose of this one too was not to evaluate the effects of the measures introduced, because this type of evaluation was considered very difficult, if not impossible to carry out. The PCVE evaluation in 2013–2014 included an evaluation of the reintegration programme for offenders on probation or parole suspected of being involved in terrorism or extremism. The main task of this evaluation was to examine whether the reintegration programme for extremists was a success (Schuurman and Bakker, 2015). This is an example of the evaluation of PCVE programmes; many such projects undergo this process at local levels.

Evaluation in the Netherlands is not only about strategies, projects, and programmes but also includes informal meetings of experts who evaluate and expand solutions at conferences and workshops. This whole ecosystem of formal and informal evolution is a strong point of the Dutch CT/PCVE system and one of the factors that has played an important role in preventing major terrorist incidents since the two killings in the early 2000s.

Currently, the M&E of counter-terrorism activities, both CT and PCVE, is in an ambivalent state. On the one hand, among Belgian, French, and Dutch experts, there is a definite prevalence of the belief that M&E is necessary and should be a permanent element of the counter-terrorism measures undertaken, while on the other hand, as our survey shows, M&E is prevalent only in the Netherlands. The development of evaluation methodologies is of interest for external specialists and experts associated with think tanks and academic centres but is rarely developed strictly in government institutions. The result is an insufficient number of competent specialists to take on the task of evaluating ongoing and implemented counter-terrorism activities in these three countries. Governments also invest far too few financial resources not only in the development but also in the implementation of evaluation programmes. While systematically implemented M&E activities cannot be seen as a panacea for the weaknesses and ills plaguing counter-terrorism programmes, they are undoubtedly capable of providing a wealth of objectified data for in-depth analyses to verify the validity and effectiveness of their application. The sources and causes of reluctance to implement M&E counter-terrorism activities that we have identified, and the benefits as well as the problematic issues arising from their application, are surmountable through activities that promote the solutions already available in this area, and those that help to develop and expand necessary knowledge. This goal should be served by further systematic scientific studies that diagnose the current state of affairs in this area and point out new perspectives for the development of this discipline of terrorism studies, which is developing dynamically today.