Review of Approaches to Assessing Anti-Corruption Measures Released

The paper stresses that the assessment of effectiveness of anti-corruption interventions is an important element of the fight against corruption that allows analysing whether the efforts made are adequate.

However, the conduct of such evaluation and subsequent employment of its findings to correct action can be hampered, inter alia, if:

anti-corruption measures were initially developed in a way that the assessment of their effectiveness is an extremely labour-intensive process;
The theory of change (ToC*) was not taken into consideration or was not employed in a correct manner in the implementation of anti-corruption measures;
The attention was focused on the very fact of adoption of anti-corruption measures rather than on their impact;
The evaluation is carried out only in the final stages of implementation of anti-corruption measures;
The impact of gender dimensions and intersectionality on anti-corruption measures was not taken into consideration;
A limited set of evaluation methods was used, in particular, only quantitative indicators that do not reflect the impact of implementation of anti-corruption measures was employed;
The monitoring data were not used in the assessment process.

The conduct of evaluation is hindered, among other things, by the fact that those responsible have to navigate different conceptual notions of corruption and take into consideration the importance of political, economic and socio-cultural factors.

What is more, deep analysis of the quality of the evaluations of anti-corruption measures has not been conducted yet.

To fill the gap, the authors of the paper scrutinized 91 sets of data concerning the evaluation of effectiveness of anti-corruption measures publicly accessible between January 2010 and October 2023** by such bodies and organisations as:

The analysis was conducted in two stages.

First stage. At this stage, U4 experts analysed the very possibility to conduct an effective evaluation of anti-corruption measures, taking into consideration, in particular, the following aspects:

Is there any conceptual understanding of the problem the solution of which implies the adoption of the anti-corruption measures assessed?
Is the theory of change used in the implementation of the anti-corruption measures assessed / is it clear how the anti-corruption measures assessed should lead to the expected outcome?
Is any data associated with the implementation of the anti-corruption measures assessed gathered to track the changes over time?

This stage concluded with the following findings:

Conceptual clarity of the problem addressed was inherent only to 12% of the evaluation systems explored;
The theory of change was taken into consideration only in 20% of cases; furthermore, only in one case the contextual analysis and the main assumptions and risks were clearly reflected;
Flexibility in the use of the theory of change is also highlighted in one case; in the other cases, the expected outcome were fixed with no possibility to revisit them in case of unforeseen circumstances;
Collection of the data necessary to conduct evaluation started only in the final stages of implementation of anti-corruption measures in 88% of cases.

Second stage. At this stage, the U4 experts analysed the quality of evaluation of anti-corruption measures, including the following facts:

Does the evaluation take into account such principles as relevance, consistency, efficiency, effectiveness, impact and sustainability?
Is the assessment based on a structured research design?
Is there any critical assessment of the theory of change employed?
Is the outcome of the evaluation reliable and do they represent a true assessment of the impact?
Does the evaluation help to understand the outcome of the impact?
Does the evaluation take into consideration gender dimensions and intersectionality?
Was the evaluation conducted in a way to enhance the probability of the subsequent use of its findings?

The authors make the following conclusions in the end of this stage:

The average duration of evaluation was of three months – in the opinion of the U4 experts, this period was not sufficient to make a high-quality analysis;
76% of cases lacked structured design to conduct evaluation;
69% of assessments are standardized and are not specific for certain anti-corruption measures;
In 80% of cases, the assessment was aimed exclusively at analysing the measures adopted in the framework of a certain anti-corruption programme, which does not make it possible to understand wider dynamics;
Contextual analysis was held only in 1/3 of the evaluations reviewed; moreover, this analysis was generally superficial and did not always relate to the findings of the evaluation;
In 92% the information on the research methods employed was published, which indicates a high level of transparency;
In 14 cases (15%), in spite of the fact that the theory of change was not employed in the beginning, evaluators made attempts to provide their own interpretations of the changes that had taken place;
In 62% of evaluations no form of qualitative measurement of corruption was employed, or the methods used had considerable flaws to understand the changes;
In 61% of evaluations, at least one representative of the state reviewed took part in the evaluation process; however, only in 1/3 of all evaluations a contract to conduct the assessment was foreseen – either with a domestic organisation or a group of individuals including a representative of the respective jurisdiction;
Only in seven assessments (8%) at all stages the gender dimensions and insersectionality were taken into consideration;
66% of assessments are presented and organised in a manner that the reader can easily comprehend the main ideas.

In order to enhance the effectiveness of evaluation of anti-corruption measures, the U4 experts put forward the following recommendations:

Develop anti-corruption measures in such a way that it is possible to conduct their subsequent assessment, including the use of the theory of change based on the contextual analysis and with the support of relevant indicators;
Revisit the existing organisational models targeted at evaluations at the final stages of implementation of anti-corruption measures in a short time, in particular, by increasing the resources allocated to the evaluations, starting the assessment process already in the beginning of implementation of anti-corruption measures, increasing the time for the assessment;
Hold training and awareness-raising initiatives for the persons responsible for conducting evaluations so that they have deeper understanding of the theory of change in developing and implementing anti-corruption measures;
Actively employ the methods that allow for establishing the cause-and-effect relationships;
Take into consideration gender dimensions and intersectionality throughout the evaluation process;
Use all available methods to measure corruption and related offences, if necessary;
Encourage the persons responsible for conducting the evaluations to take into consideration the findings in other spheres, for example, in public procurement;
Make publicly available all materials of the evaluations conducted.

* The theory of change (ToC) is an approach to developing and assessing the documents of strategic planning, measures of public policy and separate projects and programmes (for instance, to organise anti-corruption awareness-raising) aimed at getting and visualizing the most exhaustive and consistent list of actions necessary to achieve the final result (effect).

As per the ToC, every instrument of influence should be focused on the achievement of a specific change - without a reasonable assumption about how the desired change will be achieved an effective use of the instrument is unlikely. The ToC makes it possible to describe the cause-and-effect relationships in addressing a specific problem, as well as the emerging assumptions and risks and link certain actions in the implementation of a policy with the final result as planned.

The ToC includes the following key elements:

Definition of the desired outcome: clear definition of what you want to achieve as a result of change;
Analysis of the current state: understanding of the current situation and detection of the problems to be solved;
Setting of goals and objectives: formulation of specific, measurable and achievable goals and objectives that should be reached to achieved the desired result;
Determine the necessary resources: define the necessary resources (financial, human, technological etc.) to make changes;
Develop implementation strategy: draft an action plan describing how and when the objectives and goals will be reached;
Exercise monitoring and assessment: regular monitoring of the progress and assessment of effectiveness of changes to take corrective action and improve the process;
Take into consideration the feedback: collection and analysis of feedback from stakeholders to improve the change.

** The U4 experts stress that the research conducted had a number of limitations, including:

1. The research did not take into account the following evaluations:

Those that do not concern directly anti-corruption measures and regarding related areas, for example, public administration and the rule of law;
Those issued by civil society organisations in specific countries;
Those that do not imply the publication of regular reports;

2. The evaluations reviewed in the paper could not have been published in full;

3. With regard to some evaluations reviewed, there were not full data on how they had been conducted;

4. The information on the context in which the evaluations had been carried out was unavailable.