Evaluations Promote a Culture of Error: Dr. Susanne Johanna Väth
As charities or networks in the field of development cooperation do we allow some space for introspection? And consider our scope, approach and impact! Do we know if we are relevant at all? To get to the bottom of these questions that often pop-up in our field of work, Siemens Stiftung commissioned CEval GmbH to carry out an external evaluation of the empowering people. Network. We spoke to Dr. Susanne Johanna Väth, Senior Project Manager at CEval – Center for Evaluation to understand all that goes behind to yield an effective assessment.
What was the evaluation methodology used for empowering people.Network?
Evaluating the impact of complex networks needs a complex approach. To evaluate the empowering people.Network (epN) we therefore, applied a contribution analysis. In the first step we developped together with Siemens Stiftung a ‘Theory of Change’ (ToC) to understand the mechanisms through which epN aims to create an impact. In the next step we reconstructed a detailled results model with theory-based hypotheses on the links between the activities and their intended impact. This layed the foundation for an analysis grid with empirically measurable indicators. This guided data analysis and structuring of results. We undertook a mixed-methods approach using different data sources, data collection instruments and analysis methods to validate findings. We carried out an online survey among all epN members and conducted expert interviews and a focus group discussion. We selected 8 epN member organizations for in-depth case studies. Overall, more than 150 individuals from different groups like epN member’s management, staff and stakeholders from the ecosystem shared information in the course of this evaluation.
How did you assess that it was best suited for the organization?
We at CEval follow a participatory approach to make sure that our work suits the needs of our clients. At the same time an initial review of available data and documents helps us inquire which evaluation design fits best. As this was the first evaluation of the epN, comprehensive baseline data was not yet available. Thus, an ex-post facto design had to be applied. Given this limitation and the complexity of the evaluation object, a contribution analysis was the best option to derive at plausible conclusions on the effect of the network.
With respect to learning, we came to the result that the dissemination of training contents and information services among local staff of the epN members offers potential that could be used even more intensively.
Why do you think it is important for social enterprises, charities, foundations to conduct evaluation?
Beyond transparency and accountability, evaluations are helpful to foster organisational learning. They do not only analyse relevance, effectiveness, efficiency, impact and sustainability of an intervention, but also provide evidence-based recommendations on how to improve the intervention itself and the result-orientation of the client. Evaluations allow assessing performance on clearly defined criteria. Therefore, they are often a precondition to apply for grants and enhance chances when applying for loans. Moreover, evaluations help to promote a culture of error. This means that they contribute to a mindset in which mistakes are not perceived as a personal or institutional failure, but as an opportunity for future development. In this regard, evaluations have the potential to trigger innovations.
The evaluation can be downloaded here.