OeNB Financial Literacy Evaluation Series

The full report is available here in PDF and HTML formats. 

Additionally, a blog article on mixed methods can be found here (only in German available).

Authors: Theresa Lorenz
Year: 2024

Mixed-method evaluations combine qualitative and quantitative research methods, offering a deep and comprehensive analysis of complex social phenomena. This approach is widely regarded as the gold standard in evaluation research. However, effectively integrating these methods can be challenging. One common issue is the tendency to treat the findings from each method separately (e.g., in distinct publications), rather than fully integrating them. This separation can hinder a holistic understanding of the evaluation results, limiting the potential of a mixed-methods approach. In this paper, we provide an overview of various mixed-methods evaluation designs and aim to mitigate some of the challenges faced by researchers employing this approach. Our goal is to equip evaluation researchers with the knowledge and tools needed to incorporate mixed methods into their methodological toolkit, enabling them to apply this approach effectively.

The full report is available as PDF and HTML.

Authors: Maximilian Zieser, Valentin Voith, Theresa Lorenz and Sofia Anyfantaki
Year: 2024

Financial literacy has become an increasingly important research and policy topic in the past decades and has been adopted by a multitude of stakeholders. However, there is little consensus on how to define financial literacy, how to measure it or how empirically relevant it is. For related financial education interventions, evaluations and research to be successful, however, it is essential to have a clear idea of the underlying concept and its potential implications. This paper aims to provide both an overview of and guide to the concept by exploring six perspectives of financial literacy: its historical origins, its key stakeholders, its various definitions, the approaches to its measurement, its determinants and its potential impact. Indeed, we find that concepts of financial literacy, and thus its implications for research and policymaking, vary considerably. We therefore recommend using coherent definitions and a rigorous approach to measurement and reporting to ensure that future evaluation and research projects produce meaningful, interpretable and comparable results.

The full report is available as PDF and HTML.

Author: Katharina Felbermayr
Year: 2024

How can qualitative evaluations be carried out? Which are the methodological steps that need to be considered? Which qualitative methods can researchers use to collect and evaluate data? What are the advantages and disadvantages of each method? In which contexts is qualitative evaluation actually appropriate?
These are the questions this report is meant to provide answers to. It gives an overview of the main steps in the different phases of qualitative evaluations. These include planning, implementation, analysis and communication. In section 1, I introduced the characteristics of qualitative research. An important step in the planning phase is selecting adequate methods for collecting and analyzing data. Advantages and disadvantages of common qualitative surveying methods – interviews, focus groups, observations and desk reviews of documents – are mentioned in section 2. I discussed the development of appropriate survey instruments for each method in more detail in section 3. Next, section 4 dealt with a topic that is often strongly underrepresented in the literature, although it is essential for implementing evaluations or studies successfully: sampling and access to the research field. In section 5, I discussed how to apply the different methods, while section 6 focused on the analysis of qualitative data.

The full report is available as PDF and HTML.

Authors: Theresa Lorenz and Katharina Felbermayr
Year: 2025

Like all research involving individuals, evaluation research presents specific ethical and data protection challenges that must be considered throughout the entire evaluation process, from the evaluation design to the final publication. In this introductory article, we provide an overview of the key legal and ethical aspects that are typically relevant for the evaluation of financial literacy interventions. To do so, we present important legal terminology and regulations alongside fundamental ethical principles. Furthermore, we suggest potential strategies for ­addressing ethical and data protection challenges.

Financial literacy provides long-term support to individuals in their personal financial lives. Due to different target groups (e.g. students, parents, working adults and elderly people), the diversity of topics and the varying relevance of these topics depending on life situations, a wide range of financial education programs is required to address specific needs. Currently, there is limited solid knowledge about which financial education programs are effective. Whether these programs achieve their intended impact can be determined through evaluation studies. Depending on the program and context, different evaluation methods should be preferred. Choosing an appropriate method, planning and conducting an evaluation, and interpreting the obtained results each pose various challenges. There  already exist several conceptual handbooks and toolkits serving as guides for evaluation studies. Examples include the World Bank's Evaluation Toolkit¹  and OECD's² work on evaluating national financial literacy strategies. However, aspects of evaluation research, especially regarding rigorous evaluation methods, are not yet sufficiently covered in existing handbooks. The concept of "financial literacy" provides the basis for measuring financial knowledge and skills. Research in this area is still young and rapidly evolving, leading to a variety of approaches, definitions and measurements , which further complicate comparisons between evaluation results.

The OeNB Financial Literacy Evaluation Series of the Oesterreichische Nationalbank is a comprehensive guide that aims to inform researchers, policymakers and educators about the current state of research on financial literacy and education. It provides an overview on the concept of financial literacy, reviews literature addressing the effectiveness of financial education programs and gives guidelines for designing and implementing comprehensive evaluation studies. It includes, among others, guides on quantitative, qualitative and mixed-method evaluation studies. The series is currently in development, with new papers set to be published over the coming months. The series was developed by the OeNB (Theresa Lorenz, Katharina Felbermayr, Sandra Mauser, Valentin Voith and Maximilian Zieser) in collaboration with the Bank of Greece (Sofia Anyfantaki) as part of the ESCB System-wide Virtual Teams Programme.

 

_{¹ Yoong, Joanne, Kata Mihaly, Sebastian Bauhoff, Lila Rabinovich, and Angela Hung. 2013. Toolkit for the Evaluation of Financial Capability Programs in Low and Middle-Income Countries. Washington DC: World Bank. Copy at https://tinyurl.com/yd58758o}

_{² OECD (2022), OECD/INFE Toolkit for Measuring Financial Literacy and Financial Inclusion 2022, www.oecd.org/financial/education/2022-INFE-Toolkit-Measuring-Finlit-Financial-Inclusion.pdf}