What the Impact Measurement of Educational Projects Has Shown

In the recent year, we have focused on 5 selected projects, the winners of the second year of the Generation 3.0 Awards. We have worked to determine their effectiveness and dissemination potential. Below, you can read about what we have found and what we have learned in the process.

The Primary School at Dubová 1 in Bratislava is an inclusive school that inspires change. The emphasis is on an individual approach, soft skills development and teamwork. The Better Geography portal offers teachers hundreds of educational activities which make Geography the most popular subject. But how do we know if these unique approaches work?

When we start implementing a project, it is important that we believe in the meaning of our work. However, as Laura Gassner Otting from organisation Agora for Good points out: “Hope is not a strategy.” Therefore, we should think about how to collect and evaluate relevant data already in the beginning and take the results into account when planning the next steps. If we repeatedly proceed incorrectly, we are not only ineffective, our actions can also be harmful (for example, by inadvertently standing in the way of functional solutions).

Therefore, a good feeling is not enough – we need to move away from impressions to impact measurement. Although this is a complex process, which is expensive and requires expertise, at the Pontis Foundation, we believe that this is the only way to effectively change the education system in Slovakia.

How we measure the impact

We brought together five innovative educational projects, which had received the Generation 3.0 Awards in May 2018, with education research experts. Martin Kuruc from the Pedagogical Faculty of the Comenius University, Romana Kanovská from the National Institute for Certified Educational Measurements, Lucia Kováčová from the Slovak Governance Institute and Zuzana Petrová and Lukáš Vaško from the University of Trnava monitored the impact of projects on students during the school year 2018/2019. (The research was based on a logical framework matrix developed in collaboration with consultant and evaluator Andrej Salner from Basta Digital.)

Most of all, the aim was to determine whether educational approaches are effective, whether they can be disseminated to other schools taking into account costs and benefits, and what factors may help or hinder them. The researchers investigated the theoretical basis of the projects. They also looked into whether there were similar approaches at home or abroad, which had already been evaluated.  At the same time, the researchers were tasked to assess whether it was possible to evaluate the approach during the school year, to set up reliable data collection methods and possibly to carry out short-term research.

It seems simple only at the first glance

To get the answer to a seemingly simple question – whether a particular approach leads to the desired outcome is a long-term issue which must be dealt with through a comprehensive  assessment of several factors.

Let us now take a look at the Slovak Debate Association. To determine whether participating in debates has a direct impact on students’ critical thinking, we need to take into account the type of school, the region, the social situation, and also to compose the monitored groups carefully. We must also take into account other factors that may influence the development of critical thinking – usual way of teaching, family and surroundings, extra-curricular educational activities, etc. Last but not least, research must be conducted on a sufficiently large sample and include a control group. Moreover, the very definition of critical thinking, which is not unambiguous, further complicates this situation.

Even more difficult to evaluate are projects that aim to change attitudes or value settings of pupils and teachers and those which attempt to spark a comprehensive school change (Dubová Primary School). Thus, it often requires several years of research to obtain reliable results from which we could draw unambiguous conclusions on the effectiveness and impact. Lacking both time and the resources does not mean that we should resign from data collection. It is just necessary to adjust their collection methods.

What the results showed

In their final evaluation reports, our researchers focused mainly on assessing the project methodology (whether it is sufficiently standardised and provides a suitable basis for dissemination). They also evaluated the quality of implementation and set up reliable data collection for the future.

This assessment shows that out of the five approaches studied, the following four have the greatest potential for dissemination to schools:

Reflection of Freedom and Responsibility (Dubová Primary School): the project focuses on an inclusive approach at schools based on the principles of reflections, intergenerational dialogues, concentration, relationships, the expert team and dealing with errors.

Get Movin! Academy: a short course on how to start a business. The course prepares young people for a time when they may be in a situation that there is nobody to offer them a job. They must, therefore, learn how to create it themselves.

Better Geography: a methodical portal providing educational materials, class design examples and unconventional activities which can transform a typical school Geography into a subject that develops a wide range of (not only geographical) abilities.

Extension of the Debate Programme at Secondary Schools (Slovak Debate Association): the approach seeks to support young people’s critical thinking, openness and civic engagement through debate activities.

Communication helps with dissemination

According to the researchers, the key for better dissemination of these innovative approaches is mainly communication with school principals and school founders (Higher territorial units, municipalities). It is important that they develop innovation in their schools, allocate specific time for it in class, and inspire teachers to use it.

Faster dissemination can be further advanced by:

• linking project content to the focus of the school (subjects or extra-curricular activities);
• simplicity, easy training of ambassadors;
• providing free methodological guides (it is advantageous if they are available online, as in the case of projects such as the Better Geography or GetMovin! Academy);
• creating a network of schools and platforms where experience can be shared (for example in the form of EDUpoints);
• building a positive media image of innovative education;
• creating a database of good practice examples (EDUmap).

Dissemination can be hindered by:

• insufficient staff capacities of some organisations (Better Geography, Dubová Primary School);
• financially demanding and time-consuming training of expert staff, which may initially discourage school management (Dubová Primary School);
• the timeframe of the project which is not aligned with the school cycle (Get Movin! Academy);
• charging for (parts) of the content (Better Geography).

As the research shows, projects Development of the Young Physicists’ Tournament and the International Junior Science Olympiad focus exclusively on identifying and supporting the most talented students. The organisers have no ambition to change the education system to improve the results and motivation of all pupils. On the other hand, researchers do not doubt the role of this type of programmes. The competition allows gifted children to fully develop their potential and motivates them more to choose a scientific career. However, given the objectives of Generation 3.0, we have assessed that, despite their successes, it is not effective for us to focus on disseminating this type of projects.

The lessons we have learnt

Experience and know-how from research conducted in 2017 – 2019 helped us make the whole project evaluation process more effective. Already in the first stages of selection, taking into account the objectives of Generation 3.0, we focus on seeking approaches with the potential to bring systemic changes at school level and hence to reach the full spectrum of pupils and teachers (not just a specific group).

Our experience to date shows that for successful impact measurement, it is essential that projects are sufficiently structured and have clearly formulated objectives and a strategy of how to achieve them. Otherwise, it is difficult to clearly define what research should measure (e.g. the level of critical thinking, the level of pro-inclusive attitudes, the improvement of pupils’ specific knowledge). In addition to detailed preparation, it is equally important to choose the right research methods and launch the impact measurement at an appropriate stage of the project.

Hence, in the school year of 2019/2020, the researchers no longer focus primarily on conducting short-term research and fast assessment of effectiveness. Instead, as consultants at our EDUaccelerator programme, they teach organisations how to measure their impact and verify the effectiveness of their projects – whether in the short term using less costly methods or in the long term with more complicated research and data collection. Our goal is for these organisations to understand the importance of rigorous and regular evaluation of their activities and to be able to optimise their activities based on objective results – with the greatest possible quality, effectiveness and impact.

A perfect result

Of course, it is more complicated to measure the impact of educational projects which aim to shift society’s thinking and value set-up than, for example, to assess the success of sales. Still, it is not important whether we are in the business sector, in the education system or in everyday life – we rarely achieve perfect results immediately and at the first attempt. Impact measurement is an irreplaceable tool for self-control, which needs to be developed constantly. It is okay that the data or methodology will not initially be error-free. The important thing is that at the time, we do our best. As Laura Gassner Otting says: “Measuring at the right time and with the right metrics not only allows us to achieve more impact, but it is also the only way.”

Teachers can find specific inspiration in the EDUmap – a unique webpage collecting proven innovative approaches with a description of methodologies that can be tested during classes.