British Council – ‘Learn English by WhatsApp’

British Council - 'Learn English by WhatsApp'

Sub-Saharan Africa

2020 - 2021

Countries

Botswana | Lesotho | Namibia | South Africa | Zimbabwe

Lead M&E Consultant(s)

Nicky Hockly, Gavin Dudeney & Guillem Bonilla [ TCE ], John Traxler & Matthew Johnson [ University of Wolverhampton ]

Project Overview

The Learn English by WhatsApp programme aimed to support the remote delivery of English language materials during Covid-19 through a WhatsApp chatbot. The pandemic led to widespread school closures in Southern Africa, with massive learning loss amongst the most disadvantaged populations. This programme aimed to deliver a range of adapted multimedia British Council primary school level language learning materials, available in multiple languages, via the most widespread and accessible digital technology in the Southern African context – WhatsApp. A team of five M&E consultants (three from TCE; two from the University of Wolverhampton, and led by TCE) delivered summative evaluation of the programme in each of the five countries in Southern Africa, with a focus on examining programme impact on digitally excluded or disadvantaged populations.

Approaches & Outputs

Using a mixed methods approach, the team collected and analysed data via a large-scale online survey, online interviews and FGDs with selected beneficiaries, and programmatic analyses of data from the chatbot API. Outputs included: a literature review on the effects of Covid-19 on education in Southern Africa; theory of change & log frame; qualitative & quantitative evaluation instruments & protocols; graphical representations of the WhatsApp API data findings; regular progress reports; a final regional evaluation report.

Impact On

22,700+ English primary-aged language learners, many from digitally excluded and disadvantaged contexts in Southern Africa.

Challenges

These included issues with the chatbot API design, overcome by having a qualified data scientist as part of the team. Limited data collected by the chatbot API (such as not having access to geographical data in order to identify users in deprived areas) was overcome by a staged M&E design approach, from (i) large scale user metrics analysis, to (ii) survey data gathered via random stratified sampling, to (iii) online/phone interviews with beneficiaries identified via purposive sampling.

British Council – ‘Learn English by WhatsApp’