Sentiment models score lower on isiZulu and Sesotho than on English. How a brokerage should test its WhatsApp classifier before letting it reply on its own.
A client in Pietermaritzburg writes: "Sawubona, ngicela ukwazi about my policy, angikwazi ukukhokha this month." Half isiZulu, half English, one sentence, and three things a brokerage needs to catch — a greeting, a question about cover, and a payment problem. Your inbox is full of messages like this. If an AI layer is reading them to score sentiment, flag churn risk or draft a reply, the honest question is not whether it works. It is how well it works in each language your clients actually use, and what happens in the cases where it does not.
There is now solid published work on this, and it is not reassuring for anyone planning to switch on automation across a multilingual book. A 2025 study in PLOS ONE by Mabokela, Primus and Celik benchmarked pre-trained language models on sentiment in five South African languages — Sepedi, Sesotho, Setswana, isiXhosa and isiZulu. The best monolingual fine-tuned setups averaged roughly 71% weighted F1 across the set. The spread matters more than the average: the Nguni languages, isiZulu and isiXhosa, exceeded 77%, while the Sotho-Tswana group sat around 63%. The same corpus was full of code-switching with English, which the authors had to work around during annotation.
The picture across the continent is similar. The AfriSenti shared task at SemEval-2023, which the Masakhane community contributed to, built sentiment datasets across a dozen-plus African languages precisely because the resources did not exist. Purpose-built Afro-centric models outperformed general multilingual ones — which is another way of saying that a general-purpose assistant, handed an isiZulu message cold, is working further outside its training distribution than most vendors admit.
Two caveats before you quote these numbers at a supplier. They are benchmarks on social-media text, not on your inbox, and short transactional insurance messages may behave better or worse. And a system does not have to classify perfectly to be useful. The point is the gap between languages, and the fact that nobody — including your provider — knows what that gap looks like on your book until someone measures it.
An error rate is abstract until you map it onto the things your inbox actually does with a label.
| What the client sent | Likely misread | What the system does | What it costs |
|---|---|---|---|
| A bereavement notice in Sesotho | Neutral sentiment | Sends a cheerful automated reply | A family that never comes back, and a complaint you deserve |
| Polite cancellation intent in isiZulu | No churn signal | No flag, no follow-up | A lapse discovered at renewal |
| An urgent motor claim mixed with English | General enquiry | Ordinary queue, no escalation | A late notification and a hard conversation with the insurer |
| Sarcasm or an idiom about a declined claim | Positive sentiment | Auto-reply instead of a human | An escalation that starts one level angrier |
There is a regulatory edge to this too. The FAIS General Code of Conduct requires that what a provider tells a client is in plain language, is not misleading, and does not create confusion — a standard that is hard to meet with a machine-translated reply about cover terms. Treating Customers Fairly says the same thing in outcome language. Nothing in either framework prohibits AI in the inbox; both make you responsible for what leaves it.
You do not need a data scientist. You need one hundred real messages and two colleagues who speak the languages.
The point of the test is not a certificate. It is to decide, with evidence, where the machine acts and where a person does.
Worth saying plainly: the ORIS interface is in English. Your clients' conversations are not, and that asymmetry is the thing to design around — an English-speaking principal reviewing a queue of isiZulu drafts is not doing quality control, they are approving something they cannot read. Put the reviewer who speaks the language on the queue that needs them. If you are still weighing scripted automation against a supervised assistant, our comparison of chatbots and supervised AI lays out where each one belongs.
That is over-correcting. Classification is useful even when imperfect, provided it routes rather than decides: use it to surface, prioritise and draft, and keep the send button with a person for anything sentiment-bearing. The risk lies in automatic action, not in analysis.
A hundred per language is enough to see a serious gap, though not enough for a precise figure. If a language makes up a large share of your book and you cannot gather a hundred real messages in it, that itself tells you something about how well you are serving those clients.
Testing on your own client conversations is processing that needs a lawful basis and should sit within the purpose you told clients about. Keep the sample internal, do not send it to a third party without checking your operator agreements, and de-identify it where the test does not need names.
Check that assumption against your own inbox rather than against the impression of your English-speaking staff. Clients often switch to English with a brokerage while writing in another language everywhere else — and the messages that matter most, about a death or a claim, are the ones most likely to come in a home language.
Your FSP is. Regulators treat the tool as a means of delivery, not a defence: the plain-language and fair-treatment duties attach to the licensed provider whose name appears at the top of the chat.
Shared WhatsApp inbox, client records, follow-ups and opportunities for the whole brokerage. 15-minute demo.
A demo tells you nothing about where client data goes. The POPIA operator, security and cross-border questions a South African brokerage should ask in writing.
Section 35 of the Data Protection Act, the DPIA that automated decisions trigger, and the ODPC registration a small Kenyan brokerage cannot skip.
The FSCA and PA have mapped AI use in the sector. What POPIA section 71 and FAIS ask of a brokerage that scores client risk and drafts replies with AI.
