As AI systems become more ubiquitous, policymakers, technology companies, publishers, educators and students are called to play essential roles in how these tools are developed and used. In this piece, Dr Helen Beetham explores the risks AI poses to learning, expertise, creativity and educational integrity, challenging current assumptions about AI in education and arguing for the protection of what we understand learning to be.

AI Can Fail You, and You Need to Understand How That Can Happen

An interview with Dr Helen Beetham, lecturer, researcher and consultant in digital education, on criticality and AI, conducted by Carles Vidal, MSc in Digital Education, Business Director of Avallain Lab 

The following interview was initially planned to discuss the topic of critical thinking and GenAI in education, drawing on the report Avallain published last June, ‘From the Ground Up’. That text proposes 12 controls for safer, more ethical use of AI in education, and the idea of embedding critical thinking in both the design of these tools and in teaching practices is one of its core guidelines.

To explore these ideas further, we spoke with Dr Helen Beetham, a leading educational researcher and consultant whose current focus is set on criticality and AI. Right from the start, our conversation went beyond critical thinking as a product design strategy and a necessary learning skill to encompass a broader perspective on criticality.

In the following lines, Helen unpacks her views on AI and the risks its adoption implies for societies and educational systems in general, and for educators and students in particular. She points out the problematic nature of foundational models and the agendas driving them, and suggests a range of alternative policies and practices that should be considered to manage these risks. 

For those looking for a silver lining, as Helen says, this moment is a great opportunity to think about tech and what we want from it. This is why this conversation is so timely, as only by understanding the complexities at stake will we be able to address them and ensure we continue to deliver real value from our technologies for publishers and educators.

Interview Quick Links: 

1. Why is it important that all education stakeholders have a critical stance on AI?
2. Can GenAI have real transformative educational potential?
3. Can AI models be improved to generate rich, adaptive educational content?
4. Why might GenAI be counterproductive for learners without foundational knowledge?
5. Can critical thinking help us develop GenAI tools that reduce risk and foster reflection?
6. Is it possible to generate tools that prioritise the learning process over the finished product?
7. What is the future for GenAI in education, and what should we be ready to challenge?

Interview with Dr Helen Beetham

Helen, given your area of research, we would like to address the importance of criticality and critical thinking in relation to GenAI tools, particularly the main risks the educational community faces, how to address them and the opportunities you see in these technologies.

1. Why is it important that the different actors of the educational community develop a critical stance in the face of AI systems?

First, I’m glad you identify that there are different actors with different powers to act. 

Teachers, students, school and university leaders, AI developers and the foundation companies all have different responsibilities, and I wouldn’t expect the same kinds of criticality to apply. For teachers and learners, there are reasons to be critical that concern the learning process, and there are reasons to be critical that concern the systems we depend on to deliver education. 

At the level of learning, paper after research paper has shown that people who use generative AI for significant tasks – reading, writing, coding, design – are not learning to do those tasks, or not in the ways they have previously been done. Retention is poor. Subsequent performance, for example, under exam conditions, is poor. Even expert skills are degraded through persistent use of AI. This is not at all surprising. We know that learning to read and write rewires the brain, and literacy is not a one-and-done skill; it’s something we continue to develop, or that can atrophy if we stop developing it. Arguably, the whole purpose of school is to develop people who can participate in the literate practices that societies value, and university is about developing more specialist literacies such as scientific, legal, technical and so on. When generative AI is used for those tasks, the relevant development of the brain, the understanding, the practice and even the identity is not going to happen. Something else might develop, such as a facility with the AI interface. So we need to look critically at that trade-off.

Another reason to be critical is the nature of the models these technologies rely on. Most accounts of ‘critical’ AI use focus on the outputs, especially the inaccuracies, biases and errors. You can improve those issues with post-training data labour, but fundamentally, the data model is not a world model, not even a reliable model of its training data. The errors are not going to go away.  So ‘checking the outputs’ seems like an important critical response. But what does that mean? It can only mean checking against other information systems. And what happens when those other information systems are saturated with AI-generated content? The information/media literacy revolution encourages critical questions, but they mainly concern people and their motives: who authored this, when, and why, who is disseminating it, what interests are being served, with what designs on your opinions and behaviour and personal data? None of these questions can be asked of AI outputs, or really of any information in systems that are AI saturated. ‘Checking the outputs’ of AI requires us to completely reassess what is in circulation as information: what are its sources, authorities, meaning-making processes, and what systems have been involved? It’s not a simple matter of technique.

More concerning to me than the errors in outputs are the effects of stylistic and semantic flattening. Inference tends towards the middle. Everyone starts to sound and read the same. There are a few people, experts and creatives in their field, who are using generativity to push the boundaries of their own practice and good luck to them. But they have not developed in that practice by using generative AI. For learners who do not even know the range of responses that are possible, let alone how to evaluate the outlying and the innovative, the use of AI will always tend to the most standard response. In minority cultural and intellectual fields, the stereotyping effects are even more pronounced. It’s incredibly boring and demotivating, for teachers and students alike, to have rich learning materials reduced to five bullet points and for those bullet points to be expanded again to five paragraphs of entirely predictable student text. We keep asking: what do you think? What in all this material speaks to you? What do you care about? The whole point of education is to help people find the answers to these questions for themselves. I find learners increasingly reluctant to do that, because now there is always a safe answer. Of course, data models can describe different perspectives, provided these are already described in the training corpus. What they can’t do is help learners to develop their own perspective. So a critical stance might be one that asks how this kind of development, if we value it, can still take place. 

A final reason to be critical is the gap between what the AI industry promised for education and what we have actually got. I studied AI in the 1980s, and I’ve worked in education technology, broadly speaking, since the 1990s, so I’ve been around a few hype cycles, but nothing quite like this one. People in education are not enamoured of AI, of the quality of its outputs. They are increasingly concerned about its downsides, especially for learners’ development. But the idea that if you don’t love AI, then you are the problem, the idea that if you don’t inject AI into every aspect of their experience, you are failing students, these ideas are pervasive. The GenAI challenge becomes a crisis if educators collude in the magical thinking and the myth-making. When we trust our disciplinary methods to help us understand GenAI, we can be critical in a whole variety of ways. And when GenAI refuses to be understood, because it entails blackbox architectures and deliberately obscure commercial practices, this is not a mystery to bow down before but a huge risk to truth-telling and responsible thought.

2. In your opinion, can GenAI tools and AI-based technologies have real transformative educational potential?

If you mean ‘can generative AI be used for positive educational ends?’, then of course, in the hands of a dedicated educator, any material can have learning value. I’ve been in classes where generative AI benchmarks are critiqued, where students research model biases and carbon footprints, and where they do journalistic work on AI companies. I’ve had students query system prompts, learn basic ML algorithms, and try a variety of creative responses to generated outputs. Educators are constantly experimenting and constantly learning. My experience is that the wider the variety of perspectives students have for understanding generative AI, the better chance they have of making a critical response. And it’s possible to support many different perspectives in the classroom.

If you mean ‘can students’ use of generative AI in their own independent study time be good for their learning?’, then I am more sceptical. I could break down the evidence for you, but what I would observe from my own experience is that the most engaged students, the ones that tend to be most thoughtful with generative AI in their learning process, are also the ones that are most concerned about losing skills and critical perspective. They are the students we should be engaging to help us build AI-resilient assessments and learning spaces. But just because some students are navigating generative AI thoughtfully, we can’t fulfil our responsibility to the rest by showing them good examples or preaching about ‘integrity’. Like fast food, ‘fast thought’ is irresistible: it offers compulsive and addictive behaviours in place of nourishment. According to The Brookings Institution, the costs of those behaviours for individual learners are already ‘daunting’. 

And lastly, if you mean, ‘can data-based methods produce efficiencies in education systems?’ I’m sure those can be found. But the question of what we are accelerating and why is particularly pressing in education. We don’t ask students to produce assignments because we want more content in the world, but because we want students to learn. Ideally, we don’t ask academics to publish because we want more content in the world either, but because we want more meaningful research. Unfortunately, though, our systems have been set up to reward those proxies for thinking and for research, and GenAI enters those reward systems with what is, again, an irresistible promise to produce those proxies faster and get faster to the rewards. 

People are becoming very aware of the costs. The GenAI moment is an opportunity to think critically about technology and what we want from it. I see big shifts in attitudes to social media, for example, and I think that’s an example of transformational change being driven by reactions to AI. But none of these shifts happens through interactions with GenAI alone, and there is now evidence that the more time people spend with chatbots and avatars, the less critical they are inclined to be.

3. Can models be improved to generate rich educational content that can adapt to diverse contexts and learning needs?

None of the agents I have seen in use or in development has been very impressive at adapting to learners’ contexts and needs. I imagine that retraining has to go beyond extended prompting or RAG injection to get past the sycophancy and information-giving biases that are baked into the foundation models. More fundamentally, learners do not know what they do not know, or why it might be important, so learner-to-chatbot interactions on their own are very unlikely to initiate deep learning. The other approach is to involve teachers in defining the interactions they want learners to have. If teachers can define, in clear enough terms, the learning goals and needs of their class and if they can check, refine and contextualise what comes out of an AI application, I suggest they can probably adapt content and activities in more conventional ways. Many of the teacher-centred applications I’ve seen amount to good practice in lesson planning. But teachers still have to deliver on their plans. Say an AI-designed activity isn’t going as expected in class, do you fire up the AI and have another go? Teachers need to understand their materials if they are going to teach adaptively. So do we hand everything over – lesson plans, curricula, assessment rubrics, teaching materials, student feedback, and all the agency and skills and adaptability that go with really owning those things – do we hand that over to AI companies in return for fractional gains in lesson planning?

On your question about rich content, an activity I’ve shared with students for three years now is to generate images for educational use, based on a prompt from the OpenAI teaching materials. The results are always terrible, especially when compared with the resources you can find with an OER or Wikimedia search. This year, some students pushed back against the generative part of the activity on sustainability grounds. That has been an interesting development. But the positive part of the activity is that it makes students develop pedagogic judgement. The difference between the AI images and those chosen or designed by educators is a space for real learning. I’ve also had the experience of students and colleagues uploading my own materials, and generating podcasts, quizzes and gamified versions, and in one case, an app. The results were unrecognisable to me, so much design thinking and conceptual nuance were lost. But that’s not really the point. Every learner who wants one by now has an AI bot they can use to version content for themselves, whether they want it simplified, gamified, mind-mapped, transposed or translated, or given an anime makeover. There are accessibility gains here that I don’t want to trivialise. But I’m not sure where it leaves learning design. A bad outcome would be for education to disinvest in universal design and accessibility support because ‘learners are doing it with AI’. A worse outcome – just taking this to its logical conclusion – would be for learning design itself to disappear as a profession and as a shared language. If every learner is their own unique microcosm of intellectual needs and sensory preferences, if those needs can be met by ‘designs’ or ‘experiences’ generated on demand from the soup of educational content, what really is the role of the designer? Or the teacher, for that matter?

Behind the apparently empirical question ‘can GenAI applications provide effective learning support?’ are questions that might be a bit more uncomfortable. Such as: should GenAI interactions replace teaching interactions? Should teaching assistants and student support professionals become chatbots, or rather become the data workers who make the chatbots work? Perhaps most insidiously of all, should GenAI replace other students in the learning process? Alexandr Wang, head of Meta’s super-intelligence lab, has said he will wait to have kids until they can connect straight into the network mind and bypass all the messy interactions of school. Learners will kick back against further loss of contact with teachers, I suspect, but they are already voting with their feet for chatbot companions over peer learners. And that is really troubling. Social constructivist theory tells us it’s better to learn alongside other learners who are at a similar learning stage to us, who make similar mistakes and discoveries, who can be resources for our learning in their differences from us, despite the frictions and frustrations involved (that also teach us something). Yet we seem to be encouraging learners in the delusion that the perfect other is always available, and it is a chatbot. 

4. In your work, you refer to the AI expertise paradox, where if you want to use AI effectively and safely, you need to be an expert, so you can benefit from shortcuts and avoid inaccuracies. However, the same cannot be said for students who are still learning. Can you please elaborate on why GenAI might be counterproductive for those who have not yet built that foundational knowledge?

I’m not sure expert use is necessarily effective and safe. There have been several longitudinal studies now, from MIT, Bloomberg and Stanford, that have found experts are not as productive with AI as they think they are. They are certainly not as productive as their bosses think they should be. It turns out ‘checking for inaccuracies’ and ameliorating losses of quality and context are non-trivial. Experts are slower than novices when they apply AI because they are aware of these issues and have to make judgments about them. What parts of a task might lend themselves to AI efficiencies? How best to realise them? What are the likely impacts on quality, safety and professional values? Experts are finding uses, but only with significant trade-offs, some of which may be visible only at the organisational or sector level. 

From an educational point of view, yes, the worry is that novices never get to develop the judgment and expertise that is needed to work effectively with AI. Herbert Dreyfus’ original critique of the AI project, back in the 1960s and 1970s, was that it misunderstood the nature of expertise. Educators understand it better, for example, that it develops through iterative practice. That’s the point of learning spaces where novices can practice without high costs of failure, where they can get expert feedback, and develop fluency and judgement. All of those developmental processes are lost when learners use generative AI to produce what looks like expertise. At least, it looks like expertise to students, and this is another aspect of the paradox. It looks less like expertise to assessors. But we are working in a system that has previously taken that kind of evidence at face value, and where the time educators have to engage with students’ development has been pared away.

What the expertise paradox allows me to say to students is: I will not fail you for using AI. But AI can fail you, and you need to understand how that can happen.

5. You have also identified ‘Cognitive Dependence’, ‘Deskilling’, ‘Reduced Learning’ and ‘Less Personal Agency’ as potential effects for users of GenAI tools, particularly among non-experts. To mitigate these risks, do you believe that critical thinking can be effectively promoted by developing GenAI tools that foster reflection and scepticism, present alternative perspectives, and highlight that students should not take generated output at face value?

The only remedy for students who avoid practice and engagement is to practice and engage, and to have support in place to do so. Perhaps some aspects of critical thinking might be supported with GenAI tools; I don’t rule it out, particularly when it comes to the critique of AI. But criticality and scepticism are not simple ‘techniques’. I know there has been a movement in media literacy, for example, to ‘inoculate’ young people against deepfakes, which sounds like a nice, simple shot in the arm. But what this inoculation involves is really a series of engagements. Typically, learners will spend time practising and applying diagnostic methods. Then they will create their own media pieces or ‘memes’. And finally, they will strategise to spread their memes, to make them as persuasive and pervasive as possible. Basically, they learn to make clickbait so they can learn not to be so easily baited. There are analogous activities you can do with GenAI, some of which engage learners more deeply with the data structure than others. The ‘inoculation’ task might be to prompt for a particular persona and interaction style, for example, and reflect on the results. It might be vibe coding a simple app and exploring the quality of the code. I think these are productive approaches, but they are not at all simple to implement.

A common remedy proposed for these problems is to teach students ‘good’ prompting strategies. When this involves using a particular template or standard, it seems to me counterproductive. It’s unlikely learners will surface anything interesting or discover the limitations of inference for themselves if prompting is reduced to a fill-the-gaps or cut-and-paste exercise. In fact, a lot of prompt crafting now takes place in the application layer, where interfaces may be even more frictionless than the familiar prompt screen. Larry Page looked forward to an AI search tool that would ‘understand exactly what you wanted before you knew yourself. For the foundation companies, the best prompt is one that entirely pre-empts the user’s needs. One of the critical exercises I do with students is to have them review chat logs and ask whether they think the user is prompting the chatbot, or the chatbot is prompting the user. But chatbot logs these days are often hidden in the application layer.

The alternative you suggest is that inference is made more effortful, for example, by defining the chatbot persona as a reflective mentor or a Socratic partner. I think this identifies the problem correctly, and it’s one I’ve discussed a lot with colleagues. How to ‘interrupt’ the straight line to the solution and introduce a more developmental path. What I’m not sure about is the possibility of installing these solutions as dialogic techniques within a chatbot interface. As I said in a previous answer, I have not yet found an AI agent that is effective in this mode. Either the moves are generic and stereotyped, or they come from examples in the training data. If there is a generic rule for ‘scepticism’, it can be reproduced in a list of reflective questions. And for a topic-based approach to ‘challenging a student’s logic’, generative AI should not, in my view, be the first resort either. Students learn more from co-teaching other students than they learn from interacting with a chatbot, because they are also doing the thinking involved in following the other person’s logic, noticing assumptions and blind spots, and recognising there are different perspectives on the same problem. Playing both sides is far more developmental. 

The other issue is that learners turn to GenAI for complex reasons. Anxiety is often a big part of it, or the fear of missing out, or a lack of academic confidence. Dealing with these issues requires a deep engagement with learners. Learners need to understand why we ask them to do things (that they might do with GenAI), and we need to explain that better, but they also need good reasons for responding to those tasks in ways that feel challenging and uncertain. By the time learners reach the end of an undergraduate degree in the UK, they might have spent 18 years in an education system that values and measures outcomes. Valuing things that get in the way of the outcome is going to take more than individual encouragement or exhortation. It will require profound change.

Finally, much of the investment in GenAI has gone into interface effects that tend to undermine critical thinking. You can now talk to, vibe with and even engage with data models through wearables and emotion sensors. The effect is to undermine skills of mediation, or what used to be called literacy: conscious and effortful practices of engaging with other people’s thinking, and developing our own. The feeling of effortlessness and immediacy is not just about saving time. It is emotionally beguiling. You need never misunderstand or be misunderstood again. 

Cognitive offloading is natural to human beings – it is one definition of culture – but it is never entirely safe. It creates vulnerabilities and dependencies. For example, we depend on shared signifying systems and tools, cultural records, and whoever controls them. I agree with Musk on this at least, that ‘safeguards’ are always ideological. It’s just that data models are inherently ideological, from their training data through the judgements of data workers to the contents of prompts. There is no way of steering AI models that is neutral. But unlike the cultural forms we have lived with for tens and hundreds of years, how they are ideological is obscure, and how they influence us is unfamiliar. There may come a time when we have no choice but to use GenAI if we want to participate in cultural and intellectual experiences, at least if we want to participate digitally. At that point, it might make sense to align oneself with particular architectures and not with others: Grok culture or Claude culture. But until then, and in hope it never comes, the ‘safe’ option I suggest is to keep practising alternative skills, interacting with other archives besides the data archive, maintaining and valuing other media, and nurturing our own embodied memories. Then, at least, there are alternative vantage points on the obscure ideologies and behavioural effects of the data model.

6. You have pointed out that GenAI tools tend to place the value of education on the final output rather than the process. As you put it, when students are asked to create content, it is not because the world needs more content, but because the act of creating is how they acquire knowledge and skills. Do you think it is possible to generate tools that prioritise the learning process over the finished product?

We already have tools and features of tools that support a focus on the learning process. Annotation is one that can be used both for solo reflection and for shared feedback. Document sharing, design spaces and coding environments, e-portfolios, all these are useful too. But you can build a process-oriented learning environment in just about any platform, or from simple open-source tools. What matters is the pedagogic intention. If we want education to be centred on processes of reasoning, and personal development, and on the specialist practices of each subject, we have to invest attention in these things and not in proxies for them, such as grammatical sentences or test scores. As soon as we standardise what we are looking for, that standard can become a data source or a prompt for GenAI. But moving away from standardised assessments and standard proxies for learning will require the kind of transformation I mentioned at the start. It would mean learners working on authentic challenges that arise from a real context – a context that provides its own standard, its own intrinsic feedback on students’ solutions. That might be the school or university and its community, or a placement setting, or a project with stakeholders. More learning would have to happen in shared, live, embodied spaces, and that learning would have to be high-quality if learners are to invest in it rather than in the electronic angel on their shoulder. Educators would have to enable and assess students’ work on its own terms, letting go of many proxies we have relied on before. This would have implications for teaching time and attention, and the places that learning happens, and therefore its costs.

A very small part of that is about the tools that are used. There may be some technical solutions for how learning relationships are mediated and how learning environments are made. I would stick my neck out and say we have all the functionality we need; we just need it to be more modular, more open, cheaper, and more flexible. But there really is no way to automate attention, witnessing, engagement, and care. This is what students need, and when they say they want a chatbot, what I hear is that they aren’t getting enough attention and care from other sources. We have to be very careful about contradicting students’ experiences, since that is a large part of what we are working with, but I do think we can challenge what they say they want from AI. That will involve confronting their fears. They are afraid of using AI. They are afraid of not using AI. Asking students to raise their eyes from these anxieties and think about what kind of learning and working futures they want is always an intense experience, but it is powerful. These deep engagements are what we need, in my view.

It is an irony that the use of generative AI by students has produced a workload crisis for teachers, at least those who are committed to student learning. It is exceptionally hard to read through an AI-generated or partially generated assignment to discern the work students have done, the process they have undertaken, and thus what guidance they need. If you don’t care, of course, it’s a breeze. Set your AI agent onto marking theirs. But if you do care, you know that the solution is to make the process itself the focus. We need teachers and students to build those processes together, and the learning environments that can support them. That would be a lot more exciting to me than asking what workflows GenAI can come up with or what new AI-integrated mega-platforms we need.

7. What is the future for GenAI in education, and what should we be ready to challenge?

I did my fair share of foreseeing early on, and much of what I was saying now seems like common sense. So GenAI hasn’t kept improving with scale, productivity increases have been hard to demonstrate, and the promised educational benefits continue to be elusive. 

I’m not an expert on the interface between the AI industry and edtech, in terms of the business models and the distribution of income. It seems to me that a lot of what passes for AI in education is rather shallow customisation in the application layer. At the moment, the foundation companies are very keen to secure subscriptions and use cases from education organisations, and of course, educational content and data. Bespoke educational applications may be a good way to forge those relationships, for now. But the long-term vision for these companies is not AI-enabled education; it’s AI instead of education. It’s something much more like the Neuralink, or Josh Dahn’s Synthesis, or Marc Andreessen’s vision of every child having their own dedicated AI tutor from birth, bypassing the need to engage with a shared learning culture. And while I personally think that is a fantasy, a dystopia, I do think the foundation companies will want to monetise everything they have learned from education and edtech in the medium term. To mine education, if you like, in order to undermine education. 

As I said at the start, different actors in the educational space have different powers to choose from. I would love to see university leaders taking a more critical and ethically grounded position on generative AI. I’d like to see ministries and departments of education employing people who think deeply about these issues, rather than AI company secondments or industry-adjacent think tanks. I think that would make a considerable difference to how GenAI is implemented, or not implemented, in educational contexts.

I share the anxieties educators feel around generative AI, which makes us grasp at terms like ‘implementation’ so we can feel more in control. Generative AI is not being implemented. It was, as its proponents like to say, ‘unleashed’ on users and knowledge systems and cultural archives, every bit as irresponsibly as that sounds. Its developers have quickly become the richest companies in the world, dictating terms to governments, regulators and publishers. Every choice we make about AI in education is made downstream of these events, and in the face of this power. But choices are still possible. The most important, for me, is to tell the truth about GenAI, without hype or magical thinking, trusting our pedagogic and disciplinary methods to support us in that. And when those methods don’t help, to be truthful about our uncertainty. 

We should be ready to challenge developments that extend the black box of not knowing and not being accountable further into our classroom practices. That means, I think, that we should insist on alternative knowledge archives and knowledge practices being available – that do not pass through generative data architectures – to protect what we understand student learning to be. Doing this demands technical ingenuity as well as intellectual commitment, and both are valuable skills in any foreseeable future. Just as we provide alternatives to social media in school and university platforms, where different rules and norms apply, I think we can do the same in relation to GenAI.

---

About Helen Beetham

---

About Avallain

For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.

Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.

_ 

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com

Language carries assumptions, cultural context, and implicit meaning that make comprehension and translation difficult for humans and even more complex for AI to master. Expanding on the first half, this piece explores language for persuasion, showing how cultural norms, reasoning patterns and rhetorical conventions shape communication, learning and the complexities of teaching or interpreting language effectively.

AI and Why It’s Impossible to Learn or Understand Language: Cultural and Cognitive Challenges

Author: Prof John Traxler, UNESCO Chair, Commonwealth of Learning Chair and Academic Director of the Avallain Lab

This piece continues to argue that it is impossible to learn, understand or discuss what anyone else says or writes beyond the simplest, most specific and concrete level, even perhaps among people with a shared mother tongue. This makes conversation, learning, translation and reasoning more difficult than they initially seem, especially when they involve artificial intelligence. 

The piece is divided into two halves. The first, ‘AI and Why It’s Impossible to Learn or Understand Language’, already deals with language as idiom, whilst the second half deals with language for reasoning. We transition from language for description to language for persuasion. 

As I said in the first half of this piece, language as an idiom is a challenge for learners or translators outside the culture or community that actually hosts that idiom, and this clearly also applies to the chatbots of GenAI.

Sadly, but obviously, reasoning also depends on language, and reasoning is not usually about anything as concrete and specific as ‘the cat sat on the mat’. In fact, it seems safe to say, as we did in the previous piece, that language is mostly not about the specific and the concrete, rather that language, especially language that is in any sense important, is metaphor, simile or analogy, and each of these is based on implied notions or assumptions of ‘likeness’.

Deductive and Inductive Reasoning

Reasoning, or rhetoric or argumentation, in the West, is defined as either deductive or inductive.

Deduction, the former, is true by definition, like ‘2 and 2 is 4’ or ‘Socrates is a man, all men are mortal … etc, etc.’ This is because that is how ‘4’ and how ‘men’ are defined. It has to be true because that’s how the terms are defined, strictly speaking, not true but valid; it is a tautology; it is circular. On inspection, we may be unclear whether it is males, homo sapiens or hominids being discussed and unclear how some things are counted, clouds, for example. 

The latter, induction, works from specific instances towards inferences about the general, say from ‘every dog I’ve met at the park has been friendly’, to ‘all dogs are friendly always’. Somehow, those dogs in the park are ‘like’ all dogs all the time. In making this inference, we preserve or favour one aspect and neglect others, such as ‘in the park’. Even statistical inferences work the same way; the ‘sample’ being analysed is ‘like’ the ‘population’, and somehow representative of the ‘population’, to use the statistical terminology. 

But, and this is the kicker, they all depend on some tacit or shared consensus about the ‘likeness’ that is going on, that, ‘this one is like that one and like that other one and like all those others’, and they have something in general, in common, and that depends on culture, that the people within a culture or subculture basically agree. As we said, once you get slightly more abstract than cats sitting on mats, all language is metaphor, analogy or simile, sometimes in plain sight when we see ‘like’ or ‘as’ in a sentence, sometimes hidden, with only an ‘is’.

The Challenge of Abstract Thought

Plato’s ‘Allegory of the Cave’, from ‘The Republic’ (Book VII, 514a–520a), expresses the notion that we humans only experience separate poor solid instances of some higher, hidden, abstract and immutable reality. ‘The dog,’ for example, is perhaps the wrong way around; we experience each of those poor, solid real dogs and assume we can group them together as some abstract ‘dog’ and discuss them accordingly, whereas different cultures might do the grouping and thus the reasoning differently. We assume that the distinction between ‘dog’ and ‘not-dog’ is clear-cut and sharp, or the ‘park’ and ‘not-the-park’, with nothing vague and nothing in between. 

Fuzzy concepts as opposed to sharp ones are another challenge for logic and reasoning, needing the duality of ‘either/or’ with nothing smeared out in between. In fact, even ‘park’ or ‘dog’ might not be so clear; are feral dogs or wild dogs included and is Hampstead Heath a park? We could be pragmatic and use the rule-of-thumb attributed to Indiana poet James Whitcomb Riley (1849–1916), ‘When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.’ So Hampstead Heath is a park; a car park is not.

One way or another, these mental processes do not generate new knowledge; they expose and perhaps distort knowledge already beneath the words being used.

Examples of Culture Shaping Understanding

To take some specific examples where different words are used to describe basically the same process. That is, the process of culture shaping reasons, of words not describing our experiences but shaping them:

Hammer and nail: Abraham Maslow said, ‘If the only tool you have is a hammer, you tend to see every problem as a nail,’ meaning the extent to which preconceptions or interpretations shape understanding or analysis, the solution shaping the problem.  

Evolution and creation paradigms: Creationists argue that God created fossils to test the Christian faith, whilst evolutionists argue that fossils were the product of sedimentary deposition. The culture of a specific community, whether creationist or evolutionist, determines how it understands the evidence rather than the evidence determining the understanding.

Policy and evidence: The (cynical) notion is that ‘evidence-based policy formulation’ is often ‘policy-based evidence formulation’, a suspicion familiar to many of us who have worked for ministries and ministers; that the interpretation of evidence precedes the gathering of it, and of course, logically speaking, there is no evidence for evidence, that would be circular, it would be a logical fallacy.

Personal construct theory: Those ways, major or minor, that individuals use to understand or organise their experiences, those dogs in the park, for example, partial or over-simplified or over-generalised explanations that help us live our lives.

We might use different words, paradigms, policies, cultures or constructs, for example, but these are all essentially the same process at work: words actively shaping experience rather than passively describing it. I admit to being on shaky ground when analysing the workings of words with words, but what choice do I have? The aim was to point out, however weakly, the difficulty that GenAI might have in conversation, translation and education.

The Principle of Linguistic Relativity

The Sapir-Whorf Hypothesis, aka the ‘principle of linguistic relativity’, is relevant here. It is the notion that language shapes thought and perception, meaning speakers of different languages may think about and experience reality differently, in mutually incomprehensible ways, and may never truly understand each other. 

The Hypothesis suggests that a language’s structure influences how its speakers conceptualise their respective worlds. In essence, it suggests that language shapes thought and perception, meaning speakers of different languages may think about and experience reality differently. The language we learn influences our cognitive processes, including our perception, categorisation of experience and even our ability to think about certain concepts. 

There are two versions. The strong version of the hypothesis proposes that language determines thought, meaning that thought is impossible without language. A provocative view, rejected mainly by linguists and cognitive scientists, but resonating with George Orwell’s idea of Newspeak, which we mention later. The weak version, proposing that language influences thought, suggests that whilst thought is not solely determined by language, it is significantly shaped by it, a more acceptable interpretation. 

So we have differing vocabularies for snow in English. Languages like Inuit suggest that English speakers might have a less nuanced understanding of snow-related concepts. Secondly, some languages assign grammatical gender to objects, potentially influencing how speakers perceive those objects; meanwhile, the Chinese ideogram or character for ‘happiness’ was derived from ‘woman in house’, an interesting trajectory from the concrete to the abstract.

The point about Orwell is that the appendix in his novel ‘1984’ describes a political system that, by eliminating problematic or challenging words from its language, Newspeak, eliminates problematic or challenging thoughts from the population, suggesting again that the possibility that language can shape culture (or society in this case), or Orwell thought so. Any resonance with current concerns about political and corporate influence on the news media is, of course, purely coincidental.

Cultural Dimensions and the Definition of ‘Culture’

At some point, we ought to introduce ‘cultural dimensions’ and Geert Hofstede’s work, among others, as much of this piece mentions or implies culture as a fundamental mechanism in shaping language. Being abstract, we can only define ‘culture’ using either metaphors or other abstractions, so we will settle for something simple, ‘the way we do things around here’, ‘here’ being our society, our friends, our organisation, our profession or wherever else a group of people have shared values.

Cultures are obviously different from each other; ‘cultural dimensions’, based on Hofstede’s work, are a tool for describing in what respects and by how much they differ. So we might say that some cultures are risk-taking, others risk-averse; some are consensual, others authoritarian; some take the long view, others the short one; some are individualistic or even selfish, others communal and collectivist, and so on, giving us scales by which to calibrate different cultures. 

So these are alternative perspectives, language and conversation shaping culture and thought, and the opposite, culture and thought shaping conversation and language, or perhaps a dynamic between the two.

Culture, Reasoning and the Diffusion of Innovations

If we are to use language to reason, question, analyse, judge, evaluate and critique, rather than merely locate the cat, then we have to recognise how language is shaped by culture. This may be national culture, regional culture, gender culture, class culture, generational culture, ethnic culture, or indeed a mixture of all of these, as they still shape language. 

Reasoning, questioning, analysing, judging, evaluating and critiquing are essential components of higher-level learning and of higher-order language learning, if learning is to be about reasoning as well as rote reciting. Cultural dimensions, however, suggest that some cultures may be less tolerant of dissent and the outcomes of reasoning than others (and may not even have the language to express it), or might find some conclusions less palpable, less conforming or more risky. 

A further complication is the theorising behind the Diffusion of Innovations, which suggests that changes to opinions, attitudes or beliefs, in effect acceding to reasoning or argumentation, are all dependent on various factors; culture is one of these, as we can deduce from Hofstede’s ‘cultural dimensions’, for example, the risk-aversion/-acceptance and the consensual/authoritarian dimensions. There are, however, others, for example, the ‘relative advantage’ of the changed opinion, attitude or belief and its ‘trialability’, ‘observability’, ‘compatibility’ and ‘complexity’ are also factors in acceding to an argument or reason representing the changed opinion, attitude or belief. The pure logic of GenAI may well see these complications as unreasonable.

Language and Its Cultural Influence

The work of linguist Robert B. Kaplan comes at this from a different direction. Analysing essays from English, Romance, Semitic and Asian students suggested that every language is influenced by a unique thought pattern characteristic of that culture, or by the collective customs and beliefs of its people. 

Rhetoric, argumentation and thus reasoning exhibit culturally distinct patterns. Rhetorical conventions vary across cultures, affecting how students compose essays. English rhetoric in this depiction follows a linear, logical structure influenced by Western philosophical traditions. This harks back to our depiction of inductive and deductive reasoning, whilst other cultures may employ parallelism, helical, zig-zag or indirect approaches in writing, leading to different expectations in composition and argumentation, ones that make less sense to the processes of GenAI. 

Western Reasoning

Interestingly, a paper from Harvard published in November 2025 observes that, ‘LLM responses … their performance on cognitive psychological tasks most resembles that of people from Western, Educated, Industrialized, Rich, and Democratic (WEIRD)’

Returning briefly to Western, or WEIRD, reasoning: whilst we have described the established ways of reasoning correctly, the deductive and the inductive, there are also Western ways of reasoning incorrectly. When learning logic, you start with the fallacies of irrelevance: fallacies that introduce irrelevant information to distract from the main argument. 

For example:

Ad Hominem: Attacking the character or personal attributes of an opponent rather than the argument itself.  

Appeal to Emotion: Manipulating emotions, such as pity or envy, instead of using logical reasoning to win an argument.

Ad Populum: Claiming something is true or right because many people believe it. 

The Red Herring: A distracting point to divert attention from the actual issue. 

Straw Man: Misrepresenting an opponent’s argument to make it easier to attack. 

There are also Fallacies of Weak Induction, such as: 

The Post Hoc Fallacy: Assuming that because one event followed another, it must have caused it. 

The Slippery Slope: Asserting that a small first step will lead to a chain of related, often negative, events. 

Finally, there are Fallacies of Presumption, including:

Circular Reasoning: Using the conclusion as a premise to support a conclusion.

False Dichotomy: Presenting only two possible options when more options exist.

Cultural Significance of Logical Flaws

Our point is that these errors, even if valid within a Western context, may not be obvious or convincing to a non-Western language learner accustomed to reasoning differently, and that their weight may be less or different in other cultures. So, more hierarchical or authoritarian cultures may find Ad Hominem arguments perfectly valid when made by someone with sufficient status, whilst more consensual or communal cultures might be happy with Ad Populum arguments rather than standing out in the crowd. Additionally, cultures are probably each on a continuum from emotional to rational, and this, too, will determine how they react to reasoning and argument.  

Culture or individual cultures are, however, not immutable. According to historian Ian Mortimer, the Elizabethan hand mirror and the vernacular Bible, Tyndale’s in English and Luther’s in German, moved the needle towards greater individuality or individualism and lessened communality or collectivism in their societies, as the mobile phone selfie has done more recently. Relating individually to AI chatbots might have similar consequences, as individuals use them for emotional and intellectual support, or it might involve a completely different cultural dimension.

Other Linguistic Dimensions

Another attribute of language is lexical distance, the distance and differences within and between language families. So, for example, German is very close to Dutch but distant from Chinese, so German speakers might struggle to learn Chinese but not Dutch. Are GenAI chabots somewhere amongst the Anglo-American, low-context, consultative and slightly out-of-date languages, lexically distant from many other language families?

This linguistic metric might also apply to different literary genres or, indeed, different literary authors. Is James Joyce lexically distant from Ernest Hemingway, or a haiku from a sonnet, and thus more or less difficult to understand or translate, especially for chatbots rooted in the GenAI culture they inherit from their trainers?

Languages are also sometimes classified on a continuum from high-context to low-context, with greater or lesser baggage, background, assumptions and preconceptions, and metaphorically expecting more or less bandwidth for successful comprehension or translation. Clearly, low-context language learners will struggle to hear, for example, the irony, euphemism, hyperbole or sarcasm at work in high-context languages. In a high-context language, the neurodiverse will have much less metaphorical bandwidth; they, me, in this context, are low-context, missing cues and signals from their higher-context culture or colleagues.

The Challenge for Educational AI

It is difficult to imagine how to converse with each other with all of these issues going on, so perhaps we should spend more time talking about the cat sitting on the mat and less time on democracy and freedom, and perhaps, for safety’s sake, conversations with GenAI chatbots should also stick to cats. Language is not as simple as it seems, nor is learning it or teaching it, nor hoping that GenAI will be good at either.

The challenge for educational AI is how to proceed safely, from helping learners with the specific and concrete to the abstract and general, from learning about cats to learning about freedom.

The mention of ‘safely’ does, however, add the additional element of ‘harm’. This piece was not really about the ethical dimension of educational AI. This too can be tackled, like our account of deduction and induction, from the top down, from abstract general principles, such as beneficence, to the concrete and specific, such as bomb-making recipes, or from the bottom up, from a long list of concrete and specific misdemeanours, to some abstract general principle that unites them. 

Either way, we hope AI can replicate human reasoning, but human reasoning is flawed, and those flaws will likely be replicated in the training of GenAI. This is precisely the two approaches being explored and investigated at Avallain Lab.

---

About Avallain

For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.

Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.

Find out more at avallain.com

_

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com