When AI Enters the Learning Process: Design Failures, Regulatory Risk and Guardrails for EdTech

Hallucinations and biases

AI can produce confident but incorrect or biased explanations. These errors are often hard to detect and can introduce misconceptions, cognitive bias and amplify over-trust, undermining learner judgment, and reinforcing stereotypes and uneven representations.

Cognitive deskilling

When AI is available to generate answers, full solutions and completed tasks, learners offload cognitive work. This shows up as relying on full solutions rather than grappling with problems, and over time can weaken reasoning, problem-solving and persistence.

Psychological, emotional and social risk

AI learning systems built as conversational, companion-style or anthropomorphic can encourage emotional reliance, reduce peer interaction and undermine real-world support networks, especially problematic for children, distorting emotional and social development.

Context insensitivity

Without contextual adaptation, outputs can be inappropriate, misleading, or misaligned with learners’ educational context, privileging dominant knowledge frameworks and creating friction with local classroom practices.

Manipulative design

Patterns such as flattery, unjustified confidence, social pressure (“others have done this”), or reward-based engagement and monetisation-driven dark patterns can influence learner behaviour in ways not acceptable in learning contexts.

Data privacy

Continuous monitoring, collection of sensitive learner data, or reuse of data for commercial purposes or model training raise serious concerns, especially for children, treated as a high-risk group under data protection law.

Agentic autonomy

Agentic AI systems plan, adapt, and act over time. Memory and proactive decision-making can gradually shift agency away from learners (even when short-term task performance improves), creating dependency loops that are difficult to detect. Learners may rely increasingly on system guidance for task sequencing, decision-making, or problem framing, reducing opportunities for independent reasoning and productive struggle. In multi-agent systems, coordinated outputs or apparent consensus can further inflate epistemic authority, making AI guidance appear more reliable, discouraging critical evaluation.

Ground AI in learning science

AI must be grounded in human learning theory (e.g. constructivism, Inquiry-Based Learning, scaffolding, Zone of Proximal Development), not optimised for task completion, speed, fluency or output quality alone.

Why: Encourages cognitive engagement, reflection, verification and independent reasoning.

Design against over-reliance

Systems should enforce clear role boundaries, avoid authoritative or solution-first behaviour and prioritise scaffolding. Support should fade as competence increases, not persist indefinitely.

Why: Implement attempt-first flows, progressive disclosure and intentional support fading. Explicitly design for scaffolding modes (Aid/Complement) rather than replacement mode (Substitute) as the default.

Preserve agency

AI should support learners in questioning, verifying and revising, not accepting outputs at face value. Interfaces should make uncertainty visible and prompt reflection (“What would you check?”, “What do you keep and why?”).

Why: Treat agency as a core learning outcome, requiring justification, comparison and learner reasoning.

Ensure human oversight

AI should augment educator judgment, not bypass it. Responsible adoption requires explicit boundaries on which functions may be automated, which require human review and which should never be delegated to AI.

Why: Enable human review, override and escalation for consequential decisions.

Conduct longitudinal behavioural testing

Short-term confidence and efficiency gains can mask declining independent performance. Governance must include longitudinal monitoring of reliance, cognition, self-efficacy and differential impacts across learner groups.

Why: Track learning trajectories over time and monitor whether confidence and independence converge or diverge.

Design for transparency and explainability

Learners and educators should understand what the system is doing, why it intervened, what data it relied on and how confident it is.

Why: Expose reasoning and uncertainty through interaction design; make AI contributions visible in learner work.

Prevent inaccuracy and pedagogical misalignment

For AI-generated learning materials (videos, quizzes, Q&A), ensure outputs align with course content and instructional or academic goals.

Why: Use restricted retrieval from approved materials (e.g. RAG) and human oversight to prevent misinformation, biases and low pedagogical quality.

Do not dehumanize learning

Social interaction is a learning-relevant ingredient. Designs that remove it can reduce learning quality (e.g. lower perceived social presence).

Why: Position the AI as a tool or facilitator; actively redirect learners to teachers, peers or group discussion when social engagement is essential.

Treat AI literacy as a prerequisite

AI literacy for learners and educators is a prerequisite for responsible deployment; it cannot be expected to emerge automatically through use.

Why: Show what AI can/cannot do and why outputs are generated, with contextual explanations at moments of use rather than one-off onboarding. Offer distinct explanations and controls for learners, educators, parents and administrators, aligned with their decision-making responsibilities.

AI evaluations & behavioural risk assessments

We assess how learners and educators interact with your system: output quality, model and behaviour drift over time, and where over-reliance, authority transfer, misuse or unintended behaviours emerge.

Design refinement

We translate learning science and behavioural evidence into concrete design guidance (interaction patterns, autonomy boundaries, scaffolding strategies, safeguards and monitoring) that reduce risk and are developmentally appropriate while improving product value.

Regulatory alignment

We help teams align design and governance decisions with emerging standards and regulations before these become compliance or procurement blockers.