Problems → Prototypes · LXD
An EdTech Alignment Auditor: Is the App Actually Teaching?
Students can be highly fluent with devices and learning apps while the activity itself remains weakly connected to the intended lesson. Schools often evaluate tools by engagement, usability, or vendor claims rather than by the learner actions and evidence the tool produces.
This exercise runs locally using simple rules. Your entries stay on this page and clear when you reload. No AI service is called.
Confirm the actual task and collected evidence, not just words in the description. Watching a teacher explain is not learner reasoning. Completion without a learner response is not a reasoning artifact. Any text edit resets both confirmations.
Inspect the heuristic
This thin slice uses verb cues for a provisional mapping: passive, active, constructive, or interactive. Keywords cannot resolve who performs an action or whether a description denies that evidence exists. Your actor and artifact confirmations therefore gate the interpretation. Confirmations are self-reports, not validation of learning or evidence quality. The comparison and teacher override matter more than the classifier.
Critical review of this working prototype
Strongest assumption: Side-by-side objective, activity, and evidence descriptions help educators notice mismatches.
Likely failure: Keywords can mistake a teacher explanation for learner reasoning and treat explicitly absent written responses as reasoning evidence. Users can also confirm a learner artifact without inspecting it.
Test that could change the design: Compare this confirmed audit with a three-column paper review on unfamiliar lessons containing teacher demonstrations, absent learner responses and genuine independent explanations. Have blinded reviewers inspect the actual artifacts and revised tasks; reject the added controls if they only increase agreement with labels.
Non-AI comparison: A three-column lesson-review worksheet reviewed by another teacher.
Evidence to collect: Record teacher disagreement, lesson changes, learner products, and performance on a new task; classifier agreement alone is insufficient.
Human control and access: Actor and artifact confirmations are self-reports, not validated evidence. Keep unknown available and warn on uncertainty; do not infer learner reasoning from teacher/tool explanations. Native labeled controls support keyboard use. No learner text is sent remotely or interpreted as HTML.
Changes made during review
- Repair the script syntax error that prevented the app from running.
- Require all inputs and hide stale results after edits.
- Flag unclear evidence and accuracy-only evidence for reasoning objectives.
- Replace reading-specific redesign advice with task-neutral guidance.
- Add explicit reasoning-actor and learner-artifact confirmations, both defaulting to unknown; unknown, supplied reasoning and absent artifacts override reassuring keyword matches with warnings.
- Reset both confirmations and clear the old override after any source-text edit; confirmation changes invalidate displayed results.
- Replace the Evidence actually proves heading with a cues-and-confirmation label; keep no mismatch explicitly provisional and identify confirmations as self-reports.
- Next smallest test: ask educators to verify these confirmations against actual learner task/output samples before expanding the classifier.
This is a design and implementation review, not an empirical validation of learning outcomes.
Take the brief into your own AI environment
Copy a self-contained review prompt, then open the AI workspace you already use. The prompt asks for a falsification test, a non-AI alternative, evidence that distinguishes activity from learning, and human-system risks.
Preview the prompt
Act as a critical learning-engineering reviewer. Review the prototype brief below as a thin-slice experiment, not as a finished product. Return: 1. The strongest design assumption. 2. The most important plausible failure mode. 3. The highest-value falsification test for the next cycle. 4. One credible non-AI alternative that could address the same capability gap. 5. The next smallest prototype worth building. 6. What should be measured to distinguish engagement from learning, transfer, and real performance. 7. Human-agency, equity, accessibility, privacy, or governance concerns that should change the design. Be concrete. Separate evidence-backed claims from hypotheses. Do not reward novelty for its own sake, and do not assume more AI is better. COLLECTION: LXD TITLE: An EdTech Alignment Auditor: Is the App Actually Teaching? CAPABILITY GAP: Students can be highly fluent with devices and learning apps while the activity itself remains weakly connected to the intended lesson. Schools often evaluate tools by engagement, usability, or vendor claims rather than by the learner actions and evidence the tool produces. FIRST PROTOTYPE: Create an AI-assisted alignment auditor that takes a learning objective, lesson plan, and description or capture of a digital activity, then maps the activity to the cognitive work learners are actually asked to do. It suggests a stronger task or teacher prompt when alignment is weak. REPRESENTATIVE USE CASE: A literacy app asks learners to tap vocabulary definitions. The auditor identifies that the lesson target is inferential reading and proposes a short evidence-based explanation task before the app is used. LEARNING / HUMAN-SYSTEM FRAME: The design makes the learner action—not the technology—the unit of analysis. It distinguishes passive, active, constructive, and interactive engagement and asks whether feedback is connected to the learning goal. QUESTIONS ALREADY IDENTIFIED: 1. Can the auditor reliably distinguish engaging activity from useful cognitive work? 2. How should teacher judgment override automated mappings? 3. Does using the auditor improve lesson-level evidence of learning? FIRST-CYCLE FRAMING: Understand: Validate that the real gap is the ability to select and configure technology that elicits the intended learner thinking, not simply low engagement, low tool use, or a workflow inconvenience. Map: Map the system: teacher, learners, lesson objective, digital tool, AI auditor, curriculum, and classroom workflow. Identify where the capability currently succeeds, breaks down, or is masked by other constraints. Instrument: Predefine evidence: objective-task alignment, quality of learner evidence, teacher decision quality, student performance, and tool abandonment. A key disconfirming signal is: teachers produce better-looking lesson plans but learner actions and outcomes remain unchanged.
Open the underlying design brief and eight-step path
A first prototype
Create an AI-assisted alignment auditor that takes a learning objective, lesson plan, and description or capture of a digital activity, then maps the activity to the cognitive work learners are actually asked to do. It suggests a stronger task or teacher prompt when alignment is weak.
Representative use case
A literacy app asks learners to tap vocabulary definitions. The auditor identifies that the lesson target is inferential reading and proposes a short evidence-based explanation task before the app is used.
Questions worth carrying forward
- Can the auditor reliably distinguish engaging activity from useful cognitive work?
- How should teacher judgment override automated mappings?
- Does using the auditor improve lesson-level evidence of learning?
Evidence anchors
These sources motivate the design; they do not validate this prototype.
- Chi, M. T. H., & Wylie, R. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. (2014)
Designing learner actions rather than optimizing superficial engagement.
- National Academies of Sciences, Engineering, and Medicine. How People Learn II: Learners, Contexts, and Cultures. (2018)
Broad learning-science anchor for learner/context/system modeling and transfer.
- CAST. Universal Design for Learning Guidelines, Version 3.0. (2024)
Accessibility, learner agency, multiple representations, and barrier-aware design.
- Hattie, J., & Timperley, H. The Power of Feedback. Review of Educational Research. (2007)
Feedback design and evidence-producing practice.
One possible first pass
- 01 Understand
Validate that the real gap is the ability to select and configure technology that elicits the intended learner thinking, not simply low engagement, low tool use, or a workflow inconvenience.
- 02 Map
Map the system: teacher, learners, lesson objective, digital tool, AI auditor, curriculum, and classroom workflow. Identify where the capability currently succeeds, breaks down, or is masked by other constraints.
- 03 Design
Compare an AI intervention with simpler non-AI options. Specify what the human decides, what the AI may suggest, and which tradeoffs are acceptable.
- 04 Build
Create the smallest usable prototype around one representative task, with expert-curated content, visible uncertainty, and an easy human override.
- 05 Instrument
Predefine evidence: objective-task alignment, quality of learner evidence, teacher decision quality, student performance, and tool abandonment. A key disconfirming signal is: teachers produce better-looking lesson plans but learner actions and outcomes remain unchanged.
- 06 Deploy
Pilot with a small, representative group in a near-real setting; preserve a baseline or comparison condition and document implementation conditions.
- 07 Evaluate
Look for capability growth and transfer, not just satisfaction or activity. Inspect subgroup patterns, human workload, errors, and unintended adaptations.
- 08 Refine
Let the evidence change the problem model. Default back to UNDERSTAND if the assumed gap was wrong; otherwise revisit the earliest step invalidated by the evidence.