Part 4 · Defense & National Security · What worked

DARPA Digital Tutor

About this resource · How to use it · Learning engineering connections

What is this resource?

A documented case about DARPA Digital Tutor. A sponsor-commissioned IDA assessment found a 4.30 knowledge-test effect relative to a selected experienced fleet group. Practical effects varied by task.

How should I use it?

  1. Read the narrative and five beats, keeping the disclosures and scope limits in view.
  2. Compare the LE insight with the LENS approach. Identify the system condition that changed what people could do, then check the cited evidence before applying the lesson elsewhere.

How does this connect to LENS and learning engineering?

The tutor links expert teaching techniques to a measurable capability and a series of assessments. The large knowledge-test result comes from one small, nonrandom comparison. Knowledge was associated with practical performance but accounted for only about 40% of its variance. Keep individual knowledge, team exercise performance and later work separate. Specify representative IT tasks before building instruction, document participant selection and test delayed transfer. The available architecture detail is too limited to reproduce the full tutor.

2009 – 2014 defenseworkforce L&Dintelligent tutoring

Impact. A sponsor-commissioned IDA assessment found a 4.30 knowledge-test effect relative to a selected experienced fleet group. Practical effects varied by task.

Documented learning engineering. People designed or evaluated a way to develop capability. Inspect what the evidence measures and how far it transfers.

Ask AI about this case Opens a new chat prefilled with this case — explore it in your own context.
Consider the human, the science and the system
Human
Who gained or lost agency, opportunity or well-being? Whose experience is missing from the record?
Science
What evidence supports the proposed learning explanation, and what other causes could explain the outcome?
System
Which tools, responsibilities, interfaces or conditions enabled the result or prevented recovery?
Explore the three foundations →

DARPA built the Digital Tutor by studying expert tutors and modeling their practice in software. An Institute for Defense Analyses assessment for the sponsor found that graduates outscored selected experienced Navy technicians on an individual knowledge test after 16 weeks. The comparison used twelve people per group, without random assignment. Team exercises captured a different outcome, and the program’s overall design rating did not significantly exceed the fleet comparison.

This is a case in designing instruction around work people must perform and measuring different parts of capability separately. To test whether the results carry into fleet work, follow graduates into assignments and assess what they can diagnose and repair over time.

In brief

DARPA designed a digital tutor based on expert one-on-one tutoring for Navy information systems technicians. The IDA assessment for the sponsor compared 12 graduates of the 16-week program with 12 standard-course graduates and 12 fleet technicians averaging 9.1 years of experience. The groups were selected rather than randomly assigned. The digital tutor group outscored the fleet comparison group on the individual knowledge test, with an effect size of 4.30. Practical exercises were done in teams. Results varied by task, and the overall design rating was not significantly higher than the fleet group’s. The assessment did not measure sustained fleet performance.

The case in five beats

  1. The Digital Tutor modeled expert one-on-one teaching for Navy IT tasks
  2. The sponsor’s assessment compared three nonrandom groups of twelve
  3. The knowledge-test effect versus fleet technicians was 4.30
  4. Practical exercises were done in teams and results varied
  5. The assessment did not establish later on-the-job transfer
The Learning Engineering Lens

LE insight

The tutor links expert teaching techniques to a measurable capability and a series of assessments. The large knowledge-test result comes from one small, nonrandom comparison. Knowledge was associated with practical performance but accounted for only about 40% of its variance.

LENS approach

Keep individual knowledge, team exercise performance and later work separate. Specify representative IT tasks before building instruction, document participant selection and test delayed transfer. The available architecture detail is too limited to reproduce the full tutor.

The LENS competency this case exercises
  1. 1 Systems Analysis
  2. 2 Iterative Development
  3. 3 Human-System Collaboration
  4. 4 Test & Evaluation
  5. 5 Sociotechnical Constraints

Problem type · PT4

Problem type
D2/PT4
Induced
1.2
LEO
2, 4

What this case does not show

This small, nonrandom assessment cannot establish equivalence to years of fleet experience or predict sustained operational performance.

Sources and evidence