DARPA Digital Tutor
About this resource · How to use it · Learning engineering connections
What is this resource?
A documented case about DARPA Digital Tutor. A sponsor-commissioned IDA assessment found a 4.30 knowledge-test effect relative to a selected experienced fleet group. Practical effects varied by task.
How should I use it?
- Read the narrative and five beats, keeping the disclosures and scope limits in view.
- Compare the LE insight with the LENS approach. Identify the system condition that changed what people could do, then check the cited evidence before applying the lesson elsewhere.
How does this connect to LENS and learning engineering?
The tutor links expert teaching techniques to a measurable capability and a series of assessments. The large knowledge-test result comes from one small, nonrandom comparison. Knowledge was associated with practical performance but accounted for only about 40% of its variance. Keep individual knowledge, team exercise performance and later work separate. Specify representative IT tasks before building instruction, document participant selection and test delayed transfer. The available architecture detail is too limited to reproduce the full tutor.
Impact. A sponsor-commissioned IDA assessment found a 4.30 knowledge-test effect relative to a selected experienced fleet group. Practical effects varied by task.
Documented learning engineering. People designed or evaluated a way to develop capability. Inspect what the evidence measures and how far it transfers.
Consider the human, the science and the system
- Human
- Who gained or lost agency, opportunity or well-being? Whose experience is missing from the record?
- Science
- What evidence supports the proposed learning explanation, and what other causes could explain the outcome?
- System
- Which tools, responsibilities, interfaces or conditions enabled the result or prevented recovery?
DARPA built the Digital Tutor by studying expert tutors and modeling their practice in software. An Institute for Defense Analyses assessment for the sponsor found that graduates outscored selected experienced Navy technicians on an individual knowledge test after 16 weeks. The comparison used twelve people per group, without random assignment. Team exercises captured a different outcome, and the program’s overall design rating did not significantly exceed the fleet comparison.
This is a case in designing instruction around work people must perform and measuring different parts of capability separately. To test whether the results carry into fleet work, follow graduates into assignments and assess what they can diagnose and repair over time.
In brief
DARPA designed a digital tutor based on expert one-on-one tutoring for Navy information systems technicians. The IDA assessment for the sponsor compared 12 graduates of the 16-week program with 12 standard-course graduates and 12 fleet technicians averaging 9.1 years of experience. The groups were selected rather than randomly assigned. The digital tutor group outscored the fleet comparison group on the individual knowledge test, with an effect size of 4.30. Practical exercises were done in teams. Results varied by task, and the overall design rating was not significantly higher than the fleet group’s. The assessment did not measure sustained fleet performance.
The case in five beats
- The Digital Tutor modeled expert one-on-one teaching for Navy IT tasks
- The sponsor’s assessment compared three nonrandom groups of twelve
- The knowledge-test effect versus fleet technicians was 4.30
- Practical exercises were done in teams and results varied
- The assessment did not establish later on-the-job transfer
LE insight
The tutor links expert teaching techniques to a measurable capability and a series of assessments. The large knowledge-test result comes from one small, nonrandom comparison. Knowledge was associated with practical performance but accounted for only about 40% of its variance.
LENS approach
Keep individual knowledge, team exercise performance and later work separate. Specify representative IT tasks before building instruction, document participant selection and test delayed transfer. The available architecture detail is too limited to reproduce the full tutor.
- 1 Systems Analysis
- 2 Iterative Development
- 3 Human-System Collaboration
- 4 Test & Evaluation
- 5 Sociotechnical Constraints
Problem type · PT4
- Problem type
- D2/PT4
- Induced
- 1.2
- LEO
- 2, 4
What this case does not show
This small, nonrandom assessment cannot establish equivalence to years of fleet experience or predict sustained operational performance.