Summary
Good training programs for imparting Tacit knowledge are simulations.
Thoughts
- The use of video training in tennis, cricket, and baseball lines up well with Spaced repetition. However, you would need to take extra care to ensure the learner is training on what you want them to train on. That is, they may get a card correct because it’s the only video with a little watermark in the corner and what they’ve learned is “the answer for the watermark video is X” and not what you wanted them to learn (e.g. “This pitch is a fastball and I can tell from the pitcher’s stance”).
Notes
A natural response to that post is … “ok, so after you extract that tacit knowledge — now what?” This is an understandable reaction, because the next step after doing all that skill extraction is to use it to learn — to develop a training program for yourself or for others. The main problem with this is that good pedagogy is difficult to do. And, as it happens, I’ve not spent any time talking about how Naturalistic Decision Making (NDM) researchers turn their insights into actual training programs.
Why? Well, this is simple: I haven’t found a good, generalised resource in the NDM literature on how they do so. What I have found is a place in the NDM world to dig; this post is to tell you all about it.
IED defeat
One of the big findings that came out of that CTA (Cognitive Task Analysis) was that the Marines who were really doing a great job at being able to recognise a danger zone in advance were the ones who were thinking like the adversary. It sounds like a pretty simple thing, like no kidding of course they were able to put themselves in the adversary’s shoes, but we thought this is something that young Marines, the individuals who were deploying for the first time, don’t know how to do. They’ve never had the ability — outside a training environment, and it’s not something that’s really been taught in training, at least not in a standard way — they’ve never had the opportunity to think from the other guy’s perspective. So we used the VBS technology — we teamed with Bohemia Interactive — and we built out a module in VBS where the Marines were role-playing the adversaries, and they were responsible for emplacing IEDs in the environment.
So we were essentially putting them in a position where they had to think through how they were going to be effective in placing an IED. Whether they were going to use a cellphone detonator, or … with another sort of IED, they had to think through ‘When is the blue force convoy going to come through? What is the time of day right now? How can I disguise it? Where would be a good ambush zone?’ All of those kinds of considerations that are happening in the real world.
Tennis serve detection
I was working as a video coordinator with a big time football team, and we did video analysis for coaches. And I just felt that more that can be done for player training by putting video and data together […]
(…) but how are you going to systematically train it — how can you get more people over the bar faster? And the way you study it in the labs, is with these occlusion techniques, in particular, temporal occlusion, it cuts off in time. Presented by video.
So with a cricket baller, here’s the run-up, here’s the ball out of hand, boom, cuts to black. And the research subject has to say, well, was that an in-spinner, or whatever category it is, so it’s a categorisation task, and it’s a prediction task — is it going to hit the wicket, and if it’s baseball, is it a ball or a strike? So categorisation and prediction. So now we’re starting to really kind of narrow it down. You can say that the macro-cognitive skill we’re after is anticipation.
“Well, how do you train anticipation?” (And people would say): “Well, you can’t train anticipation, that comes from experience.”
Well, ok. Fine. What is the experience giving you? So we can reduce that back to, ok, it’s some pre-event cues and then it’s prediction. Ok — we can test prediction, and we can train prediction. So let’s take something like anticipation which is the quality that we’re after, and let’s try to operationalise it in some way that we can train, which is prediction.
So the thing was to take the video occlusion method that they used for testing, and turned it into a training program.
[…]
This really started in the early 80s in Australia, with a focus on cricket and tennis. This is the tennis serve return. So you’ve got video of a server, that’s recorded more or less from the point of view of a tennis receiver. So the server tosses the ball, racket comes through, and then cut to black. And you have to say what kind of serve is that?
Now that cut to black might be right after ball contact, as you see the ball coming off the strings of the racket, or right before ball contact, or right in it. So they cut it at these different places in here, and this is an expert-novice research paradigm, so they have advanced tennis players and less skilled tennis players (come in to be tested), and where does their performance diverge?
So for instance in that one we find that if they see even one frame of video — that’s the sharpest knife we have here — you’re dealing with about 33 milliseconds in there … if they see two of those (frames) coming off (the strings of the racket), so you’ve got 67 milliseconds, still a very short period of time, then the experts and the novices can pretty much say what type of serve that was — that was a kick serve, that was a flat serve, or that was a slice serve. Now you cut back to where you can only see a little bit of ball flight … well, their performance both goes down a bit, but the expert not as much. Now we cut to black before the racket hits the ball. And the experts are still able to say what type of serve that is — they haven’t even seen the racket hit the ball, and they can tell you 85% of the time whether it’s a kick, a flat, or a slice serve. And they can even predict backhand or forehand side on the return. And at that point the novices performance pretty much drops off. Now if we cut it back a little bit further, they’re both down to random (prediction).
So now we’ve defined our window of expert advantage. From 50 milliseconds before contact, to 50 milliseconds after contact. There’s a 100 millisecond window in there when the experts have a distinct advantage over the non-experts. So that’s the research finding.
Then if we repurpose that as a training technique, we say “ok, here’s the beginning of the window, you should be able to do that, everyone should be able to (detect the serve)”, and then we start cutting it back and you see less and less and less, until you, as a developing tennis player, are able to, like the experts, at 85%, identify the serve 50 milliseconds before the contact.
[…]
The experts can’t necessarily see what they’re seeing — they almost put it in the ‘ESP category’, like with the firefighters in (Gary) Klein’s story (in Sources of Power). And if you press them enough, they’ll start making stuff up. The way that experts do, because they want to give you an answer. So you really need to start ascertaining where that is.
And so this is where you can bring in another occlusion technique, which is spatial occlusion. We know we have a window in time where that expert is picking up some kind of cues. It’s not ESP, so we know they’re picking up some kind of cues (but) we don’t know what they are. So what happens if we mask out the racket?
Well, as it turns out, we can mask out the racket, and the novice’s performance will just fall off the table, and it will hardly affect the expert’s performance at all. So that tells us that they aren’t getting their cues from the racket. But if we mask out the lower half (of the server’s body) now, the experts ability to identify that serve when it’s cut off really goes down. So that tells us that somewhere in that lower half, they’re picking up very early cues, so there’s plenty of time to help you make that — what seems like an instantaneous reaction. And you can break it down enough that you realise that if someone is hitting a kick serve, they’re going to be a little deeper in their knee bend, their ball toss is going to be behind their head, if it’s a slice, the ball toss will be a little to the side. Some of those are known coaching things — the ball toss, for instance.
And you see the reverse of it. For instance, in Pete Sampras’s biography, he talks about how — when he was 12 years old, he would toss the ball into the air, and his coach would call out the type of serve to hit from there. So he would have to learn to hit kick, flat, or slice from the same ball toss. And so later, when he played in the US Open against Agassi, considered the best returner of the time, Agassi said that he couldn’t read Pete’s serve. It wasn’t the fastest flat serve, it wasn’t the spinniest kick serve, but it all looked the same.
[…]
And (my contribution) is taking these techniques from the research lab and repurpose them as drill and practice. Drill and practice: the bottom feeder of all instructional technology. What elements of extraordinary performances that we observe in many contexts — they just happen to be more measurable on TV and in sports, but they exist everywhere — how can we reduce that to something that can be drilled and practiced?
A tool for police investigations
Wong covers the story of the project in the podcast at about the 29:26 mark (a), explaining that “when we started the work, I was really intent about designing a system that would focus on actually creating an environment that would enable insight. Particularly the kind of sensemaking described by Gary Klein’s Data-Frame model [Data-frame model], and particularly his triple path model of insight [Triple path model of insight] (…) I wanted them (in the consortium) to think about sensemaking and insight, because if you don’t have that, all you’re going to build is a search and retrieval system, which is what every other intelligence analysis system is about!”
Wrapping up
If I may generalise timidly, I would say that all three show an approach where the researcher extracts the cognitive processes of experts, and then teaches the novice by making them simulate those same cognitive processes. The key word here is simulation:
- The IED defeat training program focused on high cognitive fidelity to the real-world task; the video game was simply a medium to help students acquire the mental model that seasoned marines use to predict IEDs emplacement.
- The tennis training program worked because it took apart this tacit macro-cognition aspect of tennis serve recognition, and then taught it to novices by simulating through an occlusion exercise. Fadde could have taught it via lecture. He didn’t.
- The VALCRI project extracted the insight generation process that criminal intelligence analysts used, and then designed an entire system around it.
You’ll notice that all three examples are drawn from the NDM podcast (a). This isn’t an accident! My current hack to learn the NDM method of designing training programs is to listen to as many NDM podcast interviews as possible. My goal: to collect as many prototypes of effective training interventions as I can.
‘Listen to the NDM podcast’ happens to be my recommendation if you want to do the same. It strikes me as somewhat ironic: the best practitioners of tacit knowledge extraction have yet to explicate a coherent explanation of their training methods. You’d think that they would have a generalised framework for training by now. Or at least a review paper. And, again … perhaps they have! Perhaps they have a review paper somewhere, but I haven’t found it yet.
I’ll tell you if I do. Till then, the best that we have is casual consumption of the NDM podcasts (a). I recommend you join me in doing so. If you’re as obsessed with learning as I am, I think you’ll enjoy it as much as I do.