Summary
Thoughts
Notes
[…] it’s nearly always more fruitful to mine the literature for already-extracted mental models of expertise. A few weeks ago, I learnt of Lia DiBello, an NDM researcher who specialises in the extraction of tacit mental models of business expertise.
DiBello’s findings are extremely compelling. She writes, in The Oxford Handbook of Expertise (a):
After several years using different methods, we noticed that highly talented business performers are very similar to each other. That is, they have decoded the domain of business much like chess masters had decoded chess. This led us to realise that —as a domain—business itself is an orderly closed system of relations between principles, and that so-called intuitive experts in business have an implicit grasp of this. If this is the case, it would follow that all highly skilled business experts should recognise each other and share a common mental model of these principles, although they may manifest differently in different industries. Since business has been evolving in recent years, this model would have to be very robust, running deeper than recent societal changes. (emphasis mine)
[…]
I dove into DiBello’s publication history [Deep-dive reading]. This turned out to be relatively easy — I quickly learnt that DiBello has published a relatively small number of papers, especially when compared against the research output of some expertise researchers. Initially, I found this frustrating, but then I realised that she has spent the bulk of the past 20 years consulting for big companies, applying her research to the real world. (During her interview on the NDM podcast (a), she says (54:04) somewhat ruefully, “Because of how I wanted to get a lot of data, I ended up selling my soul to big business, and I made a lot of rich people richer. But it was how I got to prove that what we do works.”). She spends more time today attempting to adapt her training methods for broader uses, to make them available to businesspeople who might not be able to afford a full consultation. To my mind, this makes her more, not less, believable (a).
DiBello’s main ideas
The bulk of her work follows from these three big ideas:
- Business expertise consists of a handful of properties: (a) it is team-based — meaning that business expertise is actually a form of distributed cognition, with expertise spread amongst an executive team; (b) it consists of two things: domain-specific mental models of the business and cognitive agility; and (c) there is a deeper mental model that underpins those domain-specific mental models, which explains how good business people are able to recognise other good business people, even if they are from different industries.
- She argues that general problem solving ability does not matter so much in business so much as a property she describes as cognitive agility — which is the ability to update mental models in light of new information.
- Her primary innovation is an intervention known as the FutureView Profiler. The Profiler is able to evaluate the shared mental models of an executive team: it is able to (a) identify blind spots in the executive team’s model of the business, (b) able to evaluate cognitive agility, and (c) able to measure expertise in business primarily because it asks business people to do year-by-year predictions of similar companies within their own industries.
I’ve highlighted the most immediately useful elements of her work above, but I want to note that the ideas sit on top of a few other contributions that are novel from an expertise research perspective:
- DiBello’s insight that domain-specific prediction tasks are a useful way to evaluate business skill is a huge contribution to the literature on expertise. She spends a few paragraphs in her 2010 paper [2] discussing why the Profiler works, and why other assessments of business skill do not. (The gist of the argument is that you want to evaluate how a business person’s mental models update in response to a dynamic context; you do not want to evaluate what currently exists in their heads).
- Her company, Workplace Technology Research Inc (WTRI), has spent years implementing company training simulations in virtual worlds. Apart from the obvious technical effort (she explains [3] that they decided to code a full world of their own, on a game engine of their choice, mostly because existing companies they collaborated with kept getting acquired or shut down) — she has novel insights on the pedagogical effectiveness of using virtual game worlds to teach business simulations.
- Last, but perhaps most importantly, DiBello argues that business expertise is primarily a matter of organisation. She writes, in the Oxford Handbook: “Like experts in other domains, business experts rely not on greater analysis or greater information, but better ways of structuring or organising their knowledge. Further, a business expert differs in the manner in which he or she looks at the business landscape, particularly with regard to what is linked with what and what is important to manipulate.” As a result, the bulk of DiBello’s training methods aim to do what is called ‘cognitive reorganisation’ — that is, WTRI does not seek to teach business people new concepts about their domain per se, but to instead reorganise existing domain-specific mental models in their heads.
The mental model of business
In the Oxford Handbook, DiBello writes:
In a study funded by the National Science Foundation conducted over four years (NSF Award ENG 9548631) we found that those who show considerable and consistent talent in business have such a mental model, shown in predictable ways of making use of information. Leveraging our relationships with companies and clients, we had unusual personal access to study a large number of highly placed leaders. Doing in-depth studies of talented business leaders who were repeat successes—even in very challenging markets—and who had risen to very high positions (such as chairmen of large corporations) and maintained that level of position even as business itself has grown much more complex, we discovered a shared mental model among all of them. [1]
That shared mental model is essentially a triad:
Our research revealed that people who have achieved a high level of business expertise have a deep understanding of the following three core areas: (1) factors involved in effective operations, (2) forces influencing the market, and (3) those driving business finance and economic climates. Consistently successful business leaders have been shown to intuitively understand these areas and their impact on each other, and to pay attention to this fundamental triad in a uniquely dynamic way within a guiding context of business strategy. For example, these experts are able to sense that market conditions change based on environmental indications that others may fail to notice. Further, they foresee the consequences to their business operations and finances, and thus make necessary adjustments proactively. Unlike most business professionals, they are attuned to the early indicators of widespread change. Beyond this, they are expert at keeping the triad in balance, or shifting the balance when external conditions are conducive to do so (e.g., focusing on marketing during favourable economic times). (emphasis added) [2]
Throughout the paper, DiBello describes the triad using different names. One way of describing it is ‘supply/demand/capital’. Another is ’leadership/strategy/finance’. To summarise:
- Supply, or leadership — These represent operational concerns related to the business, what DiBello calls ‘factors involved in effective operations’. Naturally, how the operational concerns are expressed depends on the industry (e.g. you have to be good at product development (a) if you’re in consumer software, and you must have competent manufacturing if you’re Chobani (a)). Think expansively when it comes to each category of the triad — this category includes things like org design, incentive structures, execution cadence, and even the ability to get things done within the company (e.g. execs must have good mental models of the org and its people, and the ability to push through plans given internal politics).
- Demand, or strategy — These represent the exec’s understanding of the market, what DiBello calls ‘forces influencing the market’. Of course, market dynamics is a very broad category, and includes everything from market shape, competitive analysis, positioning, changing consumer demand, and the ‘path to power’ that we’ve talked about in our discussion (a) of 7 Powers (a).
- Capital, or finance — At the basic level this represents the exec’s understanding of basic financial concepts like cash flow lockup (a), return on invested capital, margins, and their relationship with the other two categories (what DiBello calls ‘factors driving business finance and economic climates’). The factors here may be equally expansive — depending on the business, the capital leg of the triad may include expertise with raising capital (e.g. equity financing, debt) or the ability to understand the implications of changes in the capital environment (e.g. John Malone was amongst the first in the cable industry to take advantage of junk bonds in order to finance expansion (a); a more contemporary example might be startup founders who understand how to manipulate extremely liquid private markets to their advantage).
As DiBello mentions, the key property of business expertise is in understanding how a change in one leg of the triad affects the other two legs, at least within the context of one’s specific industry. Experts are able to notice cues in the environment long before novices are able to, and are then able to forward-simulate what must be done to the other two legs of the triad. So, for example, if there is a change in the capital environment, an expert businessperson might ask how that affects competition (this is the market — the demand leg of the triad). And they might immediately ask what changes the company must make in response to those market shifts? (This is the supply leg of the triad).
This mental model of business is incredibly high-level, but it tells us a lot.
For instance, it tells us that expertise may be evaluated according to the facility an exec has with each of the three categories. DiBello writes:
In contrast, competent managers (one level down from experts) tend to be very talented in only one or two of these areas; however, they often do not understand the dynamics between these areas as well as the “superstars” do. Competent managers are likely to be very successful when larger market or economic trends are favourable to their specific skills. (emphasis added) [2]
The implication is that if you want to get really good at business, you should systematically acquire skills in each of the three categories, specific to your particular business, and then — more importantly — grok the relationships between the three categories.
It also explains another observation that DiBello makes:
Experts in the same domain easily identify each other. Even if they don’t agree with another expert’s choices, they easily recognise that they share a common perceptual experience within their area of expertise. [2]
With the triad model, we may begin to understand how an experienced businessperson is able to recognise another experienced businessperson from a different industry, despite having no idea of the specifics of the other expert’s business or market. The answer to this puzzle is that they are able to recognise the intuitive facility with which the other person reasons about supply, demand, and capital in their own businesses! When I read DiBello’s articulation of the model, I thought back to certain businesspeople who have told me that they can tell if someone ‘gets it’, even if they can’t articulate how; what I now think is happening is that their brains must be attuned to the existence of a triad in the other person’s head.
Do I buy this model of business expertise? Yes, I do. Nearly everything I know about business lines up with the model. To my mind, this explains a handful of unrelated phenomena that I’ve long puzzled over. For instance:
- Why are fresh MBA grads unable to grok the implications of, say, cash flow in a given business, despite learning it in their courses? The answer is obvious: they have yet to internalise the relationships between a) market conditions and b) operational realities with c) the financial concepts they learnt in school. Every business is a system, and developing intuition for a system requires you to watch that system in action. Business students who have no real world experience of a business do not usually have such a mental model. DiBello writes, as part of a consulting engagement: ‘As a process check, we administered the same instrument to an equal of number of business-knowledgeable controls (BS or MBA business students who were not yet functioning managers or consultants and had not ever actually run a company of any size). Their performance was much lower. This, too, was an expected result.’ [2]
- How many different ways are there to build Power (a) in business? I used to think that companies with better products would win, and then I believed that companies with better distribution would win, and then I ran a business for awhile and realised that businesses have won from differentiated product or differentiated distribution or differentiated access to capital; the possibilities were more varied than I had originally imagined. With the triad, it’s clearer to me that temporary competitive advantage may emerge from any of the three categories: you could gain a temporary edge from operational excellence (supply/leadership), or from exploiting some market opportunity (through product innovation or repositioning into a new category — this is demand/strategy), or even from exploiting some change in the capital environment that others in your industry do not realise exists (e.g. Malone with debt-funded, tax-sheltered expansion — this is capital/finance).
- Buffett’s quote “I am a better investor because I am a businessman, and a better businessman because I am an investor” makes more sense when seen through the lens of the triad — it’s likely that exposure to equity investing or actual business operations would shore up one or more of the three categories. As a trivial example, a businessperson without a competent understanding of capital markets is missing one leg of the triad; an investor without a sufficiently realistic understanding of business operations (supply) is missing another.
- Everything in An MBA for Business Operators (a) may be organised neatly into one of these three categories. In fact, if you look at Permanent Equity’s checklist through the lens of the triad, you’d realise that it is a mix of supply and capital considerations … but leaves out demand. This makes sense — to a private equity fund like Permanent Equity, the demand/strategy side of things is the bit that is most specific to the individual business, and cannot be generalised.
- Can management consultants or former VCs become good business operators? With the triad in mind, we can be more specific about what must happen: management consultants are likely better equipped in the demand leg of the triad, with some understanding of the finance leg; VCs are better equipped in the finance leg, and somewhat skilled in the demand leg. How well they do is how quickly they level up on the supply leg of the triad, and how well they internalise the relationships between all three.
To reiterate the most important point again: DiBello’s explication of the triad mental model tells us that it is the relationships between the three that we need to pay attention to; expertise in each leg of the triad is more commonplace when we rise through the stages of business ability.
Create node for this triad
Cognitive agility
DiBello also points out business expertise is made up of one other quality — something she calls cognitive agility:
Given the volatility of markets and the complexity of team decision making in large organisations, we need not be as concerned with the general cognitive ability of today’s executive so much as his or her domain-specific expertise and cognitive agility, which is often associated with the kind of intuitive expertise developed through experience. Cognitive agility is what allows executives to rapidly respond to changing situations and revise guiding mental models to meet performance demands. (emphasis added) In sum, these insights require us to shift away from thinking of the cognitive ability of business leaders as fixed and static, and instead to focus on the way in which expertise evolves over time in response to dynamic environments. It also requires us to discard attempts to locate decision-making expertise as a fixed capacity that resides within the individual decision maker. [2]
Building on theories of emergence, adaptation, and flexibility from complexity science and NDM, we use the term cognitive agility to refer to the extent to which an individual revises his or her evaluation of a situation in response to data indicating that conditions have changed. The converse is cognitive rigidity, where the person is impervious to new data, being dominated by a rigid framework or paradigm that acts to filter out new, possibly relevant, information, creating blind spots. [2]
When I read this section of DiBello’s 2010 paper, my eyes basically popped out of my head.
For several years now, I’ve tried to talk about the limits of frameworks, as experienced from the perspective of a business operator. I’ve noticed that when a market changes, or when you’re operating in a local environment, no framework can fully capture everything that you’re seeing as you brush against reality. I’ve attempted to capture this idea by writing blog posts such as Reality Without Frameworks (a), and How First Principle Thinking Fails (a) and Good Synthesis is the Start of Good Sensemaking (a). More importantly, I’ve attempted to articulate DiBello’s notion of cognitive rigidity, with a trait I invented that I named ‘Dismissively Stubborn (a)’. I wrote that piece to explain the type of people I’ve learnt to avoid in startup environments.
But it turns out that DiBello had thought of this, and more, a full decade before I did. I was also surprised — and delighted! — to learn that senior business leaders have thought about this particular aspect of business expertise:
(We present) a case in which we used the Profiler to assess the senior management team of a medical device company. Our relationship with the company began when the chairman and acting CEO asked us to evaluate his senior staff for succession planning purposes. In particular, he wished to decide who, among the division presidents, would succeed him as the CEO. At the time, the company was growing very rapidly. During the four months we worked with the company between 2005 and 2006, we saw them grow from $300 million to $400 million. A pending acquisition was intended to increase the company’s size to $600 million by 2008.
The CEO of the company was aware that there is a qualitative difference between competent managers, who do very well in interviews or on personality tests, and experts, who can perform under challenging conditions. He wanted to protect himself from (mere) competence at a time when his industry was in a high growth phase. He was afraid that a favourable market might be masking a lack of skill—and cognitive agility in particular—among his senior staff. (emphasis added).
The FutureView Profiler
In their 2010 paper, DiBello, Lehmann and Missildine laid out the requirements of a good instrument to assess business expertise:
We believe that an instrument that effectively assesses business expertise must (1) be able to draw on the intuitive expertise of organisational members; (2) tap into the specific mental models individuals use to approach problems, rather than basic problem solving or generalised decision making skills; (3) locate individuals within the distributed cognition of an organisation, that is, specify the individual’s role in the organisation; (4) be able to identify strengths and blind spots of the organisation’s transactive memory system and, in so doing, highlight dimensions of expertise that contribute to high-level decision making; and lastly, (5) measure not only cognitive ability, but also cognitive agility, that is, the capacity to rapidly revise one’s mental model in the face of dynamic feedback.
The Profiler requires the user (1) to examine and analyse the same material used by actual experts (e.g., annual reports, 10K’s, analyst’s reports, etc., from actual companies) to make business decisions, (2) to make predictions about the company with respect to a number of domains (e.g., revenues), and (3) to judge various aspects of the company (e.g., evaluate the management team).
Specifically, the Profiler asks users to answer several questions about the company to predict how its finances will develop over the next five years. Users’ answers are evaluated in terms of Dreyfus’s five-stage model of expertise (a) to determine their business acumen and level of expertise within the domain of business strategy. The Profiler also helps us identify blind spots in users’ thinking; that is, we can discern areas where users seem to miss aspects critical to the company’s performance. Moreover, for each prediction or judgment users must indicate the specific materials they relied on, such as discrepancies in annual report statements or the general state of the company’s finances. This aspect of the Profiler taps into the heuristics that are guiding users’ decision making. [2]
How are users evaluated? DiBello et al note that the triad (the extracted mental model of business) isn’t directly addressed in the Profiler itself — even though it informs the design of the assessment:
These three core areas are not directly addressed in the questions we ask in the Profiler. Rather, an individual’s answers are rated and compared to those of the ideal expert. The questions are based on tangible outcomes and concern the company’s performance. The questions ask users to evaluate their company with respect to its strategy, leadership, and finances. These are areas that senior executives would be expected to make valid predictions about and are good ways to reveal their underlying business expertise. For example, a question about finances would be: As the senior manager for the business, which aspect of financial performance most urgently requires your attention? This question would then be followed by a series of options, such as “cost of goods sold,” “fixed assets,” “receivables,” or “R&D.” To answer this question—that is, to know what is troubling or of concern in the total context of the company at that time—the user must understand operations, finance, and market trends. Answer options reflect levels of expertise, 1 through 5, corresponding to the five stages in the Dreyfus model. The “correct” answer is empirically determined. Because these are actual cases, the outcomes are known and the root causes of problems have been identified.
In addition to these Dreyfus questions tapping their business expertise, users are asked to provide three predictions that refer to quantitative outcomes in the company’s performance. These questions can vary slightly, as the instrument is customised according to a company’s needs; however, they will ask about issues, such as profits and revenues, that are indicative of how well a user can synthesise information about the company to predict real-world outcomes. For example, we may ask: What is your prediction of the company’s profits for the next 12 months? Users then indicate their answers along a 5-point Likert scale whose points are labelled appropriately, for instance, in the preceding example, options may range from “down 20% or greater” to “up 20% or greater.”
After this initial set of questions, DiBello steps forward the simulation and presents the results of their first year of predictions. This is the primary way that they are able to assess cognitive agility:
Users then go on to Year 2. After reading through the Year 2 materials, they are able to see whether or not their predictions were validated by the company’s performance. In other words, they receive feedback as to the accuracy of the previous year’s predictions. They repeat this exercise for five years of material, making predictions and getting feedback as to whether or not their predictions have been validated. This iterative process taps into users’ cognitive agility, allowing us to assess the degree to which users are able to revise their thinking in the face of feedback about company performance and their own predictions. Cognitively agile individuals learn from their inaccurate predictions or judgments and improve as they go along. (emphasis added)
Furthermore, we can see persistent blind spots that may exist and occur consistently across all five years. Thus, the Profiler can provide windows into the acumen, blind spots, mental models, and agility with which users approach complex decision making for a real company. It is important to keep in mind that all of the data provided are from real companies. We do not determine the outcomes, as they are taken from actual annual reports and actual finances over a given period of time.
In addition to evaluating users’ direct responses to the Dreyfus and Likert questions, we also ask users to point to sections of the material they reviewed (including statements in the annual reports and financial summaries) that informed their decision on a particular question. For example, the annual report might mention a recent acquisition. This may reflect a strategic turning point for the company. As such, after users respond to the question on strategy, they must indicate what sections of the material influenced their decision. Users are allowed to choose up to five reasons for their decision. In this case, users should be aware that the recent acquisition plays a major role in the company’s strategy going forward, even if this is not immediately obvious. As such, when users answer this particular strategy question, they should be pointing to the acquisition as a basis for their decision. This not only allows us to evaluate the expertise and accuracy of their responses, but also gives us a window into the kinds of information they use to approach the decision-making process. Furthermore, this aspect of the Profiler allows us to determine whether users are utilising information based on the three core areas listed above: supply, demand, and finance. Specifically, we can see whether users approach the questions in an expert manner by evaluating the degree to which they are paying attention to these core areas as they make decisions.
[…]
I believe DiBello when she says that the Profiler is able to detect blindspots in an exec team, that it is able to evaluate cognitive agility, and that you may use it to assess the expertise of business decision makers. I’m just not sure how this may be used, if one isn’t able to afford DiBello’s services.