Maximising Serendipity
The ideas in Nassim Nicholas Taleb’s brilliant new book, The Black Swan: The Impact of the Highly Improbable formed the basis of a previous article Black Swan Logic: Thinking outside the Norm. Black Swans are unanticipated events that have large-scale effects. We arranged Taleb’s ideas into four levels:
I. The nature of uncertainty, unpredictability and randomness
II. How we get fooled by I
III. Why we don’t seem to learn from II
IV. And how we can.
At Level IV Taleb recommends an Apelles-style strategy: “A strategy of seeking gains by collecting positive accidents from maximising exposure to ‘good Black Swans’” (p. 307). Taleb named his strategy after Apelles the Painter, a Greek who, try as he might, could not depict the foam from a horse’s mouth. In irritation he gave up and threw the sponge he used to clean his brush at the picture. Where the sponge hit, it left a beautiful representation of what he had been trying to create — the foam. An Apelles-style strategy is an example of a more general strategy which Teleb calls maximising serendipity.
In this article we present a six-stage model of serendipity and discuss how to recognise and foster such ‘unexpected potential’.
Serendipity in Science
In Serendipity: Accidental Discoveries in Science, Royston Roberts has gathered together dozens of examples of serendipity from just about every conceivable field — from Columbus discovering the New World; to Isaac Newton and his famous apple; to Edward Jenner and the vaccination for small pox; to the discovery of many of the chemical elements; to all manner of materials (such as Celluloid, Rayon, Nylon, Polyethylene, artificial ivory, silk and safety glass); to astronomical serendipities; to the vast majority of medications — and more besides. He says:
Taleb concurs. He says almost all important discoveries are examples of positive Black Swans:
The classical model of discovery is as follows: you search for what you know (say, a new way to reach India) and find something you didn’t know was there (America). If you think that the inventions we see around us came from someone sitting in a cubical and concocting them according to a timetable, think again: almost everything of the moment is the product of serendipity. In other words, you find something you were not looking for and it changes the world, while wondering after its discovery why it “took so long” to arrive at something so obvious. (pp. 166-167)
Not only is serendipity a key component of the discovery of new things, it is also vital to the discovery of new theories. In The Structure of Scientific Revolutions, Thomas Kuhn characterised scientific development as “a succession of tradition-bound periods punctuated by non-cumulative breaks” (p. 208). Those “non-cumulative breaks” turn out to be major scientific breakthroughs which are unpredictable and not derivative of normal scientific work, but depend on an unexpected insight that leads to a better way of understanding empirical relations — what Kuhn called a “new paradigm”.
While the spectacular Black Swan catches the headlines, serendipity also operates on a smaller, more personal scale — which is just as significant to the individual.
Definitions and Types of Serendipity
Royston Roberts notes that:
The word serendipity was coined by Horace Walpole in a letter to his friend Sir Horace Mann in 1754. Walpole was impressed by a fairy tale he had read about the adventures of ‘The Three Princes of Serendip’ (or Serendib, an ancient name for Ceylon, now known as Sri Lanka), who ‘were always making discoveries, by accidents and sagacity, of things which they were not in quest of’. (p. ix)
Over the years its meaning has shifted somewhat from its earlier connotation of “accidents and sagacity” to ‘looking for one thing and finding another’. Today ‘serendipity’ is defined by theOxford English Dictionary as “the faculty of making happy and unexpected discoveries by accident.” As we shall see, this jolly definition pushes into the background a host of important features.
And there is more than one type of serendipity. To distinguish between two types Roberts coined the term pseudoserendipity to describe:
Accidental discoveries of ways to achieve an end sought for, in contrast to the meaning of (true) serendipity, which describes accidental discoveries of things not sought for. For example, Charles Goodyear discovered the vulcanization process for rubber when he accidentally dropped a piece of rubber mixed with sulfur onto a hot stove. For many years Goodyear had been obsessed with finding a way to make rubber useful. Because it was an accident that led to the successful process so diligently sought for, I call this a pseudoserendipitous discovery. In contrast, George deMestral had no intention of inventing a fastener (Velcro) when he looked to see why some burs stuck tightly to his clothing. (p. x)
This means Apelles the Painter’s accidental creation of foam was by Robert’s definition pseudoserendipity.
We conclude that there are two types of ‘classic’ serendipity and two types of ‘pseudo’ serendipity:
- Find Y unexpectedly
- Look for X and find Y unexpectedly
- Look for X and find X by unexpected means
- Find an unexpected use for X or Y.
The first two are classic serendipity while the last two are pseudoserendipitous.
Serendipity and Synchronicity
The distinction between serendipity and synchronicity is a matter of time. With synchronicity there is an immediate recognition of the ‘meaningful coincidence of two events happening close in time’. Serendipity, however, cannot be assessed until later when the consequences of events are evaluated.
Synchronicity can become serendipity if the effects of the coincidental events have large positive significance over time. However serendipity can also arise out of events that are not synchronous. This gives four possibilities:
- Synchronicity leading to serendipity
- Synchronicity leading nowhere
- Ordinary events leading to serendipity
- Ordinary events leading nowhere
The rest of this paper describes our perceptual model of serendipity and how we can prepare ourselves to maximise it.
A Perceptual Model of Serendipity
While definitions of serendipity use words like ‘sagacity’ and ‘happenstance’ we are interested in what is happening perceptually. We think there are six components which need to be in place for either classic- or pseudo-serendipity to have occurred. In the following model ‘E’ is the event which, with hindsight, proves to be serendipitous. E-1 is a time before E, E+1 is a time after E, and so on.
| E-1 | A prepared mind. |
| E | An unplanned and unexpected event happens. |
| E+1 | There is a recognition of the potential for positive significance in the future. |
| E+2 | At some point action is taken which aims to amplify the potential for positive effect of E. |
| E+3 | Over time there will be effects of the action and of other things happening, which can be utilised to further amplify the benefit of E. |
| E+4 | The value of the original event and the subsequent effects becomes apparent — at which time serendipity can be said to have taken place. |
Notes:
- Serendipity is ‘the whole shebang’. Take out any of the six components or the iterative nature of the process and it wouldn’t be serendipity.
- E+1, E+2, E+3 and E+4 together form an iterative process. This is where qualities like persistence, determination and sometimes bloody-mindedness are useful.
- Sometimes the process involves further potentially serendipitous events. And sometimes it further prepares the mind (at which time learning can said to have taken place).
A more detailed explanation of each event follows:
| E-1 | Is a mind that is prepared to recognise unexpected potential and then seize the moment. |
| E | Is an event that is “unanticipated,” “anomalous,” and “strategic”. In this context ‘strategic’ means having the potential for long-term value. |
| E+1 | Requires someone to detect or recognise the unexpected event as potentially serendipitous. As Isaac Asimov said, “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny …’.” ‘Detecting potential’ involves an evaluation. However, because ‘potential’ has not yet happened it must be a forward-facing evaluation. This creates an apparent paradox: while it is impossible to know in advance the long-term consequences of any event, a person with expertise in noticing serendipity will be more likely to recognise the ‘pattern of potential’. For example, when David Grove first presented his ideas about Clean Space to us, Penny said “This has got legs.” She recognised that Clean Space would be around for years and that great things could come of it, even though at the time she could not articulate what they would be. |
| E+2 | Choosing the appropriate action to preserve and amplify the potential, not necessarily the event, is quite a skill in itself. The action can occur immediately after the recognition of the serendipitous potential of the event, or much later. |
| E+3 | Consequences happen of their own accord. It often takes the confluence of the effects of other happenings to turn an event from an interesting anomaly into serendipity — you need to be both in the right place and at the right time. |
| E+4 | Evaluating the effects in relation to a larger system requires a value judgment. This evaluation is backward-facing and is of a different type to that in E+1. It is personal because evaluating whether something is ‘positive’ depends entirely on the perspective of the evaluator. As the old Chinese story of the farmer and his son goes, the evaluation of something as ‘good news’ or ‘bad news’ tends to alternate over time. |
Modelling the Ability to Realise the Potential of Serendipity
Louis Pasteur said, “In the fields of observation, chance favors only the prepared mind.” But what is a “prepared mind” and how can it be cultivated?
Royston Roberts looked for scientists who had a knack for turning accidents into discoveries. He identified that Joseph Priestly, Louis Pasteur and William Henry Perkin had such a talent. In studying their methods he identified two necessary qualities:
- The dominant characteristic was curiosity. “They were curious to understand the accidentthey had discovered.”
- The other characteristic was a certain way of perceiving: “They observed a phenomenon that was unexpected, and they took note of it rather than dismiss it as trivial or annoying.”
He concludes serendipity “arises not from random accidents but from circumstances brought about by unconscious motives which lead ultimately to the serendipitous event”.
Gary Fine and James Deegan studied ethnographers with a similar skill:
Fine and Deegan note that although the “unstructured, inductive quality of fieldwork often provides leeway to incorporate the power of serendipitous findings” it cannot be acquired by determination alone. In addition they identify a number of factors that depend on the ethnographer’s readiness to seize upon chance events. They categorise these factors by three “opportunities” for serendipity:
Serendipity relations – the unplanned building of social networks
Analytic serendipity – discovering concepts or theories that produce compelling claims
Temporal Serendipity
Is in part, a function of the observer’s ability to:
- Judge where the action is.
- See a pattern or implication that has gone unnoticed and, having exposed it, to find it in other social settings.
- Discover novel data sources.
Serendipity Relations
- The relationship between the researcher and the subject of the research is central in ethnographic research. Though we can find variables that contribute to the likelihoodof a social tie emerging and flourishing, relationships develop from a situationally mysterious process in which physical presence, chance remarks, and unplanned actions connect persons to each other. It is not sufficient that one makes contact (good fortune), but one must also be able to capitalize on this contact lead to other relationship opportunities.
Analytic Serendipity
- Involves the ability to establish connections between data and theory . The formal model of deductive research assumes that one knows what one is looking for before one has “found” it. Inductive research elevates “insight” or serendipity into the chosen stance of analysis, permitting numerous conclusions from the same data.
Fine and Deegan conclude that serendipity is the “interactive outcome of unique and contingent ‘mixes’ of insight coupled with chance.” They suggest the mix contains four contingencies:
- First, the researcher has previously been exposed to the relevant literature. These previous studies provide a grid in which new data can be incorporated, creating a template for the development of new theory. Theory never develops out of thin air, but is responsive to those intellectual currents that are in circulation and to which the researcher has been exposed.
- Second, the data themselves speak to the researcher. On occasion portions of the data play off each other. The unexpected similarity or dissimilarity may provoke an “ah-ha!” response. Suddenly, as in scientific revolutions, anomalous data fit because of their relations to each other or to some aspect of the literature. One example shines a light upon another.
- Third, the researcher may discover a dramatic metaphor or narrative strategy that permits him or her to conceptualize and present the problem in a novel light.
- Fourth, a researcher may be influenced by being part of a scholarly world: a system of collective action or invisible college. Through a shared intellectual community, fresh insights may arise. Becker (1982) noted that it is the network of “cooperative links” rather than the lone artist that is central to the production of art.
Jim Gritton identifies a number of other processes :
Liestman (1992) describes this latent “skill” as “intuitive sagacity”, which comes from “a random juxtaposition of ideas in which loose pieces of information frequently undergo a period of incubation in the mind and are brought together by the demands of someexternal event, such as a reference query, which serves as a catalyst.” In similar vein, Foster and Ford (2003) talk about serendipity’s role in surfacing hidden connections or hidden analogies, which are revealed through serendipitous links between information sources, much like the unexpected but useful links on one web page to a seemingly unrelated other.
It is worth noting that serendipity often derives from those unplanned happenings that stem from our own ‘mistakes’ or ‘failures’. This may explain why the metaphor ‘stumbling’ is commonly used to describe the original serendipitous act. Fine and Deegan report on the wonderfully named Hortense Powdermaker whose stumbling led to significant ethnographic work. They also recognise that stumbling is only the beginning of the process: “It matters little that Powdermaker stumbles. What matters is that she rallied as a result of her keen sense both of involvement and detachment in participant observation, turning adversity into account. Learning how to learn from mistakes is critical for using serendipity in qualitative research.”
Judith DeLozier and John Grinder tell the story of their modelling of a Congolese drummer TaTitos Sompa. When asked how he composed new drumming rhythms he said : I just play and every now and then I make a mistake. If I like the mistake I incorporate it into the rhythm.
Highlighted Qualities in Relation to the Perceptual Model
We took each of the qualities highlighted in the previous section and mapped them on to our perceptual model of serendipity:
E-1 Prepare the mind for serendipity
- Have previously been exposed to relevant literature (As Caroline Myss says, “intuition is based on acquired knowledge”)
- Learn how to learn from mistakes
- Have unconscious motives (e.g. set an intention to be open to discovery)
- Build experience of judging where the action is
- Look for novel data sources
- Juxtapose ideas in novel ways
E The unexpected event
E+1 Recognise the potential for serendipity
- In the rush of ongoing events keep your wits about you
- Take note of the unexpected, the anomalous, the surprising
- Be curious to understand the accidental discovery
- Use your ability to see a pattern or implication (tune in to your pattern-detection signals)
- Let the data speak for itself (and hold back from jumping to conclusions)
- Search for a fit which includes anomalous data
- Be open to finding hidden connections (think background rather than foreground)
- Consider ‘mistakes’ and unwanted events as opportunities rather than ‘failure’.
E+2 Seize the moment
- Be ready to seize upon chance events
- And to stumble in the process
- Capitalize on unexpected contacts and relationship opportunities
- Establish connections between data and theory
- Be on the lookout for the emergence of a metaphor
E+3 Amplify the confluence of consequences
- Look out for network effects. These can stem from the unknown links of existing contacts (i.e. ‘small worlds’ with ‘six degrees of separation’), or the possibility of establishing amplifying feedback loops or contagions.
- Share your ideas with a supportive intellectual community (i.e. Seek co-inspiration)
- Be open to external events (i.e. Make friends with the Law of Unintended Consequences)
- Remember, even a single question can start a train of thought that leads to unplanned beneficial consequences.
E+4 Evaluating the effects
None of the authors remarked on this phase. They all seemed to presuppose that when we see one we will naturally recognise and agree on what a ‘positive’ Black Swan is. There are countless examples where this is not the case. For example, some people see the discovery of nuclear power as a ‘positive’ Black Swan while other see it as a ‘negative’ Black Swan. Furthermore, some people who originally thought of it as ‘positive’ now consider it ‘negative’, and vice versa.
Note: The iterative circularity of E+1, E+2, E+3 and E+4 is vital, but receives very little press.
Preparing the Mind
The majority of the following suggestions for preparing your mind to maximise serendipity are taken from Taleb’s The Black Swan, with a few of our own observations stirred into the mix. We have organised the points to fit in loosely with our perceptual model described above.
E-1 PRIOR STUDY
Although the kind of curiosity and perception described by Roberts may be more inherent in some persons than others, he is confident that they can be encouraged and developed. Roberts quotes from an article titled “Educating for the Serendipitous Discovery” (Journal of Chemical Education, Vol. 62, 1985) where Ronald Lenox described three ways in which students can be prepared to take advantage of fortuitous accidents:
The first way is to provide training in making and recording observations, including unexpected as well as expected results.Students should be encouraged to be flexible in their thinking and interpretations. The person who sees only what is expected and discards unexpected results as “wrong” will make no discoveries.Another way in which a person can prepare to benefit from serendipity is through careful and intensive study in the field of chosen investigation. (“Due diligence” as David Grove called it.)
For example, although Fleming was not looking for an antibacterial agent at the time a spore floated into his petri dish, he was extremely well read and trained in microbiology and could readily recognize the meaning of the clear area in the bacterial culture produced by the accidental implantation of the mold.
E-1 KNOW THYSELF
Account for your emotions:
“Most of our mistakes in reasoning come from using when we are in fact thinking that we are using ” (p. 82, Taleb).“Train your reasoning abilities to control your decisions. Train yourself to spot the difference between the sensational and the empirical.” (p. 133, Taleb)Taleb’s point is that our emotional responses will unconsciously dominate unless we take this into account when we make decisions in the nonlinear world of Extremistan. He is not saying ignore your intuition, he is saying know when you are using it. Malcolm Gladwell’s bestseller,Blink, extolled the wisdom and accuracy of split-second intuitions. Recent research by Hans Welling suggests that therapists should trust their intuition but “when therapists intuit something they should consider not saying anything immediately, but observing and listening to the client instead, using their consciousness to assess what they think they’ve noticed”.
Recognize self-delusion:
“You cannot ignore self-delusion. The problem with experts is that they do not know what they do not know. Lack of knowledge and delusion about the quality of your knowledge come together — the same process that makes you know less also makes you satisfied with your knowledge.” (p. 147, Taleb)
The processes of self-delusion rely heavily on confirmation bias described below and the illusion of validity.
Define the game for yourself:
“It is more difficult to be a loser in a game you set up yourself. In Black Swan terms, this means that you are exposed to the improbable only if you let it control you. You always control what you do; so make this your end.” (p. 297, Taleb)
ADOPT META-THINKING STRATEGIES
E+1
Distinguish between behaviour and everything else:
Pay attention to what actually happens, separate this from your own and other people’s interpretations, and hold off from jumping to conclusions.
“Additional knowledge of the minutiae of daily business can be useless, even actually toxic. The more information you give someone, the more hypothesis they will formulate along the way, and the worse off they will be. They see more random noise and mistake it for information. The problem is that our ideas are sticky: once we produce a theory, we are not likely to change our minds — so those that delay developing their theories are better off. When you develop your opinions on the basis of weak evidence, you will have difficulty interpreting subsequent information that contradicts these opinions, even if this new information is obviously more accurate. Two mechanisms are at play here: Confirmation bias and belief perseverance. Remember, we treat ideas like possessions, and it will be hard for us to part with them.” (p. 144, Taleb)
Once we treat ideas as possessions, the “Endowment Effect” kicks in whereby we are reluctant to give up something we own, even if there is greater value in doing so than holding on to the possession.
Focus on Potential Consequences:
The so-called ‘serendipitous event’ is no such thing unless you have recognised its potential. And even then that’s only the beginning of the process of finding out if the potential can be realised in ways that no one could foresee. We believe it’s possible to develop a ‘potential detector’ that can be likened to dowsing.
Maintain a state of ‘not knowing’:
Surprisingly we found little mention of the importance of cultivating a state of not-knowing. In Zen it’s called a ‘beginner’s mind’. This does not mean ignoring all our acquired knowledge, nor abandoning tried and tested methods. It means fully realising at each moment that we really don’t know what is going to happen next; that what happens next may blow away everything we thought we knew; that what happens next may be the most serendipitous event of our life; or, that all our efforts might come to nought. But whatever does happen is what happens — and that’s information.
E+2
Use an unequal barbell strategy:
“A method that consists of taking both a defensive attitude and an excessively aggressive one at the same time, by protecting assets from all sources of uncertainty while allocating a small portion for high-risk strategies.” (p. 307, Taleb)
In other words, most of what you do should follow follow a safety-first principle, but a small proportion can be devoted to going for something out of the ordinary on the off chance that it points to something much more significant.
E+3
Allow for confirmation bias:
Confirmation bias means that we unconsciously seek evidence that ‘confirms’ or ‘supports’ our theories and decisions while ignoring evidence to the contrary. Confirmation bias helps us to lie to ourselves. Even more pernicious, it introduces a double bind because it makes us believe that we see the situation clearly, even if others can’t. Tavris and Aronson point out that confirmation bias is particularly useful as a way of avoiding cognitive dissonance.
Remember silent evidence:
We only ever see a fraction of the whole. Most of the evidence is ‘silent’ because it is not available to us — most research is never published, and even when it is, there are always books we haven’t read, ideas we haven’t come across, fossils that haven’t been discovered, roads never travelled, etc. etc. etc.
Avoid tunneling:
Instead of neglecting the “sources of uncertainty outside the plan itself,” go in search of those uncertainties. (p. 156, Taleb) Remember, if you want to make God laugh, tell Her your plans.
Hold hypotheses lightly:
As Hortense Powdermaker says “An over commitment to a particular hypothesis, or a rigidity in personality may prevent a fieldworker from learning as he stumbles” (quoted in Fine & Deegan). Be prepared to switch horses mid-steam, if that’s what the evidence suggests.
Use stochastic tinkering:
Taleb says to use a lot of trial and error. We prefer to call it, ‘trial and feedback’ since feedback presupposes the noticing of ‘errors’ and acting on that awareness. To do that you have to learn to “love to lose”, to be wrong, to be mistaken, to stumble — and keep trialling. This is part of the ‘iterative circularity’ in our perceptual model. Wikipedia is a human-made example of stochastic tinkering — life is Nature’s example.
Foster cognitive diversity:
“The variability in views and methods acts like an engine for tinkering. It works like evolution. By subverting the big structures we also get rid of Platonified one way of doing things — in the end, the bottom-up theory-free empiricist should prevail.” (p.224-5, Taleb)
Taleb also lists a number of “modest tricks” for maximising your exposure to positive Black Swans. He suggets the more modest they are the more effective they will be:
- Make a distinction between positive and negative contingencies.
- Remember narrow-mindedness has an analgesic effect.
- Be wary of precise plans.
- Don’t waste your time trying to fight people who forecast.
- Beware of the magic-like effect of numbers.
- The bottom line: invest in preparedness, not prediction.
- Then, seize any opportunity or anything that looks like an opportunity.
Serendipitous Learning and the WWW
Jim Gritton asked the question: Does ‘aimless’ browsing on the web lead to serendipitous learning? He suggests:
There are many who argue that aimless surfing is more productive than people might at first imagine. Callery (1996), for example, draws our attention to a largely unintended benefit of browsing – “the serendipitous discovery of related items. In cases in which the user may be looking for a specific site and doesn’t see it in its subject area, chances are that other sites grouped in the same area may have something useful”.
Gritton notes that this is not unlike the psychoanalytical concept of free association. This process assumes that all memories are arranged in a single associative network, and that sooner or later the patient will stumble across the crucial memory — or when browsing a crucial page. He goes on to quote Steven Johnson (2006):
“Thanks to the connective nature of hypertext, and the blogosphere’s exploratory hunger for finding new stuff, the web is the greatest serendipity engine in the history of culture. It is far, far easier to sit down in front of your browser and stumble across something completely brilliant but surprising than it is walking through a library looking at the spines of books.”
Gritton notes that Marchionini (1997) likens serendipitous browsing to something “like an intellectual lottery … low probability but high payoff”. A small input can give enormous results and vice versa. If we are lucky, we may stumble across that obscure but vital gem of information that has not been seen before or categorised as relevant. Gritton concludes:
The literature on serendipitous learning is still in its infancy compared to more mature theories of learning. However, it seems to have emerged as a subset of exploratory learning, which itself is rooted within the constructivist tradition. There can be little doubt that anything which encourages exploratory behaviour and therefore learning should be encouraged, but whether serendipitous browsing is a sensible or prudent learning strategy is another matter. Serendipitous browsing does however have the potential to reveal connections between ideas that may otherwise go unnoticed, to stimulate “out-of-the-box” thinking, and to challenge our mental models so that new learning can take place. In this regard, serendipity, free association and aimless browsing can lead to serendipitous learning.
And Finally …
While the so-called serendipitous event gets all the press, remember that everything that happens before and after is as important, if not more important. Forgetting this is itself is an example of Taleb’s rule that we focus on the “sensational” at the expense of the “relevant”. We hope that simply raising your awareness of serendipity and how it works will mean your mind is more prepared. As Taleb says:
“We are quick to forget that just being alive is an extraordinary piece of good luck,
a remote event, a chance occurrence of monstrous proportions.
Remember that you are a Black Swan.”
Document History
May 2011 – Diagram amended so that the iterative process always includes an evaluation of the effects, E+4; and sometimes includes further unexpected events, E, and sometimes further prepares the mind to recognise and act on potentially serendipitous events. Our reasoning is that nothing is ‘serendipity’ until someone says so. Therefore there is always an evaluative process involved.
POSTSCRIPT:
The model presented in the above paper has been used in two academic articles published in the Journal of Documentation, Vol 68 Issue 5.:
Makri, S. & Blandford, A. (2012). Coming across information serendipitously: Part 1 – A process model,
Makri, S. & Blandford, A. (2012). Coming across information serendipitously: Part 2 – A classification framework.
More recently James wrote a blog on the subject of serendipity and personal change.
