Analysing goals and potential outcomes
Six separate principles concern the analysis of the goals and potential outcomes we expect from projects to develop and deploy technology:
- Question desirability
- Clarify externalities
- Require peer reviews
- Involve multiple perspectives
- Analyse the whole system
- Anticipate fat tails
The principle of “question desirability” starts with the recognition that, just because we believe we could develop some technology, and even if we have some desires to develop that technology, that’s not a sufficient reason for us actually to go ahead and develop it and deploy it.
After all, once a project starts, with the goal of developing some technology, the project sometimes develops momentum that can carry it forward almost of its own accord:
- People involved tell themselves and each other that there’s already a commitment to continue the project.
- They may feel they’re in a race with competitors.
- They may feel a sense of loyalty to the project.
- They may feel an obligation to fellow team members, or to bosses, or to others who are assumed to be waiting for the technology.
- They may desire to ship their product to the world, to show their capabilities.
But the result of this inertia could be outcomes that we later bitterly regret.
Therefore, the principle of “question desirability” urges that we take the time to write down what we assume are the good outcomes we will obtain from the technology to be developed. Once these assumptions have been written down, it allows for a more thoughtful and considered review.
The principle also urges that we consider other methods for achieving these intended outcomes – other methods apart from the particular technology that has somehow caught our fancy.
This separation in analysis of desired outcomes, also known as “requirements”, from possible solutions, is a vital step to avoiding unintended consequences of technologies. We may find better, safer, more reliable ways of achieving the outcomes that we have in mind.
The principle of “question desirability” also recommends that we should in any case challenge assumptions about which outcomes are desirable, and we should be ready to update these assumptions in the light of improved understanding.
Finally, we should also avoid taking for granted that agreement exists on what will count as a good outcome.
That takes us to the next principle, “Clarify externalities”.
Recall that an externality is an effect of a project, or the effect of an economic transaction, that is wider than the people directly involved.
Examples of externalities include noises, smells, pollution, resource depletion, cultural chaos, and a general loss of resilience.
The principle of “clarify externalities” draws attention to possible wider impacts (both positive and negative) from the use of products and methods, beyond those initially deemed central to their operation, and seeks to ensure that these externalities are included in cost/benefit calculations.
Therefore we should not just consider metrics such as profit margin, efficiency, time-to-market, and disabling competitors. We need to consider broader measures of human flourishing.
What makes this analysis possible is the effort taken, in line with the “Question desirability” principle, to write down the intended outcomes of the technology to be developed. What makes this analysis more valuable are the principles of “Require peer reviews” and “Involve multiple perspectives”, to which we now turn.
Require peer reviews
The alternative to requiring peer reviews is that we trust the people who are behind a particular project. We may feel they have a good track record in creating technologies and products. Or that they have outstanding talent. Therefore no peer review would be required.
That may be acceptable for projects that are sufficiently similar to those undertaken in the past. However, new technologies have a habit of bringing surprises, especially when used in novel combinations.
That’s why independent peer reviews are required, involving external analysts who are not connected with the initial project team. These analysts should ask hard questions about the assumptions made by the project team.
Involve multiple perspectives
What’s more, this analysis phase, into the desirability of products and methods, in pursuit of the goals that have been stated, should involve people with multiple different skill sets and backgrounds.
This should include expertise in law, economics, and human factors, as well as designers, scientists, and engineers.
To be clear, these independent analysts won’t necessarily have a veto over decisions taken by the project team. However, what is required is that the project team, along with their sponsors, take proper account of questions and concerns raised by independent analysts.
That proper account needs to observe two further principles: analyse the whole system, and anticipate fat fails.
Analyse the whole system
What’s meant by “the whole system” is the full set of things that are connected to the technology that could be developed and deployed – upstream influences, downstream magnifiers, and processes that run in parallel. It also includes human expectations, human beliefs, and human institutions. It includes aspects of the natural environment that might interact with the technology. And, critically, it includes other technological innovations.
When analysing the potential upsides and downsides of using the new technology that we have in our mind, we need to consider possible parallel changes in that wider “whole system”.
For example, rather than simply analysing how a piece of new artificial intelligence might behave in the environment as it exists today, we should consider possible complications if other pieces of new artificial intelligence, including adversarial technology, or new forms of hacking, are introduced into the environment as well.
This analysis might lead to the conclusion that the new technology would, after all, be more dangerous to deploy than was first imagined. Or it could lead to design changes in the new technology, so that it would remain beneficial even if these other alterations in the environment took place.
Anticipate fat tails
The principle of “Anticipate fat tails” urges us to remember that not every statistical distribution follows that of the Normal curve, also known as the Gaussian bell curve.
In many circumstances, once we observe the mean of a set of observations, often denoted by the Greek letter mu, and also the standard deviation of these observations, known as sigma, we can be confident that new measurements more than three standard deviations away from the mean will be unlikely. They’ll be seen only around three times in a thousand. And for a new measurement that is more than six standard deviations away from the mean, you would have to wait on average more than one million years, if a new measurement was made every single day.
However, our initial observations of the data might lead us astray. The conditions for the distribution of results being Normal might not apply. These conditions require that the outcomes are formed from a large number of individual influences which are independent of each other. When, instead, there are connections between these individual influences, the distribution can change to have what are known as “fat tails”: outcomes that are more than six sigma away from the previously observed mean – or even more than twenty sigma away from it – can arise, taking everyone by a horrible surprise.
That possibility would change the analysis from “how might we cope with significant harm”, such as a result three sigma away from the mean, to “could we cope with total ruin”, such as a result that is, say, twenty sigma distant.
In practical terms, this means our plans for the future should beware the creation of monocultures that lack diversity – cultures in which all the variations can move in the same direction at once.
We should also beware the connection of hidden connections, such as the shadow links between multiple different financial institutions that precipitated the shock financial collapse in 2008.
If there are reasons to foresee such outcomes, it means we need to rethink our plans for the new technology. Otherwise, the world might experience a shock outcome from which there is no prospect of any recovery – perhaps for generations, perhaps indefinitely.