Desirable characteristics of technological solutions
Six separate principles concern the analysis of the characteristics that are highly desirable in technological solutions:
- Reject opacity
- Promote resilience
- Promote verifiability
- Promote auditability
- Clarify risks to users
- Clarify trade-offs
The principle of “reject opacity” means to be wary of technological solutions where we don’t understand how they work.
These solutions are called “opaque”, or “black box”, because we cannot see into their inner workings in a way that makes it clear how they are able to produce the results that they do.
This is in contrast to solutions that can be called transparent, where the inner workings can be inspected, where we understand why these solutions are effective.
The principle also means that we should resist scaling up such a solution from an existing system, where any failures could be managed, into a new, larger, system where any such failures could be ruinous.
As it happens, many useful medicinal compounds have mechanisms that are, or were, poorly understood. One example is the drug aspirin, probably the most widely used medicine in the world after its introduction by the Bayer corporation in 1897, whose mechanism of action was not understood until 1971.
Wikipedia has a category called “Drugs with unknown mechanisms of action” that, at the present moment, has 68 pages. This includes the page on “General anaesthetics”.
Many artificial intelligence systems trained by deep neural networks have a similar status. It is evident that such an AI often produces good results, in environments that are well defined, but it’s not clear why it can produce these results. It’s also unclear when such an AI system can be misled by so-called adversarial input, or what are the limits of the environments in which that AI will continue to function well.
So long as the overall process is being monitored, and actions can be taken to address failures before these failures become ruinous, or catastrophic, opaque systems can have a useful positive role in increasing human wellbeing.
But where there are risks of failures escalating, beyond the ability of any intervention to fix in a timely manner, these opaque systems need to be rejected.
Instead, more work is needed to make these systems explainable – and to increase our certainty that the explanations provided accurately reflect what is actually happening inside the technology, rather than being a mere fabrication that is unreliable.
The principle of “promote resilience” means we should prioritise products and methods that make systems more robust against shocks and surprises.
If an error condition arises, or an extreme situation, a resilient system is one that will take actions to reverse, neutralise, or otherwise handle the error, rather than such an error tipping the system into an unstable or destructive state.
An early example of a resilient design was the so-called centrifugal governor, or flyball governor, which James Watt added to steam engines. When they rotated too quickly, the flyballs acted to open a valve to reduce the pressure again.
Another example is the failsafe mechanisms in modern nuclear power generators, which force a separation of nuclear material in any case of excess temperature, preventing the kind of meltdown which occasionally happened in nuclear power generators with earlier designs.
Following the Covid pandemic and the consequent challenges to supply lines that had been over-optimised for “just-in-time” delivery, there has been a welcome rediscovery of the importance of designing for resilience rather than simply efficiency. These design principles need to be further elevated. Any plans for new technology should be suspended until a review from a resilience point of view has taken place.
The principle of “promote verifiability” states that we should prioritise products and methods where it is possible to ascertain in advance that the system will behave as specified, without having bugs in it.
We should also prioritise products and methods where it is possible to ascertain in advance that there are no significant holes in the specification, such as failure to consider interactions with elements of the environment, or combination interactions.
In other words:
- The product is specified to behave in various ways, in a variety of different circumstances, in order that particular goals or requirements are met, as previously discussed in the section on “question desirability”.
- In turn, the product is designed and implemented, using various techniques and components, in order that it behaves in line with the specification.
The principle of “promote verifiability” urges attention on ways to demonstrate the validity of both of these steps: that the specification meets the requirements, without having dangerous omissions or holes, and that the implementation meets the specification, without having dangerous defects or bugs.
These demonstrations must be more rigorous than someone saying, “well, it seems to work”. Different branches of engineering have their own sub-disciplines of verification. The associated methods deserve attention and improvement.
The principle of “promote auditability” has a similar goal to “promote verifiability”. Whereas “promote verifiability” operates at a theoretical level, before the product is introduced, “promote auditability” operates at a continuous and practical level, once the product has been deployed.
The principle urges that it must be possible to monitor the performance of the product in real-time, in such a way that alarms are raised promptly in case of any deviation from expected behaviour.
Systems that cannot be monitored should be rejected.
Systems that can be monitored but where the organisation that owns the system fails to carry out audits, or fails to investigate alarms promptly and objectively, should be subject to legal sanction, in line with the principle “Insist on accountability”.
Systems that cannot be audited, or where auditing is substandard, inevitably raise concerns. However, if auditability features are designed into the system in advance, at both the technical level and the social levels, this will help ensure that the technology boosts human flourishing, rather than behaving in deeply regrettable ways.
It’s the same with verifiability. Systems that have been designed to support verifiability are, other things being equal, more likely to produce technology that is truly beneficial.
Clarify risks to users
The principles that have been covered so far are, to be frank, challenging and difficult.
Compared to these ideals, any given real-world system is likely to fall short in a number of ways. That’s unfortunate, and steps should be taken as soon as possible to systematically reduce the shortfall. In the meantime, another principle comes into play. It’s the principle of being open to users and potential users of a piece of technology about any known risks or issues with that technology. It’s the principle of “Clarify risks to users”.
Here, the phrase “user” includes developers of larger systems that might include the original piece of technology in their own constructions.
The kinds of risks that should be clarified, before a user starts to operate with a piece of technology, include:
- Any potential biases or other limitations in the data sets used to train these systems
- Any latent weaknesses in the algorithms used (including any known potential for the system to reach unsound conclusions in particular circumstances)
- Any potential security vulnerabilities, such as risks of the system being misled by adversarial data, or having its safety measures being edited out or otherwise circumvented
When this kind of information is shared, it lessens the chances of users of the technology being taken by surprise when it goes wrong in specific circumstances. It will also allow these users to provide necessary safeguards, or to consider alternative solutions instead.
Cases where suppliers of technology fail to clarify known risks are a serious matter, which are addressed by the principle “Insist on accountability”. But first, there’s one other type of clarification that needs to be made.
The principle of “clarify trade-offs” recognises that designs typically involve compromises between different possible ideals. These ideals sometimes cannot all be achieved in a single piece of technology.
For example, different notions of fairness, or different notions of equality of opportunity, often pose contradictory requirements on an algorithm.
Rather than hiding that design choice, it should be drawn to the attention of users of the technology. These users will, in that case, be able to make better decisions about how to configure or adapt that technology into their own systems.
Another way to say this is that technology should, where appropriate, provide mechanisms rather than policies. The choice of policy can, in that case, be taken at a higher level.