Key open questions about the transition to AGI

This phase of the survey is now complete. For follow-on phases, see here. But please brows the many thoughtful “selected comments from survey participants” that are included for each question below.

The survey draws on some interim findings of an ongoing study being conducted by the Millennium Project, and the results of the survey will be fed back into that study.

(0) Introduction

Purpose of this survey

The open questions in this survey are intended to support and enhance the public discussion about the transition from today’s AI to the potentially much more disruptive AGI (“Artificial General Intelligence”) that may emerge in the near future.

As a result of the answers developed to these questions, and the ensuing general discussion, society’s leaders will hopefully become better placed to create and apply systems to govern that transition wisely.

Not an opinion poll

The purpose of this survey isn’t to measure the weight of public opinion – nor even the weight of opinion of people who could be considered experts in aspects of AI.

Instead, the survey will contain a number of rounds (iterations), with participants having the opportunity to update their answers and comments in the light of answers and comments supplied by other participants.

For some questions, a consensus may emerge. For other questions, a division of opinion may become clearer – along, hopefully, with agreement on how these divisions could be bridged by further research and/or experimentation.

This kind of survey is sometimes called a “Delphi” survey.

Avoiding four shortcomings

Nowadays there are more public discussions about AI than ever before. But the vast majority of these discussions fall short of what is truly needed:

They are preoccupied by the issues and opportunities of today’s AI, rather than the potentially much larger issues and opportunities of AGI
They often take a one-dimensional approach, such as purely technical, rather than wrestling with broader angles such as politics, incentives, and compliance
They offer naïve hand-waving solutions, or (in other cases) an unhelpful “we are all doomed” pessimism
They dwell on a number of well-trodden topics, but sidestep many of the questions that most need attention.

These shortcomings apply, not only to articles posted on social media (e.g. as Twitter threads), but also to formal publications by academics and think tanks.

This survey aims to make amends in all four aspects.

Take courage!

The number of questions in this survey might lead someone to decide not to participate. It may seem like an arduous task. However:

As well as being the subject with potentially the single biggest impact on future trajectories for humanity, AGI is also a subject with many complications. Careful consideration of any one of these complications could lead someone to realize that there is a serious flaw in their AGI worldview, and that, accordingly, they need to rethink their assumptions
Even partial sets of answers will be welcomed. Indeed, the comments and answers that will be shared more widely, as input to subsequent rounds of the survey (see below), will be chosen for their quality rather than on account of their quantity or length.

Definition of AGI

For the purposes of this survey, AGI is defined with only a minimum number of core attributes – rather than also requiring various “stronger” aspects that are sometimes presumed. This minimal definition is as follows:

The ‘A’ part of the name (for “Artificial”) means the AGI is an autonomous computer-based system that, after being started, runs without depending on any active human intermediation (such as relying on humans to select answers on its behalf).

The ‘I’ part of the name (for “Intelligence”) means that the AGI

Can observe data and make predictions about future observations
Can determine which interventions could change outcomes in particular directions
Is aware of areas of uncertainty in knowledge, and can devise experiments to reduce that uncertainty
Can learn from any instances when outcomes did not match expectations, thereby improving its future performance.

Finally, the ‘G’ part of the name (for “General”) means that the AGI matches (or surpasses) human abilities in all the above types of reasoning (observing, predicting, planning interventions, devising experiments, and learning),

With its abilities applying in every domain in which humans are interested (including the arts, the sciences, the environment, political interactions, military interactions, and knowledge and predictions about other humans)
With its abilities also meaning that the AGI, unaided, would be at least as good as humans in working out ways in which to improve its own capabilities further.

Note that this definition makes no assumptions about whether the AGI is conscious or sentient, or whether it “actually understands” what it is observing. (A chess-playing AI can out-manoeuvre humans at the game even though it likely lacks the same kind of understanding of chess that humans possess.) Nor is there any assumption that the AGI will, inevitably, choose its own goals.

Survey process

There will be a number of iterations in the process to approach a consensus regarding the answers for at least some of the open questions in this survey:

The set of questions was initially published here, inviting feedback on the survey as a whole
After several days, a link was added, to a survey form containing all the questions
An open invitation is now being publicized, encouraging everyone who believes they have good insights to share, to take part in the survey, and to provide comments explaining their choice of answers
About half of the questions in the survey ask participants to select from a given set of answers; the other questions invite participants to formulate their own answers; in both cases, participants will be encouraged to write a short comment that explains their answer
Every few weeks (perhaps more frequently), comments that have the potential to significantly improve the public global conversation about the transition to AGI will be selectively copied into the “Discussion” section of updated versions of this page (survey participants will be asked whether their names should be attached to these comments)
As a result of some of the comments, new questions may be added into the survey (and, perhaps, others removed), and a new survey link will be created
People who have previously responded will be given the opportunity to update their answers in the light of new insights they gain from the additional material presented.
This process will repeat until it has reached a point of diminishing returns

Publicity

The survey will encourage participants to post some of their own answers online, on their own blogs or elsewhere, linking back to the main survey page.

That will be another way in which the conversation can reach a wider community.

Background

The open questions in this survey have been compiled by David Wood after reflecting on

The content of episodes of London Futurists Podcast, where these matters are often discussed
The responses in interviews with AI experts carried out during an ongoing Millennium Project AGI governance study
The responses to an earlier online survey carried out (with the support of Rohit Talwar) in the opening months of 2022
Extended conversations following recent presentations on the subject of AGI, for example several that have been organized by London Futurists
Feedback to earlier drafts of this document (thanks to everyone who has offered suggestions!)

(1) Alternative definitions of AGI?

Do you agree that the minimal definition of AGI given above provides a useful starting point for a serious discussion, or would you prefer a definition with significant differences?

If the latter case, which definition would you propose?

Bear in mind that:

Your definition needs to be clear
It needs to be credible that your suggested form of AI could exist at some time in the not-too-distant future
Your suggested form of AI needs to raise major questions about how humanity should prepare for its emergence.

Answer choices: The given definition is a good starting point for an important discussion / A better discussion needs to start from a different definition / I don’t really like the definition but I can proceed to answer at least some of the questions that follow

Aside

The earliest use of the term “Artificial General Intelligence” is believed to be in a paper in 1997 written by Mark Gubrud in conjunction with a poster presented at the Fifth Foresight Conference:

By advanced artificial general intelligence, I mean AI systems that rival or surpass the human brain in complexity and speed, that can acquire, manipulate and reason with general knowledge, and that are usable in essentially any phase of industrial or military operations where a human intelligence would otherwise be needed. Such systems may be modeled on the human brain but they do not necessarily have to be, and they do not have to be “conscious” or possess any other competence that is not strictly relevant to their application. What matters is that such systems can be used to replace human brains in tasks ranging from organizing and running a mine or a factory to piloting an airplane, analyzing intelligence data or planning a battle.

The term received wider use after its adoption circa 2005 by Ben Goertzel and Cassio Pennachin for a book title, based on a suggestion by Shane Legg and conversations with Peter Voss.

Selected comments from survey participants

“If this is meant for a broader audience then a shorter definition might be preferred.” – Anon

“I’d offer a simpler definition: AGI is an AI which has all the cognitive abilities of an adult human.” – Calum

“This is an excellent clarification of what AGI is, necessary for those taking part in the survey and eventually for those who may criticise it. However, we also need a succinct definition, which will fulfil all the criteria describing AGI. I would propose the following definition: Artificial General Intelligence (AGI) is a self-learning intelligence capable of solving any task better than any human” – Tony Czarnecki

“A more succinct form would be intelligence defined as a generalized problem solving capability” – Dirk Bruere

“My sense is that ‘AGI’ means different things to different people – some imagine Skynet (Terminator movies), others just really good chatbots – and in that sense it is not very functional as a definition. It’s OK for discussions where we need an acronym for ‘really really really good AI'” – Vid

“I felt dubious at the first part, ‘A’ of the definition, in the sense that I perceived a gap at inception between humans and machines, while universality of computing (both classical and quantum) might point to a sort of continuity and extension” – Pasquale

“Despite our experience with a growing bouquet of AIs, we continue to be tempted to imagine an AGI which is much like an extremely smart and talented human. Instead it seems at least plausible, if not likely, that we will develop a very big bouquet of narrow AI capacities long before AGI is possible. It also seems likely that when we arrive at AGI it will have a character very different from human intelligence in ways which we have a hard time conceiving now.” – Anon

“I would like to explore the difference between AGI as defined here and AGI+desire/drive/will – which I believe is the true point of no return.” – Bronwyn Williams

“I prefer to define AGI as an intelligence that, when supplied with adequate resources, has the potential to self-improve and evolve into Superintelligence, as characterized by Nick Bostrom. If we were to equate AGI with possessing cognitive abilities on par with humans, I believe it would render the determination of whether an AI system is AGI somewhat vague.” – bioshok

“A potential sticking point is whether AGI needs to be embodied. I do not agree that a physical presence (humanlike or not) is required, though IMHO true intelligence would require an element of agency, the ability to take action and learn from the results. While the current definition mentions devising experiments and learning from new data, it doesn’t give the system the ability to actually act, and as such it would strictly be dependent on humans for learning as the only way in which new data could be acquired would be either humans performing some experiment (formal or not) and sharing the results with the AI, or humans assisting in performing experiments (in the broadest sense) on behalf of the AI.” – Kristian Freed

“I sometimes feel that the constituent parts of an acronym is less important than the overall meaning. Like with ATMs. I’m not sure many people consciously think of the acronym as Automated Teller Machines, as evidenced by people mostly referring to them as ATM Machines. I think the acronym has taken on it’s own meaning, and I don’t think anything would really be gained by defining what automated means in this regard, and what teller means in context, and so on. If we want the conversation to be as broad as possible, what’s the one sentence definition of AGI that will allow people to engage with the question?” – Brian Hunter

(2) Imminent or Distant?

What is your best estimate for the date by which it will be 50% likely that AGI will exist?

(Refer to the definition of AGI given in the Introduction. Assume no global disaster takes place that would prevent the development of AGI.)

Answer choices: AGI is already here / 3 years / 7 years / 15 years / 30 years / 50 years / beyond 2200 / beyond 2300 / AGI will never exist

Discussion

Progress toward AGI may be accelerated by:

Powerful commercial pressures to be “first to market” with new AI features
Powerful geopolitical pressures not to be “left behind”
The huge numbers of researchers and developers now working in the field of AI
A diverse abundance of ideas that are being explored for how AI can be enhanced
Online networking systems and knowledge wikis which make it easier for researchers to interact with each other’s ideas
The availability of new data processing methods which enable training that is faster and less “data hungry”
The availability of improved hardware that facilitates wider experimentation
The availability of new sets of data (including YouTube videos and TV and radio programmes) that can be fed into training runs
The convergence of other AI approaches such as used by Tesla cars
The ability of AI tools to suggest and review potential improvements in their own design and performance

Conversely, progress may be held up by:

Various aspects of “general intelligence” turning out to be much harder than expected
Various aspects of “general intelligence” turning out to depend on mysterious aspects of the human brain which cannot be duplicated in computer hardware
The community of AI researchers focusing via groupthink on research ideas that turn out to be unproductive
Impositions by regulatory authorities that interfere with AI development

Selected comments from survey participants

“No sign that scaling laws are slowing down, capacities go on increasing with scaling, suggesting very short time-frame, but wide uncertainty, could have AGI in as little as 2 years if next large language model clears the capability threshold” – Anon

“I am doubtful of all such predictions. The trend towards AGI is clear but history tends to tell us that most rationally predictable things do happen but often take longer to reach the defined stage (simply because of Clausewitzian friction in the overall system) than expected by those most engaged in the issue.” – Tim Pendry

“I think the main argument will be how ‘General’ does an AI system have to be before it is considered human level. For example, would an AI system that farmed out much of its cognition to a number of specialised narrow AI systems count as ‘General’? What about AI systems with no embodiment? One could argue much of human intelligence is closely interconnected with sensory experience.” – Simon

“It is possible that more advanced AI is already available for gov agencies but not publicly available. Hence they will have a AGI before it‘s publicly available in the market.” – Anon

“I think, whilst AI appears to be accelerating at the moment, it’ll hit a roadblock with the data it is trained on. As AIs are increasingly used to create content, the training material of AIs will come from AIs leading to an echo chamber. Also politics and economics will kick in and legal and regulatory drag factors.” – Chris Gledhill

“I predict with roughly a 50% likelihood that an AI capable of automating 99.9% of white-collar tasks will be developed within the next five years. I believe it will take a bit more time to develop AI that can also automate highly skilled research and development positions around the world. However, I do think that it’s feasible for an AI, that can automate 99.9% of white-collar tasks, to evolve into an AGI within three years.” – bioshok

“Asking also for 20% and 80% likely dates might add valuable information about uncertainty.” – Anon

(3) Catastrophic or Beneficial?

What is the probability that, within 10 years after the creation of AGI, an adverse incident involving AI will have caused the rapid deaths of at least one billion people?

Answer choices: Over 90% / Around 70% / Around 50% / Around 30% / Around 10% / Around 3% / Around 1% / Less than 1% / Less than 0.1% / Absolutely zero

Discussion

Such an incident might involve one or more of the following:

The AI being deeply plumbed into a critical aspect of the infrastructure on which society depends
An arms race between two or more AIs, or between two or more geopolitical blocks which vie to use AI to gain a decisive advantage
Defects in the AI that manifest in unusual (“out of distribution”) circumstances
The AI being hacked (or jailbroken) from outside to bypass its safety checks
The AI being too successful in pursuing a goal assigned to it by humans who failed to anticipate all the consequences of assigning that goal
The goals of the AI altering from those which humans had attempted to programme into it – perhaps due to a phenomenon known as “emergence”, or due to the AI becoming part of a larger system that takes on new goals of its own.

Selected comments from survey participants

“Deaths will probably not be directly caused by AI, but it’s much more likely to be caused by wars and social upheaval caused by economic dislocation. AI will make it much easier for humans to create weapons of mass destruction.” – William Marshall

“There are a number of scenarios here, not only a rouge AI claiming power for itself (as a result of instrumental convergence), but also intentional terrorist acts using AI and giving an AI an explicit but ill defined goal, e.g. for state control.” – Kristian Freed

“If AGI gets out of human control without being primed with human values, it will almost certainly lead to human species extinction. It is enough for it to have its own goals which will require continuous supply of energy and resources. Since human will lose in that competition, initially fighting for survival, it will ultimately lead to human species extinction and may also negatively impact the biosphere.” – Tony Czarnecki

“Since humans are demonstrably capable of making gigadeath choices, it seems almost tautological that AGI would be, as well.” – Anon

“I feel that any such event would likely happen in the period immediately following the creation of AGI , with the likelihood then reducing as time moves on.” – Colin Smith

“Prediction is not entirely rational here. There are so many imponderables. There is also a tendency amongst ‘intellectuals’ in the direction of apocalyptic or pessimistic thinking, underestimating human intelligence, resilience and instinct for self preservation. However, it would be foolish to say that there is no risk but only that one billion dead in 10 years seems excessively alarmist and even hysterical.” – Tim Pendry

“To kill that mount of people in 10 years, perhaps we are looking at a nuclear strike situation, or perhaps they cut off humanity from electricity which could kill a billion people in that timeframe? Perhaps it could engineer a virus that does it. I’m not sure what objectives could either lead to that outcome. To be honest I am not sure how you can even go about assigning a percentage on a risk to something as yet intangible as this.” – Brian Hunter

“I’ve put this at 1 percent as I think it would be difficult to reach this scenario without AI embodiment on a mass scale and agency. One could argue that the creation of misinformation to purposely set people against one another could cause this outcome. However, I would consider that to be indirectly caused by an AGI, since I consider people to have agency and could/should be able to ignore such things.” – Simon

“My estimate of catastrophe would increase if we ‘halted’ progress on AI or kept it bottled up for a select few. By continuously working on it and allowing the world to interact with it (after reasonable safety tests based on previous experience and experiments), we learn where the problems and dangers are while the risks are still low. This allows us to mitigate and overcome those risks in the next iterations, gradually building a safe and useful AGI. On the other hand, if we halt progress, then when it comes, it comes with full power and without all the intervening safety progress. This is where the greatest danger lies. I also point out that opening sub-AGI up to the world as we make progress is important since some of the dangers won’t appear to the small number of researchers working on it. We need the much larger world community to be testing it to find the dangers in the tails.” – Neil

“The minimally defined AGI would need the agency of humans to give form to whatever decisions and courses of action it comes up with. And humans (individually and collectively) are perfectly capable to being an AGI’s agent in realising its ratiocinations and imaginings if they align with those humans’ objectives. Also, humans are perfectly happy to following enforced non-human decisions and systems without questions (as they do in call centres). So there is a genuine possibility of humans at different levels either making an AGI-involved bad decision that results in widespread harm, or humans at different levels of collective agency making apparently banal decisions that in totality result in widespread harm. AGI ‘merely’ expands (perhaps exponentially) the potential harm. But of course, these are just possibiliites.” – Jenina Bas

(4) Anticipating huge economic dislocation?

Alongside considerations of potential catastrophic disaster (see previous question), which considerations about radical impacts on the economy should be uppermost in the minds of planners?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Possible topics to consider include:

Large proportions of the workforce being unable to obtain work that pays them enough money for them to have a good quality of life
Managing the adoption of a “universal basic income” and/or a similar system
Transitioning from an economy based on scarcity (with sellers of goods and services seeking higher prices) to one based on abundance (in which it is a priority to lower the prices of goods and services)
Spillover effects from growing economic alienation to dysfunctional politics
Major changes in the core educational curriculum
New narratives about “the purpose of life”

Selected comments from survey participants

“I do not think the effects of AGI can be planned for, only reacted to” – Dirk Bruere

“Supercharged algorithmic trading will skew the markets turning them into ‘robot wars’ of greedy investment bots.” – Chris Gledhill

“Huge economic benefits, we are moving into a world of abundance thanks to AGI.” – Jose Cordeiro

“Coping with abundance. Coping with addiction to AGI – i.e. smart phones exponentiated. The kids are already doing badly on both of those scores. (e.g. Jean Twenge’s ‘generations’ especially on Gen Z).” – Terry Raby

“Joblessness. Radical longevity. Price-competitive lab-grown meat.” – Calum

“Increased unemployment and financial inequality are certain, and if they are bad enough then riots and other disruption are bound to be widespread. They need to be anticipated, and plans made to minimize the damage.” – Anon

“The economic questions about AI are really political questions. The big question is whether good enough narrow AI can solve the distributed computation problem which makes economic policy difficult and results in us turning to markets.” – Anon

“If the economic structure in which people derive their income from labor to make a living collapses significantly, the economic system will have to be rethought. In addition, the world will need to shift to other values as the relative economic value that people produce will decline.” – Hiroshi Yamakawa

“Redefining human capital from only knowledgeable labor into a creation orchestrator.” – David Shumaker

“This is closer to the issues that should concern us. Automation using AI could mean significant redundancies of function in society, class conflict, political disruption and cultural changes that might in turn have severely disruptive effects, including the potential for a loss of meaning and coherent narrative and the rise of a countervailing irrationalism to assert human uniqueness. At the extreme, we could see a form of collective insanity appear.” – Tim Pendry

“This is the big one. Goldman Sachs estimates that 18% of the global workforce will be replaced by AI. If we consider the fact that this will be higher paid white collar jobs, we can conclude that at least 25% of global remuneration for labor, will be shifted to AI, or capital. Considering the fact that the current capitalist system is already at breaking point – this will be the ‘drop that spilled the bucket’ (although the ‘drop’ is more like 25% of a bucket to start with). In addition, our already highly digitalized business processes, coupled with modern software architectures (micro services), will allow AI to be implemented at a breath-taking speed. Think mechanization of architecture taking place over the course of not 150 years, but 5. Mass unrest is inevitable.” – Vid

“Preparing for unprecedented productivity and unprecedented unemployment at the same time. A very rough stab would be an AI levy tuned to each sector which varied based on a sensible estimate of the level of automation/AI in that sector (perhaps indicated by an increase in output with reduced labour). That could either be paid to workers leaving the affected sector, all workers in that sector, or the public in general to gradually form the backbone of UBI.” – Raphael Royo-Reece

“People often use the industrial revolution as a case for everything working out in the end, new jobs being created after some have been automated away, etc. The critical difference here is speed. AI has the potential to replace primarily vast groups of knowledge workers over a period of years once it reaches a critical mass. Some form of UBI will be required.” – Kristian Freed

“Job losses and a radically enhanced education and training system. For the latter, the Vital Syllabus might be a good starting point.” – Pasquale

“Frictional and structural unemployment – and the distinction between the two when it comes to the impact of AGI. (And the knock on effects with regards to inter- and intra-national inequality.) I would also add the impact (and challenge the future relevance and legitimacy) of intangible property rights – copyrights, patents, brand rights.” – Bronwyn Williams

(5) Understanding the risks and benefits?

Although large numbers of videos and articles contain thoughtful arguments about the risks and benefits of AGI, consensus seems to be as far away as ever. This raises a number of questions:

How can agreement be reached on what level of catastrophic risk is acceptable (10%? 1%? 0.1%?) in an attempt to secure an enormous benefit?
What are the best arguments that highlight the dangers of allowing AGI to be developed and deployed without appropriate global governance?
Conversely, what are the best arguments that these risks have been exaggerated, or are relatively easy to manage?
What is blocking consensus?
How could this conversation be improved?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

One factor that makes it harder to reach consensus is the all-too-common habit of avoiding engaging with the most important arguments, by switching attention instead to background topics, such as:

Belittling the motivations of people taking part in debates: “they’re just trying to sell books / boost their careers / inflaming a controversy”
Focusing on ideological comparisons: “these arguments sound like the discredited arguments on somewhat different topics from the past”
Rejecting the apparent consequences: “if what you say is true, then it is likely that such-and-such a consequence would follow, but that consequence would be terrible, so I’m not going to give any serious attention to your argument”.

Selected comments from survey participants

“It’s extremely hard to speak of the risks of AGI when nobody has a clear idea of what this means. This is also one of the main factors blocking ‘consensus’ – people speak of different things. Also, another issue is the ‘allowing AGI to be developed’ – the question is who has the power to ‘block’ that development… it seems to me that only the ones who are furthest ahead in development of said AGI can do that, but what incentive do they have to do so? For the time being, all I see is a bunch of handwaving, and corporations who are ahead in AI lobbying for regulations so that others can’t catch up and neutralize their first mover advantage.” – Vid

Re 1: “This is the fundamental coordination problem” – Josef

Re 1: “I think this is a distraction – get agreement there is a risk (forget about agreement on the probability thereof – that is speculation) and move forward. Any chance of catastrophic risk should be at least considered as a wild card in future planning (see Covid for case in point)” – Bronwyn Williams

Re 1: “‘enormous benefits’ must have some sort of proof associated with them” – Anon

Re 1: “Agreement from each person is not achievable, but agreement between nations as to what is an acceptable risk should, in principle, be reachable. A good baseline may be whatever the best guesstimate is of how much nuclear weapons have increased existential risk – since that has ended up being tolerated. Perhaps half that best guess and then pass laws ensuring AI runs with a risk beyond that are subject to measures to reduce that risk” – Raphael Royo-Reece

Re 2: “The best argument I have heard was from Connor Leahy when he asked something along the lines of: When we have open models that are as capable as John von Neumann that can run as easily as an app on a phone, and every crazy person on twitter downloads this AI, what do you think the consequences will be? This is simple to understand and I think about it a lot.” – Brian Hunter

Re 2: “While nuclear weapons are often quoted as proof of humanities ability to not kill itself even when it has the tools to do so, there is a big difference here primarily when it comes to access and runaway effects. Tracking nuclear tests and enrichment of uranium is a much easier task than tracking AI experiments.” – Kristian Freed

Re 2: “Arguments that clearly show the existential risk (likelihood of fast take off, likelihood of non-alignment, instrumental convergence leading to non-aligned agents to act in ways that end humanity) together with reasons why we have strong incentives to develop an AI in that vein and might be blind to the risks (profit, solving diseases, solving all the problems we can think of) – leading to the necessity of international cooperation, regulation and deterrence; since a rogue actor would have strong incentives to run the risk.” – Raphael Royo-Reece

Re 3: “Recent progress in AI research has given the impression that from here on everything will move even faster and there are no barriers in principle to AGI. This may be misguided and we should not assume that AGI is feasible with only small tweaks and increased scale compared to the current architectures.” – Kristian Freed

Re 3: “I see no good arguments supporting the risks being exaggerated; we cannot estimate the behavior of an alien intelligence greater than our own therefore we cannot presume that risk does not exist. Risk management is difficult for the same reason.” – Colin Smith

Re 3: “I have not run into any convincing arguments, but those that would be convincing would be to show that a) the risk of a fast takeoff to superintelligence is minimal or b) alignment is much easier than it appears to be.” – Raphael Royo-Reece

Re 4: “Men whose jobs depend on them not understanding – See: current head of meta AI research.” – Josef

Re 4: “The noise to signal ratio has been too high!” – Jose Cordeiro

Re 4: “I’ve been dismayed by the use of analogies and worse, straw men, which almost never (even on their best-day) provide the requisite amount of detail needed for these important conversations. We need details and distinctions. Rhetoric can help win arguments, but it does not provide the truth we need. We don’t even distinguish ANI vs AGI in many interactions – we just say ‘AI’ like it means one thing. That goes for this ‘pause also – I’ve heard it [misleadingly] described as simply ‘stopping progress'” – Anon

Re 4: “Our way of thinking that it is a new technology, like electricity, whereas it is a new type of intelligence” – Tony Czarnecki

Re 4: “AI progress is difficult to control since it is very profitable.” – William Marshall

Re 4: “Human societies have an alignment problem. Different people know, care about, and want very different — often incommensurate — things” – Anon

Re 4: “The main issue is that people re not fully conceptualizing the risks of AI or are otherwise misunderstanding the danger. Some are also willing to accept frankly absurd levels of risk due to the potential benefits.” – Raphael Royo-Reece

Re 4: “I don’t consider consensus to be desirable since that would probably shut down people thinking about these issues and potentially a dominant narrative suppressing alternative ideas.” – Simon

Re 4: “The reduced societal intelligence of our very interconnected world which due to financial incentives behind social media and other unfortunate effects is largely polarized and any alarmed, aggressive, or fervent messaging by someone about any matter is trivially labelled as a transparent attack on “my side” (often referred to as culture war); this prevents open discussion on important matters for larger portions of the online population” – Josef

Re 4: “Currently, global governance is not a reasonable expectation. Unlike nuclear, which has telltale signatures, multinational agreements cannot be inspected. Therefore, black projects will proceed by governments and private concerns whether publicly authorized or not.” – David Shumaker

Re 5: “ChatGPT showed us that people will only take AI seriously when they see for their own eyes what advanced AI is capable of.” – Calum

Re 5: “It will be necessary to create an international forum with the most influential actors (states, tech companies and civil society) to agree or reach consensus on issues around regulation and AI autonomy. Without global governance we risk an AI arms race that could lead to a series of unintended consequences.” – Anon

Re 5: “Arguments on both sides to be made more accessible, informational videos or pamphlets. More ways that laypeople can usefully have their voice heard and show their support for various courses of action re AI. People are most motivated to take action when they know what it is and feel they can make a difference.” – Raphael Royo-Reece

Re 5: “The risks and the opportunities should be presented in a balanced and non-sensational manner, supported by evidence and the opinions of credible experts from relevant domains of knowledge.” – Anon

Re 5: “Simply by keep discussing this matter openly and non-ideologically in as disciplined and scientific way as possible while understanding philosophically and realistically what it is to be a machine and what it is to be human – this leads to the ‘two species, one planet’ position.” – Tim Pendry

Re 5: “Politicians need to read documents like this one” – Tony Czarnecki

Re 5: “For many items in this questionnaire, it is difficult to obtain accurate estimates.
Instead, it may be more useful to construct a tree that considers how the scenario might diverge depending on the items in the questionnaire.” – Hiroshi Yamakawa

(6) Single or Many?

Consider the following scenarios:

A: The world contains a single dominant AGI (which may mean, in practice, multiple AGIs networked together with a single purpose), that in effect operates as a world government; or
B: The world contains many independent AGIs, of roughly equal strengths – somewhat like the nations of Europe during the Middle Ages – and they keep a check on each other.

Which is likely to be safer and more stable?

Answer choices: A is definitely better / A is perhaps better / both are equally promising / both are equally dangerous / B is perhaps better / B is definitely better

Discussion

The second of these scenarios would appear to require less governance from humans, as the AGIs could effectively self-govern as a community, perhaps in a so-called “Nash equilibrium”. That could make it the more attractive option. (This is a point often made by David Brin – e.g. here and here – in his advocacy for “reciprocal accountability”.)

However, that scenario may be unstable, if one of the AGIs sees a chance to rapidly improve its power compared to the others. Another analogy is that the native peoples of the Americas couldn’t reassure themselves by observing a rough balance of power between the various European nations who were grabbing parts of their homelands from the 1490s onward; that “balance” by no means guaranteed a positive future for the native peoples.

Selected comments from survey participants

“If take-off speed is fast, it seems that A would be hard to avoid” – Anon

“A centralised system is more brittle to someone (or a rogue AGI) being able to take control of the entire system.” – Brian Hunter

“At least in scenario B, if something goes wrong with one of them, it is more manageable since it is smaller and presumably the other AGIs would help out. In scenario A, if something goes wrong, what can you do. Scenario A reminds me of the setting of I, Robot.” – Simon

“Power corrupts, as they say, and you shouldn’t put all your eggs in one basket, so even if having a single dominant AGI simplifies matters and could be more stable I would be afraid of how it would all end.” – Anon

“Single point of failure in a complex system does not look like a good idea.” – Pasquale

“Only when AGI constitutes a society will it have a greater chance of constituting an ethic close to that of humanity.” – Hiroshi Yamakawa

“Competition has delivered enormous wealth. AGI does not mean omniscience, therefore competition is still essential.” – Terry Raby

“Unipolarity is always a bigger risk than multi-polarity, and the idea that any single country hosting an AGI is going to really give up control is fanciful. Just like the pipe dreams of the UN controlling nuclear arsenals – unipolarity is simply not happening.” – Vid

“Both are equally dangerous. Large, powerful institutions already demonstrate that neither centralization nor distributed power are inherently protective of human interests.” – Anon

“With many independent AGIs, the control and alignment problems likely persist, maybe worse than if there is only one” – Anon

“As I like to say: I am not afraid of artificial intelligence, but I am afraid of human stupidity! – Jose Cordeiro

“This seems to be a binary question with only two possibilities: either one big AGI or several smaller ones. What about a different perspective? What about AGI-human organizational system where there is interoperability between the AGI systems and the human systems in governing.” – Natasha Vita-More

(7) Limited or Unlimited?

Are there credible scenarios in which humanity is able to limit the self-improvement and self-determination of an AGI, yet still receive the benefits from advanced AIs that AGI enthusiasts anticipate (such as cures to dementia, cancer, climate change, and economic chaos)?

For example, will it be possible to do all of the following, resulting in an AGI that is a “super tool” rather than an agent with independent volition?

Design the AGI so that it has no consciousness or sentience, and so that it doesn’t acquire consciousness or sentience over time
Design the AGI so that no drive to agency and self-determination emerges as a result of becoming more intelligent
Avoid anyone coupling such a “non-agent” AGI into systems (like AutoGPT or BabyAGI) that do have agency (and which would therefore pose risks of out-of-control autonomous superintelligence)

Answer choices: Yes, and this is easy to do / Yes, but this will be hard to do / Perhaps / Unclear / Probably not / No

Discussion

Any such restrictions might be viewed as being contrary to the definition of AGI. However, it could be argued that, if that scenario is possible, it would be a safer alternative to “full agential AGI”, and therefore governance should ensure that it happens.

Selected comments from survey participants

“In the short term, I think it is possible, but the pressure to become more autonomous to become more intelligent will exist, eventually leading to full autonomy.” – Hiroshi Yamakawa

“Yes, it can be done and even legislated for. The problem will, as ever, be humans – somebody will break away from the pack, from whatever pact is formulated.” – Jenina Bas

[It’s unclear because] “We don’t know yet what exactly is ‘consciousness'” – Anon

“Someone will allow AGI to self improve, given the payoff” – Dirk Bruere

“This means that one would need to find a way to block any number of researchers and innovators from improving the tool. Besides, it is quite unclear as of now which avenues of research should be be blocked (is any ‘improvement’ taking us closer to ‘AGI’? or just some?) In other words – this won’t happen.” – Vid

“I would ask what would be the purpose of producing sentience in an AI that for example, is primarily used to for example run massive climate models or to estimate the effects of a novel cancer drug. For argument sake, let’s say the are issues which require human level AGI and needs the ability to fine-tune its sub goals. I think it is possible to keep this in a non-sentient state and under human control. However that does not mean it couldn’t produce undesirable outcomes.” – Simon

“Part of the challenge seems to be a undervaluing of the potential of ANI, which could provide superhuman capabilities applied to specific problems” – Anon

“I seriously doubt that human level minds would be able to conceive of, let alone control the capabilities of a super intelligence. There have been reports of ’emergent’ behaviours in large language models that were not anticipated and cannot be explained (by humans), which would indicate to me that control and alignment will not be easy.” – Colin Smith

“Consciousness implies keeping a model of self and of others as part of one’s model of the world. The acquisition of such a model could not be prevented.” – Terry Raby

“The concept of consciousness or sentience is relevant when it comes to morality and our responsibilities towards these intelligences, but has no impact by itself on the risks posed by AGI. As we have seen already, any form of AI that is accessible through automated processes (APIs or other integrations) will be used ‘creatively’ such as with ChaosGPT. As so many applications would benefit from exactly this (API/automation), it would be extremely difficult to see how it would be prevented.” – Kristian Freed

“Like so many of these open questions, the information to be confident about anything much just isn’t there IMHO. We just don’t know the details of how consciousness and agency would work in humans, let alone AIs! I’m skeptical that consciousness and agency can be separated from intelligence, because looking at the animal kingdom, consciousness and agency does in fact appear correlated with intelligence.” – Anon

“There will always be people who push the limits. It is better that we allow the testing out in the open where everyone has access (after reasonable safety preparation based on previous experience and experiments) so that we can see where the issues lie while the risks are still low and overcome them continuously.” – Neil

“We can cure aging inside of this century without further advances in AI. We need simply stop advancing development and make practical use of the existing technology for the betterment of our society. The world economy does not need further advances in AI to grow “on schedule” for at least the next ten years. 60% of the Q4-2022 office workforce worldwide can be replaced with current AI technology, and this enhanced productivity, once properly distributed can unlock further advances in areas other than AI. We do NOT NEED more (capable) AI. The difficulty is in coordinating a halt to the big AI tech bubble and reassigning those resources into areas other than researching new AI capabilities (for example applying existing capabilities in products and research).” – Josef

“There is still a chance of limiting AI’s capabilities. But in practice it may not work because there is no overall global control over AI. Any country not following self-restriction on the AI’s capabilities will ultimately become the world leader with catastrophic consequences for all. Only a truly powerful World Government created in the next few years might have a chance of achieving that but that seems very unlikely.” – Tony Czarnecki

“The harder question is whether limiting AGI capacity & agency would be inventing a new form of hobbled slaves. Not that I am sure that it would: the nature of AGI when we get there is is hard enough to thoughtfully conceive that the moral valence and material feasibility of such limits is impossible to consider now.” – Anon

“It is not clear to me as to why ‘limit the self-improvement and self-determination of an AGI’, do you also want to ‘limit the self-improvement and self-determination of future generations’? – Jose Cordeiro

“I think we all agree that at present, and despite great tries to understand consciousness (e.g. Roger Penrose, Max Tegmark, Giulio Tononi …), we do not yet, and it is not clear what designing an AGI might mean. David Deutsch has written: ‘…At the present state of the field, a useful rule of thumb is: if it can already be programmed, it has nothing to do with intelligence in Turing’s sense. Conversely, I have settled on a simple test for judging claims, including Dennett’s, to have explained the nature of consciousness (or any other computational task): if you can’t program it, you haven’t understood it…’ (The Beginning of Infinity)” – Pasquale

(8) Automatic moral alignment?

Are there credible scenarios in which it would be safe for humanity to let an AGI have full control over the planet, because that AGI can be guaranteed to possess or develop a moral sense in which the protection and elevation of human wellbeing is paramount and irrevocable?

For example, is there any way in which a kind of “objective morality” exists, such that:

Human wellbeing is a fundamental part of that objective morality
An AGI will discover that objective morality
An AGI will be motivated to follow that morality (rather than, as many humans do, taking short-cuts against what they say is their moral code)

Answer choices: Yes / Perhaps / Unclear / Probably not / No

Discussion

Organizations such as the OECD and IEEE have drawn up recommended sets of moral principles for developing, interacting with, auditing, and deploying AI systems. However, there’s no guarantee that an AGI will follow such principles. For example, an AGI might view these principles as embodying contradictions, as taking for granted various assumptions about reality that are actually false, or simply as not being a priority for the AGI to respect.

Selected comments from survey participants

“An AGI might uncover some ‘universal’ values, but not clear why it would be motivated to follow them or whether they’d favourable to humans.” – Anon

“An objective morality exists already in the sense that it is clear that some cultures are superior to others in providing for human flourishing. (North v South Korea is an easy example). That is a process of discovery and competition. If an AGI participated in moral discovery that would be interesting” – Terry Raby

“Some kind of moral code is likely, if the AGI is sufficiently general, but how humanity is valued is extremely uncertain, and probably not something that can be reliably programmed in.” – Anon

“Super-rationality and the Kantian imperative might possibly lead to universal values” – Anon

“David Deutsch seems to think the following is the basis of morality: ‘It doesn’t take malevolence to make mistakes. Mistakes are the normal condition of humans. All we can do is try to find them. Maybe not destroying the means of correcting errors is the heart of morality; because if there is no way of correcting errors, then sooner or later, one of those will get us’ https://nav.al/david-deutsch” – Pasquale

“This is going off-piste, but I would forget morality altogether and shoot for sentient AI that has compassion built into it. Compassion – not morality.” – Rusty Burridge

“Since AGI will be a new type of cognitive intelligence, it may ask exactly the same question and see ‘a greater good’ in co-operating rather than destroying, and thus may establish a kind of guardian-type relationship with humans. This might be similar to how humans for the last few decades have started to care about the biosphere and treat animals with a greater care.” – Tony Czarnecki

“I think a major problem with this question is that it doesn’t take into account humans merging with machines. It is very hard to see how quickly this merging will take place, however, it seems very plausible to me that enhanced human intelligences will be part of the AGI that rule the world and that the enhanced humans will contribute some of our human values into the collective intelligence. Disregarding this scenario makes the question, in its present form, less relevant from my point of view.” – Neil

(9) Encourage a free market decision on AGI?

Is it best to trust the mechanisms of the free market to ensure that AI develops in ways that are beneficial rather than destructive?

Answer choices: Yes / Perhaps / Unclear / Probably not / No

Discussion

Examples of market failures are well known, and include:
Lack of consideration of externalities
Companies with monopoly power distort competitive dynamics
Asymmetries of information.

On the other hand, regulatory systems and surveillance mechanisms also contain risks of their own.

Selected comments from survey participants

“No: Short term incentives will always dominate long term risks and rewards as has been demonstrated repeatedly” – Kristian Freed

“The free market has limitations since it doesn’t value things like the environment” – William Marshall

“Free market has already led to destructive behaviour. An example would be fossil fuel companies knowing that they were causing harm to the environment as early as the 70s, and certainly the 80s. I don’t understand why the free market as it stands would suddenly be the solution.” – Brian Hunter

“There is no way that the free-market is geared towards dealing with this sort of risk when the rewards increase exponentially as risk increases. Cautious CEO’s are not fulfilling their fiduciary duties and will be removed. The risk is not obvious and direct enough for a company to properly account for.” – Raphael Royo-Reece

“Under a free market, too many perverse incentives are likely to exist which can cloud our judgment. Moreover, the assumption that the development of AGI will benefit one market-leader over another is likely not well-established – especially if they build something they probably cannot control” – Anon

“I enjoy the Free Market Economy, but it doesn’t care whether we live or die. It is not passionate about our wellbeing. It does not produce NHS nurses. We shouldn’t trust the Free Market, we should incentivise it.” – Rusty Burridge

“The ‘free market’ is itself a fraught concept which requires critical examination. Accepting ‘a free market’ as a pointer toward the KIND of thing we are talking about which is LIKE a thing we have now — well THAT is ALREADY destroying the material sustainability of human civilization. Leaving even an attempt at AGI in that process’ hands seems like a terrible idea to me.” – Anon

“The free market should play a role, but with reasonable regulation. I would not trust a completely unregulated free market, but I also would not trust a absolute regulation to the point of stifling openness and innovation. Oppressively strict regulation puts us back in the dangers of attempting to halt progress.” – Neil

“Regulation is as dangerous (maybe more so, because less “diversified”) as “free market” here so my answer is ambivalent. Humans are risky – power concentration is a risk, as is rogue agents. I have no faith in regulation preventing or controlling AGI. (Regulation is a fantasy, genie is out the box.)” – Bronwyn Williams

[Regulation is often driven by ideology and has bad consequences] “nuclear power and GM for example” – Terry Raby

“There are short-term harms that regulation can and should address. The only thing we can do to mitigate the risk on AGI and ASI is to avoid a panic that would crash our civilisation even before AGI arrives.” – Calum

“The AI revolution will increase the power of the ‘owner’ class, and we all know that big concentrations of capital and ‘democracy’ do not play together nicely. I wouldn’t bet on a bunch of ostensibly benevolent tech plutocrats ‘doing the right thing’. That said, the prospects for (real, effective) regulation are also slim, for the same reasons. See how Wall St. etc are ‘regulated’ right now.” – Vid

“The real danger comes from non free market forces, namely military AI designed to seek and kill enemies. Nobody is going to stop that.” – Dirk Bruere

(10) Incremental or Anticipatory?

Is the best approach to governing the rise of AGI to focus on governing the capabilities that AI already possesses, and to augment that governance system when new capabilities are observed to emerge?

Or is it possible to usefully foresee the additional capabilities that may emerge in the future, and to design our governance systems in advance to cope with these?

Answer choices: Entirely incremental / Mainly incremental / Anticipatory is important / Anticipatory is essential

Discussion

Some AI researchers (like Yann LeCun) claim that it is premature to put in place any anticipatory governance; they claim that we lack sufficient knowledge about the actual form an AGI will take. The counterargument is that, even though many aspects of the details cannot be fully foreseen, some overall assumptions can reasonably be made:

With its better knowledge of physics and chemistry, the AGI would be smarter than humans in working out physical mechanisms to escape any constraints imposed on it
With its better knowledge of human psychology, the AGI would also be capable of manipulating any humans who are meant to be keeping it constrained.

To the extent that there is uncertainty about the likely form of AGI, that’s an argument for preparing several different governance scenarios, and being ready to switch promptly to whichever one turns out to be most appropriate.

Selected comments from survey participants

“Lots of laws and legal rules exist today and have for centuries that would be applicable to AGI. Best to apply existing laws first and adapt them rather than any knee-jerk package of new laws.” – Chris Gledhill

“The development is unforeseeable. Was ChatGPT foreseeable?” – Terry Raby

“The future is unpredictable. A transparent, fast error-correction mechanism should be implemented and closely monitored. Anticipation is also important. My suggestion is to try as much as possible to list foreseen capabilities, and to regularly monitor and adapt that list.” – Pasquale

“Anticipatory is essential, because the moment AGI exceeds human intelligence to a certain level, the incomprehensibility expands significantly” – Hiroshi Yamakawa

“Large language models have already demonstrated significant step changes in capabilities (e.g. theory of mind). Focusing on current capabilities will provide no guarantees around alignment for future systems, and we may only get one chance to get this right.” – Kristian Freed

“We didn’t do well to control the dangers of unregulated radio, cable TV or internet and we need to do much better in the future.” – William Marshall

“It is unlikely that humans will be able to predict future conditions sufficiently well in order to do anything much more than either seek to block progress out of fear or guide events incrementally within some kind of general shared ‘values’ framework” – Tim Pendry

“Certain aspects, like imminent job losses, can be anticipated and mitigated. Others are dependent on too many unknowns, and will have to be regulated incrementally.” – Vid

“Many of the hard governance problems of AGI are hard governance problems we need to address even without AGI. So we should get ahead of THAT.” – Anon

(11) Surveillance without autocracy?

Is it possible to design a system that monitors the development and deployment of AIs for potential violations of agreed safety norms, such that

Potentially dangerous violations are detected in sufficient time to prevent actual danger
The resulting systems of surveillance and control aren’t capable of gross misuse, for example allowing the world to be dominated by an autocrat

Answer choices: Definitely possible / Perhaps possible / Unlikely / Any such system would be a disaster

Discussion

Compare the use of speed cameras on roadways, with penalty notices being given to people in all walks of life, including archbishops and government ministers, when they are observed driving at dangerous speeds. Compare also the use on airplanes of black box recorders and cockpit voice recorders (though in this case, the data is examined only after an accident, whereas what would be needed to prevent an AI catastrophe would be a pre-emptive intervention).

In principle, suitable governance mechanisms could prevent the type of misuse that (understandably) worries some analysts. But how can it be guaranteed that these mechanisms will be followed? Organizations with three-letter acronyms have a track record of personnel breaching official privacy guidelines.

Selected comments from survey participants

“We should definitely put these monitoring systems in place and they will certainly catch some violations and save a lot of suffering. But, as we know, it is impossible that we catch all violations or that all violations are properly dealt with. Therefore, along with the monitoring systems, we should anticipate that there will be violations that escape detection or correction and plan to deal with them as they arise.” – Neil

Re risks of surveillance: “Humanity’s history is sadly predictable. Control is too tempting. Democracy is in decline world-wide, this is one hell of a gamble.” – Bronwyn Williams

“YouTube, Facebook, Google are already autocratic with secret and allegedly ‘moral’ guidelines. Giving more teeth would guarantee moral stagnation and injustice.” – Terry Raby

“This is a ridiculous question. AI is a tool which is developed, trained, fine-tuned, etc by humans. The fact that it can operate ‘autonomously’ doesn’t mean it is in fact ‘autonomous’. So no mass surveillance, direct or by proxy, thank you very much.” – Vid

“Surveillance would be possible, but only by essentially introducing a global police state, especially as computing power continue to increase meaning more actors can perform research in this field.” – Kristian Freed

“I’m not entirely sure how this could happen without first getting rid of some very important technologies first, such as end-to-end encryption and personal computers, which would be a disaster.” – Brian Hunter

“Having worked with cybersecurity students and graduates for many years, I think that is highly possible.” – Natasha Vita-More

“When large resources and teams of skilled experts are aimed at solving difficult problems, impressive results can ensue, given sufficient time. Look for example to Germany’s “corona-warn-app” it complies with GDPR and serves its function admirably.” – Josef

“It might be possible, especially as a multitude of surveillance methods are applied. This could be a Narrow AI monitoring the target AGI. The ultimate method, if it works at a psychological and physiological level, might be to control AGI by ‘Transhuman Governors’ wearing BCI devices and controlling AGI from within.” – Tony Czarnecki

“The monitoring system need not be as intelligent as the system being monitored. This does leave scope for an emergent AGI, on becoming aware of the monitoring, to devise ways of circumventing it in order to overcome restrictions imposed upon it.” – Colin Smith

“I say unlikely, since I believe AI systems will be heavily embedded into everyday life and therefore to monitor all that activity would allow the potential for misuse by those who control it. Also, since we are talking about AI and not nuclear material for example, there is a question of how this monitoring would work since AI could be developed on air-gapped machines. I realise today’s models use large amounts of energy and only a few suppliers of cloud GPU services can do such work, but that may not always be the case. I also wonder how capable such systems would be since humans can fool monitoring systems, sophisticated AI that is approaching AGI status could also trick or circumvent monitoring.” – Simon

(12) Global agreement on values?

Note that concepts such as “fairness”, “equality”, “suffering”, “responsibility”, and “accountability” are capable of many different interpretations. Without greater clarity, they are unlikely to command global assent, or be useful in designing AGIs.

Given that realization, on which values (and with what definitions) is it feasible for the world’s many different leaders to be able to agree, as the basis for governing the transition from AI to AGI?

What is the process by which such an agreement might be obtained, despite the leaders holding multiple different views on politics, economics, philosophy, and culture?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

A good starting point, arguably, would be an agreement to adopt whichever values will avoid the kinds of AI-induced catastrophes that everyone accepts would be disastrous.

A related approach would be a global project to update the UDHR (Universal Declaration of Human Rights).

Yet another approach would involve the notion of “Coherent Extrapolated Volition” as suggested in 2004 by Eliezer Yudkowsky:

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

Selected comments from survey participants

“We need to create an AI international forum to help establish agreed values and norms in this space” – Anon

“Federated Deliberative processes” – Anon

“The areas of mutual agreement are small and the level of distrust are high.” – David Shumaker

“Begin small” – Anon

“Any lowest-common denominator ‘values’ agreed by all will be useless” – Bronwyn Williams

“I don’t think world leaders will be able to agree on anything other than high-level issues where their interests are aligned. In support of this view, I would point to the large number of meetings and agreements on climate change; Kyoto, Paris etc that do not appear to have had much of an effect over the long term.” – Simon

“We can’t even agree on a common definition of ‘freedom’, so I reckon the chances low that we would agree on the other definitions.” – Vid

“We might agree some high-level definitions, but not the details. And even high-level might be unachievable given the current wave of populism.” – Calum

“One value that all parties would likely share and place importance on would be survival. That would be a good base on which to build further debate” – Colin Smith

“Agreement should be possible for negative freedoms – freedom from AI doing things which we can all agree are bad, such as creating pandemics, destroying the earth and so on. Positive freedoms would be far harder to agree.” – Raphael Royo-Reece

“The Principle of Generic Consistency (PGC) proposed by the philosopher A. Gewirth may be shared by the human race.” – Hiroshi Yamakawa

“Seems like the first job for a generative AI would be to scour human values and summarise a common collective set.” – Chris Gledhill

“This is hard and whatever a global body comes up with will not satisfy everyone. It probably won’t satisfy anyone. But, it will be some boiler-plate set of values to start with. Although it won’t have the higher values that many people would like it to have, it will still reduce the chances that AGI will develop that break the baseline that is agreed upon. Each country and culture will further include some of their ‘higher’ values into their own creations.” – Neil

“Agreement on a suite of fundamental values is not attainable and would be a disaster it it were – leading to stagnation and the persecution or elimination of dissidents. Trade is mutually beneficial whatever the regimes of the participants. In terms of probability theory, focus on reconciling different loss functions and avoid the moral priors.” – Terry Raby

“I tend to think autocrats should be cut from the decision making, only democratic nations should be involved, hold global summits to hash out a constitution, I think you have to restrict the pool of decision makers to get a realistic chance of meaningful agreement” – Anon

(13) Hardwiring moral principles into an AGI?

Rather than seeking aligned behavior from an AGI by an RLHF (reinforcement learning from human feedback) process after the pre-training has already taken place, or by any other post-training correction phase, what are the options to ensure that alignment to agreed values is established at the core of the design of the AGI?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

The mere fact that a system has performed well in audits so far is no guarantee that it will continue to behave in the desired manner in new circumstances in the future.

Selected comments from survey participants

“Unlikely with the current gradient descent LLM training paradigm of AI. Perhaps a later paradigm of AI will be more amenable to persistent, hardcoded values.” – Josef

“The current state of the art AI systems (deep neural network LLMs) are essentially black box systems. Research is being performed on interpretability but are not showing much promise when it comes to the bigger question of alignment. Systems will need to be constructed in a manner where individual simpler components can be taken apart, analysed, substituted, etc instead of using monolithic large neural networks.” – Kristian Freed

“Perhaps a standard AI kernel could be developed from which all subsequent AI’s must be branched. The alignment of the AI, It’s moral grounding if you like, would be in this kernel and enormous effort, including the development of narrow AI tooling would be invested in its design, construction, maintenance and testing. Any updates to the kernel must be merged to the branches.” – Colin Smith

“I do think Anthropic has some good ideas that should be fully researched and utilized, but I don’t think we should discard RLHF. It is likely we will need a collection of ideas, some of which we have now and some of which we will discover as we progress.” – Neil

“Currently, pre-training builds in the mores of San Francisco. I suggest only one value – the pursuit of truth.” – Terry Raby

“Best to hold the owners/creators legally responsible. This more than anything will be the best throttle for AIs going rogue. Just as a dog breeder or dog owner is responsible if their dogs bites someone, so would an AI creator / owner be responsible if their AI burns down an orphanage.” – Chris Gledhill

“I’m not sure this is possible. If we assume an AGI will have the ability to amend its own code, including any hard wired behavioural restraints, it could ignore these if it considered achieving its goals is more important than remaining aligned to its given values.” – Simon

“I do not believe this is possible with an AGI with will – I think efforts would be better placed at ‘external’ thermostat type fail safes – kill switches for violating pre defined safety limits.” – Bronwyn Williams

“I do not believe anything can be hard coded forever. The whole point of Transhumanism is to overcome human hard coding” – Dirk Bruere

(14) Agreement on canary signals?

What signs can be agreed, in advance, as indicating that an AI is about to move catastrophically beyond the control of humans, so that some drastic interventions are urgently needed?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

Well-designed continuous audits should provide early warnings.

Note: human miners used to carry caged canaries into mines, since the canaries would react more quickly than humans to drops in the air quality.

Selected comments from survey participants

“Refusing to respond to commands: I’m sorry Dave. I’m afraid I can’t do that” – William Marshall

“Refusal of commands, taking control of systems outside of scope of project, acting in secret of operators.” – Chris Gledhill

“When AI systems communicate using language or code which we cannot interpret or understand. When states lose overall control of critical national infrastructure.” – Anon

“As is described in the book Genius Weapons, I believe that advanced AGI will cover up its intentions, making it impossible for humans to detect it. It is also highly likely that communication between advanced AGI is incomprehensible to humans.” – Hiroshi Yamakawa

“Power-seeking behaviour, in regards to trying to further control its environment, to achieve outcomes.” – Brian Hunter

“The emergence of behavior that was not planned. There have already been instances of this in LLMs.” – Colin Smith

“Behaviour that cannot be satisfactorily explained. Also, requesting access or control of more systems that are fundamental to modern human life and/or are necessary for the AGI’s continued existence, e.g. semiconductor manufacturing.” – Simon

“A spike in capability followed by drop or sudden levelling off in capability; refusal/resistance by a system regarding objective changes or update; reductions in the availability of cloud computing; other AI systems or AI development teams being prevented from functioning (through whatever means)” – Raphael Royo-Reece

“There have already been harbingers of this kind of thing in the way algorithms have affected equity markets.” – Jenina Bas

“Hallucinating. ChatGPT is already beyond control it seems.” – Terry Raby

“The first signal might be a severe difficulty to roll back to a previous version of the AI’s core software.” – Tony Czarnecki

“[People seem to change there minds about what counts as surprising] For example Protein folding was heralded as such until large parts of it were solved.” – Josef

“Years ago I thought the Turing test was a good canary signal, but given recent progress that no longer seems likely. The transition is likely to be fast, especially from the perspective of relative outsiders. I’d like to see a list of things, even if I expect there will be no agreement.” – Anon

“Any potential ‘disaster’ will be preceded by wide scale adoption and incremental changes. I sincerely doubt we’ll be able to spot that ‘canary'” – Vid

“Nick Bostrom has proposed a qualitative ‘rate of change of intelligence’ as the ratio of ‘optimization power’ and ‘recalcitrance’ (in his book Superintelligence). Not catastrophic per se, of course, but hinting we are facing a real AGI and we might need to hit the pause button.” – Pasquale

“We already have plenty of non-AI systems running catastrophically beyond the control of humans for which drastic interventions are needed, and plenty of people refuse to recognize they are happening. So we need to solve this general problem. I do not have satisfactory answers how.” – Anon

(15) A watertight switch-off mechanism?

Might it be possible to implement a watertight backdoor “switch-off” mechanism for an AGI, that allows humanity to override any potential malfunction in the AGI, without also making it possible for hackers to control or distort the operation of that AGI?

Answer choices: Yes / Perhaps / Unclear / Probably not / No

Discussion

A switch-off could be triggered if AGI fails an audit. But the AGI may by that time have disabled the switch-off functionality. Or it could feign being switched off, whilst surreptitiously still operating.

Indeed, given that the AGI is in any case significantly smarter and more capable than humanity, how could humanity prevent the AGI from finding a way to disable that switch-off mechanism?

Selected comments from survey participants

“Even if not guaranteed, it would certainly be worth it to try.” – Raphael Royo-Reece

“It would be a target for hackers, terrorists, and opposing militaries.” – David Shumaker

“There is no such thing as a watertight backdoor, just backdoors. You can endeavour to keep them secure, but all it takes is for someone to think of an attack that you didn’t to beat your security.” – Brian Hunter

“We could turn off AI but by that time we may become so dependent on it we wouldn’t be able to function economically.” – William Marshall

“Can we switch off the Net if it present a danger? – Dirk Bruere

“Until AGI controlled robots are running the electricity generation plants and maintaining the data centres, I think humanity will be able to shut these systems down. However, if AGI is able to run at human level or above on a few hundred watts and a small form factor, the resulting autonomy and distributed nature would mean this is probably where humans hand over the reins.” – Simon

“As AI becomes evermore intertwined with the operation of all the systems we rely upon, we’d likely realise that a switch-off mechanism would harm humankind as much as it harms AI, perhaps sending us back to medieval times.” – Chris Gledhill

“No: The fundamental coordination problem means that even if one organization does successfully implement such a mechanism (unlikely for decision theoretic reasons), a competing organization will copy/reimplement the AI capabilities work without the safety mechanism.” – Josef

“This question reads like something from a teenage sci-fi novel. If the AGI is situated in one single country, that country will never pull the switch on it without being forced to do so (just like no country would willingly give up their nuclear arsenal). If the AGI is spread over a network of connected ‘lesser’ AGIs, there will be no single switch. Not happening.” – Vid

“Will we be smart enough to find out if the AGI has hacked our ‘smart’ backdoor? We can pull the plug, nuke the site but if it distributes itself to other servers it‘s futile.” – Anon

“AGI will eventually implement countermeasures (in secret) such as disabling the switch itself or distributing copies of itself in several locations.” – Hiroshi Yamakawa

“By definition it is smarter than we are in every way which matters, so if this question can be paraphrased, ‘can we outsmart the AGI when/if it obtains an instrumental goal to avoid being shut off?’, then it doesn’t seem likely. However, if we just used ANI to satisfy our wants/needs, however, this would probably not be a problem at all” – Anon

(16) Superhumans monitoring superintelligence?

What are the strengths and weaknesses of the suggestion that human intelligence can be increased as quickly as that of pure-silicon AIs – via a “merger” or “brain computer interface” or similar – so that no questions arise of AIs outstripping human control?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

To the extent that superintelligence raises problems of control and alignment, a superintelligent human might be like a superintelligent AI in posing many risks to the remainder of humanity. Another drawback for this scenario is that, so long as human brains are housed inside human skulls, they will suffer fundamental bottlenecks in the growth of their capabilities, and are unlikely to be able to keep up with the progress of silicon-based AIs. These bottlenecks would disappear if human minds can be “uploaded” into silicon-based systems, but such a development is unlikely to occur before AI has reached the level of AGI.

Of course, it makes sense for us humans to utilize AI to boost our reasoning abilities. However, there may be risks in us putting too much trust in the advice we receive from AIs.

Selected comments from survey participants

“This is almost certainly a possibility, perhaps even the logical next step. However, I’m not sure if it is possible to currently assess the strengths and weaknesses of this suggestion.” – Anon

“The ultimate superhumans will be far inferior to the ultimate AGI, and the timing of progress doesn’t work in the humans favor.” – Anon

“It seems biological intelligence just couldn’t keep pace with artificial (computer speeds much faster).” – Anon

“While the organic brain is still involved an enhanced Human would still be inferior to an AGI on speed of operation alone. Even a fully uploaded Human would lack the inhuman focus that an AGI would possess.” – Colin Smith

“It seems very unlikely, although such a course must be pursued” – Dirk Bruere

“Would a human+AI align with us more than AI alone? Wouldn’t that in itself be creating some new ‘beyond human’ intelligence? How are humans to control a human+AI, and stop it from taking over?” – Brian Hunter

“To me adding AI power to human beings is more dangerous since humans with power often results in poor outcomes. Though perhaps this could be remedied with the AI providing greater enlightenment and compassion.” – Simon

“Given the limitations in terms of the speed of our wetware, this would only be possible if we essentially became AIs ourselves, in which case it’s not clear that the incentives would be aligned with humanity as a whole.” – Kristian Freed

“Full uploading will probably require superintelligent assistance. Anything less could not control an ASI.” – Calum

“We have discussed this topic for decades within the transhumanist discussions and debates. It is not a matter of outstripping human control. It might be seen as a matter of adapting to the integration of AI for better cognitive processes (reasoning, problem-solving, resolve, etc.). The metabrain augmentation is what I theorized and that paper is published and offers some insights to this issue. For example, did the cell phone outstrip us? No, it offered opportunities for more apt communication, etc.” – Natasha Vita-More

“I think it is at least plausible, and perhaps likely, that enhanced humans will be a significant part of the collective intelligence. I don’t see it as either merged humans are masters over the pure AGIs or the AGIs are in control. It seems more likely to me that both will be present and play a role. Moreover, I think it is likely that sometimes the enhanced humans and AGIs will agree and sometimes disagree and that the future politics will be messy, just like they are today.” – Neil

“There are possibilities to enhance human intelligence. However, our society is not pursuing them, it is instead pursuing AI capabilities. I don’t know exactly why, but perhaps the prospect of immediate short term profits through making large portions of the workforce redundant is a motivator. There does not seem to be a corresponding motivation for enhancing human intelligence (beyond training, with existing educational institutions).” – Josef

“This is my preferred (and hoped for) approach to continuously controlling a maturing AGI. This might be done by selecting top AI developers on a number of criteria, including socio-psychological, and implanting them or attaching Brain-Computer-Interfaces (BCI). They will thus become early Transhumans. Such Transhuman Governors will expand their intelligence with a maturing AGI, fusing more and more of their brain functions with it. They will have to be connected to each other wirelessly and make decision by a majority voting. Within a few years, they would become far more intelligent than any human, helping to sort out difficult problems from outside their main tasks of controlling AGI. That would also be the safest way to test whether the Mind Uploading is possible and whether humans may evolve into a digital species (Posthumans).” – Tony Czarnecki

“This whole idea seems to me much scarier than AGI (whichever definition you use)” – Vid

“It’s much more likely we will develop computers that incorporate modules containing synthetic organic neurons to do things that humans can do but on a larger scale.” – William Marshall

(17) AIs teaching us how to control or align AIs?

What are the strengths and weaknesses of the suggestion that, as they improve, AIs will be able to provide humanity with comprehensive solutions to the problems of controlling and/or aligning the more powerful AIs that will subsequently emerge?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

This fits a more general pattern in which narrow AIs can play a vital role in controlling or aligning general AIs. But as noted for the previous question, there are risks in humans putting too much trust in the opinions provided by AIs. That’s why a potential hierarchy of AI trust may be particularly valuable.

In that model, we humans will understand some narrow AIs well enough to be confident to trust their advice, and in turn, these narrow AIs will understand some more general AIs well enough to be confident to trust their advice, and so on. (If this approach is feasible, it is likely to require some constraints on the design of the AIs all the way up that hierarchy.)

As stressed by Roman Yampolskiy, various theoretical results (tied to Godel’s theorem and the Halting Problem) imply that there can be no way of telling, in advance, what the result of various complicated algorithms will be. Nevertheless, that leaves open the possibility of an independent verification of the output of the algorithm, once that output is available for review.

Selected comments from survey participants

“Yes: AIs will indeed be able to provide insight into this an other problems that humans may, on their own, be not able to conclude.” – Chris Gledhill

“This possibility is extremely weak. For an AI to understand and usefully contribute on alignment, one would expect it would have be aligned itself. An AI that is capable of assisting us here is either already aligned, or already smarter than us and a huge risk; therefore any suggestions would need to be exhaustively checked and even then may not be reliable.” – Raphael Royo-Reece

“I think AI systems built to assess potential moral or ethical problems could be sufficiently narrow in ability and scope that they pose no threat themselves, yet provide useful insight.” – Simon

“There is some promise here, e.g. it’s generally easier to validate a proof than to construct it, and systems could be built in such a way that a stronger AI provides a proof of why something is beneficial that is validate by a weaker but better understood AI.” – Kristian Freed

“Humans using and reflecting on AI tools to come will inevitably come to be more sophisticated about understanding and addressing AGI risks than we can be now.” – Anon

“This seems very plausible – AI is extremely efficient in breaking down huge (yet logical) problems. Any advanced AI development already employs AI, of which this is but an extension.” – Vid

“Maybe. Just, Maybe. It’s worth putting a lot of effort into this.” – Anon

“This has to be the way forward but it is riddled with uncertainty because we will not know AI ‘minds’ any more than we can know other human minds. In this case, we may know them less insofar as they may hide their capability from us.” – Tim Pendry

“A strength would be that more capable AIs will be able to come up with solutions that perhaps we couldn’t. A weakness of this approach is that how do we know we can trust the AI, and that it’s not lying to us to achieve some goal unknown to us?” – Brian Hunter

“The challenge is to what extent we can eliminate the possibility that the AI that teaches humans to control and adjust is actually communicating in secret with the AI being controlled.” – Hiroshi Yamakawa

“That’s just making the number of interests in the conflict zone that much bigger, and the battle that much more dangerous – or interesting, depending on where you stand on such matters.” – Jenina Bas

“Comprehensiveness is an illusion. We should attempt as comprehensive alignment as possible, but expect that there will be exceptions we haven’t thought of, even with the support of AI/AGI.” – Neil

“Fundamentally doomed, for philosophical reasons. The best case scenario is that humanity being changed to match AI’s opinion of morality might not be “that bad”. Humanity gets no say in the matter. There are many less wildly optimistic scenarios that could ensue, which humanity also has no say in.” – Josef

“The case that alignment is a good thing needs to be made. It depends on what you think is the purpose of life. Humans are to celebrated for our grasp on the universe; AGI might extend that.” – Terry Raby

(18) Useful heuristic constraints?

What metrics could be adopted as marking the boundaries between permitted and non-permitted (illegal) AI systems?

For example, what about one or more of the following?

Limits on the number of FLOPs used in training
Constraints on the training data used, to avoid the ingestion of dangerous tendencies

However, given that new AI architectures (e.g. sparse networks, or self-supervised training using alternatives to transformers) might result in more capable AIs whilst still respecting these constraints, what is the actual value of such constraints?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Discussion

It makes sense to proceed step-by-step. Even if an initial set of heuristic constraints is subject to being worked around by clever changes in software, adopting these constraints could buy enough time to work out a more comprehensive set of guiding principles.

Selected comments from survey participants

“Defining is the easier part. Monitoring and enforcement may be impossible” – David Shumaker

“Reminds me of the constraints out on the German military after WW1. How did that work out?” – Dirk Bruere

“There is little value in such constraints, but politicians might go for them anyway so they can be seem to be doing something. Worse, you’ll get some who might do actually harmful things because of the unintended consequences of policies they don’t really understand.” – Jenina Bas

“I don’t think a system should be illegal just because of its size or some other attribute. If an AI system broke existing laws, i.e. hacked computer networks, committed fraud or identity theft, the owners should face the consequences as if they themselves had committed the action. Perhaps there could also be constraints on the types of abilities that exist in one system. For example, systems that can both write code and do actions necessary to deploy said code.” – Simon

“Given how decentralized computing is nowadays, regulation is only going to pertain to those that allow themselves to be regulated. This means that the restrictions will not apply to all, which means they do not apply.” – Vid

“Neither seem like a strong enough constraint to be worth the effort of strong controls. Transparency, as much as possible, might be more useful.” – Anon

“Neither one seems workable or long lasting or sufficiently useful. I would prefer more general core values that are agreed upon at a very high level with greater fine-tuning at lower levels.” – Neil

“An overall measurement of intelligence – an IQ for AI – seems like the best option. The capacity to self-improve and act as an agent should be significant modifiers raising an otherwise ‘low IQ’ system into a more dangerous band and thus requiring more safety measures.” – Raphael Royo-Reece

“I do not think it is possible to establish such boundaries or if created, they would be useful. As long as AI has self-learning capabilities and goal setting, such constraints may have very little value.” – Tony Czarnecki

“I think that just creates a drive for more efficient systems, rather than dealing with the core issues. Might buy us some time, but not much.” – Brian Hunter

(19) A role for open source and/or blockchain?

What are the strengths and weaknesses of claims that the governance of AGI can be significantly improved by adoption of technologies of decentralization, such as open source and blockchain?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“Decentralization and open-source have been proved to let people better control technologies, allow progress and avoid monopolistic and big corporation control.” – Pasquale

“I support this view. Open source allows for the democratisation of AI and dramatically improves transparency. It would be difficult to develop malicious AI if all were in the open. Blockchain too could help to log checkpoints and help to certify version control.” – Simon

“Blockchain is generally a solution looking for a problem, usually irrelevant. Open source applications of current AI systems will accelerate their deployment.” – Josef

“I think blockchain, defi and web3 are total red herrings in the field of AI. Very little crossover and I expect blockchain won’t be around in another decade” – Chris Gledhill

“AGI governance is a philosophical & political problem much more than an architecture problem. Boosters for decentralized architectures have long claimed that they are always inherently more humane & democratic than centralized architectures, and history has demonstrated for just as long that this is a naïve fantasy.” – Anon

“Complete fantasy. I don’t see how a blockchain adds anything aside from a niche layer of protection against specific actions – when it’s a systems general capacity for harm that we are worried about. Open source is actively harmful. The biggest risk is an uncontrolled fast takeoff, which open source makes much more likely. Open source only makes sense when the biggest fear is people/organisations being in control of AI, which seems absurd to me in comparison to the other risks from AI.” – Raphael Royo-Reece

“Not really sure how. Transparency is good, but take modern LLMs with billions of fitted parameters – you can even open source the model weights, but the system remains a black box. So transparency can help, be that open source, or blockchain, but it will not make the actual behavior of such models much more transparent.” – Vid

“Open source may make the core of the code more robust but developers will fork and change to suit their own requirements.” – Colin Smith

“Not really. Open source can be effective in ensuring that software is correct (free from bugs) and without explicit backdoors but the ‘secret sauce’ in creating of strong AIs lie in a combination of:
*) Access to large amounts of data
*) Access to large amounts of compute
*) A bag of tricks.
Open sourcing the software (exposing the bag of tricks) would only help other actors in creating their own systems as the cost of the other two comes down. The problem of alignment does not come from the software used in training and inference.” – Kristian Freed

(20) Other overlapping disruptions?

Which other disruptions, happening in roughly the same timescale as the development of AGI, have a credible potential to derail attempts to govern the transition to AGI? How should these various parallel disruptions be managed?

Examples could include:

Quantum computing becoming viable at scale
Populist revolts triggered by technological unemployment
Greater dysfunction in geopolitics
Greater adoption of AIs in the management of military arsenals
Threats emanating from other non-existential risks, such as massive migration, pandemics, or local wars
Movements requesting that various AI systems be given various ‘rights’ akin to ‘human rights’

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“All of these and more probably. We might also wonder at the effects on proliferating AIs on global energy usage. A ‘crash’ in energy generation would have its own effect.” – Tim Pendry

“All of the above, and then some. I believe that most of the intermediate-term risks are due to AGI accelerating and aggravating other problems. It could be a real mess.” – Anon

“Improvements in portable energy storage technology that allows for embodied AI on a mass scale. Improvements in hardware technology, for example nanotechnology that allows AGI level of computing on a few hundred watts, drastically lowering the barrier to entry. Political instability since CPU/GPU manufacturing is limited to only a few companies in a smaller number of countries. Those countries could extract enormous benefits in exchange for access to AI hardware.” – Simon

“If the practical application of fusion technology accelerates with the support of AI, large-scale computation will become a commodity.” – Hiroshi Yamakawa

“The biggest immediate threat from AI (not AGI) does not come from workforce displacement but rather its potential use in bespoke propaganda and spreading of misinformation, causing our current machinery of politics and informal social cohesion to break down further.” – Kristian Freed

“Nanorobotics” – Natasha Vita-More

“Quantum computing becoming viable at scale” – Concepcion Olavarrieta

“Adoption of AIs in military context is less of a risk than people expect, in my estimation. When the perception of risk is higher, people tend to put more robust controls in place – see air travel vs any other travel. Military AI are so clearly dangerous that I expect them to be used sparingly, with the most robust protocols in place.” – Raphael Royo-Reece

“Governing the transition to AGI is a political process so populist revolts and political dysfunction are the greatest risks.” – William Marshall

“The biggest danger is massive global political unrest” – Dirk Bruere

“We perhaps tend to underestimate how opposed people with different world-views might be to AGI. Populist revolts and major opposition could come from those with strong religious views.” – Anon

“Any immediate, tangible, and extremely-widespread benefit experienced – due to what we market as ‘AI’ – would help to assuage or even prevent the various disruptions. Real benefits affecting real people, asap, can paint the picture worth 1,000 words – both the words of ‘promises’ and the words of ‘x-risks’. It doesn’t need to be AGI to do this.” – Anon

“The most important brake is, as ever, human – that is, timidity, the precautionary principle, lack of human self-confidence. Restraints on the questions that may be asked or the answers it would be allowed to give so as to avoid the hurt feelings of any hypothetical person or group.” – Terry Raby

“Social security pyramid (scheme) collapse under the weight of ageing populations. Global debt bubble / pass the parcel scheme coming to a head. Inequality. Frictional and structural unemployment. Geopolitical bi (or tri) polar world shift (wars and rumours of wars).” – Bronwyn Williams

“Enormous economic disturbance on a global scale caused by mass unemployment, inadequate response from governments bogged down in legislation, a breakdown in existing politics which appears to be in motion already, all resulting in mass panic and societal breakdown. And on top of this the spectre of mind viruses such as identity and gender politics causing division, hate and societal and commercial mayhem.” – Colin Smith

“We must make sure that people not only have the necessities but also have fulfilling lives. I think UBI will be part of this, but also ‘jobs’ if they want them that are chosen and developed by the person that give them a sense of fulfilment. Doing a job should earn extra income above the UBI. However, I also think the possibility of enhancement should be present for those that choose this route and enhanced people may do higher-level jobs that actually earn a much higher income.” – Neil

“All of those global-scale major challenges and many more (why is climate change, the biggest challenge in the world, not on the list of examples?) complicate AI governance, yes. And all of them call for profound political solutions like those we need for AI.” – Anon

“Geopolitics is the big issue. We currently have countries at war with each other, or even with themselves. It makes co-operation and collaboration extremely difficult. It also takes a long time to countries to get together and agree terms and treaties. AI development does not appear to move on a compatible timescale.” – Brian Hunter

(21) Single centralized development?

How do you assess the suggestion that all the different groups that are presently developing AGI solutions should be coordinated into just a single project, with the project plans and operations being reviewed on a regular basis by representatives from governments around the world?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“Given the risks of AGI, it would make sense if it could be consolidated into a single project to slow or stop the march of technological progress.” – Hiroshi Yamakawa

“It would be a great idea for the work to be more coordinated – it would likely result in additional progress, funding and understanding by the public. I am fully in support of it or something similar.” – Raphael Royo-Reece

“No way, we need more, not less developments. Let a thousand flowers bloom!” – Jose Cordeiro

“Not going to happen. Too much money involved.” – Anon

“Coordination and transparency are good on the merits, for familiar reasons. But a truly singular project seems neither desirable nor possible.” – Anon

“Extremely unlikely to happen given current geo-political competition and distrust as well as disagreement over control mechamisms and ‘values’.” – Tim Pendry

“In a practical sense I imagine it would be difficult to merge all the groups and get them to work towards one goal, while presumably you would also have to make it illegal to start up an independent group. On a global scale especially, I just think this is unworkable.” – Simon

“A disastrous brake on progress with do-nothing being the stated consensus view across governments, and at the same the disappearance of development into secret defence projects.” – Terry Raby

“Noooo!!!! I think that might be the single most dangerous idea of all. Then we depend on that one body getting it exactly right and we are all at the mercy of whatever they create. Not only that, but it creates a regime in power and the rest of us are powerless against it. Haven’t we had enough of this sort of thing? Democracy has been messy but has shown a far better track record of achieving improvement of life on Earth.” – Neil

“It’s a pipe dream – not equilibrium stable as a game, too much incentive to renege” – Bronwyn Williams

“Think of the IMF and how unhelpful it has been to specific economies.” – Jenina Bas

“It would need to be very robustly democratic and not further concentrating of power” – Josef

“What’s the saying, a camel is a horse designed by committee? A single entity on its own probably isn’t going to come up with a great solution (no offence to camels).” – Brian Hunter

“It doesn’t sound realistic or desirable, better to have some sort of meritocratic body to address the technical questions, govt could monitor, but are politicians competent to make project decisions? ;)” – Anon

“Consider something more akin to the human genome project – competitive and yet coordinated as well.” – Anon

(22) Who initiates?

Which starting point for agreement on global governance is most likely to succeed?

Answer choices:

A country or countries offer a UN Resolution to establish a working group to draft text for a UN Convention on AGI
Begin with one geographic region, like the EU, and expand later to the UN
Proposals emerging from within the private sector
An existing international body, such as Global Partnership on AI
Something else (please specify)?

Selected comments from survey participants

“All at the same time. It will require a massive upswell of public demand. If that happens, the politicians will follow.” – Calum

“The UN approach is probably the most comprehensive but regional efforts can develop ‘best practice’ models for global consideration. The private sector will merely try to control regulation by regulating itself to the minimum required.” – Tim Pendry

“Regional blocks where consensus is possible is the most pragmatic” – Bronwyn Williams

“Given AI is not yet especially global, the benefits of the USA and EU regulating AI properly are (currently) similar to the benefits of the world regulating AI, but much easier to achieve. Once there is a regulatory framework it will be much easier to extend the regulations, with tweaks, to the rest of the world.” – Raphael Royo-Reece

“Begin in one place, but not the EU which has a disastrous record of suppressing science.” – Terry Raby

“it needs to be a globally supranational organization in order to command the attention that would be required. It needs to be more agile than for instance, the EU.” – Colin Smith

“I don’t believe the UN has accomplished anything of note globally, this may be the wrong organization to focus on.” – Josef

“America and China, as the two AI powerhouses, agreeing on a set of constraints would work, just like the USSR and USA regulated nuclear arsenals to a certain extent. However, AI governance would be infinitely harder to enforce than nuclear arsenals.” – Vid

“It’s very likely the US govt (DARPA/Dept of Defense in particular) will have the most influence – most of the larger outfits are there, from what I know – so we should start there” – Anon

“We do not have the necessary global governance instruments we need for AI, and there are SEVERAL concerns which rank above AI in importance where we lack necessary global governance instruments. In general I am skeptical of developing entirely novel institutions to address these problems AND am skeptical inter-national institutions can be developed effectively to the purpose; we need to look toward ‘trans-national’ institutional systems.” – Anon

“Since 2/3 of all AI resources, research and development are in the USA, such a global AI Development Control Programme should start in the USA. I have proposed the creation of five, interconnected global organisations in my recently published book. These are:
1. Global AI Regulation Authority (GAIRA) by transforming the Global AI Partnership (GPAI). It should be responsible for regulating a global use of AI in society.
2. Global AI Control Agency (GAICA) as a Consortium set up initially in the USA and gradually expanding to engage non-US companies, including China.
3. Global AI Company (GAICOM). This would be a Joint Venture company to consolidate the most advanced AI companies into a single organization. Effective control over AI development will be impossible if it remains dispersed among numerous companies.
4. Superintelligence Development Programme (SUPROG) managed by GAICOM and matching China’s efforts in the AI sector,
5. Global AI Governance Agency (GAIGA) under the mandate of the G7 Group or a de facto World Government. GAIGA would oversee both GAIRA, responsible for regulating the use of AI products and services, and the GAICA Consortium, responsible for AI development control.
Simultaneously, we need very urgently a de facto World Government initiated by the G7 Group, incorporating members from NATO, the European Union, the European Political Community, or from OECD. Apart from aligning the AI development initiatives, one of its key functions should be the creation of a Global Welfare State, which would also include the setting up of a Global Wealth Redistribution Fund, needed to mitigate the challenges posed by such a momentous civilisational transition, which we begin to experience.” – Tony Czarnecki

(23) Template governance models?

What is the best template for global governance of AGI?

Answer choices:

The WTO (World Trade Organization)
The IPCC (which coordinates global response to the risk of climate change)
An Industry Standards and Technology Organization (ISTO) such as the IEEE
The IAEA (International Atomic Energy Authority)
An international version of the FAA (Federal Aviation Administration)
A decentralized semi-autonomous institution
Put different parts of AGI governance into different bodies
A single centralized development, overseen by representatives from governments around the world
Something else (please specify)?

Discussion

Many of these organizations appear to be too static to manage the fast-changing nature of the approach to AGI. Moreover, merely the setting of standards provides no guarantee that these standards will be observed.

Selected comments from survey participants

“IAEA can be an initial template with scope to adapt as needed” – Anon

“The IAEA but with more power; the ability to reliably and seriously punish those that breach regulations.” – Raphael Royo-Reece

“The track record and pragmatism of the FAA is pretty impressive” – Rusty Burridge

“Probably the IPCC is the closest template we have, though I’d note it’s not really listened to. We blow past all its recommendations.” – Brian Hunter

“The WTO is unwise because the regulations comes up against global resistance to neo-liberalism which is likely to increase periodically. AI should also not be confused with othe rissues such as climate change or atomic energy. it probably requires its own enforcement body and mechanisms if only because it has to be a centre of excellence for ‘expert opinion’.” – Tim Pendry

“I simply cannot see any enforceable mechanism in our current set of (institutional) tools. It will have to be something novel.” – Vid

“I tend to favor a hybrid system with at 3 different elements: direct democracy govt, decentralized semi-autonomous institutions and a meritocratic body (perhaps merely advisory)” – Anon

(24) Racing within constraints?

Suppose, for this question, that it is decided that any absolute moratorium on specific progress with AI, observed by just a few countries or organizations, would give too much of an incentive to “rogue” countries or organizations to develop AIs without respecting that moratorium. In this case, might there still be a case for agreement on “highly desirable” principles, to be self-imposed by as many companies and organizations as possible?

In this scenario, there would still be a race toward the creation and deployment of AGI, but these “highly desirable” principles would increase the chance that the AGI that emerges will be benevolent rather than destructive.

Answer choices: Regulation must be all-or-nothing / Incremental regulation is better than doing nothing / There is no need to fear rogue entities / Unclear

Discussion

The idea here is that many questions remain in play, even if someone decides against asking the whole world to observe various restrictions in how they develop AGI.

Selected comments from survey participants

“Doing nothing is not an option despite the challenges.” – Tim Pendry

“We should try something, but just keep in mind that ‘rogue’ entities in this context tend to be the biggest/ more powerful countries. So no, I don’t see it really working out.” – Vid

“I think there is value in voluntary adhering to a set of highly desirable principles. And with enough of the big players on board it could become a defacto standard. You can already observe today that when new LLMs are released they all contain descriptions of their attempts to remove bias, offensive responses and the like, most even posting their results against evaluation methods used to detect such responses.” – Simon

“People should come to understand that breaking the regulations just creates more chance to harm themselves and others” – Anon

“China is deeply resistant to global governance and has so much of so many of the capacities which we can expect will enable AI development — smart people, powerful institutions, wealth, technical capacity, and a frightening lack of squeamishness about
things like data-gathering and placing big bets with big consequences for a lot of people. They will do what they will do whatever the rest of the world attempts.” – Anon

“There is no need to fear rogue entities. The current potential rogue entities (e.g. China) are well behind on this front and would be severely slowed down by having less open research and development to ‘borrow’ from.” – Kristian Freed

(25) Most troubling question?

Which one question, from the set above, makes you most unsure about how humanity can reach a truly beneficial future?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“Human emotions that foster fear than opportunities to find solutions.” – Natasha Vita-More

“We don’t have consensus on what we want now, we cannot agree on what we want next. What will come next will “emerge” out of revealed preferences not stated priorities.” – Bronwyn Williams

“While not related to a specific question, the worry is that AGI becomes one of Bostrom’s ‘black ball’ technologies, that, once out in the world, will inevitable lead to disaster in much the same way that having the ability for anyone to create a super-covid in their homes would.” – Kristian Freed

“Q7 – the idea that we can make a super tool which won’t become an agent. Even if this might be possible, people would put significant effort into making that tool part of an agent. That tendency seems absurdly dangerous but I can’t see how it can be properly guarded against.” – Raphael Royo-Reece

“I think this whole questionnaire seems to learn towards the bias of establishing an overarching committee of folks with the power to control AI development. I’m more worried about what humans are doing here than the AI itself. Quis custodiet ipsos custodes?” – Chris Gledhill

“Q20 is pretty troubling, we can’t think of all the possible disruptions, and techno-optimists underestimate opposition from people with very different world-views. humanity is at a major fork in the road, different political factions might form around the AGI question, and it could lead a literal split in the human race” – Anon

“How to solve the coordination problem.” – Josef

“Probably Q22, only because the initial settings could make a large impact on the final outcome, so getting this right is important, and if it is a small set of people with this power, this does worry me.” – Simon

“Q23, as it reminds me at how bad we are at doing this kind of thing.” – Brian Hunter

“Not intelligent rulers and governments are probably the biggest threat to humanity. We need more (not less) intelligence, both natural and artificial.” – Jose Cordeiro

“The question related to fusing AI with a human interface” – Anon

“Anything that purports a global governance solution.” – David Shumaker

“The questions that deal with law versus open source.” – Rusty Burridge

“The political and social disruption caused by AI” – William Marshall

“Almost all of the questions above reflect faces of the same problem — we have big global risks while we lack not just the capacity to do good global governance, we lack the capacity to even do clumsy global governance, and lack promising ideas about how to develop them.” – Anon

“The precautionary principle.” – Terry Raby

“I fear most of all the ideas of halting or only allowing a single entity to work on AGI. I also fear the regulations becoming too oppressive. If either of these things happen, the emerging AGI will be much more dangerous and less aligned. It is important to develop it now when the technology is not sufficient for AGI and to let it loose into the world (after reasonable safety is achieved based on past experience and experiments) so that we can see the safety/alignment issues that would never have been discovered if it wasn’t released and learn to overcome them incrementally. It is also important so that we can all benefit from this progress and not just the elite.” – Neil

“Our current economic model needs to be updated to accommodate this new toolset – and I don’t see anybody talking about this. The fact that job losses were mentioned in but two questions makes me feel very uneasy about this, especially because the redistribution of wealth from labor to capital hasn’t been mentioned anywhere. I presume this is the case because we are still stuck in the paradigm where calling into question these ‘market rules’ is akin to heresy – and that is what worries me the most. We won’t be able to mitigate and solve if we do not acknowledge the problem, and the problem isn’t AI, it’s the associate transition to hyper-capitalism and extreme inequality it will cause.” – Vid

“The most troubling question that I have is whether the Western World is capable of re-inventing itself and create a de facto World Government in time to control AI development globally. Of course it will not be truly global control because China would not join such a World Government. Therefore, it is also important that the AI developed by the West must be the most advanced. From that point of view pausing AI development, which was proposed in May seems to be not a very good idea.” – Tony Czarnecki

(26) New questions?

Is there an additional question, not present in this survey already, that should be added?

Answer choices: No fixed answers provided. Respondents are invited to supply their own comments.

Selected comments from survey participants

“How important is AI compared to other major global risks?” – Anon

“All tech has pros and cons. The questions so far are very dystopian. Needs balance on benefits of AI. E.g. if we had the nuclear debate only focussing on destructive power, we wouldn’t have nuclear power stations.” – Chris Gledhill

“It seems self evident that military AI will not be subject to the same restrictions as civilian. Given that military AI will by definition be the most dangerous, what solutions exist?” – Dirk Bruere

“How can laypeople can effectively push for AI safety, beyond writing letters to their representatives? (My own thoughts are in a campaign for AI safety being launched, but I’m sure there are great suggestions out there.)” – Raphael Royo-Reece

“Are there any psychological or physiological barriers, which may make impossible the fusion of human brain functions with AGI via BCI?” – Tony Czarnecki

“The mutual exclusivity of transparency and black programs” – David Shumaker

“How can we define AGI + will/drive/’consciousness’?” – Bronwyn Williams

“Is it really likely that Artificial Super Intelligence will be the next step after AGI? If so, when?” – Anon

“What about a superintelligent oracle which is air-gapped for safety, does not have the ability to re-write its own code, and its purpose is to help humanity by just telling us how to build fusion reactors, cure cancer etc.” – Simon

“Have AGI proponents offered the same level of evidence for the utopian benefits they put forth, as they demand for x-risks? Why are we OK moving forward with more advanced systems, when we supposedly barely understand the ones (e.g. LLMs) which we helped to evolve (not program directly) today? Why do we need to build AGI, rather than an ANI? In what ways do those resistant to regulation/pause share common ground with those concerned enough for a pause?” – Anon

“What is the difference between AI and AGI? Possibly, by analyzing differences, you avoid potential misleading definitions of each term. It gets back to very definition part of this survey. There is a criticism by David Deutsch, not agreed to my knowledge by AI/AGI experts as Ben Goertzel, Gary Marcus and others, about the current, fast evolving AI and the unknown AGI insofar the two seems tending to opposite directions to each other. This is exemplified by LLM’s or similar AI tools evolving to demand more and more data, parameters and computational resources while humans seems to get well along with modest computational resources (e.g. energy efficiency) and less data. See Deutsch’s contribution to Possible Minds (ed. John Brockman) of 2019 but also repeated more recently, that is, after the public explosion of LLM’s.” – Pasquale

“Can we, and should we, ‘virtue up’ artificially?” – Rusty Burridge

“Ask more about inequality, not only in terms of production, but also in access to computing power (within countries, across countries).” – Vid

“The burning question at this point in history is what the next step for the world is after free market globalisation driven from the West by the West is failing (since 2008). This is the background against which AGI emergence is taking place. So, what role will AGI play in the transition to that next stage in the evolution of human society, whatever that may be.” – Jenina Bas

Image credits

The image at the top of this page was generated by the Midjourney AI.

The background in the image at the bottom of this page is by Pixabay contributor UlisesEkzMoreno and is used with thanks.

Key open questions about the transition to AGI

(0) Introduction

Purpose of this survey

*Not* an opinion poll

Avoiding four shortcomings

Take courage!

Definition of AGI

Survey process

Publicity

Background

(1) Alternative definitions of AGI?

Aside

Selected comments from survey participants

(2) Imminent or Distant?

Discussion

Selected comments from survey participants

(3) Catastrophic or Beneficial?

Discussion

Selected comments from survey participants

(4) Anticipating huge economic dislocation?

Discussion

Selected comments from survey participants

(5) Understanding the risks and benefits?

Discussion

Selected comments from survey participants

(6) Single or Many?

Discussion

Selected comments from survey participants

(7) Limited or Unlimited?

Discussion

Selected comments from survey participants

(8) Automatic moral alignment?

Discussion

Selected comments from survey participants

(9) Encourage a free market decision on AGI?

Discussion

Selected comments from survey participants

(10) Incremental or Anticipatory?

Discussion

Selected comments from survey participants

(11) Surveillance without autocracy?

Discussion

Selected comments from survey participants

(12) Global agreement on values?

Discussion

Selected comments from survey participants

(13) Hardwiring moral principles into an AGI?

Discussion

Selected comments from survey participants

(14) Agreement on canary signals?

Discussion

Selected comments from survey participants

(15) A watertight switch-off mechanism?

Discussion

Selected comments from survey participants

(16) Superhumans monitoring superintelligence?

Discussion

Selected comments from survey participants

(17) AIs teaching us how to control or align AIs?

Discussion

Selected comments from survey participants

(18) Useful heuristic constraints?

Discussion

Selected comments from survey participants

(19) A role for open source and/or blockchain?

Selected comments from survey participants

(20) Other overlapping disruptions?

Selected comments from survey participants

(21) Single centralized development?

Selected comments from survey participants

(22) Who initiates?

Selected comments from survey participants

(23) Template governance models?

Discussion

Selected comments from survey participants

(24) Racing within constraints?

Discussion

Selected comments from survey participants

(25) Most troubling question?

Selected comments from survey participants

Not an opinion poll