We (the alignment community) think we want corrigibility with respect to some wide set of goals S, but we actually want non-obstruction with respect to S

Trying to implement corrigibility is probably a good instrumental strategy for us to induce non-obstruction in an AI we designed.

Can we get negative results, like "without such-and-such assumption on πAI, the environment, or pol, non-obstruction is impossible for most goals."

Can we prove that some kind of corrigibility or other nice property falls out of non-obstruction across many possible environments?

Given an AI policy, could we prove a high probability of non-obstruction, given conservative assumptions about how smart pol is?

Main idea: we want good things to happen; there may be more ways to do this than previously considered.

However, even if we could maximally impact-align the agent with any objective, we couldn't just align it our objective

Main idea: We only care about how the agent affects our abilities to pursue different goals (our AU landscape) in the two-player game, and not how that happens. AI alignment subproblems (such as corrigibility, intent alignment, low impact, and mild optimization) are all instrumental avenues for making AIs which affect this AU landscape in specific desirable ways.

Main idea: By considering how the AI affects your attainable utility (AU) landscape, you can quantify how helpful and flexible an AI is.

Main claim: corrigibility’s benefits can be mathematically represented as a counterfactual form of alignment.

Various biosecurity risks

there was just no way we were going to have AGI in the next 50 years

Various biosecurity risksStable dystopias, nuclear war or major power war, whole brain emulationClimate changeSuper volcanos, asteroids

So the reason that we don't have AGI is not that we could make AGI as powerful as the brain, and we just don't because it's too expensive.

Crux 1: AGI would be a big deal if it showed up here