If the research is aimed at ensuring that AI is aligned with human interests, another priority should be researching what we deem those interests to be.

The Law and Ethics research section of the research priorities document is centered on mechanisms, with a couple of nods to policy, but with the seeming assumption that that question of what is in the human interest has already been answered.

But the answer certainly isn't obvious. Different sets of humans have wildly diverging and fundamental disagreements over what is right or wrong for humans and our future.

I doubt we're going to find a consensus in the time we have. It will probably make more sense to just have smart people define a set of good principles. This is hard enough that it should be considered a research task, and I would think it would be at the top of the priority list.

Otherwise, how do we know that the research we are doing on other problems even makes sense? If you just stub out the "human wishes" component of all this as one simple "human approves" button, then you haven't solved anything. We need a robust definition of what "aligned with human interests" means.