What makes it possible to trust computations we cannot do ourselves? When are we justified in delegating? What barriers can we erect to avoid being deceived? These motivating questions in AI safety have a long tradition in computer science; I will argue that partial information plays a key role in addressing them. After a review of some celebrated mathematical ideas pertinent to AI safety (e.g., verification, debate, and interactive proofs), I will describe LawZero’s explainer-predictor architecture and some of the theory supporting consequence invariance—a certifiable blindness to certain irrelevant information—which we believe to be important for building powerful disinterested AI systems.

The second part of the talk concerns the theory of probabilistic dependency graphs (PDGs), a representation of uncertainty based on partial information. PDGs can also model epistemic conflict; in doing so, they yield a recipe for “natural training objectives”: minimize inconsistency. Time permitting, I will conclude with an interactive proof system for bounding a predictor’s inconsistency in polynomial time, despite its apparent doubly exponential complexity.