AI and the Value-Loading Problem
In his book on the future of AI (Artificial Intelligence), Life 3.0 , Max Tegmark talks of the importance of ensuring its goals are aligned with ours. This isn’t as easy as it sounds, warns Tegmark. Just think of King Midas: he got literally what he asked for, except that wasn’t what he wanted. Or all those stories where the genie grants three wishes, which almost inevitably end with the third wish being: “Please undo the first two wishes, because that’s not what I really wanted.” Be careful what you wish for: it might come true. Put differently, this is the danger with an AI that takes our instructions literally: “The real risk with AGI isn’t malice but competence.” So where does that leave us? “All these examples show that to figure out what people really want, you can’t merely go by what they say. You also need a detailed model of the world, including the many shared preferences that we tend to leave unstated because we consider them obvious, such as that ...