Humans don’t know what they care about

If you hand over the management of your civilization to machines, you then lose the incentive to understand it yourself or to teach the next generation. […] Up until now, we put a lot of our civilization into books. But the books can’t run it for us. So we always have to teach the next generation. If you work it out, it’s about a trillion person years of teaching and learning in an unbroken chain that goes back tens of thousands of generations. What happens if that chain breaks?

Stuart J. Russell