One of the country’s top mathematicians is taking a different approach to homework. It may point to the future of education.