Lauren Williams -- Human Mathematicians in the Age of AI

Summary

Lauren Williams, a Harvard mathematics professor, discusses how AI is affecting mathematical research, the First Proof benchmark, and how the profession might adapt. She uses AI modestly in her own work—for computations and proofreading—but is concerned about its wider effects on mathematics and mathematical culture.

First Proof: Purpose and Methodology

First Proof was created to test what publicly available AI systems can do with new mathematical research problems, without relying on solutions already posted online. Williams describes the process as follows:

• Choose problems with unpublished solutions. Mathematicians submitted statements they had recently proved but not yet publicized, along with their solutions. The problems were intended to involve some novelty rather than merely apply standard techniques.
• Limit solution length as a rough measure of difficulty. Human solutions were capped at five pages for the first batch, eight for the second, and 12 for the third. Williams stresses that length is an imperfect measure of difficulty.
• Run a public, one-week trial for the first batch. The problems were released before the solutions, which were encrypted and posted for later release. Anyone could try an AI model. Williams says this initial effort was an experiment rather than a controlled benchmark.
• Make later testing more controlled. For the second and third batches, teams supplied their AI systems, or “harnesses”—software and instructions for using models. First Proof organizers ran the tests themselves in cloud containers, aiming to limit human involvement and keep queries from being sent back to the companies.
• Have mathematicians assess the output. For the second batch, about 30 expert mathematicians spent two days evaluating the generated solutions, using standards similar to those for refereeing a research paper. First Proof also released materials such as logs, costs, generated solutions, and referee reports.
• Score more than correctness in the third batch. Because the second batch revealed problems with writing and attribution, the third batch scored solutions on correctness, exposition, and attribution. Each axis used a scale from −1 to 1, including negative scores to reflect the cost of spending time on incorrect or poorly written work.

Benchmark Results and Concerns

• In the second batch, systems solved about seven of the 10 problems more or less correctly, but the solutions often had poor exposition.
• Graders also found serious attribution problems. In one example, systems used the strategy and even notation from a previously published proof by Rick Schwartz without clearly citing that work. One solution included his paper in its bibliography but did not explain its relevance.
• In the third batch, attribution improved, though problems remained; exposition was still often weak.
• An open-weight model, Kimi K3, performed far behind the frontier systems on correctness. Williams presents this as a reason to invest in open alternatives, including to reduce privacy concerns and reliance on a small number of companies.
• Results depended on the harness, but higher cost did not guarantee better performance. Williams says Merlin, which used a model and Lean, cost about $5,000 to run for 24 hours. Proof Council, which combined several models, cost just under $2,000 and got nine of 10 problems correct. Running Astra alone cost under $50 and got seven correct.
• Williams distinguishes First Proof from company-reported results, such as an OpenAI release of 122 manuscripts. She says First Proof tests public models and reports failures as well as successes, while the company’s internal models and testing process were not available for the same kind of independent assessment.

AI’s Potential and Risks for Mathematics

Williams is cautiously optimistic: AI could help mathematicians do more or better work, but it can also cause harm if used indiscriminately. She compares an LLM to a bulldozer: powerful and able to cover ground quickly, but blunt enough to damage what it encounters or miss subtle patterns. A conjecture being solved does not mean an area of mathematics is finished.

She also worries that AI could change the field’s traditionally open culture. Public mathematical papers and conjectures are useful training material for models, which may make researchers more reluctant to share work openly. At the same time, access to expensive models could become important for working at the frontier, creating a new barrier even as AI lets more people generate mathematical papers. Producing a paper, she emphasizes, is not the same as understanding whether its contents are correct, useful, or interesting.

Williams argues that the field still needs trained mathematicians who can recognize valuable ideas, assess AI-produced work, and continue developing mathematics. She rejects claims that mathematics is “over” and expects both research and universities to continue.

Skills, Education, and Career Advice

• Ask good questions. Williams expects the ability to formulate original problems and conjectures to become increasingly valuable, partly because a researcher who creates a new question is less likely to be competing with an AI system working on a widely known problem.
• Develop mathematical taste through people and practice. She recommends attending seminars, talking with mathematicians, and learning what questions others find interesting. These experiences help students develop their own sense of what makes mathematics worthwhile.
• Emphasize oral communication and assessment. Since written work can be produced or assisted by AI, Williams expects interviews, oral exams, and in-person exams to become more important in courses, graduate admissions, and hiring.
• Rethink undergraduate grading. She favors reducing the weight of take-home homework—perhaps to 5–10% rather than as much as half a grade—and relying more on in-class and oral assessment. Homework can still support learning, but grades should better verify that students can engage with the material independently.
• Be selective about AI training. Williams has not settled on exactly how much students should be taught to use LLMs. She would prioritize independent work at the undergraduate level and sees more room for training on tools in graduate education, while cautioning against overreliance.
• Adapt graduate research problems. If traditional “toy” problems become solvable in one AI attempt, advisors may need to give students broader questions and let them explore toward a more specific problem.
• Don’t panic about a mathematics career. Williams advises students and early-career mathematicians not to assume the field is ending. She points to the unlimited supply of unformulated questions and the range of careers available to mathematics graduates, and warns that stopping the training of mathematicians would leave the field less able to understand both existing mathematics and AI-generated results.

Speakers and Sources Mentioned

Speakers

• Lauren Williams — Dwight Parker Robinson Professor of Mathematics at Harvard; guest.
• Alvaro Lozano-Robledo — interviewer.

People, organizations, systems, and sources discussed

• First Proof; its organizers and contributors, including Mohammad Abou Zeid, Rachel Ward, and Rick Schwartz.
• The Simons Institute and Harvard’s Center for Mathematical Sciences and Applications (CMSA).
• The Math Archive (referred to in the subtitles as the “math archive”).
• OpenAI, Anthropic, and a UCLA group associated with the SAIR model.
• AI systems and harnesses named in the discussion: Astra, Claude, Gemini, Opus 5.5, Kimi K3, Siege, Proof Council, Merlin, and Humanize; Lean was also mentioned.
• The Annals, in connection with Schwartz’s previously published work.