The author argues for evaluating Ph.D. candidates based more on the oral thesis defense than on the actual thesis.

By essentially the same reasoning, I’ve been arguing for prioritizing in-person design/code reviews over code-only async PR comments.

The important thing is to verify that the human has a coherent design in mind and can demonstrate that it got implemented, regardless of who or what was at the keyboard. “I dunno, I guess Claude thought this was a good idea” is not a coherent design.

Excellent optimistic post in a sea of negativity, and with actual suggestions, too. After reading, my mental image is this: think of Olympiads in Ancient Greece.

* A weightlifter was only awarded a laureate if he were able to lift a heavy stone (have no idea what they were lifting, for illustrative purposes only :-)

* Along comes Archimedes who invents what we would call an exoskeleton. Now any regular guy can lift twice as much as last year’s athlete.

* What to do? You can cancel the Olympiads, but they are actually useful as training, motivation, etc So now you have to give the prize on other factors, eg how well he can lift, has he opened a gym in the city, etc

BTW, physics and bio are not exempt, so those researchers better read and try to stay ahead.

> We already interview faculty hires; we must now do the same for graduate admissions

They hire PhD students without hearing them give a talk and then doing interviews? In Germany, the applicant gives a talk (30-40 min) to the research group they want to join, usually presenting their master's thesis, engage in discussion, often share lunch with the group, then do 1 on 1s with individual members of the group and a longer one with the PI. Obviously this can vary within Germany too, but I couldn't imagine hiring someone without something like this.

As someone who has a degree in math, I still can't help but think mathematicians are getting a little bit of a comeuppance. In a lot of areas of mathematics there had been little effort to make the work understandable and leaves numerous folks who could benefit from the knowledge on the outside looking in. Now AI comes along and do the same to mathematicians. Makes me chuckle a little bit.

I love how this is (without slighting the problems entailed) coming at it from a perspective of infinitude and abundance (we will always have more problems to solve) - which is the correct framing, particularly when dealing with ideas, or fields in the which ideas are the driver/product/output/material, and ideas themselves, the field itself, are infinite.-

PS. The validation problem, being one.-

I admit I just don't understand this attitude.

People had the same complaints that the code produced by early coding models was messy, lazy, poorly commented, had terrible architecture and so forth. The central complaint was that it was just too difficult for humans to review. The answer is just to improve the models and move on.

Similarly now we're getting AI doing math. The proofs compile but are a mess. So just make the models better at writing clean proofs and explaining what they're doing to humans. That's the end of it.

Rather than just go on and on about how it's the end of the world if we don't do this, why don't we just do it?

The more I see these posts about mathematics institutions reforms and challenges from AI advancements the more it looks like they may need to go through a death. Or to put it another way they may need to start again from first principles.

If math is truly about spreading intuition and understanding then our institutions have dropped the ball decades ago and have not been able to grab hold of it since (if they ever had it to begin with)

"A computer or monkey could easily start at the axioms of ZFC and iteratively apply deduction rules....simply conjecture all mathematical propositions in alphabetical order...The prospect of automating mathematics by enumerating all conjectures, and all proofs of ZFC, is probably not so disturbing to you."

I thought we were going to get at least some brief comment on Godel here?

I like the quote from Hilbert that was brought up in the article: "we must know, we will know".

With AI, it might be the case that we don't know, we won't know, but the machine does.

The central question, namely whether humans should be in the loop, will be repeated again and again in the years to come for all industries, starting with mathematics.

I am not a mathematician, but I can’t see how we are going to address the problem which we already see in coding:

Impossibility to independently validate all AI results

And in math it goes even worse. In coding code reviews are typically still the form of action you do within days. In math, historically, the lifecycle of proof is months if not years. Take as an example Millennium problems. They require at least two years of validity after publishing. Two years! In modern times with amount of output AI can produce, it feels like infinity.

We are inches close if not at the moment already when humans can’t reliable validate proofs and mathematics produced by AI. Then next research will be based on this AI-written-no-human-in-the-loop results. And we will end up in just few years in a world where novel and frontier problems will be articulated by AI and proven by AI based on AI results and humans will be incapable of understating the mere nature of the solution.

> resulting in the production of an abundance of PDFs. The contents of some of those PDFs may even have important applications.

I hope, from the depths of my soul, that the static typeset report format for transmitting knowledge and understanding will finally die and be laid to rest.

My worry is that the frontier of math is too far away for most humans to reach. AI is only going to make that worse. If today it takes twenty years of math study to reach the frontier (in a narrow field), what's it going to be like when it takes forty years? Or four-hundred years? Will the fields just get narrower and narrower to accommodate finite human intelligence?

> I think so. This machine might produce answers we value, but it would not, in itself, produce human understanding of those answers.

It's nice that the author is optimistic, but won't the AI be best placed to dumb down its increasingly complex proofs into a language us lowly humans can understand? To keep thinking until it can refactor complex proofs into ones from 'the book'?

> I propose the following reconceptualization of the goal of a mathematics PhD: to become a world expert on some interesting, deep topic, and to be able to convey that interest and understanding to others. Part of operationalizing this might be a thesis, but the degree would be awarded primarily on the basis of a rigorous defense, in which the student explains the topic to their examiners until they are satisfied.

I think this is a refreshingly forward looking idea and I agree with it 100%, especially the the "rigorous defense" part. That is a good measure of how well the topic has been researched and understood by the researcher. This is where the humans can be "in the loop".

> How different would this look from current PhDs? I think students would still meet with an advisor, who might suggest a topic. That topic could be explored with AI assistance, or not, but the student would be responsible for understanding it; it might be much more open-ended and larger than the typical PhD is currently.

Interesting point about "more open-ended" and "...larger than the typical PhD". I think the author has a point. Earlier, the bottleneck was the candidate's/researcher's understanding and knowledge. Now with AI tools, it is so much easier to zero in to relevant knowledge, get your questions answered quickly which might lead to understanding more quickly.

For e.g., before the advent of public libraries and printing press, the knowledge was inaccessible and guarded. So that was the bottleneck.

Then books became ubiquitous and the bottleneck to knowledge and understanding was people's motivation AND knowledge of WHAT books and topics to research.

Then came the internet and free PDFs of books and research articles. Now, the bottleneck was still people's motivation and a mild version of what books and topics to research. I say "mild" because one can lookup articles and newsletters, and book reviews and come up with a list of reading.

Now comes AI and it looks like the only bottleneck is people's motivation.

I believe there was also a silent, yet potent, bottleneck all along which is also removed by AI: personal tutor/coach/teacher/professor etc. Let's say if I am reading a textbook on manifolds or some research paper and I have a question about a specific theorem or even a mathematical operator being used. Before AI my only way to get my questions answered was to read more books (PDFs or print), or ask on math exchange or math overflow and wait for someone to answer, or to ask a professor. This could take up to a week.

Now all of that has been cut down to 1 hour or less with an interactive chatting session.

!!!!!

So....the only bottleneck is people's motivation! QED

Exciting time!

This is an excellent essay, and especially relevant when read together with the MIT report on the future of education in an AI era: https://aiandeducation.mit.edu/report/ (and my own https://oberbrunner.com/blog/ai-future-of-mathematics).

I'm heartened to see this kind of thoughtful, reasoned discussion around long-needed reforms in the educational community, with AI as the catalyst. We certainly have interesting times ahead.

What is the incentive for a person to sit though seminars and evaluations? People already hate redundant meetings. What is the incentive to change the system from the existing one to one that rewards this verification somehow? Who benefits from this change?

I was involved in this research a few years ago: https://metamiditoolkit.com/research-paper/

One of the things that struck me was that most people have an area of their workflow that they would be happy to hand off to an AI, so that they can focus on the thing they love. However, that area is different for each person. One person's grind-work is another person's love-work.

This is why it's so hard to build consensus around where the "red lines" are in this space.

Yes the fascinating thing is:

1. It will take much longer to understand the output of the machine that it takes to prompt and create it. 2. The only? best? one? way to /verify/ that you /in fact/ understand the output of the machine is to explain it to someone else.

So there will be a machine generating koans which need to be meditated upon and discussed with human social back-pressure validating understanding. I think this could be much more cooperative and at a minimum this will be a way different math social construct.

I’m not a mathematician, but if I were I would imagine the main attraction would be the act of thinking hard and solving problems. Maths felt like the ultimate profession for someone who loved puzzle solving. If you delegate the thinking, any percent of it to an LLM, I don’t think it’s maths anymore. Might as well join any other job and earn better money now. A completely ludicrous proposition, that maths will no longer involve mathematical problem solving.

Misanthropic and Open Artificial Ignorance are misleading the public.

They are leeching off the work of mathematicians and pretending that new math was invented.

As with the rest of the domains, AI/LLMs will do syntax and search better than any human.

In code, any developer whose differentiaton was clean code and knowledge of different technologies is now average.

In math, any mathematitian whose differentiation was to manipulate formal systems and know tricks of different domains will be average.

Fortunately, humans do more than syntax and search.

The bad news for developers is that if you know what the output of your program should be (which happens most of the time), almost all of the job is syntax and search to build the code that reproduces the output.

The good news for mathematitians is that for the majority of problems you never know the output, or you just know the output is either "True" or "False". There are some cases where you need something else, for example "a solution that blows up in finite time". For those cases AI will outperform you easily (see new Navier-Stokes solution)

So if as a mathematitian you were doing more than syntax and search, then keep doing that and use AI just for what its best.

As a recent CompSci graduate and software engineer, I can sympathise with the author’s position. At the very least, these times are highly unsettling.

Pretty neat how these abstract origins still shape our world, especially in CS. Kinda makes you appreciate the shoulders we stand on.

Always fascinated by how foundational math concepts built up. Euclid's Elements still blows my mind for its logical rigor.

Does anyone have a rough estimate of how many research mathematicians there are in the U.S. or world?

> AI does not care if you are anti-AI.

This >>>

While I broadly agree with the premise of re-directing the “purpose” of math, I quite detest the idea that judgement might be primarily based upon some in person discussion, or oral presentation, and the claim that written mathematics that is not orally communicated might be less worthwhile in some sense (i know this isn’t the exact statement of the authors proposition).

There are a good deal of people, whom, falter much more in oral discussions, whether this be for a psychological thing, stage fright, or difficulty explaining things on the spot. There are also certainly brilliant people, who can’t give an informative, discussion inviting talk to save their lives, but given enough time, can formalize their thoughts in writing at the highest levels of their field, and that writing is likewise very enlightening (sometimes).

It’s not clear to me, that, AI as is, could not pose successfully in an oral discussion of a topic. I mention this because it seems that one implication of the article is that AI might write things that are logically correct, but devoid of understanding. I suggest rather that 1) it is not extremely improbably that AI is incapable of generating mathematics that furthers human understanding and if 2) it is indeed highly likely that they cannot generate mathematics that furthers human understanding in a textual format, then surely one could also differentiate between human and AI on a textual level, and judge the contribution of a human, without the need of oral discussion?

I suppose another aside is, one might claim that the existence of AI means people have much much more text to filter for, and so, it becomes difficult to find one person’s good writing amidst a sea of, logically correct, yet understanding devoid textual content. But by and large much or mathematical academia certainly operates off of some reputation/vouching system presently anyways, that already serves as a “filter” in some sense. Perhaps the existence of such a system/culture is not a good thing, but oral discussions/seminars certainly aren’t immune from such predilections.

Perhaps I’m babbling like an idiot, but the entire and sole purpose of this comment is just to say: for the love of god please don’t let the standard be judged by oral presentation

Imagine, eventually, a large ever growing library full of unread books. One could roam around forever and discover new things.

Might be possible to generate a 3d world using some taxonomy estimate the amount of titles and generate the books as the area is explored.

Like the manuscripts in Sakya Monastery in Tibet and like the millions(?) of cuneiform clay tablets. Vast amounts of data that may contain information.

The key shift seems to be from "can you produce mathematics?" to "do you understand mathematics?"... Those used to be correlated strongly enough that a thesis or paper could serve as evidence of both. If AI breaks that correlation, evaluating people through defenses, discussion and live problem solving suddenly makes a lot more sense.

I think this goes in the right direction. You have to rethink the role of human work in math, can't put your head in the sand and cling to your comfy institutions and system just because you got to know it's ins and outs and just want it to be like that forever.

But the bigger picture is: while I understand the author know his field best and wants to keep the post focused, the same issue will hit many more fields. We need to also have a broader discussion that involves more fields of knowledge work, largely academic scholarship but also regular office work, then it will come to engineering design, medicine, it's already coming for 3d modeling and vfx, software dev, it will come for a lot of middleman services. Not at the same rate, but we have to understand that it's not just that math will change.

Change will be the default. Everything will change. It will be much easier to make math change because all things will change. You shouldn't worry and imagine that funding criteria will be like today or that journals or academia or politicians expectations will be like today. No, everything will adjust with some timing differences of course but it won't be a static world and then math changing and having to justify and fight to explain the change to other actors who are baffled. They won't be baffled they will themselves have to change.

The world will transform as much as it did when society moved from feudal agrarian to urban capitalist industrial, or from the vast majority doing physical labor to a service economy with a huge amount of desk jobs. I can't tell how it will change exactly but it will be bigger than what we have seen in the last couple of generations or maybe more.

[flagged]

[flagged]

[dead]

[dead]

Another mathematician waxing poetically about a future that will not come to happen. Call me a pessimist if you'd like, but capital has no incentive to ensure mathematicians maintain their current status in society.

If you're a mathematician you are in the same boat as the software engineer, and the Dodo.

Better learn a trade buddy /s

Better plan is to shut down PhD and make students take a oral thesis at bachelor and master level and help them become a productive economic participant as soon as possible.