Can ChatGPT solve ancient Chinese math problems? (May 4, 2025)

Since its inception, ChatGPT has been increasingly getting better at solving mathematical problems, with current iterations even surpassing the ability of most humans according to OpenAI. But does this ability also extend to pre-modern Chinese mathematics, which uses a language that differs significantly from what the model was trained on, and might rely on knowledge about a world that is unfamiliar to the model?

In order to explore this question, Florian Keßler has built a dataset of mathematical problems extracted from ancient Chinese texts, and tested ChatGPT’s performance in solving them. The results show that while ChatGPT is able to solve many problems, it does struggle with technical expressions from pre-modern Chinese mathematics, e.g. simply understanding “da ban 大半”(lit. greater half) as 1/2 instead of the correct 2/3, and has difficulties correctly handling historic units of measurements present in the texts.

The project will be presented on May 4, 2025 at the Second Workshop on Ancient Language Processing (ALP 2025). The complete program of the workshop can be found here. The dataset can be found on GitHub.