Simon Norton, who has died of a heart attack aged 66, was a world-class mathematician sometimes mistaken for a homeless man.

In the late 1960s he represented Britain at the International Mathematical Olympiads three times, scoring the top grade each time, once with 100%, another time with 99%, and winning a special prize for the elegance of his solutions. What made his work beautiful was not its complexity but its simplicity. Without drafts or false starts, he laid down his pellucid solutions to questions involving imaginary numbers, infinity and the distribution of primes with the grace of a ballerina unfolding her hands.

Simon took his first mathematics degree at Imperial College, London, while still a schoolboy. But then came Cambridge University. Rather than allowing Simon to continue his ebullient race into mathematics and start on a PhD straight away, the Cambridge mathematics department insisted he retake the final year of his degree. For the first time, Simon faltered. Mathematics legend has it that he scored a historic 52 alphas in his finals (12 is all it takes to get a first); in fact, it was 13. Simon was not even the best in his class. Bored at having to repeat material he already knew, the next year he almost failed Part III Mathematics, necessary for anyone wanting to start research.

Simon’s fortunes revived when he started to work with the charismatic John Conway, a brilliant and playful mathematician at the university. Together they worked on the Atlas of Finite Groups. Group theory concerns the study of symmetries. Turn a triangle on its side and it will still look like a triangle: that is a group theory result. At its most complex – in the rarefied landscapes where Simon gambolled – group theory underpins our understanding of the universe.

The job of the atlas was to catalogue all the fundamental types of symmetry: the atoms of the subject. Simon’s attention was caught by one of these “atoms”, known as “the Monster”. You can turn a triangle three times, and it looks the same each time. For the Monster, the equivalent number is 808,017,424,794,512,875,886,459,904,961,710,757,005,754,368,000,000,000.

Simon became the world expert on an unearthly mathematical aspect of this group called Monstrous Moonshine. “I can explain what Monstrous Moonshine is in one sentence,” said Simon. “It is the voice of God.”

Simon was playful in daily life as well as abstract thought. Backgammon and making up anagrams were regular, often riotous pastimes in the faculty common room at Cambridge. Simon was the best at both. His instant solution to “phoneboxes” (shouted out before you can reach the end of this sentence) was remembered for decades: xenophobes. He was known not only for his good looks but for his modesty (a rare trait among mathematicians): what delighted Simon was the clever defeat of a puzzle, never of a person.

One day, during the 15 years it took to write the atlas (which was published in 1985), Conway asked Simon a question. Simon gave the solution immediately, as usual, and Conway began writing the answer down … then stopped. “No,” said Conway, “you’ve got that wrong.” Simon saw that he had, and blushed. “That,” said Conway, “is the beginning of the end.”

It was. Simon continued to publish excellent work and give outstanding talks at mathematics conferences around the world, but his days of eerie brilliance were over. It is impossible to say whether this was because his genius had diminished, or his focus. Never a tidy man, he became increasingly dishevelled. He wore battered clothing and mutton-chop whiskers, and always carried a greasy holdall that Conway suspected contained the solution to Monstrous Moonshine; in fact, it was filled with bus timetables.

The youngest of three brothers, Simon was born in London. His father, Richard Norton, was a remote figure, eager for Victorian respectability, who ran SJ Phillips, the family jewellery business, and dreamed of eating supper with a tail-coated footman behind every chair.

It was Simon’s mother, Elaine (nee Manasseh), who spotted Simon’s mathematical brilliance, when he was one and a half. Instead of flinging toy bricks around the room as his elder brothers had done, Simon sorted them into patterns. When he was three, Elaine arranged for an IQ test: Simon scored 178. He could count to 100 in 2s, 3s, 5s and 10s, read books and spell “fire extinguisher”.

At five, he changed his name to 5, and sent sums to his mother, addressed “My darling 45, I cried when you went out …” All his life, he maintained that his mother embodied “loveliness”, even though he found her criticisms of his messy clothes and bumbling manners difficult. At 10, he sat the Eton entrance test. “What’s that noise?” asked a teacher during the examination. “That is Norton, doing his mathematics paper, singing for joy.” Simon’s answers were beyond words: the examiner wrote simply “!!”

In later years, he owned a house in Cambridge and was famous for his generosity. He was the only landlord in the city to reduce his rent when Margaret Thatcher brought in the poll tax. Sometimes he would set potential tenants a mathematics puzzle. One was to replace the letters with the right numbers in the following multiplication: SIMON x P = NORTON. (There are two possible solutions.) I first met him in 1995, when I became one of his tenants, and in 2011 I published a biography of him: Simon, The Genius in My Basement.

The other love of Simon’s life was public transport. Even as a boy he would rush away to ride around the country on buses and trains. As an adult, he became a vehement campaigner against cars, wrote a regular, remarkably funny newsletter for the Campaign for Better Transport, and donated £10,000 annually to fund a prize for transport activism (he was especially pleased when one of his winners superglued himself to Gordon Brown). Despite Simon’s collapse into mathematical obscurity, he was a triumphant and inspiring figure: a person unburdened by rancour, jealousy or sense of loss.

He is survived by his brothers, Michael and Francis.

• Simon Phillips Norton, mathematician, born 25 February 1952; died 13 February 2019