He was right, albeit for the wrong reason. Some 200 or more, mainly students, perished in the notorious Plaza de las Tres Culturas (Place of The Three Cultures).
On October 2, 1968, 10 days before the Games began, Government forces opened fire, some from helicopters, on a student protest in the plaza in the capital’s Tlatelolco district.
Official sources stated that the number of dead was in the dozens, but students claimed hundreds died in what has become known as the Tlatelolco massacre.
Thousands of students were beaten and jailed, and many disappeared. Almost half a century on, the final death toll remains a mystery.
It also remains one of the most distasteful episodes in Olympic history, one which shames an Olympic movement which conveniently turned a blind eye. If any Games should have been abandoned it was these.
Yet what Brasher was referring to as a potential death trap was the city’s 2,200 metre (7,200ft) altitude.
He believed those Games should have been called off because of the lack of oxygen for the athletes but as it turned out he was wrong. No-one died gasping for air.
The point I am making is that almost every Olympics after the happily unblighted Tokyo of 1964 (my first) has suffered from a succession of Cassandras like Brasher who believed there were reasons why they should have been called off or that disasters would befall when they happened.
Montreal, Munich (though no-one foresaw Black September), Moscow, Los Angeles, Seoul, Barcelona, Atlanta, Sydney, Athens, Beijing and yes, even London, endured allegations ranging from corruption, mismanagement, construction delays or deficiencies, environmental issues, or a general state of unreadiness. Somehow, all survived, albeit some by the skin of their fiscal teeth.
For instance, I recall Montreal 1976 when the mayor, Jean Drapeau, under fire for the spiralling cost of the Games, publicly proclaimed: ''The Olympics can no more have a deficit than a man can have a baby.''
M Drapeau then proceeded to to spawn an infant with a $1.5 billion (£1.04 billion/€1.3 billion) debt which the citizens of Montreal took decades to pay off. This notably included the stadium with its embarrassingly unfinished roof.
There had been those who warned that the Montreal Games would not happen, just as there were with Athens in 2004.
Among them was the equally estimable Ian Wooldridge of the London Daily Mail, a brilliantly readable journo who had calculatingly turned scaremongering into an Olympic sport in itself.
The lovely Woolers somehow managed to find pitfalls as every Olympics approached but was then generally rapturous in his acclaim for them when they happened.
Both Brasher and Wooldridge are no longer with us but it occurs to me that they would be having the proverbial field day with Rio 2016.
The harassed organisers of the impending Games have had a bellyful of pre-Olympics woe, ranging from alleged financial skulduggery to a state of unpreparedness that has been labelled as worse even than that of Athens.
Plus, of course. the outbreak of the Zika virus which is connected to a rare birth defect in infant children; a crippling recession that has left hundreds of thousands unemployed and sent inflation through the roof; and now an unprecedented situation which means that Brazil is likely to be in a state of political paralysis when the Games begin in just over three months on August 5.
Late Sunday night, Brazil's lower house of Congress voted to impeach the President, Dilma Rousseff. The charges will now go to the Brazilian senate, which will decide whether Rousseff can keep her job.
Ostensibly, the case is about a cover-up: Rousseff allegedly fudged Government accounting to hide the scope of the Government's deficit problem during the 2014 reelection campaign.
But the political crisis goes far deeper than that. The second in line, vice-president Michel Temer, is also facing impeachment, while the third, Lower House speaker Eduardo Cunha, is another politician accused of corruption.
The vast majority of Brazil's Congress is itself facing corruption charges. Of the 65 members of the impeachment commission, 37 face charges of corruption or other serious crimes, according to the Los Angeles Times.
Of the 513 members of the lower house in Congress, 303 face charges or are being investigated for serious crimes. In the Senate, the same goes for 49 of 81 members.
What a mess.
The International Olympic Committee (IOC) says it is “closely following” Brazil’s political crisis but believes nothing can knock the Games off course.
”The IOC is following closely the latest developments with regard to the impeachment of President Dilma Rousseff,” said an IOC spokesman. But he added that “preparations for the Olympic Games have now entered into a very operational phase, where these kinds of political issues have much less influence than at other stages of organising the Olympic Games.”
So yes, the Games will go on come hell or high water, because they must, even if the IOC have to bail them out.
They invariably do and I tend to agree with my insidethegames colleague Nick Butler and the latest IOC inspection committee that the sports-loving capital of carnivals will overcome these potentially crippling issues and embrace a reasonably successful Games.
But it could be a bit like Brazilian football: Touch and go.