Noel Pearson has turned on the advocate of the “modest” Indigenous recognition in the constitution, accusing Jesuit priest and law professor Frank Brennan of setting himself up “as some kind of final arbiter of what is good for the natives”.

The Cape York Indigenous leader has also strongly defended his proposal to include in a more substantial referendum question the recognition of Indigenous body to advise the Parliament.

His comments came on the eve of a weekend of high-level talks among Indigenous leaders ahead of an unprecedented summit with Tony Abbott and Bill Shorten in Sydney on Monday.

While there is no expectation that the meeting will resolve differences on the scope of the referendum question or when it should be put to the people, there is hope that it could agree on a process to settle the question, perhaps by year’s end.

“I’d like to think we’ll get to a point where we can walk away with a united voice,” declared Sean Gordon, who represents Indigenous communities on the central coast and backs the Pearson model for an Indigenous body.

The Prime Minister and Opposition Leader have spoken privately ahead of the meeting, but signalled they see their main role as to listen to the views of around 40 invited Aboriginal and Torres Strait Islander leaders.

Among them will be Patrick Dodson, who co-chaired the expert panel that proposed the prohibition on racial discrimination as part of an ambitous plan for recognition.

Mr Pearson was the initial advocate of the prohibition, but developed his proposal for an Indigenous body when he became convinced the prohibition was problematic on a number of fronts and would result in the defeat of the referendum.

Professor Brennan predicts both alternatives for substantive change face defeat and is advocating “modest” change in the form of an acknowledgement of Aboriginal history, culture, languages and land rights, the deletion of one racially discriminatory section and no constitutional ban on racial discrimination.

His case for minimalist change is spelled out in his book No Small Change, written while spending a year as a visiting professor at Boston College law school.

“This idea that somebody who has been out of the country for the last several years can come back with his black robes on and determine what is right for our people is just completely inexplicable,” Mr Pearson told Farifax Media. “I don’t know where Frank Brennan gets off.

“This is not black robe territory two centuries ago. Indigenous people are a polity. We don’t need a priest or any other person to speak on our behalf. This is about a political settlement.”

Mr Pearson dismissed Professor Brennan’s critique of his proposal for an advisory body to give Indigenous people a voice in the parliamentary process, saying the lesson from past and present experience was “if it is not chosen by Indigenous people and does not have ability to transcend the political proclivities of the government of the day, then it’s not worth doing”.

Professor Brennan replied that, if it was the view of Indigenous Australians that symbolic change was no better than no change, he would accept the judgment.

“I have great equanimity on this. I am a non-Indigenous Australian who cares passionately that Aborigines end up with a better deal under the constitution, but I am such a believer in self-determination that, of course, it’s their call.”