Former foreign secretary Boris Johnson is facing growing criticism over his remark that Muslim women wearing the burka "look like letter boxes".
Dominic Grieve, the ex-attorney general, said he would quit the party if Mr Johnson became leader.
Ex-Tory chairwoman Baroness Warsi said his remarks could trigger a rise in hate crime.
Senior Tories have urged him to apologise but Mr Johnson has not done so, and has stood by his comments.
In a Daily Telegraph article, he said full-face veils should not be banned, but it was "absolutely ridiculous" women chose to "go around looking like letter boxes". He also compared them to looking like "bank robbers".
A source close to the former London mayor has said: "We must not fall into the trap of shutting down the debate on difficult issues.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
But, speaking on BBC Radio 4's The World at One, Mr Grieve - a former Remain campaigner who has previously clashed with Mr Johnson over Brexit - said his behaviour was "very embarrassing".
Mr Grieve said he would "without the slightest doubt", quit the Tories if Mr Johnson became leader, "because I don't regard him as a fit and proper person to lead a political party".
Earlier Culture Secretary Jeremy Wright said there was no reason not to have a "robust conversation" about the subject, but added: "We're not talking to our friends in the pub. We are public figures and we have an additional obligation to be careful."
A former Tory chairman, Lord Pickles, said Mr Johnson, who was foreign secretary until resigning last month over Brexit, risked "closing down" the debate with his "illiberal language".
Supporters of Mr Johnson say the row is politically-motivated and that other senior Tories have made similar remarks without such criticism.
Writing in the Guardian, Baroness Warsi said Mr Johnson's remarks were "indefensible" and "have no place in the modern Conservative Party".
She said, although he was setting out a liberal position on the burka, he was doing it in an "alt-right" way, and using Muslim women as "political fodder… to stake out a leadership bid".
"Johnson's words… send out a message that Muslim women are fair game," she wrote.
Baroness Warsi was the first Muslim woman to sit in a British cabinet.
But mother-of-seven Tahira Noor, who has been wearing a burka for 20 years, said it was "100% my choice" and Mr Johnson's comments showed a "lack of knowledge".
She told BBC Radio 5 Live: "In today's day and age, the majority of the women who wear the burka are born and brought up in this country, are educated in this country, they've been to colleges, universities, and have understood why they want to do what they're doing.
"They're under no oppression, they're not doing it because their husbands want them to or their fathers want them to."
Ms Noor has four daughters and none of them wear a burka.
"I haven't forced my daughters into it because I don't have to," she said. "It's not a must, it's not an obligation."
Boris Johnson 'has caused offence', PM says ...
Mr Johnson's former adviser Munira Mirza said Mr Johnson's views on the burka had been "entirely consistent" and other Conservative politicians had expressed the same view, without being called Islamophobic.
In 2013 former Justice Secretary Ken Clarke - who also opposed a ban on the public wearing of burkas - said they should not be worn while giving evidence in court. He referred to burkas as a "peculiar costume" and a "kind of bag".
Ms Mirza said: "The reality is there is a political fight here.
"People who frankly couldn't care less about the issues that Muslim women face are piling into Boris because Boris said it."
Last year, Zaynab Hussein, a mother from Leicester, was left fighting for her life after a man she didn't know smashed into her with his car - and then ran over her again. She was attacked because she was a Muslim.
Tell Mama, the national organisation that collects anti-Muslim attack statistics, says that the majority of street victims of such abuse and violence are women, for the same reason that Mrs Hussein was singled out: some Muslim women are easily identifiable by their mode of dress - and therefore easy to target.
Seven years ago Baroness Warsi said prejudice against Muslims had passed the "dinner table test".
And Mr Johnson's critics regard his "letter box" and "bank robber" comments as part of the problem the peer defined: normalising prejudice and dehumanising women, rather than calmly debating the complexities of the veil in an open society.
Since Baroness Warsi's warning, there has been the launch of a cross-departmental working group to tackle anti-Muslim hate.
But it has been criticised as toothless, not least because the government can't agree a definition for Islamophobia.
What Boris Johnson said:
In his Daily Telegraph column,
Mr Johnson - who last month quit the government in protest at Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up ... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West, and could lead to "a general crackdown on any public symbols of religious affiliation".
Boris Johnson .. Burkha wearers look like letterboxes
Reply With Quote