Boris Johnson should be kicked out of the Conservative Party for his remarks about the burka, a Muslim peer says.
Conservative Muslim Forum founder Lord Sheikh told BBC Newsnight demands from the PM for an apology were not enough.
He has called for the MP to have the whip removed – meaning Mr Johnson would no longer represent the Tory party.
Mr Johnson said Muslim women wearing burkas “look like letter boxes” and compared them to “bank robbers”. A source said he stood by the comments.
Lord Sheikh, a former adviser to David Cameron, said: “Take the whip from him. Why not? He’s not a super human being, he’s a member of the party. The party chairman, the prime minister has the right to take the whip.
“It’s not out of order and that’s the thing I’d like to see.”
Speaking earlier, Lord Sheikh said Mr Johnson’s remarks would harm community relations.
Mr Johnson’s comments, in a Daily Telegraph article, have provoked criticism from Muslim groups, some Tory MPs and opposition parties.
In the article, he said full-face veils should not be banned but looked “ridiculous”.
Conservative Party chairman Brandon Lewis has said he agreed with Foreign Office minister Alistair Burt that there had been degree of offence in Mr Johnson’s comments, and called on the former London mayor to apologise.
Theresa May later backed calls for Mr Johnson to apologise, saying the remarks have “clearly caused offence”.
She said: “I do think that we all have to be very careful about the language and terms we use.
“What’s important is do we believe people should have the right to practise their religion and, in the case of women and the burka and niqab, to choose how they dress.”
But a source close to Mr Johnson said he “won’t be apologising”, adding it was “ridiculous” to attack his views.
“We must not fall into the trap of shutting down the debate on difficult issues,” the source added.
“We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists.”
By BBC political correspondent Susana Mendonca
The criticism pitted at Boris Johnson has largely been around the words that he used – “letter boxes” and “bank robbers” to describe Muslim women wearing face veils.
But Mr Johnson hasn’t directly responded to the accusation that it is his language which caused the offence.
Instead, the statement that has come from a source close to him suggests Mr Johnson wants to turn this into a debate about whether or not women should wear the burka at all.
Keeping a high profile is important to Mr Johnson.
He is seen by some in the Conservative Party as a contender for leadership – particularly since his resignation in which he criticised the prime minister’s plan for Brexit.
But he risks being seen as weak if he is forced into an apology by the party chairman.
So his refusal to apologise, and an attempt to now put himself on the side of “liberal values”, could be an example of Mr Johnson trying to save face while maintaining his profile.
Speaking on Newsnight, Shazia Awan-Scully, a Muslim who ran as a Conservative candidate in 2010 and is now a Welsh Labour member, also called for Mr Johnson to be sacked.
She said Mrs May “needs to be a leader and take action” over the “Islamophobic” comments.
Former Conservative chairwoman Baroness Warsi, the first Muslim woman to sit in a British cabinet, earlier called for disciplinary action against Mr Johnson if he did not apologise.
A debate about the burka should be had “in a serious way”, she said, rather than “trying to get airtime and attention on an issue which he knows will resonate with a certain part of the Tory Party”, she said.
Shadow equalities minister Naz Shah accused Theresa May of being “in denial” over the “offensive” comments.
She tweeted: “An apology is not enough, she needs to order an independent inquiry into Islamophobia in her party, as requested by the Muslim community, and take action against him.”
But Laura Perrins, from the Conservative Woman website described the calls for further action against Mr Johnson as “authoritarian nonsense we must resist”.
She told Newsnight Mr Johnson’s article explored “a very important debate in terms of community cohesion and integration” and his comments about the burka had been taken out of context.
Conservative backbench MP Andrew Bridgen has said Mr Johnson had raised an important subject in a “light-hearted way”.
What is Islamophobia?
By BBC home affairs correspondent Dominic Casciani
Last year, Zaynab Hussein, a mother from Leicester, was left fighting for her life after a man she didn’t know smashed into her with his car – and then ran over her again. She was attacked because she was a Muslim.
Tell Mama, the national organisation that collects anti-Muslim attack statistics, says that the majority of street victims of such abuse and violence are women, for the same reason that Mrs Hussein was singled out: some Muslim women are easily identifiable by their mode of dress – and therefore easy to target.
Seven years ago Baroness Warsi said prejudice against Muslims had passed the “dinner table test”.
And Mr Johnson’s critics regard his “letter box” and “bank robber” comments as part of the problem the peer defined: normalising prejudice and dehumanising women, rather than calmly debating the complexities of the veil in an open society.
Since Baroness Warsi’s warning, there has been the launch of a cross-departmental anti-Muslim working group to combat hate.
But it has been criticised as toothless, not least because the government can’t agree a definition for Islamophobia.
What Boris Johnson said
In his Daily Telegraph column, Mr Johnson – who last month quit the government in protest at Prime Minister Theresa May’s Brexit policy – was commenting on the introduction of a burka ban in Denmark.
He said he felt “fully entitled” to expect women to remove face coverings when talking to him at his MP surgery – and schools and universities should be able to take the same approach if a student “turns up… looking like a bank robber”.
“If you tell me that the burka is oppressive, then I am with you,” he said.
“If you say that it is weird and bullying to expect women to cover their faces, then I totally agree – and I would add that I can find no scriptural authority for the practice in the Koran.
“I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes.”
He said businesses and government agencies should be able to “enforce a dress code” that allowed them to see customers’ faces.
But he said: “Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business.”
He said a total ban on face-covering veils would give a boost to radicals who said there was a “clash of civilisations” between Islam and the West, and could lead to “a general crackdown on any public symbols of religious affiliation”.