Senior Conservatives have written off Rishi Sunak as a potential prime minister – and now believe Boris Johnson will have to remove him as chancellor in his next reshuffle – following the furore over his US green card and his wife’s tax affairs.

One former Tory minister told the Observer that the fear among Conservative MPs with small majorities was that the party was now in a “death spiral” with its two leading figures – the PM and chancellor – both having lost respect among voters.

“The worry is that this is symptomatic of a party in terminal decline and that we are in a death spiral,” said the ex-minister, after Sunak’s multimillionaire wife, Akshata Murty, agreed to pay UK tax on her global fortune, having avoided doing so under rules that apply to people who opt for non-domiciled status.

Referring to the entire controversy and revelations about the couple having had US green cards, the ex-minister added: “That is the concern of colleagues with small majorities.

“He has shown colossal naivety and the way he has arranged his affairs does not suggest he is even committed to the UK, but that he is keeping open the option of a career in finance in the United States.”

Rishi Sunak and his wife, Akshata Murthy. She has acknowledged that some people did not see her tax status as compatible with her husband’s position.

On Friday it also emerged that the chancellor and Murty had both held US green cards, meaning they had registered themselves as permanent residents in the US. In a development that shocked many Tories, it was confirmed that Sunak held the green card for a period of 19 months when he was chancellor and in charge of UK finances. The issue was even raised at a White House briefing on Friday evening.

A former Tory cabinet minister said Sunak was clearly damaged and would not stand any chance in a leadership contest if one was held in the coming months. The way he had arranged his family’s affairs had shown his political judgment to be badly flawed and raised questions about whether his leadership chances could ever be revived, he said.

A third senior figure in the party said the controversy of recent days had demonstrated how quickly a once-stellar career could go into reverse, to a point from which it would be difficult to recover. “The timing is terrible. I think his problem is that taxes have gone up a lot in his period as chancellor, which is not popular in the party or country. And we have a cost-of-living crisis. None of this looks good.”

It was reported that removal vans had been seen in Downing Street yesterday and that Murty and the two children would live at their west London home with Sunak remaining at No 11 during the working week.

Johnson is believed to be considering a ministerial reshuffle in the summer or autumn. There are known to have been tensions between No 10 and No 11 after the chancellor offered only lukewarm support to Johnson at the height of the scandal over lockdown-busting parties, before Russia’s invasion of Ukraine. A government source said: “I don’t see the PM resisting this opportunity to move Rishi now.”

The latest Opinium poll for the Observer finds that Sunak’s approval rating has dropped to a new low of -15, making him only slightly less unpopular than Johnson himself.

Only four months ago, when he was firm favourite to succeed Johnson if the PM was toppled over parties at No 10, the chancellor’s approval rating stood at a very healthy plus 11.

All opposition parties are now determined to keep up pressure on Sunak over in the coming weeks. Labour and the Liberal Democrats called on Murty, who remains an Indian citizen, not just to begin paying tax on her overseas earnings but also to pay back what she had saved over recent years.

Shadow transport secretary Louise Haigh said that while her “non-dom” status had been legal, Sunak, who claimed last week that she was victim of a smear campaign, had failed to be transparent about his family’s tax status at a time when he was raising taxes for millions of people. “He has come out on a number of occasions to try and muddy the waters around this and to obfuscate,” she said.

In a statement on Friday, Murty, who is to keep India as her “place of domicile”, said she had done nothing wrong but acknowledged some people did not see her tax status as being compatible with her husband’s position. “I understand and appreciate the British sense of fairness and I do not wish my tax status to be a distraction for my husband or to affect my family,” she said. “I do this because I want to, not because the rules require me to.”

By retaining her non-domicile status she will in future be able to save huge sums in inheritance tax. Murty has assets of at least £690m in her father’s company, Infosys.

The Liberal Democrats have drawn up draft legislation aimed at forcing the chancellor and any other government ministers to reveal whether they or their spouses claim non-domiciled status or have holdings in overseas tax havens.

Daniel Beizsley, a researcher with the charity Spotlight on Corruption, said the fact that Sunak had a blind trust through which his financial interests were handed over to an independent trustee raised issues over potential conflicts of interest. “By having this arrangement in place, Sunak is preventing any proper scrutiny of his financial interests and is the only current member of the cabinet taking this approach. The system of blind trusts needs to be rethought to include at least a summary of their contents if they are relevant to a minister’s brief to give the public a better understanding of any potential conflicts.”