Theresa May is under pressure to get an agreement from the Democratic Unionist Party (DUP) on the status of the Irish border when the UK leaves the EU.
The prime minister pulled out of a deal with Brussels that would have kick-started trade talks after meeting fierce resistance from the DUP.
The party said it would not accept a deal which saw Northern Ireland treated differently from the rest of the UK.
Mrs May is due to update ministers at a cabinet meeting on Tuesday morning.
Former Brexit minister David Jones described Monday as a “very difficult day” and said the prime minister now had “a lot of talking to do” with DUP leader Arlene Foster.
He believed the DUP had been spoken to beforehand about what was being proposed, but that the precise wording had not been explained and the Northern Ireland party “clearly thought twice”.
Mr Jones, one of the Eurosceptic MPs on the Conservative benches, said he too was not happy with the key phrase the DUP objected to in the draft agreement, which apparently guaranteed “regulatory alignment” between Northern Ireland and the Republic of Ireland.
The prime minister needs the support of the DUP, which is Northern Ireland’s largest party and has 10 MPs at Westminster, because she does not have a majority to win votes in the House of Commons.
Former top Foreign Office official Lord Ricketts said the row was “damaging” for Mrs May, adding that it was “pretty extraordinary that this wasn’t all stitched up with the DUP beforehand”.
“We’re used to prime ministers going to Brussels and having a row with the EU and coming back without an agreement, but to go to agree with the EU and then have a row on your own side is inconvenient,” he said.
The UK is due to leave the EU in March 2019 having voted for Brexit in last year’s referendum.
Mrs May wants to begin drawing up a deal on post-Brexit trade relations between the UK and the EU but first needs to get an agreement on “divorce” issues before European leaders meet on 14 December.
The three issues that need to be resolved are the Northern Ireland border, citizens’ rights and the amount of money the UK will pay as it leaves.
She will return to Brussels as early as Wednesday to try again.
Talks with European Commission president Jean-Claude Juncker broke up without agreement on Monday, although both sides said they were hopeful of getting a deal by the end of the week.
Mrs May had been expecting to make a major statement to MPs about a breakthrough in Brexit talks on Tuesday, but that has now been cancelled.
Instead, it is thought she will hold meetings with the DUP, as she attempts to get them to support her.
Ireland’s Prime Minister Leo Varadkar is understood to have insisted on the clause to prevent the return of a “hard border” on the island of Ireland, amid concern it could undermine the 1998 peace treaty – the Good Friday Agreement that brought an end to 30 years of sectarian conflict in Northern Ireland.
But the DUP says it would prevent Northern Ireland from leaving the EU “on the same terms” as the rest of the UK.
Northern Ireland would effectively have remained in the EU’s customs union and single market in all but name.
Scotland’s First Minister Nicola Sturgeon said that if Northern Ireland was allowed to operate under different rules there was “surely no good practical reason” why other parts of the UK could not do the same – a message echoed by Welsh First Minister Carwyn Jones and London Mayor Sadiq Khan.
Leo Veradkar said he had been “surprised and disappointed” by the failure to get a deal on Monday and claimed the UK had changed its mind at the last minute after the DUP raised objections.
Downing Street has not responded to Mr Varadkar’s claim.
But it said the border was not the only outstanding problem and disagreement remains over the role of the European Court of Justice in overseeing EU citizens’ rights in the UK after Brexit.
The prime minister is expected back in Brussels for further talks before the end of the week.
Sources declined to confirm reports that she would meet Mr Juncker and chief Brexit negotiator Michel Barnier on Wednesday.
The government’s chief whip, Julian Smith, is understood to have held talks with his DUP opposite number, Sir Jeffrey Donaldson, on Monday evening, as efforts began to get the party on side.