The Scottish Government is terminating the contract to run the Caledonian Sleeper service seven years early.
Serco will stop operating the cross-border rail service in June next year.
The company said it had tried to renegotiate the contract to put the loss-making service on "a more sustainable financial footing".
But Transport Minister Jenny Gilruth said its proposals had been rejected because they did not represent value for money to the public.
She added that Serco had "delivered well" and said it had significantly improved the service over the last seven years.
The Caledonian Sleeper, which has been operating in various forms since 1873, runs overnight trains between Scotland and London.
Different routes
There is a Lowlander route between London and Glasgow and Edinburgh, and a Highlander route to Aberdeen, Inverness and Fort William.
The current 15-year franchise was awarded to outsourcing giant Serco in a deal worth £800million. It had been due to run from 2015 to 2030.
In 2019, Serco unveiled a new £150million fleet of trains which the company said would "transform" the journey. It said 75 new carriages were now in service.
John Whitehurst, managing director of its transport business, said it had inherited an "unreliable and outdated" fleet of carriages which dated back to the 1970s.
"We are extremely proud that under our leadership and management we have introduced new rolling stock and other significant innovations that have completely transformed the service," he told the BBC.
There was a clause in the contract which allowed Serco to try and renegotiate the terms through a process known as rebasing.
No agreement
However, Mr Whitehurst said the two sides had been unable to reach an agreement.
Ms Gilruth said work was underway to determine arrangements for the continued provision of Caledonian Sleeper rail services beyond June 25, 2023.
Meanwhile, a union representing ScotRail workers is to consider a new pay offer aimed at avoiding more strikes.
The undisclosed deal was offered on Tuesday during talks between the train operator and the RMT.
The union had rejected an earlier 5% pay rise as an effective wage cut for its members because of the soaring rate of inflation.