obbie Williams and Ayda Field quit "The X Factor" after Simon Cowell refused to increase their £10 million salary.
The husband-and-wife team walked away from the reality TV show judging panel after just one series and although they remain on good terms with Simon, they were reportedly unhappy that their pay demands were not met.
A source told The Sun newspaper: "The loss of Robbie is a big blow, given his huge star power. But the "X Factor" is being overhauled and we are likely going to go back to three judges, which is how we started.
"Robbie and Ayda were already on a £10million deal and there was no way we were going to be able to increase that sort of money.
"They haven't fallen out with Simon - they were all on holiday together recently - but there are business realities at play."
Robbie previously insisted that they would only return to the show if they got a pay rise.
He said: "We would love to come back. But we need to be treated kindly, so we'll see. Is that my pitch for a pay rise? Abso-f***ing-lutely.
"Honestly, "The X Factor" has been my favourite thing that I've done in showbiz and a welcome break from the thing that I've been doing for the last 30 years."
Robbie and Ayda announced their departure in a joint post on Robbie's Instagram page, revealed that they had other commitments.
The statement read: "Sadly though it's impossible to do everything ... and we're gutted to tell you we won't be able to return to The X Factor this year."
But the married couple will still be working with show boss Simon Cowell on other projects.
They added: "We will, however, be working on projects with Simon going forward - and as huge fans of the show, we will still be tuning into The X Factor, voting and winding Simon up...x (sic)"