Simon Cowell decided to change "The X Factor" because he felt they had "lost sight of the contestants".
The 59-year-old music mogul has decided to overhaul the talent show in favour of a new celebrity and all stars version as he felt the old format was focusing on the judges rather than the auditionees.
Speaking about the judging panel costs, he said: "It was no secret it was millions and millions of pounds. I am going to be really honest with you - and this is no disrespect to Robbie or anyone else who has been on the panel - my focus this year is mainly about the people who are coming on the show to compete. I'm probably talking myself out of a job here, but I think there has been so much emphasis on the judges.
"We've lost sight of the contestants. It got crazy, I never thought this would happen 20 years ago, when I started this show, that we would be talking more about the judges than the contestants, it's complete nonsense. Somehow that's the way it's gone. So that's why me and Rob were talking, and he could see where I was coming from. It's about getting the right people to come on the show, that was it."
Simon isn't sure whether Louis Tomlinson would return to the panel this year but the music mogul would "respect" his decision if he needed to take a break.
He told The Sun newspaper: "If he said to me, I am taking time off I would respect that. He's a great judge, by the way. He worked so hard for it.
"What he has gone through, you don't even think about work, you just think about how you're going to be there for him as his friend."