Caroline, 35, has been blasted for ‘having no personality’, while Olly, 31, has suffered a series of blunders, most notably when he accidentally told Monica Michael she was going home before a decision had been made.
Now American Idol host Ryan Seacrest is rumoured to be taking over from the pair, and he’s even got 56-year-old Simon’s approval.
The 40-year-old told music mogul Simon: “Part of me wanted to come and host a segment of [The X Factor UK], but nobody made the offer and there’s two hosts.”
Simon replied: “You have to ask yourself to come to the party, you could have done it.”
It was recently announced that season 16 of American Idol, set to premiere on 6 January 2016, will be the last, meaning Ryan, 40, will definitely have enough time to front The X Factor.
But don’t worry – Simon hasn’t gone and abandoned poor Olly. He recently praised the Dear Darling singer for his constant improvement.
Si told the Daily Star: “The way he handled it was great, it’s part of his charm. Every week he’s getting better and better.”
Do you think Olly and Caroline need to be replaced? Let us know on Facebook and Twitter.