After axing his friend from the judging panel earlier this year, Simon, 55, is said to have warned the current line-up that nobody is safe, and that Louis, 62, could reclaim his seat at any time.
Rita Ora and Nick Grimshaw were added as mentors for the new series, joining returning judges Cheryl Fernandez-Versini and Simon.
But the music mogul wants to keep things controversial, and won’t hesitate to make changes should the show become ‘boring’.
Simon told the Daily Star: “If they’re not talking about you then you’ve got a problem. I think it’s a good thing. I think we’re lucky.
“When you have the opportunity to make a could of shows like ours then there’s more fun than problems.”
Rita, 24, also revealed that, following the news that One Direction are to go on hiatus in 2016, they might have found the next big band.
She said: “We have three groups that I would take right now from their first audition straight through to the live shows and I would have no qualms about that. No fear.”
The X Factor returns on Saturday at 8pm on ITV – are you excited? Let us know now on Facebook and Twitter.