And it looks like he won't be returning.
The 62-year-old appeared on The Seven O'Clock Show and admitted he's tired of the ITV singing competition.
He said: "I'm done with X Factor, honestly."
When asked if he would consider retuning, he added: "I'd really have to think about it. I've done 11 seasons and that's a long time in television – I don't think I'll be going back."
Music manager Louis also hinted that there could be a few more changes on the panel.
He said: "Simon will probably be back because it's his show. Cheryl, I'm not sure. I don't know who else will be back."
There's been a lot of speculation over which judges will be returning, and who will be replaced.
While it's been reported that Cheryl, 31, will definitely be taking her seat on the panel again, former judge Tulisa is heavily rumoured to be replacing Spice Girl Mel B.
A source told the Mirror: "They haven't signed a deal yet but Simon, Cheryl and ITV are all on the same page. But that's really bad for Mel. She had a good first year but bosses want a new line-up."
Will you miss Louis on the show? Let us know via the comments box below.